Letters to the AOC – Risk to court assets / first written notification of something afoul in contracting practices

Posted on December 23, 2010


First written notification of something afoul in the OCCMs contracting practices.

Background: This email was in response to new directions from OCCM management. Initially, there was a plan to use ARRA funds to survey and retrofit every building in the state for energy and building management and control systems.

It was about this time that all had learned that no federal funds would be coming and this was being done with state money because it was still a worthy, commonsense plan. The e-mail from Mr. Leung was his response to his new instructions – pick a subsetset of buildings – those over 45,000 square feet.

Mr. Paul and Mr. Leung strongly objected to the approach being utilized, which was to commission costly studies that time and time again had proven to be inadequate or worse yet, plagarized and not at all applicable to the structure being evaluated. These studies cost anywhere from $20,000 to $50,000.00. None of the previous studies were usable. A much quicker methodology of doing a quick survey to qualify for the funding as opposed to over 500 expensive individual studies being commissioned slowly on buildings across the state was rejected by OCCM management. They never wanted to qualify for federal funds so they had Mr. Leung and Mr. Paul engaged in busy work for two years defining and reviewing methodology, sending it up to OCCM management for long term evaluations and ultimate rejection and back down to the Paul/Leung engineering drawing board. This occurred for the two years previous to this email. You’ll notice it references building systems integration V(ersion)6A. This was version 6 with the 5 previous versions being rejected.

If you are a presiding judge or a facilities manager, there is some really interesting information in this email for your Information Technology manager to review as they will likely wish to act upon it to protect your courts. 

As of the date of this post nearly eighteen months later, NONE of the issues denoted by Michael Paul have been addressed, leaving all of these systems at risk across the state. As a point of reference, Michael Paul co-designed and co-deployed the entire Microsoft network utilized by the AOC, all courts of appeal and the California supreme court. He designed most of the new IT facilities in the branch so he knows a thing or two about computers and architecture.

I’m told the mail never garnered any response from OCCM management.

From: Paul, Michael
Sent: Friday, July 24, 2009 8:24 AM
To: Leung, Dennis; McGrath, Patrick; Mullen, James
Subject: RE: RE: Strategic Plan for the Judicial Branch Building Systems Integration Revision V6.A

To my associates in occm:

First, at this rate, you will complete all buildings about the time I am ready to retire. This will not work. You are several hundred buildings short.

You inherited hundreds of server or workstation based building management systems that need pre-assessments of the IT portion of these systems. yesterday. At a rate of 52 scope assessments per year, 80% of your systems remain at risk for systematic IT failures. At 30-50K per system, that is a huge chunk of change that the taxpayer may unnecessarily be responsible for.

Now, I know I have been warning FMU and FMU mamagement about this and it has been well documented for over two years now. I’ve also been mostly ignored for most of that two years about the need for pre-assessment because you guys indicate various things like “your service provider jacobs will do this for us” and your service provider 1) has little to no proven competence in OPERATING these systems and 2) none whatsoever in deploying or maintaining these systems.

In fact, they have been BYPASSING building management systems due to a lack of knowledge of these systems and this has caused numerous other problems, including the destruction of many other parts of the HVAC systems and the buildings falling into “manual control mode” where your contractor has bypassed so much of a building they require a staff of people onsite to manually control the building. More revenue dollars for them, more bills to the taxpayer. This is just the tip of the iceberg on what is currently happening that I have firsthand knowledge of and the complaints I am personally fielding – remember, IS was in the courts 10 years before OCCM was introduced to the courts. This puts the public and our employees at risk because these are also life safety systems (like smoke dampers) that are getting bypassed.

A lack of pre-assessments of the IT portions of these systems places you at significant risk for costly systemic IT failures.

What are systematic IT failures?

Hard disks carry incredibly inflated figures for mean time between failures. Published ratings give you 114 years for a mean time between failure. If that were true, how come on average the AOC replaces two hundred and fifty hard disks per year on 1400 computers? We use 10000 hours as MTBF as some of us have worked for hard drive manufacturers and others (like me) spent hundreds of hours doing mil-spec certification of hard drives. When the components that comprise the drive only have a published MTBF of 10000 hours, only a fool would believe manufacturers published MTBF figures.

1. Statistically, during the enovity assessments 5% of the non-assessed systems per year will experience hardware failures. 90% of those hardware failures will be related to hard discs. If you have a full system copy of said system on a USB drive, you can prevent generating a 30-50K bill on these systems due to nothing more than a hard disk failure as these systems currently deployed do not meet our current spec which requires mirrored hard drives. If they did have mirrored hard drives they would be far more robust and redundant. A case where spending 200 on a hard drive going in may save you 50K on the back-end. A good investment in new systems. OR spending 200.00 for me to make a USB key system copy on a pre-assessment might just save you 50K during the process of the enovity assessments.

2. All of the network connected systems require a full security suite protecting the base operating system and perpetual updates of the operating system and the security suite. Antivirus is insufficient. You need whole system protection including firewalls and internet browsing protection. Because many of these services were previously provided by the county and we update your own systems silently and in the background every time you hook your computer to our network, you develop the risk of false assumption that these systems “will be okay”. They won’t be okay. Once the protection definition files cease to get pushed to the server, these systems scan only for known vectors. Now given there is 3,000 new computer viruses written every day, that’s a pretty substantial risk not to be perpetually updating the BMS system in accordance with the BMS manufacturers guidelines. During the course of the enovity assessments, statistically, you can count on losing 15% of your systems per year due to a lack of updates of the operating system and the underlying security suite and not patching the BMS system or licensing falling out of compliance. A pre-assessment would define where and how this system is currently configured to recieve the updates, what OS platform and software is on the system and most importantly, what is missing and what you need to do to protect the IT portion of these systems. If it is something as simple as loading a security suite as an interim solution until enovity does their fly-by I would do that in the pre-assessment and just move on to the next building, knowing I protected the system for at least a year. That buys you time for enovity to get to it.

3. I’m not sure you all have a handle on administrative permissions for these systems. Who administers these systems right now. Are they locked down? Can a Jacobs guy use this BMS system to surf the net? Does the person sitting at the terminal happen to have administrative-level permissions that would allow them to erase the BMS system off the server in its entirety. Without a pre-assessment you have no idea the state of the systems. Given that many of these systems were previously operated by the county with county administrators, that institutional knowledge to gain permissions to these systems may evaporate over time with people retiring and the county deleting their data because you haven’t contacted them in years about the permissions attached to the system. End result? Nobody can do anything to the systems because no AOC IT guy got administrative access to the system itself. End result? The system has to be hacked to gain administrative access, install any updates or patch anything because the institutional knowledge evaporated during the period enovity was scheduled to do their assessments. Or, administrative permissions could have been granted to someone the caliber of Steve Sylvester and its loaded with file sharing software to update his ipod, has a few games on it to entertain him at work and has never been updated…. or worse, has been set to automatic updates and one day the BMS system stops working because a microsoft update stopped a service running on the BMS side. And you have not been paying for licensing of the BMS software, so you are not entitled to the patch that would be required because of the update to get your system working agian.

You desperately need pre-assessments at a rate 20 times what enovity is doing and they need to start yesterday. You need to assess all IT systems soon as currently they are degrading and falling apart due to the three factors named above. This is a wise and prudent strategy that would be recommended by literally every single BMS manufacturer we do business with and if you want proof of that, I can provide proof of that. As it were, the proposed strategy of enovity only touching 52 buildings a year leaves all others at risk and I speak with absolute certainty that Enovity too would recommend this “bare minimum” course of action.

It isn’t a matter of affording to do the pre-assessments. Its a matter of the resultant costs of not doing the pre-assessments. You can’t afford NOT to do them.

In fact, if you can find a BMS manufacturer that will tell you in writing that this is not a prudent “bare minimum” strategy you should immediately undertake to protect the taxpayers, then I will shut up but without a strategy for pre-assements forthcoming or willingness to even consider it, I must escalate this matter because FMU risks setting millions of dollars worth of taxpayer money on fire in unnecessary utility bills, system failures and system replacements.

And lastly…

It seems like all work is routed through JACOBS as an exclusive service provider, sometimes at a cost 5 times as high as the next closest bidder. I have had complaints to me about this and I don’t know how to handle them. Furthermore, I am fielding complaints that they are contracting without a contractors license doing millions of dollars worth of work when the law requires all work over 600.00 has a contractors license. I checked the CSLB. I see no contractors license for jacobs anything. I am fielding complaints from qualified, licensed contractors that the people you are using have no license to be doing what they are doing. I have also fielded complaints from various county entities that Jacobs is going on its own without pulling permits and installing systems in a manner that does not meet code requirements or breaches other system warranties. Again, I get these calls and emails and I don’t know how to handle them. In all cases, they said they already brought it to the attention of someone in OCCM and were ignored or told there was nothing that could be done about fraud, waste and abuse because we in the judicial branch are not covered under the same laws as the rest of california.

This is a mistake. We are covered by the exact same fraud, waste and abuse laws that govern the rest of california. We have our own internal staff that investigates it. Since I just found out about the hotline, I will be distributing that number shortly and encouraging the auditing section to put a fraud, waste and abuse form up on the internal AOC intranet and the courtinfo site in an effort to curtail the possibility of fraud, waste and abuse and build public trust that we are willing to do something about it.

From: Leung, Dennis
Sent: Tuesday, July 07, 2009 4:47 PM
To: McGrath, Patrick; Pfab, Gerald
Cc: Paul, Michael
Subject: RE: Strategic Plan for the Judicial Branch Building Systems Integration Revision V6.A


I have incorporated 52 buildings that have been identified by CAFM that exceed 45,000 and 0 sq ft2 in area . I have modified the Phase 0 cost budget to $ 4.4 million dollars over 2.5 years to complete approximately 52 building scope assessments by the end of 2012


Dennis Leung, MEP Engineering Group
Facilities Management / Design and Construction Services
Office of Court Construction and Management
Judicial Council of California – Administrative Office of the Courts
455 Golden Gate Ave., Suite 8331-03
San Francisco, California, 94102-3688
415-865-4045 Fax 415-865-8885

“Serving the courts for benefits of all Californians”

This message was also forwarded to Eric Pulido and John Judnick of the IAD.