Isaac: the rest of the story, Part 1

Updated SCADA failure section June 2, 2013. See below.

Last October, I requested via FOIA the emails of a number of key personnel in the Corps' New Orleans District. I confined my request to emails from August 27th to August 30th of 2012, the days Hurricane Isaac came ashore. I wanted to know exactly what happened at the Corps' gate structures at the mouths of the three New Orleans outfall canals under real storm circumstances. After a delay of over 200 days, I finally received the bulk of those emails this month. They help fill in the other half of the story. The first half - that told by the pump logs of the city pump operators, was covered in my October 6, 2012 post, "Isaac in New Orleans - what we know so far." That post includes links to all the city pump logs.

I went after the emails of a number of folks in the Operations Division of the New Orleans District who would be on the front lines during a storm. Raymond Newman, Donald Constantine, and Carl Robinson serve as "canal captains" during storm events at - respectively - the 17th Street canal, the Orleans Avenue canal, and the London Avenue canal. As such, they are the personnel with the freshest information during a storm. Donald Schneider, also within the Operations Division, has popped up repeatedly in past outfall canal communications. Chris Accardo is head of Operations at the New Orleans District, and would have information flowing through him from all over the District (I am still awaiting Accardo's emails, but so many of them are included in everyone else's that I do not anticipate much new information).

I also wanted to know what was going on outside Operations, so I asked for a few others' emails. Ken Holder is the head of Public Affairs at the Corps' New Orleans District, and would be the public face to the press during the event. Bac Nguyen has, in the past, had responsibility for specifying and coordinating pump repairs and other work at the closure structures.

The headlines that emerge from these emails are the following:

A) The Corps could not lock one of the gate segments in place at the London Avenue site before the storm hit, and actively decided NOT to drop large sandbags in front of the gates, despite their own procedures saying they were required to do so. The trench the gates drop into had partially filled in with silt just a few hours after being cleaned, and they decided to just let it ride. This particular incident is detailed below.

B) Confirmation that the 2011 remediation projects failed at the 17th Street and London Avenue canals. According to the "Outfall Canal Reference" spreadsheet, the maximum operating water level (MOWL) at 17th Street is 6.5 feet and is 5 feet at London Avenue. Before the remediation projects, the 17th Street MOWL was 6 feet and the London Avenue canal MOWL was 5 feet. The Corps had never fully confirmed this. Literally tens of millions of dollars were wasted in those projects. The Corps has also announced separately there is more work coming on both canals.

C) While the Corps believes the MOWL at the Orleans Avenue canal is 8 feet, there is a "secret" trigger along that canal which prevents its full utilization. According to that same spreadsheet, no more than 5 feet of water can be put in the canal or S&WB pump station 7 at the canal's south end will flood. I believe this is possibly the true reason the gates were dropped at Orleans Avenue during Isaac. Here's a screenshot of the spreadsheet:

D) At one point during Isaac, there were 8 pumps out of commision at 17th Street, representing nearly 15% of that structure's supposed capacity. This is far more extensive than previously reported. I'll have more details on this in the next post.

E) The level gauge and remote system control and monitoring failures - previously reported by the Lens as occurring during the storm - were apparently already occurring two days before the storm ever hit, and they were across the entire system. The Corps was reduced to trading emails back and forth during the storm's passage to let each other know what was going on with water levels and pump statuses, because there was no remote visibility. My guess is nearly the entire SCADA system failed at the height of the storm, and none of the backup systems worked either (for example: the SCADA backup generator for the London Avenue canal system was out of service at city pump station 3).

F) Electrical problems included the complete failure of the gate hoists at Orleans Avenue on August 27th (the gates had to be lowered by crane), and an hours-long failure of part of the fuel supply system at the London Avenue site which left half the pumps at that site with just four hours worth of fuel on the morning of August 29th. That last one was later corrected - a gecko was found across some wires.

I'll get to some of these in a later post.

In sum, the emails reveal that far more happened during Isaac than the Corps publicly revealed, even to reporters (the NY Times' John Schwartz and the Times-Picayune's Becky Mowbray) and a Senator (David Vitter) embedded in the Corps' Emergency Operations Center (EOC) during the key overnight and morning key hours August 28-29, when the storm was at its peak. But first we'll go back about a week before that.

There are a few attachments to the emails, which I also received. The first key one is this, the Corps' weekly "pump tracker." It is an Excel workbook containing readiness data for every major piece of mechanical equipment at all three sites, including every pump, every gate segment, and all the generators, fuel tanks, and city power lines coming into the station. According to the August 24, 2012 pump tracker workbook, every piece of equipment at all three sites was "operational," except for pumps E4 and E5 at the Orleans Avenue site. Those two pumps were out for repairs due to corrosion-caused hydraulic system failures. I wrote about them August 28, 2012 here.

Theoretically, that "operational" status is borne of what the Corps claims is extensive testing throughout the year, especially during hurricane season. I have serious doubts of the efficacy of the testing, because even before the systems were stressed, they were falling apart.

The troubles started on the evening of August 27th, over a day before Isaac even came ashore. As I noted in my earlier entry, the gates at the Orleans Avenue structure were dropped during a tropical event for the first time August 27th. An email from Chris Accardo sent at 9:19 AM on August 28th provides some detail:

"due to an electrical problem, the gates had to be lowered with a crane last night."

It is unclear whether a) the gates were lowered because there was an electrical problem, so they were lowered out of an abundance of caution, or b) they were going to be lowered anyway, and Accardo is only referring to the style of their lowering (crane versus electrically-actuated hydraulic winches). It is also unclear if the gates were lowered because the Corps was trying to avoid getting more than five feet of water in the canal in order to avoid flooding the city's pump station, but it's a good guess considering predicted storm surge levels in Lake Pontchartrain were higher than five feet at that point. The "crane" is one of three rental cranes the Corps has, each stationed at each site. The Corps has paid so much in annual rental fees for those cranes at this point they could have easily purchased them, and they are continuing to rent them rather than buy them.

There are two later emails referencing this problem. Donald Constantine emailed Donald Schneider at 10:04 AM on August 30th:

"Don - do we have anyone working on getting the gate hoists working again? Will it be another crane operation to pull the gates?"

The response came from Donald Schneider at 7:49 PM on August 30th:

"Prime Controls determined that the hydraulic gate lift problem was caused by failed contacts. They're searching for a replacement part; however, I think they rigged the unit to operate on a temporary basis."

This certainly brings into question whether these gates were exercised at all as part of the reporting in the August 24th pump tracker workbook, when they were said to be "operational."

[Begin June 2, 2013 update]
The system problems extended far beyond just the Orleans Avenue gate hoists, though. Based on email traffic before and during Isaac's landfall, it appears the Corps' monitoring and control system for the gates experienced a total failure some time on or before August 27th, two days before the storm hit. The Supervisory Control and Data Acquisition - or "SCADA" - system allows the Corps to remotely control activities at the gate structures such as running pumps without having to have personnel there. It also allows for real-time monitoring of a series of 8 level gauges placed along each canal. There are SCADA terminals in Sewerage and Water Board drainage pump stations 3 (at the foot of the London Avenue canal), 7 (at the foot of the Orleans Avenue canal), and 6 (at the foot of the 17th Street canal), as well as in the Corps' Emergency Operations Center in their District Office on Leake Avenue in New Orleans.

The SCADA system was designed and installed by Prime Controls of Lewisville, Texas. Prime Controls was also awarded a five year maintenance contract for the system in late 2008, a contract still in effect at the time Isaac came ashore. The Corps' Bac Nguyen was emailing back and forth with Prime Controls' Metairie, LA-based project manager Steve Baker August 27th, trying to get one of Prime's employees (project engineer Randy Oliver) to drive in from their home office in the Dallas area for the duration of the storm.

At the end of that email chain, Nguyen wrote Baker at 4:25 PM:

"We lost communicaiom [sic] to all three sites. Is Jim still working on these?

Thanks."

That is confirmation of the systemwide SCADA failure on the 27th.

Prime Controls' Baker wrote back at 4:29 PM:

"Yes Jim is there. When Deep South got the microwave working there is a conflict in Communications.

Jim is working with Randy on the phone till Randy gets here."

"Jim" is a Prime Controls employee. "Randy" is Prime's Randy Oliver. "Deep South" is very likely Baton Rouge-headquartered Deep South Communications, who specializes in microwave radio communications. Baker is referring to the backup microwave towers (one at each gate structure and one at each city pump station) that are supposed to allow communications if the main fiberoptic communications cables along the canal beds fail. The fact that Deep South had to come out and get "the microwave working" indicates that not only did the primary fiberoptic system failed, but the backup microwave system also failed. And this was all two days before the storm even hit!

The SCADA problems continued into the evening of the 27th. At 9:09 PM, the Corps' London Avenue canal captain Carl Robinson sent this:

"London Ave team status

The team reported to the site at 1800. We're standing by for orders. We're checking equipment and securing loose items. Station is operational with SCADA Level indicator problems at PS3, Harrison, Mirabeau, Filmore, Prentiss.

Team members on site:

Carl Robinson
Randy Faherty
Kevin Calico
Larry Trout
Shane Thames"

Robinson's August 27th email puts the lie to Corps reassurances about the severity of level gauge failures during Isaac. Those statements are within reporting about the level gauges done by Steve Meyers at the Lens in October, 2012. At that time, the Corps said:

"Moreover, the corps was getting more timely, reliable information than was available to the public on Rivergages.com. Gauge readings are sent via satellite to that website every hour. The corps gets readings several times a minute via fiber-optic cable laid on the bottom of the canal. If the cable is cut, they can get readings via microwave dishes.

Ray Newman, the corps’ 17th Street Canal captain, said gauges 'were coming and going' during the storm, but for less than a minute at a time — not long enough for the water level to rise dramatically. 'I still knew what the level was,' he said."

In fact, according to Robinson's email above, the Corps' supposed better internal level gauge visibility was nonexistent on five of eight London Avenue gauges two days before the storm's peak hit.

The SCADA problems continued as Isaac came ashore. Robinson reported from the London Avenue canal at 7:17 AM on the 29th during the storm's highest intensity:

"Two problems popped up within the last hour. We are having problems with our refueling system for the 4 west direct drive pumps. The problem is an electrical problem that happened when the station loss power yesterday. We'll need an electrician to correct the problem. We have 4 to 5 hours of fuel in the engine tanks for these 4 pumps.

Next, I've lost access to SCADA at Pump Station 3. The Corps standby generators are not available due to previous work at the station. These generators are tied to the battery back up power which is depleted due to the loss of power at PS3. Its a 100 amp ckt so just plugging into an outlet won't due.

The only solution I see is for me to call in what the SWB is pumping and let the station match it. I think that Don S. Should be at the London Station in case we can't keep up with SWB. He can make the call to me to ask the SWB to cut back.

With [Corps mechanic] Randy [Faherty, normally assigned to London Avenue] going to 17th street we are short handed.

Please advise.

Carl"

We'll get to the fuel system problems in a later entry. I have to say though this email encapsulates all the things wrong with how the Corps has decided to run the outfall canals: 1) They apparently don't have someone in authority at the gates and at the city pump stations during storms; 2) after six years of construction and maintenance by the same company (Prime Controls), their data communication system still fails under even the stress of a category 1 storm; 3) the canal walls are still so fragile that they cannot put all the city's water in the canals without risking a breach, so the idea of cutting back on the city pumps is a routine way of doing business. That is, the problems that led to the flooding in 2005 have yet to be addressed.

Back to the SCADA failures. Robinson's 7:17 AM email on the 29th coincides with the first news articles describing communications and pump losses at the 17th Street canal, leading one to believe the whole SCADA system was again down. (Note: Robinson's email came over two hours before pump station 3 lost power at 9:55 AM, so it is unclear why he was talking about the Corps' backup generators). Indeed, less than hour after Robinson's email - at 8:02 AM - the Corps' Operations number two, Jerry Colletti, sent an email to all three canal captains requesting updates on which pumps at each site were working and which weren't:

"I know you guys are real busy and are busting your butts; but as always is expected, we are asked for current status of operations. So, if you can quickly shoot me an email saying which pumps are NOT (NOT) operational, I will try to take it from there. If you have total current operating capacity that you can quickly provide, that's fine. If not, I will estimate it based on the pump capacity of those pumps that are operational.

For example, [Orleans Avenue canal captain] Donald [Constantine] just had W5 go down to fire plus he had 2 others down, so I can report 10 pumps total, 9 operational [sic - shoudl read 7 operational, since two were already out for repairs before the storm hit]. If I knew the others I could estimate current capacity. You can just list them like

17-W2, 17-W4, 17-LP6, ... I can then update on my total list and keep it current."

(We'll get to the Orleans Avenue fire in the next part)
This points to a systemwide failure because if the EOC were receiving SCADA data from the three gate sites - data which includes which pumps were on or off - such emails would be completely unnecessary.

Later on, at 12:30 PM, Corps Operations Chief Chris Accardo requested level gauge information via email from all three sites, opening up a second email chain for transmission of data which should have been instantly visible via SCADA. There are pages and pages of such emails from all three canal captains Accardo - who would have been at the Corps' Emergency Operations Center inside their building on Leake Avenue throughout the day of August 29th - at the height of the storm. Those emails, sent every half hour, contain readings from level gauges on each side of the gates (i.e the only readings directly visible to Corps personnel) from about 12:30 PM onward on August 29th. Such emails should have been completely unnecessary, because personnel in the EOC are supposed to be able to see all 24 level gauge readings minute by minute, including those at the gates.

Final confirmation of the loss of SCADA data is provided by follow-up work. A few months after Isaac, the Corps gave a $3570 task order to Prime Controls to do the following:

"TASK Order 18 - Provide labor and materials (if needed) to set up SCADA system to automatically cut over to the backup Metro Ethernet Service at Pumping Station # 3 when the main Metro Ethernet Service at Pumping Station # 6 (which provides communication between ICS and Emergency Operations Center located at the District headquarters) loses the connection with EOC office."

The impression conveyed through all these emails - as well as the news coverage of the SCADA outages on the morning of the 29th that led to pumps being shut down in the city pump station at the foot of the 17th Street canal - is that the SCADA system along the outfall canals did not withstand a category 1 storm. That is a much larger loss than simply some level gauges going "in and out."

[End June 2, 2013 update]

And while the electronic failures are bad enough, there were actual physical problems that were even worse. At 5:10 PM on August 28th, long before the storm hit with its full fury, Carl Robinson reported from the London Avenue site to Corps Operations Chief Chris Accardo and Corps Emergency Management Chief Michael Stack:

"Chris/Mike

The following gates did not fully pin at London.

Gate #8 - couldn't insert neither pin. 1/2 to 1 inch off.

Gates 6,7,9, 10 only one pin inserted.

These gates were cleaned out yesterday. Looks like debris made it's way in overnight or this morning. I suggest we make several smaller pins that we can insert in an emergency. They could be secured using the existing threaded rods and nuts.

Carl"

I had actually posted a reminder post that morning which directly referenced the possibility of gates not getting secured with their pins. It wasn't idle speculation - it had already happened at least once before, in 2009, as I detailed in this 2010 post. (for those looking to catch up, there's further detail available on individual components of the gate structures including the trenches, the gates themselves, the pins, and the seals). If the gates were not properly secured, failure of the closure structures was a distinct possibility. That would allow storm surge directly into the outfall canals, with a failure of still unrepaired wall segments likely to follow.

It is the follow up emails that really drive home how nonchalantly the Corps takes its storm-protection responsibilities. Stack wrote back to Robinson just minutes later:

"Are we good to get through this event or do we need to drop bags?"

Stack is referring to the requirement in the closure structure operating manual that massive pre-staged sandbags be dropped by the rental crane in front of gate segments which cannot be secured in advance of a storm. Instead of following procedure and safeguarding the citizens of the city by doing everything possible, Accardo - the head of Operations - waived Stack off:

"We are OK, Some gates pinned on one side only, but they are only off by fractions of an inch."

Robinson chimes in five minutes later:

"I agree.
Also If we need to, we may be able to find something at the ICS to insert and secure in number 8.
Carl"

And that's that. The Corps entered Isaac with five of the eleven gates at the London Avenue site not completely secured, including one that was utterly unsecured; the Corps was relying on gravity to keep it in place. Put simply, the Corps had closed the barn door at the London Avenue canal (the weakest canal structurally speaking), but they hadn't bothered to lock it. The possibility of storm surge loosening the gates was apparently too inconceivable for the people directly responsible for making sure the city was protected. Procedures - such as calling out divers that were supposed to be on call, or failing that, dropping massive sandbags - were simply brushed off. It is in-the-moment exchanges like this that undermine whatever bluster the Corps puts out about safety being their top priority.

In the next part we'll detail the further unreported failures that occurred during Isaac, including much wider pumping problems than the Corps reported at the time.