Fix the pumps

Wednesday, July 24, 2013

Isaac: the rest of the story, Part 3

Part 1
Part 2

NOTE: I am currently awaiting more material from my Isaac email FOIA request, including a PowerPoint presentation from the evening of August 30, 2012 which was supposed to include a candid assessment of pumping at the outfall canals. In the interim, I've pulled together what I could regarding events along the 17th Street canal during Isaac. This post may change as the remaining information is received.

When I placed my FOIA request for the Isaac emails, I had hoped to not only uncover unknown events during the storm, but also to get more details on the pump failures that occurred at the 17th Street canal on the morning of August 29th. Those failures were so extreme the Corps had to report them to the media. Here's the timeline from my October 6, 2012 entry, "Isaac in New Orleans - what we know so far:"

The PS#6 logs for August 29th (here and here) confirm the outlines of this story, while filling in some detail. At 4:35 AM, the station staff asked to "load" (that is, bring to full capacity) 1100 cfs pump "F." They were refused. Then, at 4:55 AM the Corps told the S and WB not to "load" any more pumps in PS#6:
"Note: Army Corps of Engineers said we cannot load any more pumps. [T]hey are at their max of what they can pump"
At that time, the station was pumping approximately 6200 cubic feet per second, or cfs. The station's capacity among its main rainstorm drainage pumps is 9200 cfs.

This initial notice to the S and WB to hold off on pumping appears to be 1.5 to 2 hours before the Corps alerted the media of pumping restrictions, according to the chronology of the NY Times article. Then, at about 6:30 AM, the station operators received another Corps warning not to load additional pumps beyond what was already running, despite the fact that in the intervening hours, water in the station's inlet basin had risen over four feet. Ten minutes later, the station received orders from S and WB's Central Control to break the prime on 1000 cfs pump "D," effectively losing most of its capacity and sending into a pump status called "light." So there were not only Corps orders to keep some pumps from running, but one pump was effectively ordered turned off with the water coming up. This left the station running at about 5200 cfs.

At 7:55 AM, with the water still very high in the inlet basin, the Corps 17th Street canal captain in the station (Ray Newman) gave an all clear to the S and WB personnel in the station to load additional pumps. Five minutes later, 1000 cfs pump "D" was authorized to be loaded by the S and WB Central, and was running in a loaded condition within a few minutes. Another 1000 cfs pump ("C") was brought on line shortly afterward, increasing the station's flowrate to 7200 cfs. Inlet basin levels dropped four feet in about an hour after those two pumps were added. This chronology roughly tracks what was playing out in the media that morning.

Oddly, there's zero Corps email traffic among the key personnel (Ray Newman, Don Schneider, and Chris Accardo) through that whole time. I would expect something to have been written by someone over that three-plus hour span, but perhaps they were just so busy they didn't have time to type on their Blackberrys. Or perhaps they were all communicating either over the radio or in person, so electronic communication was unnecessary. When there was a pump drive skid fire at the Orleans Avenue site a little over an hour after this event subsided, there was an email trail immediately created, so it's very strange there's no trail for the biggest event during the storm.

How high did the water get in the 17th Street canal on August 29, 2012?

The first question to be asked after an event like this is whether the 17th Street canal maximum operation water level of 6.5 feet (first revealed to the public during the press coverage of this mishap) was exceeded. If it was, that means the walls and levees were loaded beyond their safe capacity. To determine this, normally we would examine the outfall canal level gauge records kept at the Corps' rivergauges.com website, but the Corps' level gauges along the canal were not responding reliably on August 29th.

However, there is another level gauge that did not fail: the Sewerage and Water Board's gauge at drainage pump station #6, located at the southern end of the canal.

That gauge uses a different datum for its zero point, an antiquated thing called "Cairo Datum." It must be converted to normal water levels, and sometimes figuring that conversion out is difficult. I've gotten different responses over the years. However, I just recently found the definitive source for the conversion. Take a look at this photo:


This was taken by an architectural graduate student affiliated with a group project called "Gutter to Gulf." They visited New Orleans in February, 2009 and stopped in at pump station 6. One of them snapped this photo inside the pump station control room. The digital readout on the right shows the water levels on the inlet (top row) and outlet (bottom row left) of the station. Of far more interest is the paper above that display:


There's our conversion factor for the discharge water levels at DPS#6: 6.0 feet in normal terms equals 27.6 feet in Cairo Datum terms. The safe water elevation was raised since 2009 to 6.5 feet, which was the number in effect when Isaac hit. So anything over 28.1 feet as measured by the DPS#6 discharge water gauge indicates the 17th Street safe water elevation was exceeded.

Examining the DPS#6 discharge level gauge readings on the morning of August 29, 2012, as recorded in the station's logs:



The logs show the discharge gauge read above 28.1 feet for almost two hours, at one point exceeding it by a full 0.5 feet, or six inches. That is, the water got to 7 feet in the canal, placing the walls and levees in danger. This is the first time such information has been reported publicly.

How bad were the Corps pump failures?

The next question to ask is how this happened. We know there were sizable pump failures at the Corps end of the canal, but how sizable were they? Let's go back two days before the storm.

Despite Corps Operations chief Chris Accardo's assurances to the locals and others in the days leading up August 29th, all was not well at the Corps' 17th Street site. Just before noon on August 27th, the Corps' 17th Street canal captain Ray Newman sent an email to Kirk Bowman of Point Eight Power in Belle Chasse, LA, in nearby Plaquemines Parish. Point Eight was (and remains) responsible for the pump controls on the 11 direct drive pumps at 17th Street and the 8 direct drive pumps at the London Avenue canal site. Newman's email reads,

"Kurt,
This message is your authorization to have your field service tech. override the shutdown safety devices at our request, as listed below. These are temporary measures to provide vital pumping capacity for our operations during TS/Hurricane Isaac. Our operators will monitor the equipment manually while these shutdowns are inactive.

a) Override low gearbox oil pressure shutdown on Unit No. 19, 17th St. Canal
b) Override gearbox vibration shutdown on Unit No. 18, 17th St. Canal

Thanks,
Ray"

This email is revealing in a number of ways. On the surface, it points to problems on two of the supposedly more reliable direct drive pumps just ahead of a hurricane. Overriding the automatic shutdown commands could point to the Corps wishing to temporarily bypass a nuisance interlock that was not indicating a real problem. That's the most generous explanation. The more likely - and less charitable - explanation was that pump W19 was experiencing low oil pressure on its' gearbox and pump W18 was experiencing high vibrations on its' gearbox. The Corps needed these pumps to run, so they told Point Eight to pull those interlocks out of the program.

But in a greater sense, it points to how thin the Corps itself is on expertise in their own equipment. For an experienced industrial programmer, bypasses like these should be a piece of cake. But the Corps obviously has no such person on staff. They had to summon a contractor to make programming changes on the eve of a major storm. We see this over and over - the Corps themselves doesn't just contract out duties like engineering, quality control, and quality assurance. They also farm out their emergency response activities under the pressure of a storm, leaving until the last minute the call to these folks, hoping the companies have decided to keep their employees around. The Corps can order their own personnel to stick around; their power over contractors is much more murky and lends a sense of unease to citizens depending upon the correct functioning of life-critical storm protection systems.

But back to the events surrounding the morning of August 29th. As noted above, S and WB drainage pumping station 6 was limited to 5200 cfs at one point. This means the Corps pumps were likely pumping at approximately the same rate. What does that mean in real terms?

Here's the layout of the pumps at the Corps 17th Street site:


Here's the numbering of each pump at 17th Street:



According to the Corps' "pump tracker" spreadsheet, the 18 "Phase 1" and "Phase 2" pumps, which are the 60" hydraulic pumps made and supplied by MWI, each have a nominal rated capacity of 200 cfs. The 11 larger direct drive pumps (abbreviated "DD" in Corps pump status emails) each have a nominal rated capacity of 364 cfs. The 14 "Phase 3," or "bridge," pumps - also hydraulically powered units made by MWI - each have a nominal rated capacity of 114 cfs.

We can get an idea of what was going on early on the morning of the 29th by examining what happened afterward. You see, the Corps' pump problems did not disappear at 8 AM. The problems just got small enough, and the rainfall lessened up enough, that the Corps pumps could barely squeak by. In fact, emails subsequent to the morning crisis reveal substantial problems remained. Just after 10 AM, Corps Operations Assistant Chief Jerry Colletti sent out an email requesting pump statuses from all three gate sites.

At 11:20 AM, the Corps' Michael Sullivan sent back the first update for 17th Street:

"No change.
ICS Capacity - 7700cfs
PS6 Capacity - 6200 cfs, will increase to 7300cfs w in 10 min."

Unfortunately, we don't have what Sullivan had possibly previously sent when he referred to "no change." Also, it is unclear if Sullivan, when he wrote "ICS capacity," was referring to the maximum capacity the Corps' station could pump, or what the station was currently pumping. However, the lack of clarity was eliminated in subsequent updates. A 12:03 PM email said:

"3 phase 1, 1 DD, 3 Bridge - DOWN.
ICS - 7894cfs
PS6 - 6200cfs, going to 7200cfs in 5."

So with 3 phase 1 pumps, 1 direct drive pump, and 3 bridge pumps down at noon on August 29, 2012, the theoretical capacity of the Corps 17th Street ICS was 9200 minus (3 times 200 + 1 times 364 + 3 times 114), or 7894 cfs, which matches what Sullivan reported. Put another way, the Corps's pump station - which does not have any spare capacity when the nearly 10,000 cfs city pump station #6 is going all out - was only running at 86% at noon on the 29th

What that means is that if the city pumps had needed to put more than 7894 cfs down the canal (and they can, and have before), the crisis would have restarted again.

From this sort of information, we can backtrack and figure out how poorly the Corps pumps were doing between 6:30 AM and 8 AM. We know they had 7 pumps down at noon and that resulted in a capacity of 7894 cfs. Let's round that to 7900 for ease of calculation.

Between 6:30 and 8 they couldn't pump more than between 5200 and 6200 cfs, or an additional 1700 to 2700 cfs deficit. At a minimum, this means at least 5 additional direct drive pumps, or some combination of an even larger number of pumps - some with smaller rated flows - were out of commission at that time.

Assuming the pumps offline at noon were also down earlier in the morning, this would mean at least 12 - a dozen - and possibly many more pumps were not available for nearly two hours during the height of Isaac. Given there's 43 pumps total at the site, just on a raw pump count basis that means nearly 28% of the pumps were non-functional. On a flowrate basis, it means the Corps was down between a third and nearly half of their rated throughput.

Nearly third to a half the Corps' pumping capacity was offline for hours... at the very time such pumps were needed. That allowed the 17th Street canal walls to have their safe level exceeded for hours as well. Let that sink in. At the moment the Corps was supposed to be providing maximum storm protection through pumping, they were failing so miserably the outfall canals were threatened with another breach.

Pumping failures continued throughout Isaac

Frankly, it was pure luck there wasn't more rain that day, and that is what saved the Corps' (and by extension, the city's) bacon. Because while the Corps got some of their stuff in order, after noon the news got worse at the 17th Street ICS.

At 2:04 PM, they lost another phase 1 hydraulic pump:

"4 phase 1, 1 DD, 3 Bridge - DOWN.
ICS - 2900 cfs
PS6 - 3000 cfs"

[Note this flow report for the ICS refers to Corps pumps turned on, not those actually ready to run]
Again - I cannot say this enough - it was pure luck that the city was only sending 3000 cfs down the canal at that time. The loss of a fourth hydraulic pump dropped the Corps' pumping capacity to 7694 cfs, or 83.6% - its lowest point outside the crisis period in the morning. The capacity at pump station 6 remained at its maximum of 9300 cfs - every city pump was available. The Corps pumps would never reach their maximum capacity during Isaac.

That situation continued for about 4 hours, during which the city fortunately pumped a steady 3000 cfs down the canal. The 6:01 PM update showed some improvement on the Corps end of things:

"(2) Phase 1 pumps out (change), (1) DD pump out, (2) Bridge pumps out (change)
17th ICS: 2700, 0.5 gage
PS 6: 2200, 0.7 gage"

So they got 2 of the phase 1 pumps and one of bridge pumps going, raising the ICS capacity to 8208 cfs, or 89%. The ICS remained at that pumping level until at least 10 PM.

Then, at 11:43 PM things appeared to get worse again. Donald Schneider sent the following email:

"11W and now 20W are now malfunctioning the same fault. Start the engine the point 8 screen goes to reboot and the engine dies. Never saw this one before. Gotta let point 8 wrestle this one its programing."

Once again, the Corps was dependent on a contractor - Point Eight Power - to make things run. This time - it was during the storm. As soon as they ran into the smallest problem that appeared to be "computerey," they were forced to throw up their hands and send up the bat-signal because they had no programming knowledge.

This problem is possibly the source of a report from WWLTV titled "17th street canal pumps forced to be started manually Tuesday night" Most of the article is a confused rehash of what had happened Tuesday morning. But the first line says:

"Pumps at the 17th Street canal failed to turn on automatically Tuesday night, forcing pump operators to manually start the system, the Corps of Engineers said Thursday."

Notice how the emergency role of contractors in bypassing the pump control programming was not even mentioned.

The final report on pump statuses came the following morning at 5:06 AM

Pumps out: 1 Phase 1, 2 DD, 2 Bridge
17th:1400 cfs, 0.2 gage
PS6: 2200 cfs, 0.2 gage

So even then, the Corps was down 5 pumps. This and the other updates through the afternoon and evening of August 29th completely contradict reporting by the New York Times' John Schwartz, who claimed regarding the pump status at 8 AM on the 29th: "Within a couple of hours, workers would get all of the pumps running and bring the water levels down, and the sense of potential crisis eased." The Corps never got "all" the pumps running. They got enough running at 8 AM to barely do the job at the same time rainwater in the city's drainage system was slackening off.

Other stuff going on at the same time

The events on the periphery of the Corps failures at the 17th Street canal - and they were failures, which were eventually only remedied by the luck of the weather - are just as interesting. I'd like to highlight two email chains from that day, both of which only came to light with the receipt of Chris Accardo's emails.

The first was an email chain involving the then-New Orleans District commander Colonel Ed Fleming. Fleming has since moved on to Corps HQ in Washington. That morning, Fleming responded to an inquiry from the Army in Washington DC asking what was happening at 17th Street:

"- We never had to tell SWB to slow down. They wanted to bring another 500cfs on line and we asked them to wait a little while - which they did. We started the rest of the pumps manually. Then we told them to give us everything they could.
- Our pumping percent capacity is interesting but not determining. Our goal is to stay ahead of SWB. If we are pumping at 80% capacity and they are pumping at 50% capacity then we are in good shape. Our goal is to out-pump them."

I'll deal with the misinformation in the first passage later. But first, I'd like to opine on the second statement that the Corps' "goal is to stay ahead of SWB." This reeks of the "it's close enough, we don't need any extra" philosophy that appears to permeate the New Orleans District. They always appear to be looking for a corner to cut or a penny to pinch, designing or building or operating things to the gnat's ass as if there's never going to be a circumstance where extra anything will be needed. Meanwhile, they ignore gaping holes (sometimes literally) and try to cover them with duct tape and press releases.

What happens when the SWB is pumping at 80% of capacity and the Corps only has 50% available? You get the freak-out we saw on the morning of August 29, 2012. The goal for the Corps' pumps should not be just to stay ahead of the SWB, because no one can predict the weather. The goal should be 100% reliable pumping at any time, for any duration, with spare capacity built in to overcome unforeseen failures. The SWB pump stations were built this way, but the Corps decided to go with "probably good enough" and "trust us." Their calculations for safe water elevations within the canals are of the same flavor - they only calculated the strength of the walls up to 10 feet, even though the walls are actually 15 feet or taller. What would happen in the case of a failure of a gate segment at the lake and the inrush of storm surge, as easily could have happened at London Avenue with the unpinned gates? Would the Corps say "we built stuff good enough to handle a scenario we deemed most likely, despite the dire circumstances a less likely scenario would present?" Phrased another way, that question would be "Katrina? Never heard of her."

At the very least, these massive pump failures show all the supposed exercising of pumps during and outside of storm season is clearly garbage. Under fewer than 12 hours of stress, their system nearly collapsed. The ability to run a single pump on a sunny day for ten minutes in front of TV cameras and a few dignitaries (as the Corps does every May) simply cannot compare with the need to run 43 pumps on the rainiest day of the year for 12 to 24 hours.

It's like someone telling you, "It'll work until it breaks." We saw how well that philosophy worked on August 29th, 2005, and it does not appear to have been excised from the New Orleans District at all.

Now, to the half-truths in Fleming's first passage. We know the Corps told the SWB to shut down 1000 cfs pump "D" in PS6 at about 6:40 AM on August 29th. That came after a an earlier refusal - at 4:35 AM - to allow the SWB to run 1100 cfs pump "F." A second refusal on "F" came at 6:50 AM. In addition, the SWB was told not to load any more pumps above the 5200 cfs they were flowing at 6:40. That's a potential 4000 cfs kept out of the canal, not 500 cfs.

Also, the first refusal came at 4:35 AM. The SWB was not allowed to turn on any pumps until 7:55 AM. That's well over three hours, not "a little while."

The second email chain of interest involves Corps engineering personnel within and outside the New Orleans District on August 29th.

At 9:33 AM, Nancy J. Powell, Chief of the Hydraulics and Hydrologic Branch at the Corps' New Orleans District, emailed Walter Baumy, the then-chief of the New Orleans' District's Engineering Division. Also copied was David Ramirez, the New Orleans District Chief of Water Management. By this time, multiple media reports were already mentioning the faulty level gauge readings within the canals, readings available to the public at rivergauges.com. During storms, Powell is a point person for storm modeling data, including predicted surge levels. She wrote:

"David can call MVR and see if they can stop malfunctioning gages from reporting"

"MVR" is the Corps' Rock Island District, headquartered in Rock Island, Illinois. They run the riverguages website. Powell was attempting to find out if the Corps could stop the bad readings from being reported to the public.

A little over two hours later, at 11:48 AM, Baumy forwarded Powell's email to Denny A. Lundberg, the engineering chief at the Rock Island District, with the additional message:

"Denny, potential issues on outfalls, especially 17th. Public calling on safe water elevations."

Oh no! The public was calling about safe water elevations! We can't have that.

Lundberg wrote back about an hour later, at 12:54 PM:

"Whoops

No one has called MVR as of an hour ago. Any station can be easily turned off using the RiverGages management page. Takes about 30 seconds. I'll have my WC [Water Control] staff contact your staff."

"Whoops" indeed.

Baumy wrote back a few minutes later with the last message we have, copying Chris Accardo, Powell, Ramirez, Emergency Operations honcho Michael Stack, and - most tellingly - two members of the New Orleans District's Public Affairs staff: chief Ken Holder and contractor Rachel Rodi:
"If outfall gages are unreliable, they should be turned off in my opinion.

Will need to defend the unreliability as public will want access. Please provide your assessment.

Walter"

Based on the records on the rivergauges website, the Corps never carried out this plan. And with squinty eyes, I can see a sorta justification for not putting messed up information out on a public website. But the overriding concern here doesn't appear to be misinforming the public - or even what was being done to address the problem (something one would expect Baumy, the head of engineering for the District, to be asking), but rather exposure of the Corps' missteps to public scrutiny in real time. Having seen the Corps' penchant for keeping outfall canal information away from the public, my view may be a bit colored, but the plain language of the email exchange certainly doesn't cover the Corps in glory and worry for the flood-endangered public.

Summary

The events at the 17th Street canal on August 29, 2012 appear to be more serious and longer lasting than the reporting that morning and afterward indicated. The water level in the canal exceeded the Corps' safe water elevation for over two hours in the morning because approximately a dozen Corps pumps failed and could not remove the rainwater being put into the canal by city pumps. The Corps' pumping capacity was cut nearly in half at the height of the need for pumping. Those Corps pump failures prompted orders to the locals to not activate some pumps and in one case to turn one off, causing stormwater to back up into city streets.

After the heaviest rain abated and the Corps got a handfull of their pumps working, water levels inside the canal and in the city streets dropped. However, many Corps pumps continued out of commission throughout the event, raising the probability that if rainfall had again picked up, a crisis identical to that of the morning of August 29th would again occur.

Occurring in parallel with these events were Corps efforts to shape the narrative, often with faulty information. One effort even contemplated cutting the public (and the local pumping authority) off from level gauge data.

Strictly due to cost, the Corps has decided to make this tandem pumping system with weakened canal walls permanent, their so-called "Option 1." Ground has broken for permanent pump stations to replace the clearly faulty Interim Closure Structures, but no information has been released regarding further work to make the canals themselves sturdier. Taken together, the outfall canals remain a serious Achilles heel within the confines of the City of New Orleans and Jefferson Parish.

Monday, June 24, 2013

Isaac: the rest of the story, Part 2

In Part 1 of this series, we reviewed what internal Corps emails written during 2012's Hurricane Isaac revealed about failures of the gate hoists at the Orleans Avenue canals and the SCADA system across all three outfall canal sites. We also saw how Corps personnel decided to not secure the gates at the London Avenue canal, in contravention of their own procedures.

In this entry, I'm going to focus on events along the Orleans Avenue canal. As we've mentioned before (including on the day Isaac started making its presence known), the Corps' Orleans Avenue gate structure was down to 80% of its rated pumping capacity as Isaac moved ashore because two of its ten pumps were rusted out. They had been yanked out weeks before Isaac. I thought it was pretty clear this would cause problems, since even with all ten pumps installed at the gates, the city Sewerage and Water Board (SWB) pump station #7 at the canal's inlet can put out about 2700 cubic feet per second (cfs) of rainwater, while the Corps structure at the canal's outlet can only pump 2000 cfs into the lake. Take two pumps out of the Corps end of things, and it is a guarantee water will climb in the canal if the SWB station pumps as hard as it can. If water got too high (i.e. above the maximum operating water level), it was likely the Corps would turn off Sewerage and Water Board pumps that drain the city's streets. That was the setup going into Isaac.

Corps internal allowable water levels lower than what the public was told

Let's go back to the previous coverage around the Orleans Avenue canal. There was the Lens article on August 27, 2012 about those two rusty pumps, which included these quotes from Corps New Orleans District Operations Chief Chris Accardo:

"Further, Accardo said that the Orleans Canal has the highest-rated water level of the three.

'We’ve never had to close the (Orleans Avenue) gates during a storm. Not even during Ike, not even during Gustav,' he said. 'So the threat of these pumps being down and causing any problems is extremely remote. There is hardly any threat to City Park or Mid-City.

'We are not where we want to be at Orleans currently. I would like to be a little higher, but it is not something people need to be worried about,' he said. 'My standard is to have as many pumps functional at all times.'"

That "highest rated water level" is supposedly 8 feet. If water gets above that level (the walls are 14 feet tall) bad things could start happening.

Accardo sent an email to local stakeholders and a number of Corps personnel at 9:19 AM on August 28 with a summary of pump statuses at the three outfall canal sites (this email was part of the first batch linked in Part 1 of this series). Included in that email was this:

"Orleans Ave - Max. Water Level is 8.0. The Corps has 2 pumps that are down. With 100% pumping capability, the Corps has a rated capacity of 2200 cfs. Since 2 pumps are down, the rated capacity is 1800 cfs. The SWB reports that they have a 2,700 cfs rated capacity when all their pumps are working. The SWB is currently at 100%. Since the maximum water elevation is higher at Orleans [maximum at 17th Street is 6.5 feet, at London Avenue 5 feet - ed.], we have never dropped gates before at Orleans. However, due to an electrical problem, the gates had to be lowered with a crane last night. The canal is currently at elev. 1.5 with the gates closed and we have a corps employee at the SWB pump station. We will turn pumps on and try to maintain a 1.5 canal elevation as soon as the SWB turns their pumps. Even though the SWB has greater pumping capacity, they will have to out-pump the Corps such that the water level in the canal will rise 6.5 ft. before there would be a problem. I believe the chances of that happening is remote. If it did happen, we would request that the SWB reduce pumping so as to preserve the maximum water level."

As we learned from the Corps' internal emails, this was untrue. Accardo sent another email at 9:18 PM the same day which included a spreadsheet with the real maximum water levels. This is it:

First off, the max water level at Orleans Avenue is not 8 feet, it's 5 feet (ignore the "trigger" language), because above that level the SWB pump station floods (!). That's as low as the the very weak London Avenue canal, which has many sections of very short sheet pile in sandy soil that have not been upgraded.

So right off the bat, we know the Orleans Avenue canal, even if it had all its Corps pumps, would be a trouble spot. Down two of those pumps, it would be even worse.

Now, thanks to additional emails I received this past Friday, we have conclusive proof of exactly how little faith the Corps has in the Orleans Avenue canal.

Corps actually runs Orleans Avenue canal at water levels even lower than their internal numbers

Donald Constantine is a member of the Corps Operations team. During tropical events, Constantine is assigned as the "canal captain" for the Orleans Avenue canal. This means he monitors all aspects of the Corps' side of the the canal operations (the pumps at the lakefront and the level gauges along the canal length). He is stationed in SWB pump station 7 on Marconi at the south end of the canal, so he has a direct contact with with SWB pump station operators. This was the situation during Isaac.

As I detailed back in October, those operators were refused permission to turn on SWB pumps at various times during Isaac, presumably under direction by the Corps' Constantine. The number of refusals documented by the SWB operators in their logs for August 29 and 30 - four - was surprising, given the supposed large amount of storage capacity (or "freeboard") in the canal. At the time, I thought it was evidence the eight Corps pumps could not keep up with the SWB pumps, but I didn't then know about the 5 foot restriction.

As it turns out, the Corps kept the level in the Orleans Avenue canal even lower than that.

I want to focus on the final SWB pumping refusal of Isaac, which took place in the wee hours of the overnight between August 29th and August 30th. At the time, pump station 7 was putting out its maximum amount of flow from its three main pumps. Pump "A" was flowing 550 cfs, and pumps "C" and "D" were flowing 1000 cfs each, for a total of 2550 cfs (there is no pump "B"). "A" had just started flowing at 12:40 AM after water in the pump station inlet basin had started rising. At 11:30 it was 10.1 feet, at midnight it had gotten up to 10.7 feet, and at 12:30 AM it was 11.0 feet. While not as high as during the late morning on the 29th (when the inlet basin had climbed to over 15 feet during a SWB power outage), it was still high enough to make the SWB operator feel it was necessary to run all three main pumps.

The SWB pump log shows the following at 12:55 AM on the 30th:

"O/ Donald Constantine w/C.O.E. Break prime on C pump. N/cc"

That translates to "Ordered by Donald Constantine with the Corps of Engineers to break prime on (effectively turn off) C pump. Notified SWB central control on South Claiborne Avenue"

Thanks to the new emails, now we know exactly what led to Constantine's order. He had sent the following email to Accardo about an hour and half before the refusal order, at 11:22 PM on the 29th (emphasis mine):

"From: Constantine, Donald A MVN
To: Accardo, Christopher J MVN
Subject: FW: Gage Reading Orleans (UNCLASSIFIED)
Date: Wednesday, August 29, 2012 11:22:24 PM

Classification: UNCLASSIFIED
Caveats: NONE

Chris - we are raising the level in the canal. PS#7 asked to add their last 1000 cfs pump to drop the suction side water level. I am monitoring the levels throughout the canal and we will discontinue one of the pumps when we hit 3 to 4 foot. In 30 minutes they dropped their suction by more than a foot.

The canal has been raised by 1 foot (PS#7 - 2.57, ICS - 2.22).

Even though they are pushing more water than we can handled [sic], we still have the freeboard in the canal to handle the rise.

Donald"

For context, according to the August 29, 2012 PS#7 log, pump "C" (the "last 1000 cfs pump" referred to in Constantine's email) had been turned on at 10:45 PM, and - as Constantine noted - the inlet basin levels dropped from 11.2 feet at 10:30 PM to 10.1 feet at 11:30 PM. Of course, there was a corresponding increase in water levels in the canal, which is what concerned Constantine. For an unexplained reason, 550 cfs pump "A" was dialed back at 11:35 PM, likely because the inlet basin level had gotten low enough for the SWB operator's comfort. When it started rising again an hour later, "A" was restarted, maximizing the flow out of the SWB station. The SWB pump station 7 outlet basin level got to 23.5 feet (Cairo Datum). This is approximately equivalent to 3.4 feet on the Corps' level gauge (another Constantine email provides the conversation factor: subtract 20.1 feet from the SWB canal levels) resulting in Constantine's order to shut down pump "C," just as he said he would.

Constantine's email is remarkable on a number of counts. First, it details how low the Corps actually kept the water level in the Orleans Avenue canal during a tropical event: 3 to 4 feet. This is half of what the Corps was telling the public and their supposed local parters less than 48 hours previously, and is two feet lower than what they were even saying to each other internally just two hours earlier. For perspective, remember that the tops of the canal walls are actually 14 feet tall.

Secondly, it completely contradicts what Accardo had said in his email to the locals earlier. Constantine admits the SWB "are pushing more water than we can handle," and by virtue of his 3 to 4 foot cutoff, cuts the amount of freeboard available (think of it as head space) from Accardo's 6.5 feet to a maximum of 2.5 feet.

Third, it is abundantly clear there is no partnership between the Corps and the SWB; the Corps orders the SWB what to do during tropical storms, and it does so according to its own, secret criteria unknown to the public, or perhaps even the SWB. Note there are no cc's on the email, to the SWB or even to any other Corps personnel. In fact, I did not even receive this email as part of my request for Constantine's emails. If I hadn't asked for Accardo's emails, I wouldn't have even known about it.

So let's review:

August 27th, Corps Operations Chief Chris Accardo comments to the Lens about water levels in the Orleans Avenue canal: "it is not something people need to be worried about."

August 28th, 9:19 AM, Accardo emails to local stakeholders: "Orleans Ave - Max. Water Level is 8.0 ... We will turn pumps on and try to maintain a 1.5 canal elevation as soon as the SWB turns their pumps. Even though the SWB has greater pumping capacity, they will have to out-pump the Corps such that the water level in the canal will rise 6.5 ft. before there would be a problem."

August 28th, 9:18 PM: Accardo forwards spreadsheet internally to Corps personnel showing 5 foot maximum operating water elevation "trigger" on Orleans Avenue canal, caused by potential flooding of SWB pump station at that level.

August 29th, 2:20 AM: Corps Orleans Avenue canal captain Donald Constantine denies SWB's request to turn on 1000 cfs pump "C" in SWB station 7. Canal level is at about 3 feet at PS#7. PS#7 inlet basin (an indication of how much water is heading to the station from the streets) measures approximately 12.0 feet deep.

August 29th, 3:05 AM: SWB again denied pump "C." Canal level is about 3.1 feet at PS#7. PS#7 inlet basin measures approximately 11.3 feet deep.

August 29th, 4:45 AM: Heavy rainfall is sending lots of water toward the SWB pump stations throughout the city. SWB at PS#7 is again denied pump "C." Canal level is about 3.1 feet at PS#7. PS#7 inlet basin measures approximately 13.5 feet and rising sharply (it would climb another foot fifteen minutes after this). That represents water in the streets drained by PS#7. Pump "C" remains off (in part due to SWB power problems) the next 18 hours until...

August 29th, 10:45 PM: For the first time during Isaac, all three major pumps - including "C" - are turned on at PS#7.

August 29th, 11:22 PM: Constantine emails only Accardo, reporting rising canal water level. He informs Accardo he is keeping the canal level between 3 and 4 feet and will turn off a SWB pump if the level gets beyond that. Says the SWB is "pushing more water than we can handle" but "we still have the freeboard in the canal to handle the rise." Note this is within minutes of the SWB first turning on all their pumps.

August 30th, 12:55 AM: SWB ordered by Corps' Constantine to turn pump "C" off. Canal level is 3.4 feet. PS#7 inlet basin measures approximately 12.2 feet. The inlet basin would ultimately climb to 13.6 feet before starting to come down with a lessening of rainfall.

As we can see, the Corps' method of operating the Orleans Avenue canal during Isaac was not to have its pumps keep up with whatever the SWB could send their way. In fact, it was quite the opposite. They insisted the SWB keep one of their biggest pumps off whenever it was most needed. When that pump did finally get turned on, the Corps had the SWB turn it off less than two hours later. They even memorialized their intention to do so in an email just 37 minutes after the pump came on! This is because the Corps was operating the canal at a water level of about 3 feet, not the 8 feet they claimed to the public and their local stakeholders. That is, the Corps was actively keeping water out of the Orleans Avenue drainage canal during the entire time Isaac was over the city.

So that's the very wide difference between what the Corps presents to the public and what actually happens. Let's look at another aspect of Isaac along the Orleans Avenue canal, the fire at the Corps' pumps on the morning of August 29th, during the heaviest rainfall.

The Corps Pump Fire During Isaac

I first got wind of that fire from the Sewerage and Water Board's pump log for pump station #7. At 9:40 AM, the log reported:

"Donald Constantine w/C.O.E. reported a Fire w/C.O.E pump"

The Corps' emails provide more detail. Constantine sent this email at 9:25 AM on the 29th:

"[Corps mechanic] Robert Gauthreaux just reported that the turbo on pump W5 caught on fire. Fire was put out and pump is being checked out for operability.
Orleans is currently down to 7 pumps at this point."

About an hour later, at 10:21 AM Constantine provided follow-up

"Looks like a hydraulic leak spraying on turbo that caught fire. Engine will run after short term fix of leak.
Crew working on it."

At 11:08 AM, the problem had been addressed, according to this email from Constantine:

"Pump W5 has been repaired and runnIng again. We are back to 80 percent."

Recall two of the 10 hydraulic pumps at the Orleans Avenue site had been pulled out before Isaac for corrosion rebuilding and had not been returned to the site by August 29th. This left just 8 of 10 pumps, or 80%.

So what exactly happened?

A few weeks after Isaac, Sandy Rosenthal of levees.org was given a tour of the Orleans Avenue gate structure by the Corps. She took pictures, including this one titled "piece w/ fracturedfailed:"

What you are looking at is an Omega Engineering Incorporated model PSW-194 hydraulic pressure switch, with some hardware screwed into the left hand side of it. You can see a picture of the entire switch at Omega's page here. Specifically, this is the switch that was installed on the engine skid for pump W5 at the time of the fire on August 29, 2012. It was meant to switch on or off at a specified hydraulic fluid pressure, sending a signal to a butterfly valve on the skid to divert hydraulic fluid to a bypass/recirculation line, thus decreasing the pressure of the fluid going to the W5 water pump in the canal. Further details on how the engine skids work can be found at this post. Such switches are installed on all 40 hydraulic pumps across all three gate sites.

The switch appears to be missing a piece to the right of the label. Ms. Rosenthal took a picture of an undamaged switch on the engine skid for pump W4:


Here's the two switches side by side:


The blue highlighted area is the portion of the switch missing from the W5 unit. I've drawn in what appears to be the line where the switch may have fractured.

I want to caution I am proceeding strictly on the photographs, their titles, and Ms. Rosenthal's relation of her October 18, 2012 conversations with the Corps via email. She said the Corps told her the fire was caused by "a fracture, a flaw in the fitting." She said the first picture above - showing the pressure switch - was a picture of "the piece that failed."

I contacted Omega, asking for clarification or any backgrund on the failure, but their in-house counsel B. Christine Riggs refused all comment, saying only, "While we appreciate your effort to obtain further information, Omega does not disclose information on its customers and their orders to third parties."

In a follow-up conversation with Corps New Orleans District Public Affairs chief Ken Holder last week, the Corps confirmed the substance of their October statements to Ms. Rosenthal, saying the fire was caused by the Omega switch. Holder said the failure would better be described as an "internal failure," likely from a seal or membrane, rather than a "fracture." This leaves me somewhat mystified what I'm looking at in the pictures, but it does get me 90% of the way to the nut of the story: the fire was caused by the hydraulic switch failing.

According to Ms. Rosenthal, the Corps said pressurized, very hot hydraulic fluid sprayed out the switch as a mist on to the switch's flexible conduit and the adjacent diesel engine turbocharger, also known as a "turbo." The conduit melted and the insulation caught fire. Ms. Rosenthal took pictures of the damage:




Here's a picture of the undamaged turbo on the W4 skid for comparison:


The fire was put out using a fire extinguisher in about a minute. The approximate pressure of the hydraulic fluid at the time was about 2500 pounds per square inch, or psi (indicating the water pump was not pumping at full speed, which would correspond to about 3000 to 3200 psi). According to the manufacturer's information, the switch is rated for pressures up to 3000 psi.

In late September, 2006, while the pumps were still very much in a "testing" phase, the Corps requested from the pumps' manufacturer - MWI of Deerfield Beach, Florida - the addition of hydraulic pressure switches to all 40 units across the three sites. After some back and forth, the Corps gave the go-ahead to MWI on October 18, 2006 as modification 23 to the original pump contract (entire contract here). One can assume the switches in 2006's modification 23 are same ones in service on August 29, 2012. Holder confirmed the Corps has not changed the style or make of switch since 2006.

Notably, a few months prior to the hydraulic pressure switch modification - in June and July of 2006 - the Corps had requested the installation of hydraulic pressure transmitters and temperature on the pump skids as add-ons to the original design. It is unusual that the pumps were originally ordered with no electronic capability to monitor hydraulic temperature and pressure, or with the capacity lent by the hydraulic pressure switches, which was presumably to keep the system from overpressurizing. Given the sensitivity to the entire hydraulic system at higher pressures (around 3000 psi), these switches were likely meant to keep the pumping systems from failing.

So with all that background in hand, let's do another review of the Corps' contemporaneous public statements about one of their pumps catching fire during a hurricane in New Orleans:

"                                                                                                                  "

Oh right. They never said a word. Just like they never let the public know they'd be operating the canal at a water level 5 feet below what they said publicly, and 11 feet below the tops of the 14 foot high walls.

That's just how they roll. You almost have to admire their ability to keep stuff hidden, even with two reporters and a Senator looking over their shoulders (the NY Times' John Schwartz, the Times-Picayune's Katy Reckdahl Rebecca Mowbray, and Senator David Vitter were all "embedded" with the Corps August 28th and 29th). It appears to be the only thing you can count on happening.

Next up in Part 3: the events along the 17th Street canal during Isaac (which even the Corps couldn't hide), along with more behind-the-scenes shenanigans with the level gauges.

Note regarding the emails linked in this part:
The 8/29/12 11:22 PM email from Donald Constantine to Chris Accardo quoted above is part of the latest tranche of emails received via my October, 2012 FOIA request. I got all of Accardo's inbox from August 29th and 30th, 2012 and what I presume are most of his "sent" box from August 27th through the 30th. Also included were most of the attachments to those emails, though some key attachments remain unsent. Also missing are Accardo's inbox emails from the 27th and 28th, though some of them reside in the ealier releases by virtue of carbon copies to others. I have asked the Corps to rectify these oversights.

These emails were sent in an odd format. While all the inbox emails were gathered into a single file, all the sent emails were sent individually. I have linked to the inbox emails, but I'm still collating the sent emails into a single file for downloading.

Thursday, May 30, 2013

Isaac: the rest of the story, Part 1

Updated SCADA failure section June 2, 2013. See below.

Last October, I requested via FOIA the emails of a number of key personnel in the Corps' New Orleans District. I confined my request to emails from August 27th to August 30th of 2012, the days Hurricane Isaac came ashore. I wanted to know exactly what happened at the Corps' gate structures at the mouths of the three New Orleans outfall canals under real storm circumstances. After a delay of over 200 days, I finally received the bulk of those emails this month. They help fill in the other half of the story. The first half - that told by the pump logs of the city pump operators, was covered in my October 6, 2012 post, "Isaac in New Orleans - what we know so far." That post includes links to all the city pump logs.

I went after the emails of a number of folks in the Operations Division of the New Orleans District who would be on the front lines during a storm. Raymond Newman, Donald Constantine, and Carl Robinson serve as "canal captains" during storm events at - respectively - the 17th Street canal, the Orleans Avenue canal, and the London Avenue canal. As such, they are the personnel with the freshest information during a storm. Donald Schneider, also within the Operations Division, has popped up repeatedly in past outfall canal communications. Chris Accardo is head of Operations at the New Orleans District, and would have information flowing through him from all over the District (I am still awaiting Accardo's emails, but so many of them are included in everyone else's that I do not anticipate much new information).

I also wanted to know what was going on outside Operations, so I asked for a few others' emails. Ken Holder is the head of Public Affairs at the Corps' New Orleans District, and would be the public face to the press during the event. Bac Nguyen has, in the past, had responsibility for specifying and coordinating pump repairs and other work at the closure structures.

The headlines that emerge from these emails are the following:

A) The Corps could not lock one of the gate segments in place at the London Avenue site before the storm hit, and actively decided NOT to drop large sandbags in front of the gates, despite their own procedures saying they were required to do so. The trench the gates drop into had partially filled in with silt just a few hours after being cleaned, and they decided to just let it ride. This particular incident is detailed below.

B) Confirmation that the 2011 remediation projects failed at the 17th Street and London Avenue canals. According to the "Outfall Canal Reference" spreadsheet, the maximum operating water level (MOWL) at 17th Street is 6.5 feet and is 5 feet at London Avenue. Before the remediation projects, the 17th Street MOWL was 6 feet and the London Avenue canal MOWL was 5 feet. The Corps had never fully confirmed this. Literally tens of millions of dollars were wasted in those projects. The Corps has also announced separately there is more work coming on both canals.

C) While the Corps believes the MOWL at the Orleans Avenue canal is 8 feet, there is a "secret" trigger along that canal which prevents its full utilization. According to that same spreadsheet, no more than 5 feet of water can be put in the canal or S&WB pump station 7 at the canal's south end will flood. I believe this is possibly the true reason the gates were dropped at Orleans Avenue during Isaac. Here's a screenshot of the spreadsheet:


D) At one point during Isaac, there were 8 pumps out of commision at 17th Street, representing nearly 15% of that structure's supposed capacity. This is far more extensive than previously reported. I'll have more details on this in the next post.

E) The level gauge and remote system control and monitoring failures - previously reported by the Lens as occurring during the storm - were apparently already occurring two days before the storm ever hit, and they were across the entire system. The Corps was reduced to trading emails back and forth during the storm's passage to let each other know what was going on with water levels and pump statuses, because there was no remote visibility. My guess is nearly the entire SCADA system failed at the height of the storm, and none of the backup systems worked either (for example: the SCADA backup generator for the London Avenue canal system was out of service at city pump station 3).

F) Electrical problems included the complete failure of the gate hoists at Orleans Avenue on August 27th (the gates had to be lowered by crane), and an hours-long failure of part of the fuel supply system at the London Avenue site which left half the pumps at that site with just four hours worth of fuel on the morning of August 29th. That last one was later corrected - a gecko was found across some wires.

I'll get to some of these in a later post.

In sum, the emails reveal that far more happened during Isaac than the Corps publicly revealed, even to reporters (the NY Times' John Schwartz and the Times-Picayune's Becky Mowbray) and a Senator (David Vitter) embedded in the Corps' Emergency Operations Center (EOC) during the key overnight and morning key hours August 28-29, when the storm was at its peak. But first we'll go back about a week before that.

There are a few attachments to the emails, which I also received. The first key one is this, the Corps' weekly "pump tracker." It is an Excel workbook containing readiness data for every major piece of mechanical equipment at all three sites, including every pump, every gate segment, and all the generators, fuel tanks, and city power lines coming into the station. According to the August 24, 2012 pump tracker workbook, every piece of equipment at all three sites was "operational," except for pumps E4 and E5 at the Orleans Avenue site. Those two pumps were out for repairs due to corrosion-caused hydraulic system failures. I wrote about them August 28, 2012 here.

Theoretically, that "operational" status is borne of what the Corps claims is extensive testing throughout the year, especially during hurricane season. I have serious doubts of the efficacy of the testing, because even before the systems were stressed, they were falling apart.

The troubles started on the evening of August 27th, over a day before Isaac even came ashore. As I noted in my earlier entry, the gates at the Orleans Avenue structure were dropped during a tropical event for the first time August 27th. An email from Chris Accardo sent at 9:19 AM on August 28th provides some detail:
"due to an electrical problem, the gates had to be lowered with a crane last night."

It is unclear whether a) the gates were lowered because there was an electrical problem, so they were lowered out of an abundance of caution, or b) they were going to be lowered anyway, and Accardo is only referring to the style of their lowering (crane versus electrically-actuated hydraulic winches). It is also unclear if the gates were lowered because the Corps was trying to avoid getting more than five feet of water int eh canal in order to avoid flooding the city's pump station, but it's a good guess considering predicted storm surge levels in Lake Pontchartrain were higher than five feet at that point. The "crane" is one of three rental cranes the Corps has, each stationed at each site. The Corps has paid so much in annual rental fees for those cranes at this point they could have easily purchased them, and they are continuing to rent them rather than buy them.

There are two later emails referencing this problem. Donald Constantine emailed Donald Schneider at 10:04 AM on August 30th:
"Don - do we have anyone working on getting the gate hoists working again? Will it be another crane operation to pull the gates?"

The response came from Donald Schneider at 7:49 PM on August 30th:
"Prime Controls determined that the hydraulic gate lift problem was caused by failed contacts. They're searching for a replacement part; however, I think they rigged the unit to operate on a temporary basis."

This certainly brings into question whether these gates were exercised at all as part of the reporting in the August 24th pump tracker workbook, when they were said to be "operational."

[Begin June 2, 2013 update]
The system problems extended far beyond just the Orleans Avenue gate hoists, though. Based on email traffic before and during Isaac's landfall, it appears the Corps' monitoring and control system for the gates experienced a total failure some time on or before August 27th, two days before the storm hit. The Supervisory Control and Data Acquisition - or "SCADA" - system allows the Corps to remotely control activities at the gate structures such as running pumps without having to have personnel there. It also allows for real-time monitoring of a series of 8 level gauges placed along each canal. There are SCADA terminals in Sewerage and Water Board drainage pump stations 3 (at the foot of the London Avenue canal), 7 (at the foot of the Orleans Avenue canal), and 6 (at the foot of the 17th Street canal), as well as in the Corps' Emergency Operations Center in their District Office on Leake Avenue in New Orleans.

The SCADA system was designed and installed by Prime Controls of Lewisville, Texas. Prime Controls was also awarded a five year maintenance contract for the system in late 2008, a contract still in effect at the time Isaac came ashore. The Corps' Bac Nguyen was emailing back and forth with Prime Controls' Metairie, LA-based project manager Steve Baker August 27th, trying to get one of Prime's employees (project engineer Randy Oliver) to drive in from their home office in the Dallas area for the duration of the storm.

At the end of that email chain, Nguyen wrote Baker at 4:25 PM:

"We lost communicaiom [sic] to all three sites. Is Jim still working on these?

Thanks."

That is confirmation of the systemwide SCADA failure on the 27th.

Prime Controls' Baker wrote back at 4:29 PM:

"Yes Jim is there. When Deep South got the microwave working there is a conflict in Communications.

Jim is working with Randy on the phone till Randy gets here."
"Jim" is a Prime Controls employee. "Randy" is Prime's Randy Oliver. "Deep South" is very likely Baton Rouge-headquartered Deep South Communications, who specializes in microwave radio communications. Baker is referring to the backup microwave towers (one at each gate structure and one at each city pump station) that are supposed to allow communications if the main fiberoptic communications cables along the canal beds fail. The fact that Deep South had to come out and get "the microwave working" indicates that not only did the primary fiberoptic system failed, but the backup microwave system also failed. And this was all two days before the storm even hit!

The SCADA problems continued into the evening of the 27th. At 9:09 PM, the Corps' London Avenue canal captain Carl Robinson sent this:
"London Ave team status

The team reported to the site at 1800. We're standing by for orders. We're checking equipment and securing loose items. Station is operational with SCADA Level indicator problems at PS3, Harrison, Mirabeau, Filmore, Prentiss.

Team members on site:

Carl Robinson
Randy Faherty
Kevin Calico
Larry Trout
Shane Thames"

Robinson's August 27th email puts the lie to Corps reassurances about the severity of level gauge failures during Isaac. Those statements are within reporting about the level gauges done by Steve Meyers at the Lens in October, 2012. At that time, the Corps said:
"Moreover, the corps was getting more timely, reliable information than was available to the public on Rivergages.com. Gauge readings are sent via satellite to that website every hour. The corps gets readings several times a minute via fiber-optic cable laid on the bottom of the canal. If the cable is cut, they can get readings via microwave dishes.

Ray Newman, the corps’ 17th Street Canal captain, said gauges 'were coming and going' during the storm, but for less than a minute at a time — not long enough for the water level to rise dramatically. 'I still knew what the level was,' he said."

In fact, according to Robinson's email above, the Corps' supposed better internal level gauge visibility was nonexistent on five of eight London Avenue gauges two days before the storm's peak hit.

The SCADA problems continued as Isaac came ashore. Robinson reported from the London Avenue canal at 7:17 AM on the 29th during the storm's highest intensity:

"Two problems popped up within the last hour. We are having problems with our refueling system for the 4 west direct drive pumps. The problem is an electrical problem that happened when the station loss power yesterday. We'll need an electrician to correct the problem. We have 4 to 5 hours of fuel in the engine tanks for these 4 pumps.

Next, I've lost access to SCADA at Pump Station 3. The Corps standby generators are not available due to previous work at the station. These generators are tied to the battery back up power which is depleted due to the loss of power at PS3. Its a 100 amp ckt so just plugging into an outlet won't due.

The only solution I see is for me to call in what the SWB is pumping and let the station match it. I think that Don S. Should be at the London Station in case we can't keep up with SWB. He can make the call to me to ask the SWB to cut back.

With [Corps mechanic] Randy [Faherty, normally assigned to London Avenue] going to 17th street we are short handed.

Please advise.

Carl"

We'll get to the fuel system problems in a later entry. I have to say though this email encapsulates all the things wrong with how the Corps has decided to run the outfall canals: 1) They apparently don't have someone in authority at the gates and at the city pump stations during storms; 2) after six years of construction and maintenance by the same company (Prime Controls), their data communication system still fails under even the stress of a category 1 storm; 3) the canal walls are still so fragile that they cannot put all the city's water in the canals without risking a breach, so the idea of cutting back on the city pumps is a routine way of doing business. That is, the problems that led to the flooding in 2005 have yet to be addressed.

Back to the SCADA failures. Robinson's 7:17 AM email on the 29th coincides with the first news articles describing communications and pump losses at the 17th Street canal, leading one to believe the whole SCADA system was again down. (Note: Robinson's email came over two hours before pump station 3 lost power at 9:55 AM, so it is unclear why he was talking about the Corps' backup generators). Indeed, less than hour after Robinson's email - at 8:02 AM - the Corps' Operations number two, Jerry Colletti, sent an email to all three canal captains requesting updates on which pumps at each site were working and which weren't:
"I know you guys are real busy and are busting your butts; but as always is expected, we are asked for current status of operations. So, if you can quickly shoot me an email saying which pumps are NOT (NOT) operational, I will try to take it from there. If you have total current operating capacity that you can quickly provide, that's fine. If not, I will estimate it based on the pump capacity of those pumps that are operational.

For example, [Orleans Avenue canal captain] Donald [Constantine] just had W5 go down to fire plus he had 2 others down, so I can report 10 pumps total, 9 operational [sic - shoudl read 7 operational, since two were already out for repairs before the storm hit]. If I knew the others I could estimate current capacity. You can just list them like

17-W2, 17-W4, 17-LP6, ... I can then update on my total list and keep it current."

(We'll get to the Orleans Avenue fire in the next part)
This points to a systemwide failure because if the EOC were receiving SCADA data from the three gate sites - data which includes which pumps were on or off - such emails would be completely unnecessary.

Later on, at 12:30 PM, Corps Operations Chief Chris Accardo requested level gauge information via email from all three sites, opening up a second email chain for transmission of data which should have been instantly visible via SCADA. There are pages and pages of such emails from all three canal captains Accardo - who would have been at the Corps' Emergency Operations Center inside their building on Leake Avenue throughout the day of August 29th - at the height of the storm. Those emails, sent every half hour, contain readings from level gauges on each side of the gates (i.e the only readings directly visible to Corps personnel) from about 12:30 PM onward on August 29th. Such emails should have been completely unnecessary, because personnel in the EOC are supposed to be able to see all 24 level gauge readings minute by minute, including those at the gates.

Final confirmation of the loss of SCADA data is provided by follow-up work. A few months after Isaac, the Corps gave a $3570 task order to Prime Controls to do the following:
"TASK Order 18 - Provide labor and materials (if needed) to set up SCADA system to automatically cut over to the backup Metro Ethernet Service at Pumping Station # 3 when the main Metro Ethernet Service at Pumping Station # 6 (which provides communication between ICS and Emergency Operations Center located at the District headquarters) loses the connection with EOC office."

The impression conveyed through all these emails - as well as the news coverage of the SCADA outages on the morning of the 29th that led to pumps being shut down in the city pump station at the foot of the 17th Street canal - is that the SCADA system along the outfall canals did not withstand a category 1 storm. That is a much larger loss than simply some level gauges going "in and out."

[End June 2, 2013 update]

And while the electronic failures are bad enough, there were actual physical problems that were even worse. At 5:10 PM on August 28th, long before the storm hit with its full fury, Carl Robinson reported from the London Avenue site to Corps Operations Chief Chris Accardo and Corps Emergency Management Chief Michael Stack:
"Chris/Mike

The following gates did not fully pin at London.

Gate #8 - couldn't insert neither pin. 1/2 to 1 inch off.

Gates 6,7,9, 10 only one pin inserted.

These gates were cleaned out yesterday. Looks like debris made it's way in overnight or this morning. I suggest we make several smaller pins that we can insert in an emergency. They could be secured using the existing threaded rods and nuts.

Carl"

I had actually posted a reminder post that morning which directly referenced the possibility of gates not getting secured with their pins. It wasn't idle speculation - it had already happened at least once before, in 2009, as I detailed in this 2010 post. (for those looking to catch up, there's further detail available on individual components of the gate structures including the trenches, the gates themselves, the pins, and the seals). If the gates were not properly secured, failure of the closure structures was a distinct possibility. That would allow storm surge directly into the outfall canals, with a failure of still unrepaired wall segments likely to follow.

It is the follow up emails that really drive home how nonchalantly the Corps takes its storm-protection responsibilities. Stack wrote back to Robinson just minutes later:
"Are we good to get through this event or do we need to drop bags?"

Stack is referring to the requirement in the closure structure operating manual that massive pre-staged sandbags be dropped by the rental crane in front of gate segments which cannot be secured in advance of a storm. Instead of following procedure and safeguarding the citizens of the city by doing everything possible, Accardo - the head of Operations - waived Stack off:
"We are OK, Some gates pinned on one side only, but they are only off by fractions of an inch."

Robinson chimes in five minutes later:
"I agree.
Also If we need to, we may be able to find something at the ICS to insert and secure in number 8.
Carl"

And that's that. The Corps entered Isaac with five of the eleven gates at the London Avenue site not completely secured, including one that was utterly unsecured; the Corps was relying on gravity to keep it in place. Put simply, the Corps had closed the barn door at the London Avenue canal (the weakest canal structurally speaking), but they hadn't bothered to lock it. The possibility of storm surge loosening the gates was apparently too inconceivable for the people directly responsible for making sure the city was protected. Procedures - such as calling out divers that were supposed to be on call, or failing that, dropping massive sandbags - were simply brushed off. It is in-the-moment exchanges like this that undermine whatever bluster the Corps puts out about safety being their top priority.

In the next part we'll detail the further unreported failures that occurred during Isaac, including much wider pumping problems than the Corps reported at the time.

Wednesday, May 15, 2013

There's more to come

While it has been a long time since I posted, I wanted to let people know there is much more to come. I have received a raft of internal Corps emails from August 27, 2012 through August 30, 2012, which is when Isaac came ashore. They detail exactly what was happening along the outfall canal during those events.

Saturday, October 06, 2012

Isaac in New Orleans - what we know so far

The first round of primary documents from Hurricane Isaac's hit on New Orleans have arrived. They are the Sewerage and Water Board logs for drainage pump stations throughout the city.

I got the logs for Pump Stations 6 (at the south end of the 17th Street canal), 7 (at the south end of the Orleans Avenue canal), 3 (at the south end of the London Avenue canal), 4 (about half way along the east side of the London Avenue canal), and 1 (in the middle of the city, just lake side of the Broadmoor neighborhood, feeds station 6). Links for these files can be found at the end of this entry. The findings are depressingly predictable. But first let's review the information the Corps did not hold back.

17th Street canal - Still a probem seven years post-Federal Flood

The top news on the morning of August 29, 2012 - when Isaac was lashing the city - as it related to the drainage flow out of the city was that the Corps of Engineers lost communications with their lakefront pump station along the 17th Street canal, at the height of the heaviest rainfall. What emerged from coverage by the Times-Picayune (initial report at 7:41 AM, follow-ups at 8:30 AM, 9:41 AM, and 11:51 AM), the New York Times, and WWL-TV was the SCADA (Supervisory Control And Data Acquisition) system that allows the Corps to run their lakefront pumps remotely had failed early that morning. The Corps could not turn some of the lakefront pumps on, and in order to keep water below the safe water elevation for the canal, they refused to allow the Sewerage and Water Board to turn on more pumps at PS#6 as the water rose in the city's interior. Eventually, the Corps started their pumps at the lakefront manually (an hours-long delay in doing so - despite Corps procedures calling for staffing of the lakefront stations - was never explained publicly) and authorized the Sewerage and Water Board to pump as much as they liked.

The PS#6 logs for August 29th (here and here) confirm the outlines of this story, while filling in some detail. At 4:35 AM, the station staff asked to "load" (that is, bring to full capacity) 1100 cfs pump "F." They were refused. Then, at 4:55 AM the Corps told the S and WB not to "load" any more pumps in PS#6:
"Note: Army Corps of Engineers said we cannot load any more pumps. [T]hey are at their max of what they can pump"
At that time, the station was pumping aproximately 6200 cubic feet per second, or cfs. The station's capacity among its main rainstorm drainage pumps is 9200 cfs.

This initial notice to the S and WB to hold off on pumping appears to be 1.5 to 2 hours before the Corps alerted the media of pumping restrictions, according to the chronology of the NY Times article. Then, at about 6:30 AM, the station operators received another Corps warning not to load additional pumps beyond what was already running, despite the fact that in the intervening hours, water in the station's inlet basin had risen over four feet. Ten minutes later, the station received orders from S and WB's Central Control to break the prime on 1000 cfs pump "D," effectively losing most of its capacity and sending into a pump status called "light." So there were not only Corps orders to keep some pumps from running, but one pump was effectively ordered turned off with the water coming up. This left the station running at about 5200 cfs.

At 7:55 AM, with the water still very high in the inlet basin, the Corps 17th Street canal captain in the station (Ray Newman) gave an all clear to the S and WB personnel in the station to load additional pumps. Five minutes later, 1000 cfs pump "D" was authorized to be loaded by the S and WB Central, and was running in a loaded condition within a few minutes. Another 1000 cfs pump ("C") was brought on line shortly afterward, increasing the station's flowrate to 7200 cfs. Inlet basin levels dropped four feet in about an hour after those two pumps were added. This chronology roughly tracks what was playing out in the media that morning.

What also emerged from this story was that the 2011 remediation project along the 17th Street canal had failed to raise the Safe Water Elevation along the canal from 6 feet to 8 feet, as had been publicly promised repeatedly. Instead, it appears to have only been raised six inches, to 6.5 feet. This number was mentioned in the intial 17th Street reporting from the T-P on the morning of August 29th:
"Engineers were able to start the pumps manually, but because of the rainfall, the water piled up inside the canal a little higher than the corps would like to see, and is currently between six and six and a half feet."
Was further hinted at in subsequent T-P updates:
"The 17th Street Canal is now functioning properly, and water levels have dropped to safe levels... [W]ater levels in the canal have dropped from more than 6 feet to 4.3 feet at 9:30 this morning."
Was nearly confirmed by the NY Times later that afternoon:
"Chris Accardo, chief of operations for the district, sitting at a conference table with Colonel Fleming, quietly explained that gauges in the 17th Street Canal suggested the water had risen to six feet — maybe even six and a half — a height at or even beyond the margin of safety for those flood walls."
And was finally confirmed by WWL-TV the following morning:
"when water reached the 6½ foot maximum operating level in the canal, operators attempted to start the pumps from a remote control system but it didn’t work."

Had the 17th Street SWE been 8 feet, there may have been no crisis that morning. The remediation project on that canal had cost $15 million, and it all appears to have been wasted.

Many other Corps pumping restrictions revealed by logs

What has not been reported until now was that similar stories were also playing out at the other two outfall canals, with perhaps more spectacular events.

At the Orleans Avenue canal, the Corps had pulled two of their ten lakefront pumps out in early August, and had not repaired them by the time Isaac came ashore. In an article published in the Lens on Monday, August 27, 2012, the Corps gave some bland, content-free reassurances about how everything was fine with only 80% pumping capacity available at the lakefront:
"We’ve never had to close the (Orleans Avenue) gates during a storm. Not even during Ike, not even during Gustav," [Accardo] said. "So the threat of these pumps being down and causing any problems is extremely remote. There is hardly any threat to City Park or Mid-City.

"We are not where we want to be at Orleans currently. I would like to be a little higher, but it is not something people need to be worried about," he said. "My standard is to have as many pumps functional at all times."
[...]
"as long as I can stay ahead of the S and WB, it’s not a problem."
The first part of the quote was proven untrue just hours after the Lens published their article, when the gates were lowered at the Orleans Avenue site at around 11 PM on the 27th. That was the first time that had happened since the gates were put in place in mid-2006 and the Corps' pumps were accepted for service in September, 2007. The gates at Lomdon Avenue and 17th Street were dropped the following day, the 28th.

The truly worrisome activity began the morning of the 29th, during the heaviest of Isaac's rain. S and WB pump station 7 at the south end of the canal houses five pumps:

"A" (550 cubic feet per second, or cfs, powered by S and WB-generated 25 cycle power),
"C" (1000 cfs, 25 cycle),
"D" (1000 cfs, Entergy supplied 60 cycle, with diesel generator backup),
"1" (70 cfs, 25 cycle), a so-called constant-duty, or "CD" pump that normally only runs during dry times to pump out runoff from sprinklers and the like
"2" (70 cfs, 25 cycle), the other "CD" pump.

At about 2 AM on the 29th, as the water started rising in the stations' inlet basin, all five pumps were running, though only four were loaded. Pump "C" had yet to be loaded. 60 cycle pump "D" was running on generator backup, after Entergy power had been lost shortly after midnight.

According to the log, the station operators asked to load pump "C" at 2:20 AM, and were denied by the Corps of Engineers. Subsequent requests to load "C" at 3:05 AM and 4:46 AM were also denied, though the log seems to imply it was the S and WB's central control station ("c/c") in both those cases. However, it seems unlikely the S and WB do such a thing voluntarily, so it's sensible to presume the Corps refused to allow the city to run all the pumps at PS#7 for over two hours during Isaac (the results of FOIA rquests to the Corps will reveal the exact reason). Level gauges show the water levels in the canal were between 2 and 3 feet, well below the supposed canal Safe Water Elevation of 8 feet. Thus the only reason to cut back on the water going in the canal is because the Corps couldn't pump it out fast enough. This puts the lie to the Corps' Accardo when he said his pumps could stay ahead of the S and WB. They obviously couldn't.

But things would get even worse later that morning. At 9:40 AM, the log reported:
"Donald Constantine w/C.O.E. reported a Fire w/C.O.E pump"
You read that right.

At least one of the Corps' pumps Caught On Fire During Hurricane Isaac.

Thirteen minutes later, pump "C" - still unloaded - was turned off completely. The logs do not note who ordered that shutoff, but it's not hard to guess.

Then things went completely to pot in PS#7:
"9:55 AM: Lost 25 Hz power. Stop A, CD#1 and 2, MG 1, 2, 3 n/cc"
Since 25 cycle-powered pump C was already off, it didn't have to be shut down when they lost power. But all the other major equipment in the station was lost, including the three motor-generator ("MG") sets that provided power for supporting equipment.

At that point, the station was down from a potential flowrate of 2690 cfs to a generator-backup-powered 1000 cfs, or a 63% drop. I bet even the Corps' pumps could keep up with that.

The problems were not confined to just PS#7 though. Nearly the same script was playing out just to the east along the London Avenue. Two pump stations feed that canal: PS#3 and PS#4. Here's the major pump makeup at each:

PS#3
"A" (550 cfs, 25 cycle)
"B" (550 cfs, 25 cycle)
"C" (1000 cfs, 25 cycle)
"D" (1000 cfs, 25 cycle)
"E" (1000 cfs, 25 cycle)

PS#4
"1" (320 cfs, 60 cycle)
"2" (320 cfs, 60 cycle)
"C" (1000 cfs, 25 cycle)
"D" (1000 cfs, 25 cycle)
"E" (1000 cfs, 25 cycle, out of service long before Isaac approached)

So the potential total flow into the London Avenue canal was 6740 cfs. It should have been higher, but with pump "E" at PS#4 out of the game, the city was starting in the hole.

Nevertheless, the London Avenue pump operators were also refused the privelege of pumping water out of the city on the morning of the 29th. At 4:03 AM the operators in PS#3 logged the following:
"N/cc [notified S and WB central control] to load C pump (12' 25'5") [these numbers are the depths of the stations' inlet and outlet basins respectively], L/rain, Refused"
The operator also drew an arrow pointing to this entry and wrote "NOTE" in dark letters at the end of the arrow, highlighting the refusal to load pump C. Half an hour later, they were refused again. At that point, water in the inlet basin had risen another foot.

Meanwhile, just north at PS#4, the operators were also getting refusals. Remember that one of PS#4's three main pumps - 1000 cfs pump "E" - was already out of service before the storm. Also, they lost Entergy's 60 cycle power at 7:45 PM on the 28th, so their two 60 cycle pumps ("1" and "2") were also never turned on. I am unsure whether there is generator backup at PS#4, but it certainly doesn't look like it. Even with all these losses, when they lost load on pump "D" (that happens occasionally, sometimes because the pump prime is lost) and they called Central at 6:35 AM to reestablish the load, they were refused. A half hour later, however, they reloaded the pump.

What one notices when putting all these logs together was that requests to pump more water into the outfall canals from pump operators across all three canals were repeatedly refused at the height of the storm. Effectively, the Corps' inability to pump out the canals placed an hours-long cork in the city's drainage system. And this is the pumping system the Corps - because of cost reasons - has saddled the city with for years to come.

The exact reasons for individual refusals are unclear, but we can speculate. There's only two obvious causes to stop the S and WB from pumping into the canals: 1) Water levels were above the Safe Water Elevations in the canals; 2) the Corps' lakefront pumps could not keep up. (2) does not seem far fetched at all, considering we already know of Corps pump outages at 17th Street (SCADA system down) and Orleans Avenue (fire). (1) would come about if the 2011 remediation projects along the Orleans Avenue and London Avenue projects were also failures, and the Safe Water Elevations had not been reaised to the promised 8 feet, or had perhaps even been lowered. This also seems likely, since we know there is more remediation work coming on all three canals. I bet the reasons were a combination of both, with multiple Corps pump outages beyond what we know now.

Sewerage and Water Board power outage across three pump stations cripples pumping for hours

Though the refusals to authorize loading of city pumps throughout the early morning of the 29th are quite significant (and point to the hazard of the the Corps' permanent plant to keep the exactly same system in place for at least the next 5 decades), the effects paled in comparison to the loss of 25 cycle power later that morning. The same power outage that struck PS#7 at 9:55 AM also hit PS#3 and PS#4. This had the effect of shutting pumping at both those stations off completely. Among the three stations, only 1000 cfs was pumping, which was at PS#7. As at PS#6, inlet basin levels at all three stations shot up two to four feet very quickly with nowhere for the water to go.

The effects of the outage lasted different times at different stations. At PS#4, they were back up with their two functioning 1000 cfs pumps about an hour later. At PS#3 they were able to get everything running again in about 90 minutes. But at PS#7 it appears they were unable to restart pumps "A" and "C" until much later in the afternoon because of subsequent electrical problems.

In the middle of all that, the Corps reported at 11:10 AM they had put out the pump fire at the Orleans Avenue lakefront gates. But with all but one of the city's pumps in PS#7 down at that point, it was pretty irrelevant. The Corps' pumps were only receiving about 1000 cfs of water, so they could keep up with just five of their 10 lakefront pumps if they wished. So when the Corps reports they were able to keep up with what the city sent them, take it with a grain of salt. They couldn't, as the repeated refusals earlier in the morning of August 29th amply demonstrated.

Corps pumping refusals along Orleans Avenue canal continue nearly 1 day after initial incidents

One might think after all that activity, the worst had passed. And for the most part that was true. However, in the dead of night, at 12:55 AM on the 30th, the Corps once again stuck a plug in the city's Orleans Avenue canal drainpipe. From the PS#7 logs:
"O/ Donald Constantine w/C.O.E. Break prime on C pump. N/cc"
That is, the S and WB operators in PS#7 were ordered ("O/") directly by the Corps' Orleans Avenue canal captain to remove most of the flow from 1000 cfs pump "C," leaving just the flow from 550 cfs pump "A" (which had been loaded minutes earlier) and 1000 cfs pump "D." Inlet basin levels rose 2.5 feet in an hour. The station continued to run in this diminished state for four more hours, eventually lowering the inlet basin level back to where it had began before the Corps order to stop pumping. This is more evidence the Corps' Orleans Avenue canal lakefront pumps - two of which were removed from service - could not keep up with the city's pumps.

In fact, at no time during Isaac did city pump station PS#7 run at maximum capacity, a remarkable fact that blows to bits the ridiculous pre-storm reassurances from the Corps. That is, the city was refused permission by the Corps to run its drainage pumps at PS#7 as fast as it wanted to for the entire storm.

Wrapping up

In sum, as with every story involving the Corps of Engineers, there was much more happening behind the scenes than what the public was told during Hurricane Isaac's trudge across New Orleans. Restrictions on city pumping into the outfall canals - imposed by the Corps - occurred earlier than was previously reported and extended across all four major stations putting water into those canals. Specifically at the Orleans Avenue canal, the Corps appears to have limited city pumping for the entire duration of the storm, in part due one of the Corps' pumps catching on fire at the height of the storm. A Sewerage and Water Board power outage at the stations on the Orleans and London Avenue canals - also at the height of the storm - masked the effects of some of the pumping restrictions.

The Corps issued a $630 million contract on September 28, 2012 to construct permanent pump stations at the lakefront on all three outfall canals. The winner was a coalition called PCCP Constructors - formed specifically for the project - made up of Kiewit, Traylor Brothers, and local contractor M.R. Pittman . In part they will be supplying $70 million worth of Patterson Pumps, likely very similar to the direct drive pumps currently installed at the 17th Street and London Avenue gate structures and also supplied by Pittman and Patterson [edit, 10/8/12 - the 17th Street direct drive pumps are from Fairbanks-Morse, not Patterson]. The permanent stations are intended to replace the current unreliable stations by 2016. But the overall tandem pumping scheme, dubbed Option 1, will remain in place. As Isaac demonstrated amply, that is a dangerous way to operate. As more documents roll in, I will continue to add to this story.

Links to Isaac Sewerage and Water Board pump logs

Zipped files containing daily logs for each station from Sunday, August 26, 2012 through Tuesday, September 4, 2012:

PS#1 (in city interior, north of Broadmoor neighborhood)
PS#6 (at south end of 17th Street canal)
PS#7 (at south end of Orleans Avenue canal)
PS#3 (at south end of London Avenue canal)
PS#4 (half way along east side of London Avenu canal)

Individual daily log sheets for each station:

PS#1
PS#6
PS#7
PS#3
PS#4

Go to older posts