Isaac: the rest of the story, Part 2

In Part 1 of this series, we reviewed what internal Corps emails written during 2012's Hurricane Isaac revealed about failures of the gate hoists at the Orleans Avenue canals and the SCADA system across all three outfall canal sites. We also saw how Corps personnel decided to not secure the gates at the London Avenue canal, in contravention of their own procedures.

In this entry, I'm going to focus on events along the Orleans Avenue canal. As we've mentioned before (including on the day Isaac started making its presence known), the Corps' Orleans Avenue gate structure was down to 80% of its rated pumping capacity as Isaac moved ashore because two of its ten pumps were rusted out. They had been yanked out weeks before Isaac. I thought it was pretty clear this would cause problems, since even with all ten pumps installed at the gates, the city Sewerage and Water Board (SWB) pump station #7 at the canal's inlet can put out about 2700 cubic feet per second (cfs) of rainwater, while the Corps structure at the canal's outlet can only pump 2000 cfs into the lake. Take two pumps out of the Corps end of things, and it is a guarantee water will climb in the canal if the SWB station pumps as hard as it can. If water got too high (i.e. above the maximum operating water level), it was likely the Corps would turn off Sewerage and Water Board pumps that drain the city's streets. That was the setup going into Isaac.

Corps internal allowable water levels lower than what the public was told

Let's go back to the previous coverage around the Orleans Avenue canal. There was the Lens article on August 27, 2012 about those two rusty pumps, which included these quotes from Corps New Orleans District Operations Chief Chris Accardo:

"Further, Accardo said that the Orleans Canal has the highest-rated water level of the three.

'We’ve never had to close the (Orleans Avenue) gates during a storm. Not even during Ike, not even during Gustav,' he said. 'So the threat of these pumps being down and causing any problems is extremely remote. There is hardly any threat to City Park or Mid-City.

'We are not where we want to be at Orleans currently. I would like to be a little higher, but it is not something people need to be worried about,' he said. 'My standard is to have as many pumps functional at all times.'"

That "highest rated water level" is supposedly 8 feet. If water gets above that level (the walls are 14 feet tall) bad things could start happening.

Accardo sent an email to local stakeholders and a number of Corps personnel at 9:19 AM on August 28 with a summary of pump statuses at the three outfall canal sites (this email was part of the first batch linked in Part 1 of this series). Included in that email was this:

"Orleans Ave - Max. Water Level is 8.0. The Corps has 2 pumps that are down. With 100% pumping capability, the Corps has a rated capacity of 2200 cfs. Since 2 pumps are down, the rated capacity is 1800 cfs. The SWB reports that they have a 2,700 cfs rated capacity when all their pumps are working. The SWB is currently at 100%. Since the maximum water elevation is higher at Orleans [maximum at 17th Street is 6.5 feet, at London Avenue 5 feet - ed.], we have never dropped gates before at Orleans. However, due to an electrical problem, the gates had to be lowered with a crane last night. The canal is currently at elev. 1.5 with the gates closed and we have a corps employee at the SWB pump station. We will turn pumps on and try to maintain a 1.5 canal elevation as soon as the SWB turns their pumps. Even though the SWB has greater pumping capacity, they will have to out-pump the Corps such that the water level in the canal will rise 6.5 ft. before there would be a problem. I believe the chances of that happening is remote. If it did happen, we would request that the SWB reduce pumping so as to preserve the maximum water level."

As we learned from the Corps' internal emails, this was untrue. Accardo sent another email at 9:18 PM the same day which included a spreadsheet with the real maximum water levels. This is it:

First off, the max water level at Orleans Avenue is not 8 feet, it's 5 feet (ignore the "trigger" language), because above that level the SWB pump station floods (!). That's as low as the the very weak London Avenue canal, which has many sections of very short sheet pile in sandy soil that have not been upgraded.

So right off the bat, we know the Orleans Avenue canal, even if it had all its Corps pumps, would be a trouble spot. Down two of those pumps, it would be even worse.

Now, thanks to additional emails I received this past Friday, we have conclusive proof of exactly how little faith the Corps has in the Orleans Avenue canal.

Corps actually runs Orleans Avenue canal at water levels even lower than their internal numbers

Donald Constantine is a member of the Corps Operations team. During tropical events, Constantine is assigned as the "canal captain" for the Orleans Avenue canal. This means he monitors all aspects of the Corps' side of the the canal operations (the pumps at the lakefront and the level gauges along the canal length). He is stationed in SWB pump station 7 on Marconi at the south end of the canal, so he has a direct contact with with SWB pump station operators. This was the situation during Isaac.

As I detailed back in October, those operators were refused permission to turn on SWB pumps at various times during Isaac, presumably under direction by the Corps' Constantine. The number of refusals documented by the SWB operators in their logs for August 29 and 30 - four - was surprising, given the supposed large amount of storage capacity (or "freeboard") in the canal. At the time, I thought it was evidence the eight Corps pumps could not keep up with the SWB pumps, but I didn't then know about the 5 foot restriction.

As it turns out, the Corps kept the level in the Orleans Avenue canal even lower than that.

I want to focus on the final SWB pumping refusal of Isaac, which took place in the wee hours of the overnight between August 29th and August 30th. At the time, pump station 7 was putting out its maximum amount of flow from its three main pumps. Pump "A" was flowing 550 cfs, and pumps "C" and "D" were flowing 1000 cfs each, for a total of 2550 cfs (there is no pump "B"). "A" had just started flowing at 12:40 AM after water in the pump station inlet basin had started rising. At 11:30 it was 10.1 feet, at midnight it had gotten up to 10.7 feet, and at 12:30 AM it was 11.0 feet. While not as high as during the late morning on the 29th (when the inlet basin had climbed to over 15 feet during a SWB power outage), it was still high enough to make the SWB operator feel it was necessary to run all three main pumps.

The SWB pump log shows the following at 12:55 AM on the 30th:

"O/ Donald Constantine w/C.O.E. Break prime on C pump. N/cc"

That translates to "Ordered by Donald Constantine with the Corps of Engineers to break prime on (effectively turn off) C pump. Notified SWB central control on South Claiborne Avenue"

Thanks to the new emails, now we know exactly what led to Constantine's order. He had sent the following email to Accardo about an hour and half before the refusal order, at 11:22 PM on the 29th (emphasis mine):

"From: Constantine, Donald A MVN
To: Accardo, Christopher J MVN
Subject: FW: Gage Reading Orleans (UNCLASSIFIED)
Date: Wednesday, August 29, 2012 11:22:24 PM

Classification: UNCLASSIFIED
Caveats: NONE

Chris - we are raising the level in the canal. PS#7 asked to add their last 1000 cfs pump to drop the suction side water level. I am monitoring the levels throughout the canal and we will discontinue one of the pumps when we hit 3 to 4 foot. In 30 minutes they dropped their suction by more than a foot.

The canal has been raised by 1 foot (PS#7 - 2.57, ICS - 2.22).

Even though they are pushing more water than we can handled [sic], we still have the freeboard in the canal to handle the rise.

Donald"

For context, according to the August 29, 2012 PS#7 log, pump "C" (the "last 1000 cfs pump" referred to in Constantine's email) had been turned on at 10:45 PM, and - as Constantine noted - the inlet basin levels dropped from 11.2 feet at 10:30 PM to 10.1 feet at 11:30 PM. Of course, there was a corresponding increase in water levels in the canal, which is what concerned Constantine. For an unexplained reason, 550 cfs pump "A" was dialed back at 11:35 PM, likely because the inlet basin level had gotten low enough for the SWB operator's comfort. When it started rising again an hour later, "A" was restarted, maximizing the flow out of the SWB station. The SWB pump station 7 outlet basin level got to 23.5 feet (Cairo Datum). This is approximately equivalent to 3.4 feet on the Corps' level gauge (another Constantine email provides the conversation factor: subtract 20.1 feet from the SWB canal levels) resulting in Constantine's order to shut down pump "C," just as he said he would.

Constantine's email is remarkable on a number of counts. First, it details how low the Corps actually kept the water level in the Orleans Avenue canal during a tropical event: 3 to 4 feet. This is half of what the Corps was telling the public and their supposed local parters less than 48 hours previously, and is two feet lower than what they were even saying to each other internally just two hours earlier. For perspective, remember that the tops of the canal walls are actually 14 feet tall.

Secondly, it completely contradicts what Accardo had said in his email to the locals earlier. Constantine admits the SWB "are pushing more water than we can handle," and by virtue of his 3 to 4 foot cutoff, cuts the amount of freeboard available (think of it as head space) from Accardo's 6.5 feet to a maximum of 2.5 feet.

Third, it is abundantly clear there is no partnership between the Corps and the SWB; the Corps orders the SWB what to do during tropical storms, and it does so according to its own, secret criteria unknown to the public, or perhaps even the SWB. Note there are no cc's on the email, to the SWB or even to any other Corps personnel. In fact, I did not even receive this email as part of my request for Constantine's emails. If I hadn't asked for Accardo's emails, I wouldn't have even known about it.

So let's review:

August 27th, Corps Operations Chief Chris Accardo comments to the Lens about water levels in the Orleans Avenue canal: "it is not something people need to be worried about."

August 28th, 9:19 AM, Accardo emails to local stakeholders: "Orleans Ave - Max. Water Level is 8.0 ... We will turn pumps on and try to maintain a 1.5 canal elevation as soon as the SWB turns their pumps. Even though the SWB has greater pumping capacity, they will have to out-pump the Corps such that the water level in the canal will rise 6.5 ft. before there would be a problem."

August 28th, 9:18 PM: Accardo forwards spreadsheet internally to Corps personnel showing 5 foot maximum operating water elevation "trigger" on Orleans Avenue canal, caused by potential flooding of SWB pump station at that level.

August 29th, 2:20 AM: Corps Orleans Avenue canal captain Donald Constantine denies SWB's request to turn on 1000 cfs pump "C" in SWB station 7. Canal level is at about 3 feet at PS#7. PS#7 inlet basin (an indication of how much water is heading to the station from the streets) measures approximately 12.0 feet deep.

August 29th, 3:05 AM: SWB again denied pump "C." Canal level is about 3.1 feet at PS#7. PS#7 inlet basin measures approximately 11.3 feet deep.

August 29th, 4:45 AM: Heavy rainfall is sending lots of water toward the SWB pump stations throughout the city. SWB at PS#7 is again denied pump "C." Canal level is about 3.1 feet at PS#7. PS#7 inlet basin measures approximately 13.5 feet and rising sharply (it would climb another foot fifteen minutes after this). That represents water in the streets drained by PS#7. Pump "C" remains off (in part due to SWB power problems) the next 18 hours until...

August 29th, 10:45 PM: For the first time during Isaac, all three major pumps - including "C" - are turned on at PS#7.

August 29th, 11:22 PM: Constantine emails only Accardo, reporting rising canal water level. He informs Accardo he is keeping the canal level between 3 and 4 feet and will turn off a SWB pump if the level gets beyond that. Says the SWB is "pushing more water than we can handle" but "we still have the freeboard in the canal to handle the rise." Note this is within minutes of the SWB first turning on all their pumps.

August 30th, 12:55 AM: SWB ordered by Corps' Constantine to turn pump "C" off. Canal level is 3.4 feet. PS#7 inlet basin measures approximately 12.2 feet. The inlet basin would ultimately climb to 13.6 feet before starting to come down with a lessening of rainfall.

As we can see, the Corps' method of operating the Orleans Avenue canal during Isaac was not to have its pumps keep up with whatever the SWB could send their way. In fact, it was quite the opposite. They insisted the SWB keep one of their biggest pumps off whenever it was most needed. When that pump did finally get turned on, the Corps had the SWB turn it off less than two hours later. They even memorialized their intention to do so in an email just 37 minutes after the pump came on! This is because the Corps was operating the canal at a water level of about 3 feet, not the 8 feet they claimed to the public and their local stakeholders. That is, the Corps was actively keeping water out of the Orleans Avenue drainage canal during the entire time Isaac was over the city.

So that's the very wide difference between what the Corps presents to the public and what actually happens. Let's look at another aspect of Isaac along the Orleans Avenue canal, the fire at the Corps' pumps on the morning of August 29th, during the heaviest rainfall.

The Corps Pump Fire During Isaac

I first got wind of that fire from the Sewerage and Water Board's pump log for pump station #7. At 9:40 AM, the log reported:

"Donald Constantine w/C.O.E. reported a Fire w/C.O.E pump"

The Corps' emails provide more detail. Constantine sent this email at 9:25 AM on the 29th:

"[Corps mechanic] Robert Gauthreaux just reported that the turbo on pump W5 caught on fire. Fire was put out and pump is being checked out for operability.
Orleans is currently down to 7 pumps at this point."

About an hour later, at 10:21 AM Constantine provided follow-up

"Looks like a hydraulic leak spraying on turbo that caught fire. Engine will run after short term fix of leak.
Crew working on it."

At 11:08 AM, the problem had been addressed, according to this email from Constantine:

"Pump W5 has been repaired and runnIng again. We are back to 80 percent."

Recall two of the 10 hydraulic pumps at the Orleans Avenue site had been pulled out before Isaac for corrosion rebuilding and had not been returned to the site by August 29th. This left just 8 of 10 pumps, or 80%.

So what exactly happened?

A few weeks after Isaac, Sandy Rosenthal of levees.org was given a tour of the Orleans Avenue gate structure by the Corps. She took pictures, including this one titled "piece w/ fracturedfailed:"

What you are looking at is an Omega Engineering Incorporated model PSW-194 hydraulic pressure switch, with some hardware screwed into the left hand side of it. You can see a picture of the entire switch at Omega's page here. Specifically, this is the switch that was installed on the engine skid for pump W5 at the time of the fire on August 29, 2012. It was meant to switch on or off at a specified hydraulic fluid pressure, sending a signal to a butterfly valve on the skid to divert hydraulic fluid to a bypass/recirculation line, thus decreasing the pressure of the fluid going to the W5 water pump in the canal. Further details on how the engine skids work can be found at this post. Such switches are installed on all 40 hydraulic pumps across all three gate sites.

The switch appears to be missing a piece to the right of the label. Ms. Rosenthal took a picture of an undamaged switch on the engine skid for pump W4:

Here's the two switches side by side:

The blue highlighted area is the portion of the switch missing from the W5 unit. I've drawn in what appears to be the line where the switch may have fractured.

I want to caution I am proceeding strictly on the photographs, their titles, and Ms. Rosenthal's relation of her October 18, 2012 conversations with the Corps via email. She said the Corps told her the fire was caused by "a fracture, a flaw in the fitting." She said the first picture above - showing the pressure switch - was a picture of "the piece that failed."

I contacted Omega, asking for clarification or any backgrund on the failure, but their in-house counsel B. Christine Riggs refused all comment, saying only, "While we appreciate your effort to obtain further information, Omega does not disclose information on its customers and their orders to third parties."

In a follow-up conversation with Corps New Orleans District Public Affairs chief Ken Holder last week, the Corps confirmed the substance of their October statements to Ms. Rosenthal, saying the fire was caused by the Omega switch. Holder said the failure would better be described as an "internal failure," likely from a seal or membrane, rather than a "fracture." This leaves me somewhat mystified what I'm looking at in the pictures, but it does get me 90% of the way to the nut of the story: the fire was caused by the hydraulic switch failing.

According to Ms. Rosenthal, the Corps said pressurized, very hot hydraulic fluid sprayed out the switch as a mist on to the switch's flexible conduit and the adjacent diesel engine turbocharger, also known as a "turbo." The conduit melted and the insulation caught fire. Ms. Rosenthal took pictures of the damage:

Here's a picture of the undamaged turbo on the W4 skid for comparison:

The fire was put out using a fire extinguisher in about a minute. The approximate pressure of the hydraulic fluid at the time was about 2500 pounds per square inch, or psi (indicating the water pump was not pumping at full speed, which would correspond to about 3000 to 3200 psi). According to the manufacturer's information, the switch is rated for pressures up to 3000 psi.

In late September, 2006, while the pumps were still very much in a "testing" phase, the Corps requested from the pumps' manufacturer - MWI of Deerfield Beach, Florida - the addition of hydraulic pressure switches to all 40 units across the three sites. After some back and forth, the Corps gave the go-ahead to MWI on October 18, 2006 as modification 23 to the original pump contract (entire contract here). One can assume the switches in 2006's modification 23 are same ones in service on August 29, 2012. Holder confirmed the Corps has not changed the style or make of switch since 2006.

Notably, a few months prior to the hydraulic pressure switch modification - in June and July of 2006 - the Corps had requested the installation of hydraulic pressure transmitters and temperature on the pump skids as add-ons to the original design. It is unusual that the pumps were originally ordered with no electronic capability to monitor hydraulic temperature and pressure, or with the capacity lent by the hydraulic pressure switches, which was presumably to keep the system from overpressurizing. Given the sensitivity to the entire hydraulic system at higher pressures (around 3000 psi), these switches were likely meant to keep the pumping systems from failing.

So with all that background in hand, let's do another review of the Corps' contemporaneous public statements about one of their pumps catching fire during a hurricane in New Orleans:

"                                                                                                                  "

Oh right. They never said a word. Just like they never let the public know they'd be operating the canal at a water level 5 feet below what they said publicly, and 11 feet below the tops of the 14 foot high walls.

That's just how they roll. You almost have to admire their ability to keep stuff hidden, even with two reporters and a Senator looking over their shoulders (the NY Times' John Schwartz, the Times-Picayune's Katy Reckdahl Rebecca Mowbray, and Senator David Vitter were all "embedded" with the Corps August 28th and 29th). It appears to be the only thing you can count on happening.

Next up in Part 3: the events along the 17th Street canal during Isaac (which even the Corps couldn't hide), along with more behind-the-scenes shenanigans with the level gauges.

Note regarding the emails linked in this part:
The 8/29/12 11:22 PM email from Donald Constantine to Chris Accardo quoted above is part of the latest tranche of emails received via my October, 2012 FOIA request. I got all of Accardo's inbox from August 29th and 30th, 2012 and what I presume are most of his "sent" box from August 27th through the 30th. Also included were most of the attachments to those emails, though some key attachments remain unsent. Also missing are Accardo's inbox emails from the 27th and 28th, though some of them reside in the ealier releases by virtue of carbon copies to others. I have asked the Corps to rectify these oversights.

These emails were sent in an odd format. While all the inbox emails were gathered into a single file, all the sent emails were sent individually. I have linked to the inbox emails, but I'm still collating the sent emails into a single file for downloading.