Isaac: the rest of the story, Part 3
Part 1
Part 2
NOTE: I am currently awaiting more material from my Isaac email FOIA request, including a PowerPoint presentation from the evening of August 30, 2012 which was supposed to include a candid assessment of pumping at the outfall canals. In the interim, I've pulled together what I could regarding events along the 17th Street canal during Isaac. This post may change as the remaining information is received.
When I placed my FOIA request for the Isaac emails, I had hoped to not only uncover unknown events during the storm, but also to get more details on the pump failures that occurred at the 17th Street canal on the morning of August 29th. Those failures were so extreme the Corps had to report them to the media. Here's the timeline from my October 6, 2012 entry, "Isaac in New Orleans - what we know so far:"
Oddly, there's zero Corps email traffic among the key personnel (Ray Newman, Don Schneider, and Chris Accardo) through that whole time. I would expect something to have been written by someone over that three-plus hour span, but perhaps they were just so busy they didn't have time to type on their Blackberrys. Or perhaps they were all communicating either over the radio or in person, so electronic communication was unnecessary. When there was a pump drive skid fire at the Orleans Avenue site a little over an hour after this event subsided, there was an email trail immediately created, so it's very strange there's no trail for the biggest event during the storm.
How high did the water get in the 17th Street canal on August 29, 2012?
The first question to be asked after an event like this is whether the 17th Street canal maximum operation water level of 6.5 feet (first revealed to the public during the press coverage of this mishap) was exceeded. If it was, that means the walls and levees were loaded beyond their safe capacity. To determine this, normally we would examine the outfall canal level gauge records kept at the Corps' rivergauges.com website, but the Corps' level gauges along the canal were not responding reliably on August 29th.
However, there is another level gauge that did not fail: the Sewerage and Water Board's gauge at drainage pump station #6, located at the southern end of the canal.
That gauge uses a different datum for its zero point, an antiquated thing called "Cairo Datum." It must be converted to normal water levels, and sometimes figuring that conversion out is difficult. I've gotten different responses over the years. However, I just recently found the definitive source for the conversion. Take a look at this photo:
This was taken by an architectural graduate student affiliated with a group project called "Gutter to Gulf." They visited New Orleans in February, 2009 and stopped in at pump station 6. One of them snapped this photo inside the pump station control room. The digital readout on the right shows the water levels on the inlet (top row) and outlet (bottom row left) of the station. Of far more interest is the paper above that display:
There's our conversion factor for the discharge water levels at DPS#6: 6.0 feet in normal terms equals 27.6 feet in Cairo Datum terms. The safe water elevation was raised since 2009 to 6.5 feet, which was the number in effect when Isaac hit. So anything over 28.1 feet as measured by the DPS#6 discharge water gauge indicates the 17th Street safe water elevation was exceeded.
Examining the DPS#6 discharge level gauge readings on the morning of August 29, 2012, as recorded in the station's logs:
The logs show the discharge gauge read above 28.1 feet for almost two hours, at one point exceeding it by a full 0.5 feet, or six inches. That is, the water got to 7 feet in the canal, placing the walls and levees in danger. This is the first time such information has been reported publicly.
How bad were the Corps pump failures?
The next question to ask is how this happened. We know there were sizable pump failures at the Corps end of the canal, but how sizable were they? Let's go back two days before the storm.
Despite Corps Operations chief Chris Accardo's assurances to the locals and others in the days leading up August 29th, all was not well at the Corps' 17th Street site. Just before noon on August 27th, the Corps' 17th Street canal captain Ray Newman sent an email to Kirk Bowman of Point Eight Power in Belle Chasse, LA, in nearby Plaquemines Parish. Point Eight was (and remains) responsible for the pump controls on the 11 direct drive pumps at 17th Street and the 8 direct drive pumps at the London Avenue canal site. Newman's email reads,
This email is revealing in a number of ways. On the surface, it points to problems on two of the supposedly more reliable direct drive pumps just ahead of a hurricane. Overriding the automatic shutdown commands could point to the Corps wishing to temporarily bypass a nuisance interlock that was not indicating a real problem. That's the most generous explanation. The more likely - and less charitable - explanation was that pump W19 was experiencing low oil pressure on its' gearbox and pump W18 was experiencing high vibrations on its' gearbox. The Corps needed these pumps to run, so they told Point Eight to pull those interlocks out of the program.
But in a greater sense, it points to how thin the Corps itself is on expertise in their own equipment. For an experienced industrial programmer, bypasses like these should be a piece of cake. But the Corps obviously has no such person on staff. They had to summon a contractor to make programming changes on the eve of a major storm. We see this over and over - the Corps themselves doesn't just contract out duties like engineering, quality control, and quality assurance. They also farm out their emergency response activities under the pressure of a storm, leaving until the last minute the call to these folks, hoping the companies have decided to keep their employees around. The Corps can order their own personnel to stick around; their power over contractors is much more murky and lends a sense of unease to citizens depending upon the correct functioning of life-critical storm protection systems.
But back to the events surrounding the morning of August 29th. As noted above, S and WB drainage pumping station 6 was limited to 5200 cfs at one point. This means the Corps pumps were likely pumping at approximately the same rate. What does that mean in real terms?
Here's the layout of the pumps at the Corps 17th Street site:
Here's the numbering of each pump at 17th Street:
According to the Corps' "pump tracker" spreadsheet, the 18 "Phase 1" and "Phase 2" pumps, which are the 60" hydraulic pumps made and supplied by MWI, each have a nominal rated capacity of 200 cfs. The 11 larger direct drive pumps (abbreviated "DD" in Corps pump status emails) each have a nominal rated capacity of 364 cfs. The 14 "Phase 3," or "bridge," pumps - also hydraulically powered units made by MWI - each have a nominal rated capacity of 114 cfs.
We can get an idea of what was going on early on the morning of the 29th by examining what happened afterward. You see, the Corps' pump problems did not disappear at 8 AM. The problems just got small enough, and the rainfall lessened up enough, that the Corps pumps could barely squeak by. In fact, emails subsequent to the morning crisis reveal substantial problems remained. Just after 10 AM, Corps Operations Assistant Chief Jerry Colletti sent out an email requesting pump statuses from all three gate sites.
At 11:20 AM, the Corps' Michael Sullivan sent back the first update for 17th Street:
Unfortunately, we don't have what Sullivan had possibly previously sent when he referred to "no change." Also, it is unclear if Sullivan, when he wrote "ICS capacity," was referring to the maximum capacity the Corps' station could pump, or what the station was currently pumping. However, the lack of clarity was eliminated in subsequent updates. A 12:03 PM email said:
So with 3 phase 1 pumps, 1 direct drive pump, and 3 bridge pumps down at noon on August 29, 2012, the theoretical capacity of the Corps 17th Street ICS was 9200 minus (3 times 200 + 1 times 364 + 3 times 114), or 7894 cfs, which matches what Sullivan reported. Put another way, the Corps's pump station - which does not have any spare capacity when the nearly 10,000 cfs city pump station #6 is going all out - was only running at 86% at noon on the 29th
What that means is that if the city pumps had needed to put more than 7894 cfs down the canal (and they can, and have before), the crisis would have restarted again.
From this sort of information, we can backtrack and figure out how poorly the Corps pumps were doing between 6:30 AM and 8 AM. We know they had 7 pumps down at noon and that resulted in a capacity of 7894 cfs. Let's round that to 7900 for ease of calculation.
Between 6:30 and 8 they couldn't pump more than between 5200 and 6200 cfs, or an additional 1700 to 2700 cfs deficit. At a minimum, this means at least 5 additional direct drive pumps, or some combination of an even larger number of pumps - some with smaller rated flows - were out of commission at that time.
Assuming the pumps offline at noon were also down earlier in the morning, this would mean at least 12 - a dozen - and possibly many more pumps were not available for nearly two hours during the height of Isaac. Given there's 43 pumps total at the site, just on a raw pump count basis that means nearly 28% of the pumps were non-functional. On a flowrate basis, it means the Corps was down between a third and nearly half of their rated throughput.
Nearly third to a half the Corps' pumping capacity was offline for hours... at the very time such pumps were needed. That allowed the 17th Street canal walls to have their safe level exceeded for hours as well. Let that sink in. At the moment the Corps was supposed to be providing maximum storm protection through pumping, they were failing so miserably the outfall canals were threatened with another breach.
Pumping failures continued throughout Isaac
Frankly, it was pure luck there wasn't more rain that day, and that is what saved the Corps' (and by extension, the city's) bacon. Because while the Corps got some of their stuff in order, after noon the news got worse at the 17th Street ICS.
At 2:04 PM, they lost another phase 1 hydraulic pump:
[Note this flow report for the ICS refers to Corps pumps turned on, not those actually ready to run]
Again - I cannot say this enough - it was pure luck that the city was only sending 3000 cfs down the canal at that time. The loss of a fourth hydraulic pump dropped the Corps' pumping capacity to 7694 cfs, or 83.6% - its lowest point outside the crisis period in the morning. The capacity at pump station 6 remained at its maximum of 9300 cfs - every city pump was available. The Corps pumps would never reach their maximum capacity during Isaac.
That situation continued for about 4 hours, during which the city fortunately pumped a steady 3000 cfs down the canal. The 6:01 PM update showed some improvement on the Corps end of things:
So they got 2 of the phase 1 pumps and one of bridge pumps going, raising the ICS capacity to 8208 cfs, or 89%. The ICS remained at that pumping level until at least 10 PM.
Then, at 11:43 PM things appeared to get worse again. Donald Schneider sent the following email:
Once again, the Corps was dependent on a contractor - Point Eight Power - to make things run. This time - it was during the storm. As soon as they ran into the smallest problem that appeared to be "computerey," they were forced to throw up their hands and send up the bat-signal because they had no programming knowledge.
This problem is possibly the source of a report from WWLTV titled "17th street canal pumps forced to be started manually Tuesday night" Most of the article is a confused rehash of what had happened Tuesday morning. But the first line says:
Notice how the emergency role of contractors in bypassing the pump control programming was not even mentioned.
The final report on pump statuses came the following morning at 5:06 AM
So even then, the Corps was down 5 pumps. This and the other updates through the afternoon and evening of August 29th completely contradict reporting by the New York Times' John Schwartz, who claimed regarding the pump status at 8 AM on the 29th: "Within a couple of hours, workers would get all of the pumps running and bring the water levels down, and the sense of potential crisis eased." The Corps never got "all" the pumps running. They got enough running at 8 AM to barely do the job at the same time rainwater in the city's drainage system was slackening off.
Other stuff going on at the same time
The events on the periphery of the Corps failures at the 17th Street canal - and they were failures, which were eventually only remedied by the luck of the weather - are just as interesting. I'd like to highlight two email chains from that day, both of which only came to light with the receipt of Chris Accardo's emails.
The first was an email chain involving the then-New Orleans District commander Colonel Ed Fleming. Fleming has since moved on to Corps HQ in Washington. That morning, Fleming responded to an inquiry from the Army in Washington DC asking what was happening at 17th Street:
I'll deal with the misinformation in the first passage later. But first, I'd like to opine on the second statement that the Corps' "goal is to stay ahead of SWB." This reeks of the "it's close enough, we don't need any extra" philosophy that appears to permeate the New Orleans District. They always appear to be looking for a corner to cut or a penny to pinch, designing or building or operating things to the gnat's ass as if there's never going to be a circumstance where extra anything will be needed. Meanwhile, they ignore gaping holes (sometimes literally) and try to cover them with duct tape and press releases.
What happens when the SWB is pumping at 80% of capacity and the Corps only has 50% available? You get the freak-out we saw on the morning of August 29, 2012. The goal for the Corps' pumps should not be just to stay ahead of the SWB, because no one can predict the weather. The goal should be 100% reliable pumping at any time, for any duration, with spare capacity built in to overcome unforeseen failures. The SWB pump stations were built this way, but the Corps decided to go with "probably good enough" and "trust us." Their calculations for safe water elevations within the canals are of the same flavor - they only calculated the strength of the walls up to 10 feet, even though the walls are actually 15 feet or taller. What would happen in the case of a failure of a gate segment at the lake and the inrush of storm surge, as easily could have happened at London Avenue with the unpinned gates? Would the Corps say "we built stuff good enough to handle a scenario we deemed most likely, despite the dire circumstances a less likely scenario would present?" Phrased another way, that question would be "Katrina? Never heard of her."
At the very least, these massive pump failures show all the supposed exercising of pumps during and outside of storm season is clearly garbage. Under fewer than 12 hours of stress, their system nearly collapsed. The ability to run a single pump on a sunny day for ten minutes in front of TV cameras and a few dignitaries (as the Corps does every May) simply cannot compare with the need to run 43 pumps on the rainiest day of the year for 12 to 24 hours.
It's like someone telling you, "It'll work until it breaks." We saw how well that philosophy worked on August 29th, 2005, and it does not appear to have been excised from the New Orleans District at all.
Now, to the half-truths in Fleming's first passage. We know the Corps told the SWB to shut down 1000 cfs pump "D" in PS6 at about 6:40 AM on August 29th. That came after a an earlier refusal - at 4:35 AM - to allow the SWB to run 1100 cfs pump "F." A second refusal on "F" came at 6:50 AM. In addition, the SWB was told not to load any more pumps above the 5200 cfs they were flowing at 6:40. That's a potential 4000 cfs kept out of the canal, not 500 cfs.
Also, the first refusal came at 4:35 AM. The SWB was not allowed to turn on any pumps until 7:55 AM. That's well over three hours, not "a little while."
The second email chain of interest involves Corps engineering personnel within and outside the New Orleans District on August 29th.
At 9:33 AM, Nancy J. Powell, Chief of the Hydraulics and Hydrologic Branch at the Corps' New Orleans District, emailed Walter Baumy, the then-chief of the New Orleans' District's Engineering Division. Also copied was David Ramirez, the New Orleans District Chief of Water Management. By this time, multiple media reports were already mentioning the faulty level gauge readings within the canals, readings available to the public at rivergauges.com. During storms, Powell is a point person for storm modeling data, including predicted surge levels. She wrote:
"MVR" is the Corps' Rock Island District, headquartered in Rock Island, Illinois. They run the riverguages website. Powell was attempting to find out if the Corps could stop the bad readings from being reported to the public.
A little over two hours later, at 11:48 AM, Baumy forwarded Powell's email to Denny A. Lundberg, the engineering chief at the Rock Island District, with the additional message:
Oh no! The public was calling about safe water elevations! We can't have that.
Lundberg wrote back about an hour later, at 12:54 PM:
"Whoops" indeed.
Baumy wrote back a few minutes later with the last message we have, copying Chris Accardo, Powell, Ramirez, Emergency Operations honcho Michael Stack, and - most tellingly - two members of the New Orleans District's Public Affairs staff: chief Ken Holder and contractor Rachel Rodi:
Based on the records on the rivergauges website, the Corps never carried out this plan. And with squinty eyes, I can see a sorta justification for not putting messed up information out on a public website. But the overriding concern here doesn't appear to be misinforming the public - or even what was being done to address the problem (something one would expect Baumy, the head of engineering for the District, to be asking), but rather exposure of the Corps' missteps to public scrutiny in real time. Having seen the Corps' penchant for keeping outfall canal information away from the public, my view may be a bit colored, but the plain language of the email exchange certainly doesn't cover the Corps in glory and worry for the flood-endangered public.
Summary
The events at the 17th Street canal on August 29, 2012 appear to be more serious and longer lasting than the reporting that morning and afterward indicated. The water level in the canal exceeded the Corps' safe water elevation for over two hours in the morning because approximately a dozen Corps pumps failed and could not remove the rainwater being put into the canal by city pumps. The Corps' pumping capacity was cut nearly in half at the height of the need for pumping. Those Corps pump failures prompted orders to the locals to not activate some pumps and in one case to turn one off, causing stormwater to back up into city streets.
After the heaviest rain abated and the Corps got a handfull of their pumps working, water levels inside the canal and in the city streets dropped. However, many Corps pumps continued out of commission throughout the event, raising the probability that if rainfall had again picked up, a crisis identical to that of the morning of August 29th would again occur.
Occurring in parallel with these events were Corps efforts to shape the narrative, often with faulty information. One effort even contemplated cutting the public (and the local pumping authority) off from level gauge data.
Strictly due to cost, the Corps has decided to make this tandem pumping system with weakened canal walls permanent, their so-called "Option 1." Ground has broken for permanent pump stations to replace the clearly faulty Interim Closure Structures, but no information has been released regarding further work to make the canals themselves sturdier. Taken together, the outfall canals remain a serious Achilles heel within the confines of the City of New Orleans and Jefferson Parish.
Part 2
NOTE: I am currently awaiting more material from my Isaac email FOIA request, including a PowerPoint presentation from the evening of August 30, 2012 which was supposed to include a candid assessment of pumping at the outfall canals. In the interim, I've pulled together what I could regarding events along the 17th Street canal during Isaac. This post may change as the remaining information is received.
When I placed my FOIA request for the Isaac emails, I had hoped to not only uncover unknown events during the storm, but also to get more details on the pump failures that occurred at the 17th Street canal on the morning of August 29th. Those failures were so extreme the Corps had to report them to the media. Here's the timeline from my October 6, 2012 entry, "Isaac in New Orleans - what we know so far:"
The PS#6 logs for August 29th (here and here) confirm the outlines of this story, while filling in some detail. At 4:35 AM, the station staff asked to "load" (that is, bring to full capacity) 1100 cfs pump "F." They were refused. Then, at 4:55 AM the Corps told the S and WB not to "load" any more pumps in PS#6:
"Note: Army Corps of Engineers said we cannot load any more pumps. [T]hey are at their max of what they can pump"At that time, the station was pumping approximately 6200 cubic feet per second, or cfs. The station's capacity among its main rainstorm drainage pumps is 9200 cfs.
This initial notice to the S and WB to hold off on pumping appears to be 1.5 to 2 hours before the Corps alerted the media of pumping restrictions, according to the chronology of the NY Times article. Then, at about 6:30 AM, the station operators received another Corps warning not to load additional pumps beyond what was already running, despite the fact that in the intervening hours, water in the station's inlet basin had risen over four feet. Ten minutes later, the station received orders from S and WB's Central Control to break the prime on 1000 cfs pump "D," effectively losing most of its capacity and sending into a pump status called "light." So there were not only Corps orders to keep some pumps from running, but one pump was effectively ordered turned off with the water coming up. This left the station running at about 5200 cfs.
At 7:55 AM, with the water still very high in the inlet basin, the Corps 17th Street canal captain in the station (Ray Newman) gave an all clear to the S and WB personnel in the station to load additional pumps. Five minutes later, 1000 cfs pump "D" was authorized to be loaded by the S and WB Central, and was running in a loaded condition within a few minutes. Another 1000 cfs pump ("C") was brought on line shortly afterward, increasing the station's flowrate to 7200 cfs. Inlet basin levels dropped four feet in about an hour after those two pumps were added. This chronology roughly tracks what was playing out in the media that morning.
Oddly, there's zero Corps email traffic among the key personnel (Ray Newman, Don Schneider, and Chris Accardo) through that whole time. I would expect something to have been written by someone over that three-plus hour span, but perhaps they were just so busy they didn't have time to type on their Blackberrys. Or perhaps they were all communicating either over the radio or in person, so electronic communication was unnecessary. When there was a pump drive skid fire at the Orleans Avenue site a little over an hour after this event subsided, there was an email trail immediately created, so it's very strange there's no trail for the biggest event during the storm.
How high did the water get in the 17th Street canal on August 29, 2012?
The first question to be asked after an event like this is whether the 17th Street canal maximum operation water level of 6.5 feet (first revealed to the public during the press coverage of this mishap) was exceeded. If it was, that means the walls and levees were loaded beyond their safe capacity. To determine this, normally we would examine the outfall canal level gauge records kept at the Corps' rivergauges.com website, but the Corps' level gauges along the canal were not responding reliably on August 29th.
However, there is another level gauge that did not fail: the Sewerage and Water Board's gauge at drainage pump station #6, located at the southern end of the canal.
That gauge uses a different datum for its zero point, an antiquated thing called "Cairo Datum." It must be converted to normal water levels, and sometimes figuring that conversion out is difficult. I've gotten different responses over the years. However, I just recently found the definitive source for the conversion. Take a look at this photo:
There's our conversion factor for the discharge water levels at DPS#6: 6.0 feet in normal terms equals 27.6 feet in Cairo Datum terms. The safe water elevation was raised since 2009 to 6.5 feet, which was the number in effect when Isaac hit. So anything over 28.1 feet as measured by the DPS#6 discharge water gauge indicates the 17th Street safe water elevation was exceeded.
Examining the DPS#6 discharge level gauge readings on the morning of August 29, 2012, as recorded in the station's logs:
The logs show the discharge gauge read above 28.1 feet for almost two hours, at one point exceeding it by a full 0.5 feet, or six inches. That is, the water got to 7 feet in the canal, placing the walls and levees in danger. This is the first time such information has been reported publicly.
How bad were the Corps pump failures?
The next question to ask is how this happened. We know there were sizable pump failures at the Corps end of the canal, but how sizable were they? Let's go back two days before the storm.
Despite Corps Operations chief Chris Accardo's assurances to the locals and others in the days leading up August 29th, all was not well at the Corps' 17th Street site. Just before noon on August 27th, the Corps' 17th Street canal captain Ray Newman sent an email to Kirk Bowman of Point Eight Power in Belle Chasse, LA, in nearby Plaquemines Parish. Point Eight was (and remains) responsible for the pump controls on the 11 direct drive pumps at 17th Street and the 8 direct drive pumps at the London Avenue canal site. Newman's email reads,
"Kurt,
This message is your authorization to have your field service tech. override the shutdown safety devices at our request, as listed below. These are temporary measures to provide vital pumping capacity for our operations during TS/Hurricane Isaac. Our operators will monitor the equipment manually while these shutdowns are inactive.
a) Override low gearbox oil pressure shutdown on Unit No. 19, 17th St. Canal
b) Override gearbox vibration shutdown on Unit No. 18, 17th St. Canal
Thanks,
Ray"
This email is revealing in a number of ways. On the surface, it points to problems on two of the supposedly more reliable direct drive pumps just ahead of a hurricane. Overriding the automatic shutdown commands could point to the Corps wishing to temporarily bypass a nuisance interlock that was not indicating a real problem. That's the most generous explanation. The more likely - and less charitable - explanation was that pump W19 was experiencing low oil pressure on its' gearbox and pump W18 was experiencing high vibrations on its' gearbox. The Corps needed these pumps to run, so they told Point Eight to pull those interlocks out of the program.
But in a greater sense, it points to how thin the Corps itself is on expertise in their own equipment. For an experienced industrial programmer, bypasses like these should be a piece of cake. But the Corps obviously has no such person on staff. They had to summon a contractor to make programming changes on the eve of a major storm. We see this over and over - the Corps themselves doesn't just contract out duties like engineering, quality control, and quality assurance. They also farm out their emergency response activities under the pressure of a storm, leaving until the last minute the call to these folks, hoping the companies have decided to keep their employees around. The Corps can order their own personnel to stick around; their power over contractors is much more murky and lends a sense of unease to citizens depending upon the correct functioning of life-critical storm protection systems.
But back to the events surrounding the morning of August 29th. As noted above, S and WB drainage pumping station 6 was limited to 5200 cfs at one point. This means the Corps pumps were likely pumping at approximately the same rate. What does that mean in real terms?
Here's the layout of the pumps at the Corps 17th Street site:
Here's the numbering of each pump at 17th Street:
According to the Corps' "pump tracker" spreadsheet, the 18 "Phase 1" and "Phase 2" pumps, which are the 60" hydraulic pumps made and supplied by MWI, each have a nominal rated capacity of 200 cfs. The 11 larger direct drive pumps (abbreviated "DD" in Corps pump status emails) each have a nominal rated capacity of 364 cfs. The 14 "Phase 3," or "bridge," pumps - also hydraulically powered units made by MWI - each have a nominal rated capacity of 114 cfs.
We can get an idea of what was going on early on the morning of the 29th by examining what happened afterward. You see, the Corps' pump problems did not disappear at 8 AM. The problems just got small enough, and the rainfall lessened up enough, that the Corps pumps could barely squeak by. In fact, emails subsequent to the morning crisis reveal substantial problems remained. Just after 10 AM, Corps Operations Assistant Chief Jerry Colletti sent out an email requesting pump statuses from all three gate sites.
At 11:20 AM, the Corps' Michael Sullivan sent back the first update for 17th Street:
"No change.
ICS Capacity - 7700cfs
PS6 Capacity - 6200 cfs, will increase to 7300cfs w in 10 min."
Unfortunately, we don't have what Sullivan had possibly previously sent when he referred to "no change." Also, it is unclear if Sullivan, when he wrote "ICS capacity," was referring to the maximum capacity the Corps' station could pump, or what the station was currently pumping. However, the lack of clarity was eliminated in subsequent updates. A 12:03 PM email said:
"3 phase 1, 1 DD, 3 Bridge - DOWN.
ICS - 7894cfs
PS6 - 6200cfs, going to 7200cfs in 5."
So with 3 phase 1 pumps, 1 direct drive pump, and 3 bridge pumps down at noon on August 29, 2012, the theoretical capacity of the Corps 17th Street ICS was 9200 minus (3 times 200 + 1 times 364 + 3 times 114), or 7894 cfs, which matches what Sullivan reported. Put another way, the Corps's pump station - which does not have any spare capacity when the nearly 10,000 cfs city pump station #6 is going all out - was only running at 86% at noon on the 29th
What that means is that if the city pumps had needed to put more than 7894 cfs down the canal (and they can, and have before), the crisis would have restarted again.
From this sort of information, we can backtrack and figure out how poorly the Corps pumps were doing between 6:30 AM and 8 AM. We know they had 7 pumps down at noon and that resulted in a capacity of 7894 cfs. Let's round that to 7900 for ease of calculation.
Between 6:30 and 8 they couldn't pump more than between 5200 and 6200 cfs, or an additional 1700 to 2700 cfs deficit. At a minimum, this means at least 5 additional direct drive pumps, or some combination of an even larger number of pumps - some with smaller rated flows - were out of commission at that time.
Assuming the pumps offline at noon were also down earlier in the morning, this would mean at least 12 - a dozen - and possibly many more pumps were not available for nearly two hours during the height of Isaac. Given there's 43 pumps total at the site, just on a raw pump count basis that means nearly 28% of the pumps were non-functional. On a flowrate basis, it means the Corps was down between a third and nearly half of their rated throughput.
Nearly third to a half the Corps' pumping capacity was offline for hours... at the very time such pumps were needed. That allowed the 17th Street canal walls to have their safe level exceeded for hours as well. Let that sink in. At the moment the Corps was supposed to be providing maximum storm protection through pumping, they were failing so miserably the outfall canals were threatened with another breach.
Pumping failures continued throughout Isaac
Frankly, it was pure luck there wasn't more rain that day, and that is what saved the Corps' (and by extension, the city's) bacon. Because while the Corps got some of their stuff in order, after noon the news got worse at the 17th Street ICS.
At 2:04 PM, they lost another phase 1 hydraulic pump:
"4 phase 1, 1 DD, 3 Bridge - DOWN.
ICS - 2900 cfs
PS6 - 3000 cfs"
[Note this flow report for the ICS refers to Corps pumps turned on, not those actually ready to run]
Again - I cannot say this enough - it was pure luck that the city was only sending 3000 cfs down the canal at that time. The loss of a fourth hydraulic pump dropped the Corps' pumping capacity to 7694 cfs, or 83.6% - its lowest point outside the crisis period in the morning. The capacity at pump station 6 remained at its maximum of 9300 cfs - every city pump was available. The Corps pumps would never reach their maximum capacity during Isaac.
That situation continued for about 4 hours, during which the city fortunately pumped a steady 3000 cfs down the canal. The 6:01 PM update showed some improvement on the Corps end of things:
"(2) Phase 1 pumps out (change), (1) DD pump out, (2) Bridge pumps out (change)
17th ICS: 2700, 0.5 gage
PS 6: 2200, 0.7 gage"
So they got 2 of the phase 1 pumps and one of bridge pumps going, raising the ICS capacity to 8208 cfs, or 89%. The ICS remained at that pumping level until at least 10 PM.
Then, at 11:43 PM things appeared to get worse again. Donald Schneider sent the following email:
"11W and now 20W are now malfunctioning the same fault. Start the engine the point 8 screen goes to reboot and the engine dies. Never saw this one before. Gotta let point 8 wrestle this one its programing."
Once again, the Corps was dependent on a contractor - Point Eight Power - to make things run. This time - it was during the storm. As soon as they ran into the smallest problem that appeared to be "computerey," they were forced to throw up their hands and send up the bat-signal because they had no programming knowledge.
This problem is possibly the source of a report from WWLTV titled "17th street canal pumps forced to be started manually Tuesday night" Most of the article is a confused rehash of what had happened Tuesday morning. But the first line says:
"Pumps at the 17th Street canal failed to turn on automatically Tuesday night, forcing pump operators to manually start the system, the Corps of Engineers said Thursday."
Notice how the emergency role of contractors in bypassing the pump control programming was not even mentioned.
The final report on pump statuses came the following morning at 5:06 AM
Pumps out: 1 Phase 1, 2 DD, 2 Bridge
17th:1400 cfs, 0.2 gage
PS6: 2200 cfs, 0.2 gage
So even then, the Corps was down 5 pumps. This and the other updates through the afternoon and evening of August 29th completely contradict reporting by the New York Times' John Schwartz, who claimed regarding the pump status at 8 AM on the 29th: "Within a couple of hours, workers would get all of the pumps running and bring the water levels down, and the sense of potential crisis eased." The Corps never got "all" the pumps running. They got enough running at 8 AM to barely do the job at the same time rainwater in the city's drainage system was slackening off.
Other stuff going on at the same time
The events on the periphery of the Corps failures at the 17th Street canal - and they were failures, which were eventually only remedied by the luck of the weather - are just as interesting. I'd like to highlight two email chains from that day, both of which only came to light with the receipt of Chris Accardo's emails.
The first was an email chain involving the then-New Orleans District commander Colonel Ed Fleming. Fleming has since moved on to Corps HQ in Washington. That morning, Fleming responded to an inquiry from the Army in Washington DC asking what was happening at 17th Street:
"- We never had to tell SWB to slow down. They wanted to bring another 500cfs on line and we asked them to wait a little while - which they did. We started the rest of the pumps manually. Then we told them to give us everything they could.
- Our pumping percent capacity is interesting but not determining. Our goal is to stay ahead of SWB. If we are pumping at 80% capacity and they are pumping at 50% capacity then we are in good shape. Our goal is to out-pump them."
I'll deal with the misinformation in the first passage later. But first, I'd like to opine on the second statement that the Corps' "goal is to stay ahead of SWB." This reeks of the "it's close enough, we don't need any extra" philosophy that appears to permeate the New Orleans District. They always appear to be looking for a corner to cut or a penny to pinch, designing or building or operating things to the gnat's ass as if there's never going to be a circumstance where extra anything will be needed. Meanwhile, they ignore gaping holes (sometimes literally) and try to cover them with duct tape and press releases.
What happens when the SWB is pumping at 80% of capacity and the Corps only has 50% available? You get the freak-out we saw on the morning of August 29, 2012. The goal for the Corps' pumps should not be just to stay ahead of the SWB, because no one can predict the weather. The goal should be 100% reliable pumping at any time, for any duration, with spare capacity built in to overcome unforeseen failures. The SWB pump stations were built this way, but the Corps decided to go with "probably good enough" and "trust us." Their calculations for safe water elevations within the canals are of the same flavor - they only calculated the strength of the walls up to 10 feet, even though the walls are actually 15 feet or taller. What would happen in the case of a failure of a gate segment at the lake and the inrush of storm surge, as easily could have happened at London Avenue with the unpinned gates? Would the Corps say "we built stuff good enough to handle a scenario we deemed most likely, despite the dire circumstances a less likely scenario would present?" Phrased another way, that question would be "Katrina? Never heard of her."
At the very least, these massive pump failures show all the supposed exercising of pumps during and outside of storm season is clearly garbage. Under fewer than 12 hours of stress, their system nearly collapsed. The ability to run a single pump on a sunny day for ten minutes in front of TV cameras and a few dignitaries (as the Corps does every May) simply cannot compare with the need to run 43 pumps on the rainiest day of the year for 12 to 24 hours.
It's like someone telling you, "It'll work until it breaks." We saw how well that philosophy worked on August 29th, 2005, and it does not appear to have been excised from the New Orleans District at all.
Now, to the half-truths in Fleming's first passage. We know the Corps told the SWB to shut down 1000 cfs pump "D" in PS6 at about 6:40 AM on August 29th. That came after a an earlier refusal - at 4:35 AM - to allow the SWB to run 1100 cfs pump "F." A second refusal on "F" came at 6:50 AM. In addition, the SWB was told not to load any more pumps above the 5200 cfs they were flowing at 6:40. That's a potential 4000 cfs kept out of the canal, not 500 cfs.
Also, the first refusal came at 4:35 AM. The SWB was not allowed to turn on any pumps until 7:55 AM. That's well over three hours, not "a little while."
The second email chain of interest involves Corps engineering personnel within and outside the New Orleans District on August 29th.
At 9:33 AM, Nancy J. Powell, Chief of the Hydraulics and Hydrologic Branch at the Corps' New Orleans District, emailed Walter Baumy, the then-chief of the New Orleans' District's Engineering Division. Also copied was David Ramirez, the New Orleans District Chief of Water Management. By this time, multiple media reports were already mentioning the faulty level gauge readings within the canals, readings available to the public at rivergauges.com. During storms, Powell is a point person for storm modeling data, including predicted surge levels. She wrote:
"David can call MVR and see if they can stop malfunctioning gages from reporting"
"MVR" is the Corps' Rock Island District, headquartered in Rock Island, Illinois. They run the riverguages website. Powell was attempting to find out if the Corps could stop the bad readings from being reported to the public.
A little over two hours later, at 11:48 AM, Baumy forwarded Powell's email to Denny A. Lundberg, the engineering chief at the Rock Island District, with the additional message:
"Denny, potential issues on outfalls, especially 17th. Public calling on safe water elevations."
Oh no! The public was calling about safe water elevations! We can't have that.
Lundberg wrote back about an hour later, at 12:54 PM:
"Whoops
No one has called MVR as of an hour ago. Any station can be easily turned off using the RiverGages management page. Takes about 30 seconds. I'll have my WC [Water Control] staff contact your staff."
"Whoops" indeed.
Baumy wrote back a few minutes later with the last message we have, copying Chris Accardo, Powell, Ramirez, Emergency Operations honcho Michael Stack, and - most tellingly - two members of the New Orleans District's Public Affairs staff: chief Ken Holder and contractor Rachel Rodi:
"If outfall gages are unreliable, they should be turned off in my opinion.
Will need to defend the unreliability as public will want access. Please provide your assessment.
Walter"
Based on the records on the rivergauges website, the Corps never carried out this plan. And with squinty eyes, I can see a sorta justification for not putting messed up information out on a public website. But the overriding concern here doesn't appear to be misinforming the public - or even what was being done to address the problem (something one would expect Baumy, the head of engineering for the District, to be asking), but rather exposure of the Corps' missteps to public scrutiny in real time. Having seen the Corps' penchant for keeping outfall canal information away from the public, my view may be a bit colored, but the plain language of the email exchange certainly doesn't cover the Corps in glory and worry for the flood-endangered public.
Summary
The events at the 17th Street canal on August 29, 2012 appear to be more serious and longer lasting than the reporting that morning and afterward indicated. The water level in the canal exceeded the Corps' safe water elevation for over two hours in the morning because approximately a dozen Corps pumps failed and could not remove the rainwater being put into the canal by city pumps. The Corps' pumping capacity was cut nearly in half at the height of the need for pumping. Those Corps pump failures prompted orders to the locals to not activate some pumps and in one case to turn one off, causing stormwater to back up into city streets.
After the heaviest rain abated and the Corps got a handfull of their pumps working, water levels inside the canal and in the city streets dropped. However, many Corps pumps continued out of commission throughout the event, raising the probability that if rainfall had again picked up, a crisis identical to that of the morning of August 29th would again occur.
Occurring in parallel with these events were Corps efforts to shape the narrative, often with faulty information. One effort even contemplated cutting the public (and the local pumping authority) off from level gauge data.
Strictly due to cost, the Corps has decided to make this tandem pumping system with weakened canal walls permanent, their so-called "Option 1." Ground has broken for permanent pump stations to replace the clearly faulty Interim Closure Structures, but no information has been released regarding further work to make the canals themselves sturdier. Taken together, the outfall canals remain a serious Achilles heel within the confines of the City of New Orleans and Jefferson Parish.
posted by mcbrid35 at
1:31 PM
|
0 comments
