Imperative

After reading about all the problems with the gates, it's fair to say they're not the most robust systems possible.

But in addition to all the technical pieces of the gates, there's other parts: the Corps' procedures and the supervisors and operating crews charged with following them. And here too, the possibility of risk is introduced.

The procedure for lowering the gates

In addition to preparing the sites before storm season, including the annual demonstration for the press in late May, the operators and supervisors are responsible for a number of tasks immediately in advance of a storm. They have to:

a) Remove the dogging pins which are supporting the gates in their stored position
b) Lower the gates into the trench via the hydraulic winches
c) Attempt to reinsert the dogging pins to determine if the gates are properly seated in the trench
d) If the dogging pins won't go in, the diving contractor Independent Divers (aka H.J. Merrihue) is supposed to remove the obstructions from the trench
e) Once the obstructions are removed, they then must lower the gates and try again to insert the dogging pins to secure the gates.
f) If the obstructions can't be removed, large sandbags are to be dropped in front of the gates (also called needles) that are not completely seated.

[Pictures and more details on the pins are available at my earlier entry, "The Pins."]

It is the insertion of the dogging pins that lets the operators and supervisors know the needles are properly placed to resist storm surge. The Corps Operations & Maintenance Manual for the gates says,

"If the dogging pins can be inserted when the gate is down, this will confirm that the notch in the sill is free from obstructions."

So if the dogging pins are not inserted completely, that would indicate an obstruction in the trench (or "notch" in Corps-ese) at that gate section. That would mean the gates are not contacting the inner surface of the trench fully - an unacceptable condition, as emphasized in the Corps' Manual :

"Because the notches in the sill structure provide the reaction for the bottom of the gates it is imperative that these notches be free of all foreign material and obstructions."

That is, the structures are designed to have surge force distributed in a certain way between the trench and the structure:



If the bottom of the gate is not in contact with the back of the trench, almost all the surge force is then going through the structure, with a large chunk falling against the unreinforced guide beams.

These forces from storm surge are huge. If they weren't, the sill under the trench at the London and Orleans sites wouldn't have sheet piling extending down 48 feet below the bottom of the canal and the structure wouldn't have pilings going down 100 feet below the canal bottom.

In sum, if the dogging pins are not completely inserted and locked, there's a problem in the trenches that has not been solved. That means the gates are not fully seated, and the city is placed at risk.

The procedure in practice - September 12, 2009

So how has this procedure worked during real events?

Gates have been closed for storm events on four occasions since the structures began construciton in 2006:

1) Hurricane Gustav in September, 2008 - 17th Street and London Avenue gates were closed
2) Hurricane Ike in September, 2008 - 17th Street and London Avenue gates were closed
3) September, 2009 - London Avenue gates were closed
4) Hurricane/Tropical Storm Ida in November, 2009 - London Avenue gates were closed

A fifth event, on December 12-13, 2009, should have triggered gate closures at London and 17th Street, but did not, likely due to a systemwide SCADA failure four days earlier during a very heavy rainstorm. Details are here. The Corps continues to refuse to release FOIA-requested emails from before, during, and after this event.

Pins unpinned

It's very difficult to get contemporaneous accounts of what happened at the sites during these events. The Corps can put out press releases, and the media can do one- or two-minute stand-up reports standing hundreds of feet away, but actually knowing what happened in detail is tough. In addition to my FOIA request for the December, 2009 emails, the Corps is continuing to refuse release of emails relating to the November, 2009 London Avenue closure during Ida.

Fortunately, the Corps themselves dispatched someone to take pictures during at least one of these events. On September 12, 2009, heavy rains hit the New Orleans area. As a result of coincident rising lake levels, the gates were closed starting around 6 PM.

There weren't a lot of pictures from that Corps photographer, but one only needs a single good one.

Here is that one:


This photo was taken on September 12, 2009, after the gates were closed. It's the area inside the red square that should concern everyone in the New Orleans metro. Let's zoom in:


This shows that Corps operators and supervisors apparently left at least four dogging pins uninserted into the gates, a major violation of procedure. Those pins appear to be on needles 4, 5, 7, and 8. There are 11 needles in total.

The Times-Picayune also sent a photographer, Scott Threlkeld, out to London Avenue that afternoon to photograph the gate closing process. Mr. Threlkeld snapped two photos whch might partially confirm the pins' status.

The caption on the first (linked here) reads, "Randy Faherty of the U.S. Army Corps of Engineers shoves in a steel pin to lock a lowered flood gate in place Saturday, September 12, 2009, at the London Avenue Outfall Canal structure."

The caption on the second (linked here) reads, "Randy Faherty of the U.S. Army Corps of Engineers checks the water level after dropping the flood gates Saturday, September 12, 2009, at the London Avenue Outfall Canal structure."

I believe the captions on these photos are likely wrong. I do not believe the operator is checking the water level or shoving in a pin. I believe he is trying to figure out why a pin won't go in. And I believe that pin - on needle #8 - is the same one visible out of place in the deep background inside the blue frame in the Corps photo above. It appears Corps supervisors allowed the needles to remain unsecured.

Divers undived

So multiple gates were very likely not seated properly, due to obstructions in the trench. In that case, the Corps was supposed to have called Independent Divers (who are supposed to be on standby during storm events) to clear the obstructions. Let's take a look at the task orders given to Independent Divers (aka H.J. Merrihue) around September 12, 2009:

Task order 60
August 24, 2009

"For an emergency Dive required on 22 Aug 2009 due to a seal blow-out on the dewatering needles. Divers will be directed as needed."

The location of this task order's work is unknown. However, it is definitely not the outfall canals, since there are no dewatering needles at those sites. It was likely one of the Corps' locks.

Task order 61
August 31, 2009

"4 Man dive team required at Port Allen Lock for rewatering Process."

The Port Allen lock is across the Mississippi River from Baton Rouge.

London Avenue gates closed
September 12-14, 2009

Task order 62
September 24, 2009

"Dive contractor is required to perform magnetometer surveys to locate unknown sunken objects."

Even without a location, this is clearly not for trench cleanouts.

London Avenue gates closed
Hurricane/Tropical Storm Ida
November 9-11, 2009

Task order 63
November 18, 2009

"Four Man Dive team to investigate loud noise at gates at Algiers Lock.

The Algiers Lock is across the Mississippi River from St Bernard Parish.

As we can see, at least according to the contract record, Independent Divers was not summoned by Corps supervisors for trench cleaning during the time of the September London Avenue gate closure, even through photos seem to show an inability to close multiple gate sections. It also doesn't appear they made it out any time between that September event and the November London Avenue gate closure. This appears to be another violation of the Corps' own procedure.

Frankly, it appears there was little diving activity at any of the outfall canals through the 2009 hurricane season. Here's all the confirmed task orders for those sites during 2009:

Task order 54
May 18, 2009

"PROVIDE DIVING SERVICES TO CLEAR DEBRIS FROM GATE SILLS WITH THE USE OF A WATER JET PUMP, WITH GENERAL GATE INSPECTIONS REQUIRED AT EACH OF THE THREE LOCATIONS.
LOCATION: 17TH ST. CANAL, ORLEANS AVE. CANAL, & LONDON AVE. CANAL FLOODGATES."

This is the annual cleanout of the trenches before hurricane season

Task order 56
June 3, 2009

"Inspect the bridge pumps 1 and 6 and their foundations and remove any existing obstructions.
The dive work is estimated to take approximately five (5) hours.
LOCATION: 17th Street Canal
WORK TO START AT HRS: 0800 HOURS, 04 JUNE 2009."

This was diving associated with the bridge pump repair work at 17th Street.

Task order 59
August 17, 2009

"Emergency Dive at 17th St Canal Floodgate to clear debris from gates sills with the use of a water jet pump, with general gate inspections required at each of the two locations. Verbal NTP was issued on Aug 17, 2009."

This would appear to be preparation associated with Tropical Storm Claudette. Claudette was a minor threat that formed west of Tampa on August 16th and dissipated the next morning as its remnants came ashore in Alabama. By the time the Corps issued the verbal notice to proceed ("NTP") on the 17th, the storm was already past. Tides ran about a foot above normal.

Since the gates at the London Avenue canal, with its lower Safe Water Level, would be closed before 17th Street, it seems unlikely this task order only applied to 17th Street. The mention of "two locations" at the end of the description hints at the possibility that divers were supposed to also visit London, but it's impossible to know whether that occurred. It could simply refer to obstructions under two separate gate sections at 17th Street. In any case, the inability of the operators to completely secure multiple gate sections at London Avenue on September 12th makes the question moot. The London Avenue trenches do not appear to have been clear on the 12th for a real storm event.

Those three task orders are it for 2009. It does not appear the divers were regularly cleaning out the trenches at all during 2009, despite the Corps Operations & Maintenance Manual calling for the gates to be tested every three weeks during hurricane season:

"Once hurricane season has started, certain activities should be performed at regular intervals so that issues that could potentially interfere with the operation of the gates can be identified and addressed without the threat of an approaching storm.

Operation of the gates should be performed every three (3) weeks. Gate operation should be performed in the same manner as described in paragraph 3.2.1.2 for the check of gate operation prior to the hurricane season. A diver should be brought in to inspect and assist in removing obstructions if they are encountered."

It seems unlikely, if the gates are being tested every three weeks, that there would be no call for removal of obstructions on any of the gates across all three sites. City pumps are run often during the rainy New Orleans summer, sending all sorts of debris down the canals. Some of that debris would inevitably get caught in the trenches. This near complete lack of diving activity brings into question if the gates are truly being tested every three weeks during hurricane season, and if not, why?

It would have been Corps supervisory personnel in the Operations Division making these decisions regarding divers.

Sandbags unplaced

So we've got gates not being secured and divers not being called in to clean the trenches. What else happened - or didn't happen - during that September 2009 event?

The 2009 operations manual says if the obstructions in the trenches cannot be dislodged by the divers, massive sandbags are supposed to be dropped by the on-site rental crane in front of the needle where the obstruction lies. This is supposed to close the gap along the trench. Just for reference, here's the sandbags at the London Avenue canal earlier in 2010:


I previously established that this Google satellite shot was from the spring of 2010.

Zooming into the area inside the red box:


On the ground shots around the same time as the Google shot:



I can't speak to whether those sandbags were placed or not, or if they were even on site. However, the lack of photographic evidence both on September 12th (from two independent sources) as well as no Corps photos of sandbags getting removed from the water on September 14th certainly leads one to believe that no sandbags were placed. That's a third violation of procedure.

1) Gates unpinned
2) Divers not summoned
3) Sandbags not placed

The Corps "responds"

I repeatedly asked for comment on this story from the Corps' New Orleans District Public Affairs office. Here's what they sent me:

"Mr. McBride -- The gates do not have to be pinned to be sealed. Everything worked as it should.

V/R
Ken [Holder, Chief, Public Affairs, Corps New Orleans District]"

This semi-confirms the gates were unpinned last September, and certainly doesn't deny it. The mention of them being sealed is not relevant, because the seals at London Avenure are just for show. The importance of pinning is in effective transfer of surge forces to the foundation, not sealing, which is made abundantly clear in the Corps' own Operations & Maintenance Manual for the gates.

Summing up

So, it looks pretty bad. Compared to other storm events, the September, 2009 closure was a minor event. The lake never got above about 3 feet, and that happened just two hours after the gates were closed on September 12th. Such an event would act as a real world drill, so you'd think the Corps supervisors would try to get everything right with a bit of the pressure off. That way, when the Big One comes with its Big Pressure, they'd be able to handle it. Instead, it appears they viewed this as a nothing event when key parts of procedure could be skipped, which makes one wonder what's happening the rest of the time, and what will happen when the Big One hits.

Next up: How the pumps performed during this event.

Karen Gadbois of The Lens, and Molly Peterson of Southern California Public Radio contributed to this report.

The gates story so far

In examining the Interim Closure Structures, we've looked at the trench the gates sit in when lowered, the gates themselves, the guide columns, the seals, the pins that are meant to lock the gates in place, the winches that raise and lower the gates, and the SCADA system that controls and monitors the whole thing.

Along the way, we've learned about these unresolved problems:

- The Corps relies on divers to clean out the trenches in which the gates sit, rather than an automated cleaning system. Over the years, the use of those divers has been somewhat spotty, as I'll show in my next post.

- During construction in 2006, the trenches at Orleans Avenue and 17th Street were filled and covered with jet grout to a depth of at least three feet. All that had to be jackhammered out only after the Corps discovered the problem in advance of a tropical system. One of the needles at the site was lowered in the trench at the time they found this problem. Here's a picture of that needle after it was freed:


Two years later, they were still chiseling the stuff out. Inserts meant to keep silt from clogging the trenches, promised in 2006, were never installed. The upshot is the public does not know what condition the trenches are in currently, though smart money suggests "not good."

- The gates themselves are not built in accordance with the Corps' own design standards. Among the weaknesses are vertical framing, unreinforced guide columns, and point concentration of storm surge forces into a handful of locations on the main frames of the structures.

- The gates are lowered through the guide beams with no means to ensure they (the gates) don't rattle around as water flows around them. This increases the potential of damage to the gates and the guides, as well as creating the possibility of the gates binding up. The Corps' Operation and Maintenance Manual for the gates recognizes this as a possibility in the instructions for lowering the gates:

"If during this process it is observed that as the gate is lowered it does not continue at a constant rate, the top elevation of the gate where the interruption occurred shall be noted and the gate shall go through the operation procedure a second time. If an interruption occurs again at the noted location then an inspection of the guide columns along their full height, as well as the portion of the gate within the guide columns shall be performed to determine what is causing the gate to hang up."

The Corps' engineering manual and their own consultant's report calls for a series of rollers on the guides to ensure against these problems. There are no rollers.

And it's not like the Corps doesn't know rollers are needed. They are building ten of them into each of the two vertical lift gates at the Seabrook closure just a couple of miles from the outfal canals, also along the Lake Pontchartrain lakefront. That design also includes two pours of concrete around the guide beams, horizontal reinforcement for the gate segments, and basically everything the current outfall canal gates do not. Frankly, the Seabrook design amply demonstrates how flimsy the outfall canal gates are.

- At 17th Street, the gates are sealed only on one side, and at London and Orleans Avenues, there is no sealing (a cosmetic seal on the lake side of the structures at those two sites is purely decorative). Thus, the gates are actually - as built - leaky.

Here's the location of the "seals" on the front of the London Avenue structure, away from the sealing surfaces between the needles and the guide beams (photo here):


A detail from the applicable London Avenue as-built drawing shows the seals from above:


This design includes a 4.5" gap between the needles and the structure:



Not good.

- At all three sites, the hydraulic power units and control boxes for the winches - which lower and raise the gates - are placed directly in the line of fire of storm-driven waves and overwash.



At 17th Street, the units are actually closer to the water than the gates themselves:


These winch control units are not enclosed in storm-hardened enclosures and remain completely exposed to waves and storm-thrown debris. This arrangement places the equipment and the operators - and by extension, the city - at risk.

- The SCADA control system at all three sites appears to be extremely vulnerable to weather induced failures, demonstrated by the five day outage in December, 2009. During this outage, water levels in both the 17th Street and London Avenue canals exceeded the criteria for closure of the gates on each canal, but neither set of gates was dropped. On the London Avenue canal, the water depth exceeded the Safe Water Elevation during this outage as well.

In the next part, we'll take a look at how some of these moving parts interact during a real event. In this case, it'll be the closure of the London Avenue gates on September 12-14, 2009

SCADA

I've been working on entries about the September 12-14, 2009 closure of the gates along the London Avenue canal. That particular closure is instructive since it arose not from a tropical storm, but from a heavy non-tropical rainfall event. The gates closed just days after the Corps issued new procedures specific to London Avenue which called for dropping the gates when the water got to just 2.5 feet, down from the previous 4 feet. I'll bring that information out soon.

When I say it is instructive, I mean that it should have set the tone for future events. Indeed, when Hurricane Ida approached New Orleans in early November, 2009, the London Avenue gates dropped again when water got to 2.5 feet.

But then came the very serious rains in mid-December, 2009, and something very odd happened. Despite water getting close to, and then eventually exceeding the London Avenue Safe Water Elevation of 5 feet, the gates did not drop. Indeed, despite numerous, repeated instances of waters exeeding the 2.5 foot criterion in the London Avenue canal over a seven day period from December 8th through the 15th, never did the gates drop.

This perplexed me. For a while, I figured it was because the Corps was scared of their own rusty pumps at the gates, none of which had been fully repaired at that point (they still haven't). But that didn't make sense, because they knew the condition of those pumps in September and November, yet they still proceeded with gate closures and pump operations during those events.

It wasn't until I started reviewing the actual SCADA data for all three events that I found a clue to why the December, 2009 rainfall was treated differently from the September and November events.

First, "SCADA" stands for Supervisory Control and Data Acquisition. It is the computerized control system that alows the Corps to monitor and control many aspects of operations at the gates, including water levels within the canals, starting and stopping of their pumps, and even weather (just the monitoring on that one).

Along the London Avenue canal, the Corps has placed 10 level gauges which are connected to the SCADA system and provide near-real-time feedback on the water level within the entire length of the canal. Four of the gauges are at the gates, while the other six are placed along the canal, at the bridge crossings and Sewerage & Water Board drainage pump stations. Water levels are recorded every ten seconds from these gauges and are transmitted either by submerged fiberoptic cable or by microwave radio to the SCADA system. There are display terminals for the system at the gate structures, in the Sewerage & Water Board pump stations, and at the Corps' building in Leake Avenue.

The level gauges also have two backup systems in their individual control boxes. The secondary system polls the gauges a set number of times in an hour (maybe four or six) and takes the highest reading for the hour. That number is transmitted via satellite to a website the Corps maintains called Rivergauges, which is publicly available. A tertiary system involves satellite phone lines.

Obviously, due to its near-real-time basis, the Corps relies on the primary system for gauge readouts along the canal, especially during storms. The SCADA system also allows the Corps to run their pumps remotely, and monitor them for problems. Presumably, the status of gate segments (i.e. open/closed) is also included in the SCADA system. All these things can be done manually by crew at the gate sites, but the SCADA system provides a great deal more visibility at a glance.

Early on the morning of December 8, 2009, it appears the Corps' SCADA system suffered a systemwide failure across all three canals. Level gauges stopped sending updated readings to the system, and it is likely the Corps lost the ability to remotely start, stop, and monitor all their pumps. This occurred during a very heavy rainstorm that caused widespread street flooding, as the Sewerage & Water Board (S&WB) was pumping water into the canals. For some reason, the secondary level gauge reporting system continued to work, reporting canal water levels throughout.

This SCADA system failure appears to have continued for five days, until the afternoon of December 13th when the level gauges started reliably reporting water depths again. In between, New Orleans saw massive rainfall. On the evening of December 12th, as torrents lashed the city, water levels in the London Avenue canal rose above the 5 foot safe water elevation. Remember that Corps protocol called for gates to be dropped when water hit just 2.5 feet. The Corps knew this was serious, because their London Avenue canal captain showed up in the S&WB pumping station at the south end of the canal, Drainage Pumping Station #3, at around 6:30 that evening.

Meanwhile, a couple of miles west at the 17th Street canal, waters were also rising. Along that canal, the safe water level is six feet, and the criteria for closure is a level of five feet. As S&WB personnel turned on nearly every pump in Drainage Pumping Station #6 at the south end of the canal, the water level rose past the closure criteria along that canal as well. That is according to the Rivergauges readings coming from the backup level signal system, since the primary SCADA system was still out along all three canals.

On the 17th Street canal, the water appears to have stayed at or above the five foot closure criterion for at least 3 hours on the evening of the 12th. There's no evidence of Corps presence in DPS#6 that evening.

In neither of these cases were the gates dropped and the Corps pumps started, as called for in their own procedures. At 17th Street, water was within inches of the safe water level for hours. At London Avenue, it exceeded the safe water level for approxmately six hours, and was above the closure criteria of 2.5 feet for the better part of 10 to 16 hours, well into the morning of the 13th.

The denoument of this tale came on the 16th, when the Corps issued a $57,388.07 task order to their SCADA system maintenance contractor, Prime Controls (the same firm that installed the SCADA system) for maintenance of the system along all three canals. I believe it is a fair assumption that task order was for emergency work Prime performed after the system went down on the 8th. I have placed a FOIA request for the complete contract, including the December 16th task order.

I noted these events when they happened, but it is only recently that I have gotten the complete SCADA record for all three canals, which allowed me to see exactly what was happening minute by minute. However, what would truly allow clarity would be the emails of all the principal Corps personnel during the period around the December events.

I placed a FOIA request for those emails, as well as any after-action reports, almost nine months ago. It remains unfilfilled, along with about a half dozen other months-old requests. It is understandable that the Corps would be reluctant to discuss what happened that week in December, but it doesn't stop them from spinning it.

In multiple recent public relations documents, starting in March of this year, they actually point to the walls seeing more water than they should have in December as a good thing. From their March, 2010 "Teammate Update:"

"Since their initial construction in the mid 1990s, the outfall canal walls have performed well in removing rainwater from the city, including the record rainfall events in May 1995 and December 2009."

That's chutzpah!

When (if?) I get those emails, I'm sure I'll have a better perspective on events. However, I do not believe it is a coincidence that of the three major rainfall or surge events to occur in New Orleans in the last third of 2009, the only one when the gates should have been dropped - but weren't - was the one when the Corps' entire control system appears to have been out of commission for nearly a week. Add in the over $57,000 in repairs to that system just days after the event, and there appears to be a scary story there.

As I mentioned at the beginning, I've also got stuff on the September, 2009 event that I'll be bringing out, but this seemed too large to hold back until I get every single bit of information. After all, I've been waiting nine months for about a week's worth of emails confined to a very small section of the Corps' business, and I don't feel like holding back any more.