Ready? Ummm

Quick primer

The three outfall canals in New Orleans are the 17th Street, Orleans Avenue, and London Avenue canals.

They drain rainfall from the city to Lake Pontchartrain via a needlessly complex system of pumps. Pumps at the south end of each canal - run by the local authorities - collect rainwater from subsurface drains throughout the city and place the water in the canals. Those stations are a century old and proved very reliable over the years, until they were completely flooded following the August 29, 2005 failure of the Corps of Engineers-designed and Corps of Engineers-built floodwalls along the 17th Street and London Avenue canals. The city stations were repaired and stormproofed since then.

At the north end of the canals are the Corps-built and Corps-operated interim closure structures, constructed following the 2005 floodwall failures. They are large gates meant to keep hurricane-driven storm surge out of the canals. They also have ancillary pumps to drain the rainwater put into the canal by the city pumps. They are... not great.

In addition, the floodwalls along the canals - ten years after the deadly failures - still cannot withstand water to their design heights at their tops. Over the past few years, the Corps has installed a variety of reinforcements to selected sections of the walls and their levees to raise the "safe water levels." Those safe water levels are supposedly 8 feet now (wall tops are about 13 to 15 feet), but the Corps has claimed that before and then later admitted (or were exposed) the levels were far below 8 feet. Limits on how high water can get against the walls limit the amount of water pumped down the canal. There are eight level gauges along each canal to measure the depth of the water. The Corps is supposed to use the data from those gauges to ensure the water doesn't get too high. Unfortunately, the only way to limit the amount of water going into a canal during a storm is for the city pumps to shut down. Thus there has to be tight coordination between the Corps and the locals, which can be very difficult during a hurricane.

The obvious solution to this needless complexity - elimination of the southern stations in favor of the northern stations and lowering of the canals to allow gravity flow all the way to the lake - was eliminated from consideration by the Corps in the years following the 2005 floodwall failures for unknown but certainly peevish reasons. Instead, they are leaving the city stations and are building permanent pumping stations adjacent to the interim structures, due to be finished sometime in 2017, about eight years after they were initially promised. Because of these delays, the interim structures and their pumps are now being used long past their expected life.

The only sizable test the interim facilities have faced came in 2012, when Hurricane Isaac was targeting New Orleans. And that's where our story begins...

Back to Isaac

At the beginning of every hurricane season, as well as when storms approach New Orleans, the Corps of Engineers New Orleans District goes before the press to claim they're more "ready" than ever. For example, as Hurricane Isaac approached the city in 2012, the Corps' 17th Street canal captain Raymond Newman claimed on August 28th to the assembled press and government officials, "We're ready. We expect to do well... We have a lot of confidence in the system."

This was a system which:

- had already experienced near total failure of data collection, control and communications on August 27th - a day before the press conference - and which continued to fail throughout the storm.

- at the peak of the storm on August 29th, experienced at a loss of one third to one half of Corps pumping capacity at the 17th Street site at the very time maximum pumping was needed.

- before, during, and after the storm's peak - due to Corps pumping outages - prevented the locals to from feeding water to it on TWELVE separate occasions (five on the Orleans Avenue canal, three on the London Avenue canal, and four on the 17th Street canal)

- had most of its level gauges fail throughout the storm event

- could not even be fully secured against storm surge

The true state of readiness was so poor that in a remarkably undisciplined move, Corps New Orleans  District Commander Colonel Edward Fleming left his command and control bunker at the height of the storm - where he is supposed to be supervising operations of his entire District - to ... check the water level or something ... at the 17th Street canal site. The whole thing was a mess. Only by virtue of luck - the storm dropping less rain when the Corps' pump failures were at their peak - was a possible collapse of the canal walls along the 17th Street canal or devastating flooding within the city avoided. Corps readiness had nothing to do with it.

Afterwards, thanks to adept massaging of the press by the New Orleans District's Public Affairs office, very little of this was revealed until I published the Corps internal emails from Isaac. In the meantime, the Corps got their preferred, talking points-driven narrative cemented: Isaac was a huge test that the Corps passed with flying colors! I'll talk more in a later post about how exactly the Public Affairs staff performed that bit of magic, but for now, the question needs to be asked whether the outfall canals' readiness is any better for a storm three years later.

Looking under the hood

I always find it's instructive to look under the hood at the guts of a system to see how well the outward appearance reflects reality. If the nitty gritty stuff is being attended to, then it's likely the shiny exterior is legit. However, if the internal mechanism is in shambles, it's an indication there's no there there.

So let's look at one of those definitely less glamorous things - the backup system for the readouts of the outfall canal level gauges. There were problems with those level gauges during Isaac, but we first need to understand how the system works.

The outfall canals each have eight level gauge stations along their length:

[The gauges are numbered 1 to 8 working from the canal outlets at the Lake on the north to the canal inlets at the city's pump stations on the south. In each case, gauge 1 is located outside the Corps' gates, while the other 7 gauges are within the canal walls.]

The level gauge stations were designed by Corps contractor Sutron and installed in coordination with the New Orleans District's favorite SCADA (Supervisory Control and Data Acquisition) contractor, Prime Controls. Sutron brags about the level stations on their website, but the New Orleans gauges are just tiny portion of their business. Their relationship with the Corps is substantial. In a 2014 summary of past projects, they claim installation of 1435 level gauge stations for the Corps nationwide, with a total contract value over $20 million. Beyond that, they do work all over the country for federal, state, and local governments, as well as work around the world for governments from Afghanistan to Venezuela.

In New Orleans, Sutron was not responsible for the level gauges themselves, but they provided everything to transmit the signals from those gauges to a diverse set of users. Inside the Sutron boxes (referred to as "DCP's", short for Data Collection Platform) are communications links to three different systems:

a) The primary system, which is the SCADA system. Data is relayed via fiber optic cable buried beneath the silt at the canal bottoms to stations at each end of each canal, as well as back to the Corps District HQ in New Orleans. Level readings are relayed every minute, providing - theoretically - a great deal of granularity in the data available to Corps decision-makers and personnel at the canals during a storm.

b) The backup system, which uses a satellite uplink to send data every hour via NASA's GOES satellite system. Let's let Sutron provide detail:

"GOES provides hourly transmission of 15 minute interval stage data. GOES also provides a wide-area dissemination of the water level data and is extremely reliable, an important element especially since no land lines or terrestrial repeaters are involved. Additionally, the data received from GOES is monitored not only in New Orleans but also monitored at [Corps of Engineers] regional headquarters in Vicksburg, MS"

c) The backup to the backup, which uses Iridium satellite phone modems. There's a lot about Iridium out there on the Internet: they filed for bankruptcy within nine months of starting operations in 1999; their satellites were already near the end of their anticipated lifespan when the Corps bought the Sutron DCP's for New Orleans in 2006; and Iridium's network only supports a network speed of 10 kbps, or about the speed of a 1990's dialup modem. They had captured the Pentagon's interest after their restart in 2001 because of their total global coverage.

Getting data on the performance of the SCADA system or the Iridium system is quite difficult. But the performance of the backup system is there for all the public to see, since the hourly readings from it are published on the Corps' Rivergages website. That is, there's data for all 24 gauges every hour of every day.

Quantifying "readiness"

How to measure the readiness of the backup system? Since we cannot tell the future, we should examine how the equipment is performing in the present. For some pieces of the outfall canal system, present condition is measured only by intermittent testing. The pumps are a good example. They are turned on for a few minutes every couple of weeks, a test which bears little resemblance to real world conditions.

However, the level gauge backup systems are on all the time. Effectively, they are undergoing continuous testing, so their present and past conditions represent - at least - the best one can expect. That is, the level gauge backup systems aren't generally going to fix themselves during a storm if they are broken going into it. We saw that with the gauges along the London Avenue canal during Isaac, where three gauges were not reporting to Rivergages for over two months before the storm even existed, and a fourth didn't report for over a week beforehand. None of those gauge systems reported to Rivergages during the storm either.

To measure the condition of the gauge backups, I settled on "Uptime." Simply, if the gauge is reading a number, it's counted as "Up" for that hour. If it is reading "M" (short for "data Missing") or is not reading at all, it is counted as "Down" for that hour. In the aggregate, 50% uptime for one canal - or all three - could represent:

- all gauges working 50% of the time, or
- 50% the gauges working all the time and the other 50% never working, or
- something in between.

I collected data from the beginning of the 2012 hurricane season on June 1, 2012 through the beginning of the peak of the 2015 hurricane season on August 15, 2015 for all 24 gauges [Data was not available for the "London ICS Canal" gauge for dates before September 10, 2012, so calculations for 2012 earlier than that date were were adjusted to compensate]. Overall it shows a significant dropoff in uptime for this year compared to the prior three years.

If the outfall canal level gauge backup systems were truly ready each year before hurricane season, one would expect high degrees of uptime in the month before June 1. So here's the uptime numbers for each May back to 2012:

May, 2012: 51.7%
May, 2013: 72.8%
May, 2014: 88.3%
May, 2015: 46.6%

One could argue the less than stellar uptime in May 2012 reflected activities to get ready for the peak hurricane season (generally August 15 through September 30). In that case, there should have been a substantial jump in the uptime from June through August 15:

May, 2012: 51.7%
June, 2012: 55.6%
July, 2012:  63.5%
August 1, 2012 - August 15, 2012: 68.2%

So a bit of a jump, to about 70% uptime for the first part of August, 2012. Not great, but better than 52% three months earlier. For the final few days of August before Isaac blew in, they got a little bit higher:

August 16, 2012 - August 26, 2012: 75.7%

Of course, that wasn't good enough, as many gauges went out during Isaac. In addition to the five gauges (four at London Avenue and a fifth at Orleans Avenue) that were already not reporting through their backup systems before the storm and continued to fail, 7 of the 8 gauges at the 17th Street canal failed at the peak of the storm. It is unclear what the mode of failure was on those gauges.

Still, the 2012 uptime was better than this year:

May, 2015: 46.6%
June, 2015: 52.6%
July, 2015: 55.6%
August 1, 2015 - August 15, 2015:  49.7%

That's awful. Effectively 50% of the level gauge data backup system is out of commission. Digging deeper pinpoints where the problems are:

Level gauge backup system uptime for each canal, May 1, 2015 - August 15, 2015:
17th Street: 47.2%
Orleans Avenue: 63.5%
London Avenue: 42.8%

The red-yellow-green classification scheme is my own, but I think it applies fairly well considering the vital importance of the equipment. London Avenue and 17th Street are definitely in the worst shape, though Orleans Avenue is no great shakes.

The picture along the London Avenue canal is somewhat mixed. One gauge (London 4) didn't report at all from May through August 15, while another (London 7) only reported for 112 hours scattered over the 3.5 month period. Two others - London 3 and London 8 - have also had serious problems. In fact, other than a brief burst of data on July 13th, London 8's backup system hasn't been up consistently since May 29th this year. The other 4 gauges have at least been "up," though hardly at stellar rates.

Along the 17th Street canal, it's much clearer. While the four northernmost gauges' backup systems have pretty much been up near 100% of the time, the four southernmost gauges' backup systems have been out of commission the entire period except for some very stray readings from 17th 7. In fact, 17th 5 and 17th 6 appear to have been abandoned, since their backup systems haven't transmitted since last October. In addition, the backup systems for 17th 7 and 17th 8 have been out since early January this year.

Why does it matter?

After Isaac in 2012, The Lens' Steve Myers produced two reports about the failure of the level gauges. In the first article, the Corps in full spin mode claimed all they really needed during a storm were the gauges at the southern inlet and northern exit of the canals (that would be numbers 8 and 2 for each canal respectively):

"At minimum, the corps needs two gauges in each canal: one at the Sewerage and Water Board pumping station and another inside the lake floodgate. Chris Accardo, chief of operations for the corps’ New Orleans district, said even if one of those gauges goes out, employees at each location can look outside to see how high the water is." 

So what does it mean that on two of the three canals in 2015, the backup systems for gauges at the Sewerage and Water Board pumping stations have been either broken or not maintained for months, and we are now in the height of hurricane season? And what about the eight other gauges with similarly minuscule or nonexistent uptimes? It probably means those ten gauge systems - like the four London Avenue gauges that were not working ahead of Isaac in 2012 - won't work if a storm comes in this year.

So for the Corps, if (when?) their primary SCADA system goes down during a storm, they will have very poor visibility on canal water levels other than sending people out in the middle of a hurricane. It's hard to believe Corps management can be that callous, but there it is.

Practically, looking at gauges during a storm might not even work. The Corps' own operating manual for the interim closure structures says Corps personnel are to abandon the structures for any storm of category 3 and above, meaning there would not be anyone left to look at gauges.

It also means that during a storm, parties besides the Corps New Orleans District - the public, local government officials, Corps personnel outside New Orleans, the media, and state and federal officials - will not have a good, unbiased, nearly realtime measure of what is happening along the locations of the worst engineering failures to strike the United States in its history. That's a very poor way to communicate risk.

Finally, internal systems in shambles indicate there's no there there. And the publicly available outfall canal level gauge backup systems indicate serious shamblization has occurred in recent months. The damage appears severe enough that storm readiness must be seriously questioned for all the other systems along the outfall canals, including the pumps and the gates themselves.