Fix the pumps

Wednesday, August 26, 2015

Ready? Ummm

Quick primer

The three outfall canals in New Orleans are the 17th Street, Orleans Avenue, and London Avenue canals.

They drain rainfall from the city to Lake Pontchartrain via a needlessly complex system of pumps. Pumps at the south end of each canal - run by the local authorities - collect rainwater from subsurface drains throughout the city and place the water in the canals. Those stations are a century old and proved very reliable over the years, until they were completely flooded following the August 29, 2005 failure of the Corps of Engineers-designed and Corps of Engineers-built floodwalls along the 17th Street and London Avenue canals. The city stations were repaired and stormproofed since then.

At the north end of the canals are the Corps-built and Corps-operated interim closure structures, constructed following the 2005 floodwall failures. They are large gates meant to keep hurricane-driven storm surge out of the canals. They also have ancillary pumps to drain the rainwater put into the canal by the city pumps. They are... not great.

In addition, the floodwalls along the canals - ten years after the deadly failures - still cannot withstand water to their design heights at their tops. Over the past few years, the Corps has installed a variety of reinforcements to selected sections of the walls and their levees to raise the "safe water levels." Those safe water levels are supposedly 8 feet now (wall tops are about 13 to 15 feet), but the Corps has claimed that before and then later admitted (or were exposed) the levels were far below 8 feet. Limits on how high water can get against the walls limit the amount of water pumped down the canal. There are eight level gauges along each canal to measure the depth of the water. The Corps is supposed to use the data from those gauges to ensure the water doesn't get too high. Unfortunately, the only way to limit the amount of water going into a canal during a storm is for the city pumps to shut down. Thus there has to be tight coordination between the Corps and the locals, which can be very difficult during a hurricane.

The obvious solution to this needless complexity - elimination of the southern stations in favor of the northern stations and lowering of the canals to allow gravity flow all the way to the lake - was eliminated from consideration by the Corps in the years following the 2005 floodwall failures for unknown but certainly peevish reasons. Instead, they are leaving the city stations and are building permanent pumping stations adjacent to the interim structures, due to be finished sometime in 2017, about eight years after they were initially promised. Because of these delays, the interim structures and their pumps are now being used long past their expected life.

The only sizable test the interim facilities have faced came in 2012, when Hurricane Isaac was targeting New Orleans. And that's where our story begins...

Back to Isaac

At the beginning of every hurricane season, as well as when storms approach New Orleans, the Corps of Engineers New Orleans District goes before the press to claim they're more "ready" than ever. For example, as Hurricane Isaac approached the city in 2012, the Corps' 17th Street canal captain Raymond Newman claimed on August 28th to the assembled press and government officials, "We're ready. We expect to do well... We have a lot of confidence in the system."

This was a system which:

- had already experienced near total failure of data collection, control and communications on August 27th - a day before the press conference - and which continued to fail throughout the storm.

- at the peak of the storm on August 29th, experienced at a loss of one third to one half of Corps pumping capacity at the 17th Street site at the very time maximum pumping was needed.

- before, during, and after the storm's peak - due to Corps pumping outages - prevented the locals to from feeding water to it on TWELVE separate occasions (five on the Orleans Avenue canal, three on the London Avenue canal, and four on the 17th Street canal)

- had most of its level gauges fail throughout the storm event

- could not even be fully secured against storm surge

The true state of readiness was so poor that in a remarkably undisciplined move, Corps New Orleans  District Commander Colonel Edward Fleming left his command and control bunker at the height of the storm - where he is supposed to be supervising operations of his entire District - to ... check the water level or something ... at the 17th Street canal site. The whole thing was a mess. Only by virtue of luck - the storm dropping less rain when the Corps' pump failures were at their peak - was a possible collapse of the canal walls along the 17th Street canal or devastating flooding within the city avoided. Corps readiness had nothing to do with it.

Afterwards, thanks to adept massaging of the press by the New Orleans District's Public Affairs office, very little of this was revealed until I published the Corps internal emails from Isaac. In the meantime, the Corps got their preferred, talking points-driven narrative cemented: Isaac was a huge test that the Corps passed with flying colors! I'll talk more in a later post about how exactly the Public Affairs staff performed that bit of magic, but for now, the question needs to be asked whether the outfall canals' readiness is any better for a storm three years later.

Looking under the hood

I always find it's instructive to look under the hood at the guts of a system to see how well the outward appearance reflects reality. If the nitty gritty stuff is being attended to, then it's likely the shiny exterior is legit. However, if the internal mechanism is in shambles, it's an indication there's no there there.

So let's look at one of those definitely less glamorous things - the backup system for the readouts of the outfall canal level gauges. There were problems with those level gauges during Isaac, but we first need to understand how the system works.

The outfall canals each have eight level gauge stations along their length:

[The gauges are numbered 1 to 8 working from the canal outlets at the Lake on the north to the canal inlets at the city's pump stations on the south. In each case, gauge 1 is located outside the Corps' gates, while the other 7 gauges are within the canal walls.]

The level gauge stations were designed by Corps contractor Sutron and installed in coordination with the New Orleans District's favorite SCADA (Supervisory Control and Data Acquisition) contractor, Prime Controls. Sutron brags about the level stations on their website, but the New Orleans gauges are just tiny portion of their business. Their relationship with the Corps is substantial. In a 2014 summary of past projects, they claim installation of 1435 level gauge stations for the Corps nationwide, with a total contract value over $20 million. Beyond that, they do work all over the country for federal, state, and local governments, as well as work around the world for governments from Afghanistan to Venezuela.

In New Orleans, Sutron was not responsible for the level gauges themselves, but they provided everything to transmit the signals from those gauges to a diverse set of users. Inside the Sutron boxes (referred to as "DCP's", short for Data Collection Platform) are communications links to three different systems:

a) The primary system, which is the SCADA system. Data is relayed via fiber optic cable buried beneath the silt at the canal bottoms to stations at each end of each canal, as well as back to the Corps District HQ in New Orleans. Level readings are relayed every minute, providing - theoretically - a great deal of granularity in the data available to Corps decision-makers and personnel at the canals during a storm.

b) The backup system, which uses a satellite uplink to send data every hour via NASA's GOES satellite system. Let's let Sutron provide detail:
"GOES provides hourly transmission of 15 minute interval stage data. GOES also provides a wide-area dissemination of the water level data and is extremely reliable, an important element especially since no land lines or terrestrial repeaters are involved. Additionally, the data received from GOES is monitored not only in New Orleans but also monitored at [Corps of Engineers] regional headquarters in Vicksburg, MS"

c) The backup to the backup, which uses Iridium satellite phone modems. There's a lot about Iridium out there on the Internet: they filed for bankruptcy within nine months of starting operations in 1999; their satellites were already near the end of their anticipated lifespan when the Corps bought the Sutron DCP's for New Orleans in 2006; and Iridium's network only supports a network speed of 10 kbps, or about the speed of a 1990's dialup modem. They had captured the Pentagon's interest after their restart in 2001 because of their total global coverage.

Getting data on the performance of the SCADA system or the Iridium system is quite difficult. But the performance of the backup system is there for all the public to see, since the hourly readings from it are published on the Corps' Rivergages website. That is, there's data for all 24 gauges every hour of every day.

Quantifying "readiness"

How to measure the readiness of the backup system? Since we cannot tell the future, we should examine how the equipment is performing in the present. For some pieces of the outfall canal system, present condition is measured only by intermittent testing. The pumps are a good example. They are turned on for a few minutes every couple of weeks, a test which bears little resemblance to real world conditions.

However, the level gauge backup systems are on all the time. Effectively, they are undergoing continuous testing, so their present and past conditions represent - at least - the best one can expect. That is, the level gauge backup systems aren't generally going to fix themselves during a storm if they are broken going into it. We saw that with the gauges along the London Avenue canal during Isaac, where three gauges were not reporting to Rivergages for over two months before the storm even existed, and a fourth didn't report for over a week beforehand. None of those gauge systems reported to Rivergages during the storm either.

To measure the condition of the gauge backups, I settled on "Uptime." Simply, if the gauge is reading a number, it's counted as "Up" for that hour. If it is reading "M" (short for "data Missing") or is not reading at all, it is counted as "Down" for that hour. In the aggregate, 50% uptime for one canal - or all three - could represent:

- all gauges working 50% of the time, or
- 50% the gauges working all the time and the other 50% never working, or
- something in between.

I collected data from the beginning of the 2012 hurricane season on June 1, 2012 through the beginning of the peak of the 2015 hurricane season on August 15, 2015 for all 24 gauges [Data was not available for the "London ICS Canal" gauge for dates before September 10, 2012, so calculations for 2012 earlier than that date were were adjusted to compensate]. Overall it shows a significant dropoff in uptime for this year compared to the prior three years.

If the outfall canal level gauge backup systems were truly ready each year before hurricane season, one would expect high degrees of uptime in the month before June 1. So here's the uptime numbers for each May back to 2012:

May, 2012: 51.7%
May, 2013: 72.8%
May, 2014: 88.3%
May, 2015: 46.6%

One could argue the less than stellar uptime in May 2012 reflected activities to get ready for the peak hurricane season (generally August 15 through September 30). In that case, there should have been a substantial jump in the uptime from June through August 15:

May, 2012: 51.7%
June, 2012: 55.6%
July, 2012:  63.5%
August 1, 2012 - August 15, 2012: 68.2%

So a bit of a jump, to about 70% uptime for the first part of August, 2012. Not great, but better than 52% three months earlier. For the final few days of August before Isaac blew in, they got a little bit higher:

August 16, 2012 - August 26, 2012: 75.7%

Of course, that wasn't good enough, as many gauges went out during Isaac. In addition to the five gauges (four at London Avenue and a fifth at Orleans Avenue) that were already not reporting through their backup systems before the storm and continued to fail, 7 of the 8 gauges at the 17th Street canal failed at the peak of the storm. It is unclear what the mode of failure was on those gauges.

Still, the 2012 uptime was better than this year:

May, 2015: 46.6%
June, 2015: 52.6%
July, 2015: 55.6%
August 1, 2015 - August 15, 2015:  49.7%

That's awful. Effectively 50% of the level gauge data backup system is out of commission. Digging deeper pinpoints where the problems are:

Level gauge backup system uptime for each canal, May 1, 2015 - August 15, 2015:
17th Street: 47.2%
Orleans Avenue: 63.5%
London Avenue: 42.8%

The red-yellow-green classification scheme is my own, but I think it applies fairly well considering the vital importance of the equipment. London Avenue and 17th Street are definitely in the worst shape, though Orleans Avenue is no great shakes.

The picture along the London Avenue canal is somewhat mixed. One gauge (London 4) didn't report at all from May through August 15, while another (London 7) only reported for 112 hours scattered over the 3.5 month period. Two others - London 3 and London 8 - have also had serious problems. In fact, other than a brief burst of data on July 13th, London 8's backup system hasn't been up consistently since May 29th this year. The other 4 gauges have at least been "up," though hardly at stellar rates.

Along the 17th Street canal, it's much clearer. While the four northernmost gauges' backup systems have pretty much been up near 100% of the time, the four southernmost gauges' backup systems have been out of commission the entire period except for some very stray readings from 17th 7. In fact, 17th 5 and 17th 6 appear to have been abandoned, since their backup systems haven't transmitted since last October. In addition, the backup systems for 17th 7 and 17th 8 have been out since early January this year.

Why does it matter?

After Isaac in 2012, The Lens' Steve Myers produced two reports about the failure of the level gauges. In the first article, the Corps in full spin mode claimed all they really needed during a storm were the gauges at the southern inlet and northern exit of the canals (that would be numbers 8 and 2 for each canal respectively):

"At minimum, the corps needs two gauges in each canal: one at the Sewerage and Water Board pumping station and another inside the lake floodgate. Chris Accardo, chief of operations for the corps’ New Orleans district, said even if one of those gauges goes out, employees at each location can look outside to see how high the water is." 

So what does it mean that on two of the three canals in 2015, the backup systems for gauges at the Sewerage and Water Board pumping stations have been either broken or not maintained for months, and we are now in the height of hurricane season? And what about the eight other gauges with similarly minuscule or nonexistent uptimes? It probably means those ten gauge systems - like the four London Avenue gauges that were not working ahead of Isaac in 2012 - won't work if a storm comes in this year.

So for the Corps, if (when?) their primary SCADA system goes down during a storm, they will have very poor visibility on canal water levels other than sending people out in the middle of a hurricane. It's hard to believe Corps management can be that callous, but there it is.

Practically, looking at gauges during a storm might not even work. The Corps' own operating manual for the interim closure structures says Corps personnel are to abandon the structures for any storm of category 3 and above, meaning there would not be anyone left to look at gauges.

It also means that during a storm, parties besides the Corps New Orleans District - the public, local government officials, Corps personnel outside New Orleans, the media, and state and federal officials - will not have a good, unbiased, nearly realtime measure of what is happening along the locations of the worst engineering failures to strike the United States in its history. That's a very poor way to communicate risk.

Finally, internal systems in shambles indicate there's no there there. And the publicly available outfall canal level gauge backup systems indicate serious shamblization has occurred in recent months. The damage appears severe enough that storm readiness must be seriously questioned for all the other systems along the outfall canals, including the pumps and the gates themselves.

Monday, July 13, 2015

A $6.68 million question for the Corps

Updated 7/24/15. See bottom of post.

Imagine you are a person, made of flesh and blood. You have hopes and dreams, likes and dislikes, needs and desires. Imagine further that you lived in the greater New Orleans area on August 29, 2005. Perhaps you lost your home, or your business, or worst of all, someone you loved. Before the levees failed you had a relationship of "protected" to the Corps of Engineers' "protector." Afterward, you struggled for months or years to return and then rebuild, and during that process you came to understand the Corps bore ultimate responsibility for your losses, since their storm protection failed well below their design limits. While you sweated, and cried, and laughed, and stressed through the rebuilding, you maybe - no, definitely - signed up for a class action lawsuit against the Corps.

As your personal recovery - and that of your neighbors and your city or parish - dragged on, you occasionally took note of the legal cases wending their way through the courts. Never was there talk of a global settlement; it was always the Corps fighting tooth and nail. Even when the head of the Corps, Carl Strock, admitted on June 1, 2006 they screwed up (but notably did not apologize), it didn't make a difference in the courts. Eventually all the cases died, leaving the Corps untouched. As a real flesh and blood person, you wondered how such a thing could happen.

Now imagine you are a person of a different sort - a corporation. Imagine further you're a corporation specializing in infrastructure construction with a lot of money coming in from government contracts. You too have hopes and dreams (beating analyst projections and having your patron agencies get large congressional appropriations), likes and dislikes (profits and costs), needs and desires (cash and more cash). Before the levee failures you had a similar relationship to the Corps as flesh and blood people. The Corps - and other federal agencies - protected you from the vicissitudes of the free market with a constant stream of contracts.

If you're big enough, the losses suffered by the unprotected human persons on August 29, 2005 turned out to be terrific opportunities to fulfill your desires. Unlike the flesh-and-bloods, your "recovery" began almost immediately, with four massive no-bid contracts going out two weeks after the levees failed. Those contracts got bigger and bigger, and if some of those millions of dollars were later deemed "wasteful," so be it. You stayed protected while you recovered.

So when - over four years after August 29, 2005 - another opportunity to enhance your "recovery" presented itself in the form of the permanent pumps project, you assumed you would remain safe under the watchful gaze of your protectors, the Corps. But when they inevitably screwed up and hurt you, you were sad. So like the flesh-and-bloods, you too went to court. But you're a better person than the puny humans, so of course you won. Repeatedly. And also unlike the flesh-and-bloods, the Corps felt bad about their mistakes and made up for your troubles in the form of taxpayers' cash. And you felt better and safe again.

Meanwhile the flesh and blood people again wondered how such a thing could happen. How could the Corps make mistakes that destroyed a city and its actual inhabitants, and not be found financially liable even a decade later, while when they made mistakes that harmed contractors' bottom lines, they chose to help out those contractors within months? Let's examine that.

First a little background...
The bid process for the permanent pump stations now under construction took place in two stages. In the first stage that ran from March, 2010 to June, 2010, anyone could submit a bid. At the end of that process, the Corps shortlisted five of the seven bidders (I know - not a very short list), who proceeded to the second stage where an eventual winner was chosen.

Dramatis Personae
The five bidders chosen for the second round were:

CBY Builders

This group included CDM Smith, Brasfield and Gorrie, and Yates Construction. According to Engineering News-Record, the three firms combined reported $3.1 billion in contracting revenue in 2011, the year the permanent pumps were originally awarded.

Bechtel team

Bechtel partnered with New Orleans-area engineering firms Waldemar S. Nelson; Brown, Cunningham and Gannuch (aka "BCG"); N-Y Associates; and Eustis Engineering. During the bidding process, Bechtel had a website up at, but it went away after the contract was awarded. It's still visible at the Internet Archive.

Bechtel is the largest contractor in the nation, reporting over $25 billion in 2011 contracting revenue according to ENR. They are a behemoth astride the planet. Fun fact: they hired the head of the Corps at the time of the Federal Flood, Carl Strock.

PCCP Constructors

PCCP Constructors is made up of Kiewit, Traylor Brothers, and the New Orleans-area based M.R. Pittman group. This is the same set of companies that made up the joint venture "Gulf Intracoastal Constructors," who had the prime contract for building the West Closure Complex. Kiewit and Traylor Brothers are national firms with a very large footprint. According to ENR, Kiewit - the number 3 contractor in the country in 2011, behind only Fluor and Bechtel - brought in $8.4 billion in contracting revenue that year, and Traylor Brothers generated an additional $470 million.

Louisiana Canal Constructors

"Louisiana Canal Constructors" appeared to be Alberici and Archer-Western (a subsidiary of Chicago-based Walsh Group), the same team that worked levee project LPV-111 along the Gulf Intracoastal Waterway in New Orleans East. Parsons was also involved. We know this because all three firms are listed on the Louisiana Secretary of State entry on LCC. Combined, Alberici and Parsons had just under $2 billion in contracting revenues in 2011.

Weston Solutions group

The Weston group also included design heavyweight HNTB. Weston brought in $338 million in 2011 contracting revenue

Altogether, the five bidders had over $39.2 billion in contracting revenue in 2011 alone. That is, all the bidders were national heavyweights used to going after huge projects, including two of the top three contractors in the country. They were used to getting their way.

A long and winding road

After almost a year of review and back and forth, the Corps made an initial $675 million award of the permanent pumps project to CBY in April, 2011. Nearly instantly, it was protested by two of the losing bidders - PCCP and Bechtel (Weston and LCC sat out the protest process). On a project this large, bid protests were almost a given. But the Corps didn't do themselves any favors. Through published rulings during the various protests, we learned a bunch about the inner workings of the bid process. Included in those documents were the following findings:

- A top civilian with intimate knowledge of the permanent pump project by virtue of being in charge of it, Richmond Kendrick, retired from the Corps and took a job with CBY during the bid process. The Corps - not an independent outsider - investigated themselves three times to determine whether there was a conflict of interest in Kendrick's move. Unsurprisingly, they exonerated themselves each time.

- The Corps review of the foundation engineering of the various bids appears to have been done with no engineering expertise whatsoever. A Corps witness at one bid protest hearing said the conversation about the bidders' foundation proposals took up no more than five minutes. More telling was the fact the Corps presented absolutely no contemporaneous documentary evidence at that protest hearing on how they arrived at their decision on who should get the job during their initial awarding. There were apparently no notes kept during the discussions, and the Corps could not come up with a structural engineer as a witness.

After a relatively brief protest process, and then a federal legal process that dragged on through the rest of 2011 and into 2012, the original award to CBY was thrown out and the bidding process restarted in May, 2012. This time, PCCP won in September, 2012. The new contract cost was revised downward to $629.5 million. This second award begat protests from CBY and Bechtel in October, 2012.

Now the good stuff

And this is where our story actually gets started. We know very little about what exactly happened after the September, 2012 award to PCCP. We can surmise there must have been deeper problems with the second round of bidding, because CBY's and Bechtel's October, 2012 protests resulted in the Corps very quickly agreeing to redo the bid process yet again, starting in November, 2012.

The activity in early 2013 is even murkier. All we know is that the Corps received yet another set of proposals from the bidders in February, 2013, and announced two months later that PCCP had won once again, albeit for a slightly smaller sum: $614 million. We'll get to the sequence of events surrounding the final award in a bit. But first some background on the dollars floating around this process before even the first shovel of dirt was turned.

Before the final award was announced in April, 2013, the Corps had already been paying bidders. Unmentioned in all the articles and press releases about the permanent pump project was a provision in the original solicitation which promised a $500,000 "stipend" to each of the bidders that made it to the second round but didn't win. Since there were five bids in the second round, and four of the bids would belong to the losers, $2 million total was promised to be sent out before construction began. However, that was not the only money transferred to bidders for non-construction reasons.

The Corps New Orleans District paid the legal costs for all the successful bid protests too. On January 15, 2013, the New Orleans District wrote a check for $244,179.06 to Bechtel (corrected downward on January 23 to $239,205.52 due to a clerical error). The Corps then cut another check, this time to PCCP (i.e. Kiewit) for $312,056.28 on February 23, 2013. And then they cut yet another check to PCCP - after awarding the $614 million contract to them - for more protest costs, this one for $134,679.27 on July 18, 2013. Altogether that's another $685,000 sent out to contracting firms as compensation for the Corps' errors, all before anything was built. To be clear, it appears successful protestors can file a claim to recover legal costs from protests, though the Corps is under no obligation to pay such costs, even if GAO recommends it (as they did for the 2011 round of protests). It is galling that the Corps chose to be more generous paying for their errors to well heeled contractors than to the hundreds of thousands of New Orleans, Jefferson, Plaquemines and St Bernard residents who suffered from much more significant mistakes in 2005.

That brings us to the very end of the bidding process. The chronology of the final award is of great interest, because it marks the high water mark of the Corps New Orleans District's largesse toward contractors. It's also typical of how the New Orleans District attempts to engage in image upkeep. So I've reconstructed the narrative after each bit of information was revealed, working from the most publicized chronology to the least publicized one.

Chronology 1, published in the Times-Picayune May 6, 2013
The official final contract award to PCCP Constructors came on April 17, 2013. The Corps announced it in a press release, and a corresponding article was written by the Times-Picayune's Mark Schleifstein.

On May 6, 2013, another Corps press release was issued announcing that the Corps had issued a Notice to Proceed to PCCP Constructors, the official green flag to start work. Schleifstein's companion article the same day had this to say about events between April 17 and that day:
"The Army Corps of Engineers has approved the start of construction of the $614.8 million permanent canal closures and pump stations at the mouths of the 17th Street, Orleans Avenue and London Avenue canals in New Orleans, after no objections were filed to the corps' April 17 award of the construction contract to PCCP Constructors... CBY and Bechtel had 10 days to appeal this latest corps contract decision."

Chronology 2, published by the Corps May 7, 2013
The next day, the Corps issued a much less noticed statement on the permanent pumps. In their May 7, 2013 Task Force Hope newsletter, they provided a chronology of the permanent pump bidding process. The next to last entry of that chronology reads,
"April 2013: Following the completion of corrective action procedures, the Corps reaffirms the award to PCCP JV, and their contract, previously awarded in Sept. 2012, is modified to incorporate the new PCCP JV Feb. 2013 revised proposal, and the sole Agency protest is voluntarily withdrawn."

[To clarify, there are two levels of protests bidders can file on contracts. They can go to the Government Accountability Office (GAO), which has a very formal, independent process and will publish notifications and decisions for all the public to see. Or they can also go to the Agency - the Corps in this case - where the protest is decided behind closed doors in a very informal process without any publication of results or even a notification a protest exists. In fact, even the protestors and other bidders themselves might be kept in the dark by the agency during an agency protest.]

This was the first indication there had been a protest of the final award, which contradicted the story given the Times-Picayune just one day earlier. Given the tenacity and infinitely deep pockets of the bidders throughout the two year long protest and rebid process, it seemed extremely unlikely one of them would "voluntarily" withdraw that same protest within days without receiving something in exchange.

Chronology 3, derived from FPDS-NG on August 1, 2013
We learned what that "something" was three months later. Due to the 90 day FPDS-NG publication hold on all Pentagon contracting actions, the public did not discover until early August that losing bidder Bechtel had been paid exactly $4 million on May 2, 2013, just four days before the Notice to Proceed was issued to PCCP Constructors. The description for the $4 million on FPDS was "PCCP Stipend Settlement for Unsuccessful Offeror." The use of the word "settlement" indicated something besides the standard $500,000 stipend was involved.

It seemed a good bet Bechtel was paid $3.5 million on top of their previously promised $500,000 in order to drop a protest, allowing the project to proceed more than three years after the original bid solicitation was issued. But without knowing what had happened with the other losing bidders, we could not confirm Bechtel was the bidder mentioned in the Corps May 7, 2013 newsletter who "voluntarily" withdrew a protest.

Chronology 4, derived from FPDS-NG on August 26, 2013
The final pieces of the puzzle were placed August 26, 2013, when the transactions from May 28, 2013 became public on FPDS-NG. Both CBY and Louisiana Canal Constructors had been paid their standard $500,000 stipends on May 28, 2013. With the already known information that Weston had gotten their $500,000 all the way back in January, 2012, this left Bechtel as the only firm that could have been the protestor. The only thing left was to get the documentation.

Which is what I did. It's here. The document confirms Bechtel was paid $4 million to "to settle and amicably resolve all differences, to avoid the uncertainties and expenses of litigation associated with the filing of its Agency Protest and any other litigation arising out of or related to the solicitation."

So, to summarize:
- The Corps appears to have withheld information from a reporter about how the final days of the permanent pumps bid process played out
- Then in their own newsletter, while revealing the protest, they claimed the firm (still unnamed) voluntarily dropped the protest without mentioning any recompense for the firm.
- Three months later the truth came out: It was largest U.S. contractor Bechtel that had protested, and the Corps had paid an extra $3.5 million above the standard $500,000 to make Bechtel drop its protest.
- In total, the Corps chose to pay out over $6.68 million in losing and winning bidders' protest costs, losing bidders' stipends, and losing bidders' settlement costs to companies whose combined annual contracting revenue was at least $39 billion.

For perspective on what $6.68 million can buy, each of the 54 rusting temporary pumps at the canals costs about $170,000 to refit for about two years' service. Given the Corps still has $3 million left in pump repair funds sitting unused, all of the existing pumps could have been made ready for the two extra years they must now serve with $6.68 million, with cash left over. Since the permanent pumps bid process took three years (two years longer than originally scheduled), that would have been the right thing to do to ensure proper storm protection. Instead, the Corps ensured contractor protection.

The question

Did you think no one would notice?

Did you think no one would notice nearly $7 million going out to bidders on the permanent pumps project - over $6 million of it to losing bidders - to cover legal bills, bidding incentives, and payoffs to make some of them just go away? Did you not think such grandiose treatment of multibillion dollar contractors was distasteful in light of your fighting against any compensation for those hundreds of thousands of actual human beings harmed due to your monumental screwups in 2005? Or did you think you could just hide the whole thing?

Update 7/24/15

The Times-Picayune's Mark Schleifstein wrote about this on July 22, 2015. Schleifstein got statements from Bechtel and the Corps. The Corps' statement was literally a copy-and-paste from the $4 million contract action, so that's useless. Bechtel's statement wasn't much better:
"Bechtel spokesman Warren Getler said Tuesday that the company challenged the award 'because of lingering concerns about the bid process.'
'Following discussion with the corps and to avoid further delay in this critical project, Bechtel accepted a stipend of $4,000,000, which included $500,000 for submitting a responsive bid and granting the corps the rights to all of Bechtel's work products in preparing its bid proposal,' Getler said in an email message responding to questions about the bid appeal.
Getler pointed out that a corps financial settlement with Bechtel was listed as one of a number of recommendations made by the Government Accountability Office in a 2011 report that recommended the corps throw out its first decision to award a $675 million contract to CBY Design Builders to build the pump stations.
'We also recommend that PCCP and Bechtel be reimbursed for the reasonable costs of filing and pursuing the protests, including reasonable attorneys' fees,' said the GAO opinion."

I call BS on Bechtel trying to characterize the $4 million as the same thing as the protest fee reimbursement. For one thing, Bechtel was already reimbursed $239,205.52 for those fees four months before the $4 million payoff. For another, the GAO caps the costs for legal fees at $150/hr. If we were to take Bechtel's statement at face value, this would mean Bechtel's lawyers had expended almost 27,000 hours on a protest which was filed on April 26, 2011 and decided 100 days later on August 4, 2011. That is, they would have had 11 attorneys working around the clock with no break for 100 days straight. Unlikely.

In an email exchange I had with Mr. Getler following the publication of the article, he did indeed backtrack from the linkage between the protest fees and the stipend:

"Thanks for asking for clarification on the reimbursement of legal fees. I did not mean to imply that the reimbursement was related to the $4 million stipend. As you know, it was a separate issue associated with the protest of the initial award of the contract. The reimbursement was recommended by the Government Accountability Office (GAO) in its decision sustaining Bechtel and PCCP’s protest."

When I asked for more detail regarding the $4 million payment, Mr. Getler demurred:

"Regarding Bechtel’s agency protest of the April 17, 2013, contract award, I don’t have anything to add to the information found in the government form that describes the terms of the stipend payment. If you have further questions, please direct them to the U.S. Army Corp [sic] of Engineers."

Finally, Schleifstein notes the most important detail: the extra $3.5 million came out of the money budgeted for the permanent pumps. Nice to know money meant to provide vital storm protection instead went to a multibillion dollar corporation that built no storm protection. 
[end 7/24/15 update]

Overall, I think Jeffrey over at Library Chronicles put it best when reacting to all this:
"Maybe there's stuff going on that I don't quite get. That does happen on a regular basis. But I swear it looks here like Bechtel had been holding the city's safety hostage and the Corps finally decided just to pay the ransom. But, again, this is all complicated engineering stuff and maybe I don't quite get it."

I think he got it just right.

Sunday, May 31, 2015

Rusty pumps keep rusting, Corps yawns

This post has been updated in January and February, 2017 with extra detail around pumps that were pulled and repaired in 2013 through 2016. Updates are in bold.

With the 2015 hurricane season upon us, and the continuously rusting pumps at the three interim closure structures (17th Street, Orleans Avenue, and London Avenue) well beyond their predicted life by any reasonable measure, I thought it would be time to check back in and see if the Corps has stepped up - or actually done any - preventative maintenance on them. For an overview of the rusty pumps saga from 2006 through 2012, see this wrap up post.

The pumps were put into the water in 2006, pulled out, and then put in for good in 2007. They were not expected to be in service more than a couple of years before they would get replaced by sturdier, bigger pumps. But planning for their replacement dragged on for years. The Corps, starting in 2009, stated the life of the entire structures - slyly without specifying the exact life of the pumps themselves - was "five to seven years." They said maintaining the pumps beyond 2013 would be difficult. They did this mostly to bring pressure to bear on locals to approve their permanent pumping scheme. But taking them at their word, if we started the lifespan clock in even 2007 - generous given construction of the structures began January of 2006 - we are now beyond even the worst case for the structures' and pumps' lives. One would have expected to see some effort toward life extension on these pumps.

And we did for a little while, with the replacement of many carbon steel components on the pumps with stainless steel between 2010 and 2012 (Note: Two pumps - E5 and E7 at the 17th Street site - remain untouched by those refit efforts). During that 2010-2012 period, the pumps were not expected to be needed past the 2014 storm season. However, in the intervening time the end date of the pumps has been extended even further into the future. Now they are expected to be in service until 2017, or 11 years after they first saw brackish lakewater. So what has been happening with the pumps since my last update during the 2012 hurricane season? Has the Corps done anything to make them last the additional three years?

In a word, no. Since 2012, the Corps has pulled six pumps across the three sites, almost all in reaction to oil leaks. There does not appear to have been any proactive moves to ensure the pumps can last until 2017. Here's what has happened:

2012-2013 off season: no pumps pulled.

May 20, 2013: Pump W10 at the 17th Street site sprung a hydraulic oil leak. The pump was pulled out a couple of weeks after the oil leak, just as the 2013 hurricane season was starting. It likely went back in the water some time in August, 2013 July 19, 2013 [see update below]. This failure was not a surprise, since W10 had not received the stainless steel upgrade in 2010-2012, but had been part of a group of six pumps given inferior carbon steel replacement repairs during a misbegotten effort in 2009.

Update, 2/5/17

The contract task order for this work is here, while the repair report is here. They confirm that this pump received stainless steel piping and coolers during this 2013 refit, which was essentially a complete rebuilding of the pump. Here's a picture of a severely corroded Rineer hydraulic motor, the device at the heart of pump, being pulled out:

July 2, 2013: Pump W9 at the 17th Street site sprung a leak. This failure is far, far more concerning. In fact, it's a canary in the coal mine. That's because W9 had been upgraded with the stainless steel [see update below] suite of repairs during the summer of 2011, just two years earlier. Now in 2013 it was leaking once again, causing a third round of repairs (W9 also got the 2009 carbon steel repairs).

This is troubling in the extreme. It means the stainless steel upgrades are all suspect, which makes the integrity of all the pumps suspect. This would be confirmed just a few months later on...

Update 2/5/17

The story on this pump is somewhat complicated. The pump had been part of the original set of half repairs in 2009, when only its internal carbon steel piping was replaced with stainless steel. The external oil coolers remained carbon steel. A picture of the new stainless internal piping appeared in the 2009 report:

Then in 2011 the pump had to be pulled out on an emergency basis when it started leaking during hurricane season, on July 5. Conhagen's work was very fast, from July 5th to July 15th that year. In fact, the task order for the work explicitly mentions overtime being required:

"Remove pump # 9W at 17th St. Canal to inspect, recommend what need to be repaired and reinstall it.

1. Pump # 9W was overhauled in Aug 2009 and is currently leaked [sic] outside of the pump.
2. After the contractor inspects and finds out what the pump problems are, we will provide Contracting Division a separate scope of work for repair.
3. Estimated date to start is 7/8/ 2011 and overtime for the contractors to work on Saturday 7/9/11 and Sunday 7/10/11 should be authorized."

Part of those repairs was replacement of rusted out oil coolers, seen here in a "before" picture from the 2011 report:
However, a picture of the internal piping at the completion of Conhagen's repairs from the same report appears to show the 2009-installed internal piping was reused. New stainless steel piping would have been shiny. Instead, the piping is black (a process known as passivation):
This is backed up by the scope of work in the 2011 report, which makes no mention of replacement of the internal piping, only the oil coolers. The reason for the piping reuse is likely time. Pump W9 was only out of the water for about a week in 2011, and the piping probably looked fine at that time. So rather than replace it preventively, it was left in place.
Then came the third removal in 2013. The contract task order for this work is here, while the repair report is here. Rust was again the cause for failure:

Note the spots of rust on the Rineer itself.

The pump was completely overhauled, including new stainless steel piping inside:

and outside:

The pump was accepted back into service during the height of hurricane season, on August 23, 2013.

So the removal in 2013 was an indication of the fragility of the repairs, but also one that demonstrated a lack of forethought. If the Corps had a complete spare pump assembly ready to go in 2011, they would not have had to rush those repairs. They could have simply replaced the entire pump and worked on it in a more reasonable timeframe, and then placed it back into service with a complete suite of repairs.

Not that any of that really matters, because the complete suite of repairs done in 2013 would prove inadequate three years later in July, 2016, when the pump would again get pulled. In fact, 17th Street pump W9 holds the record for most repairs to date since 2009, with four. It's neighbor, W8, is in a three way tie for second with three pulls. The other three time failures are London Avenue pumps E2 and W5. New Orleans is protected by phalanxes of Edsels and Pintos.

December 19, 2013: Pump E8 at the 17th Street site sprung a leak. It was pulled out for repairs in January of 2014 and went back in the next month. The canary started chirping louder, since E8 had also received the stainless steel upgrade, in this case three years earlier, during the spring of 2010. While we don't know the exact problem in 2013, there is some indication the pump did not need a complete overhaul. The original task order for the 2013 E8 repair was for $172,934.74, but a modification subtracted $130,693.20 of that. Full pump overhauls usually cost around $160,000 to $170,000. If there was only $42,000 of work done on the pump - and considering some of that was for crane rentals on both ends of the repair - it makes sense to think the problem was relatively small but critical. There's really only a few possibilities: a) a piping failure, b) a seal failure, or c) a failure of the Rineer hydraulic motor. [See update below] But it doesn't really matter, because there is no spare pumping capacity built into the interim closure structures, so a failure of any pump will hurt a storm response.

That was the only pump pulled in the 2013-2014 off season.

Update - 2/5/17

The contract modification covering the 2013 work on this pump is here, while the repair report showing the damage is here. Much of the damage was from corrosion, as can be seen in this picture of a very rusty Rineer motor:

In fact, the reduction in price for the rebuild of pump E8 at 17th Street was not due to a reduction in scope, but to a warranty claim related to the rusty Rineer. A rebuild of the bearings and seals was performed, along with weld repairs and presumably replacement of the Rineer. No piping was replaced. The Corps clawed back $130,693.20 because "the contractor installed steel plugs in lieu of stainless steel plugs in the previous overhaul which caused corrosion and damaged the seal/accessories." That appears to refer to carbon steel plugs installed on the Rineer motors and left in when the motors are supplied to Conhagen back in 2010. Conhagen apparently forgot to replace them with stainless steel plugs. 

The 2013 repair report doesn't clearly show the plugs on the Rineer motor when it came out. The picture above is the best one, and it doesn't clearly show the plugs. However, Conhagen actually took a picture of the motor being installed on this pump in 2010 and included it in their repair report:

It is impossible to tell if these plugs are stainless or carbon steel, or if they are indeed the final, as-built plugs (other documents and photos show that hex head plugs were used on other Rineer motors). Generally, stainless is shinier than carbon steel, and these appear to be stainless, but one cannot be absolutely sure.

And then late last year...

October 29, 2014: Two pumps at the London Avenue site sprung leaks. They were pulled out and repaired late in the year. The exact identification of the pumps is not known, but it is irrelevant, since all ten of the London Avenue hydraulic pumps underwent the stainless upgrade between 2010 and 2012. This means that the lifespan of the refitted pumps is about 3 years. Given that we are now in 2015, and the refits took place from 2010 to 2012, that means all the pumps - except the ones repaired since 2012 - are likely at the end of their lives due to corrosion. Again.

Update - 1/15/17

The London Avenue pumps that failed in late 2014 were E2 and W5. E2 received a stainless steel upgrade in late 2010, while W5 had gotten the same upgrade in spring 2011. The task order for the 2014 rework on both pumps is here, while the repair report is here. The work performed on both pumps was substantial, requiring repair or replacement of nearly every component of the pumps except for the external coolers.

Finally, a sixth pump was pulled for repairs last December. There are no details on which pump this was, not even a spill report. It likely went back in this past February. There do not appear to have been any other pumps pulled, but Corps contracting records on FPDS only extend through the beginning of March, 2015.

Update - 1/15/17

The pump pulled for repairs in December, 2014 was 17th Street pump E1. It got a full refit (except for piping repalcement), as documented in the task order here and repair report here. It was back in the water for acceptance testing March 12, 2015. It had received the stainless steel overhaul in the spring of 2010. A notable line in the scope of the 2014 repairs is the explicit callout, "Change out carbon steel plugs with S/S [stainless steel - ed.] in customer furnished motor." This step was missed on the 17th Street pump E8 repair earlier in 2014 that resulted in the warranty reimbursement to the Corps.

Update - 1/28/17

Just three days after the original publication of this post, yet another pump sprung a leak - W7 at the 17th Street site. In this case, there was a spill report filed on June 3, 2015 and a cleanup costing $6706.53 was performed by QRI. The pump appears to have come out of the canal about a week and a half later.

It had been refitted in spring 2011, as documented here. The repair report for the 2015 repairs calls out the same cause of failure as the last pump that failed:
"This pump was pulled for hydraulic oil loss. After being pulled, it was found that one of the plugs in the motor had corroded away."

The pump was refit again at an expense of $163,546.47. Everything was repaired or replaced except the stainless piping and coolers, though one stainless line did have to be replaced. Presumably the Rineer motor got stainless plugs this time around. The pump was back in the water and accepted back for service July 23, 2015.

This would turn out to be the only pump pulled in 2015, though two of its neighboring pumps - W8 and W9 at 17th Street - would be pulled the following year. For W9, it would be the fourth time out of the water since 2009.

What's going on with these latest failures? Without the Conhagen repair reports, I can only venture a guess. There's the possibility of galvanic corrosion on the remaining carbon steel components, such as the deeply critical Rineer motors. Stainless steel plus carbon steel plus seawater sets up a battery, which can corrode things even faster than all-carbon steel units. There's also the possibility of inadequate amount of zinc anode protection placed during the refits. The anodes are supposed to corrode preferentially to the pumps, but who knows if that's taking place? The pumps aren't being pulled out preventatively to check such things, only when they are already corroded beyond use. Effectively, the Corps is driving their car with the engine light on.

Update 1/28/17

With the Conhagen reports and the Corps task orders, we know corrosion of the Rineer motors has been at the center of many of the failures, with Conhagen explicitly calling it out in their 2015 repair report of 17th Street pump W7 and the Corps calling it out in the 2014 contract action for repair of 17th Street pump E8. Additionally, the Corps has continued a constant rotation of Rineer motors, issuing a $9365.21 task order in August 2014 for repair of three of them, and then a $12,525.11 task order in February, 2015 for repairs to three more. I don't have the supporting documents for the 2014 repairs, but I did get the purchase order to the Harvey, LA office of Hydradyne for the 2015 repairs.While the repairs did not call for the replacement of the vane cartridges, which are the moving heart of the pumps, there were callouts for seal and bearing and cover replacements. There were also explicit callouts for stainless steel plugs. This points at water ingress and corrosion.

This is of a piece with the historical trend of heavy use of Rineer motors. As noted in this 2012 entry, the Corps rebuilt or purchased 34 Rineer motors between July, 2010 and March, 2012. Tellingly, one of the task orders then explicitly called out replacement of the housing plugs, though they were not called out to be stainless steel:
"All motors: Replace all motor housing plugs with new solid hex head plugs."
But that's really just details. The overall arc of Rineers failing due to corrosion and seal failures - either of the Rineers or the components connected to them - hasn't changed. What I wrote back then still applies:
"The bottom line is that these Rineer motors are obviously ill suited to salt water exposure ... Leaving them where they are is simply asking for big trouble ... I've published a slew of pictures showing Rineers rusting to bits. They've been getting salt water into them through the rusty pipes, the seals, and every other leak path. The Corps has tried replacing them and repairing them. Nothing works. They all get eaten up. These motors are the heart of the pumping system at all three Lakefront closure structures, and they are unreliable and rusting to this day."

So that's it. There does not seem to be any movement on the Corps' part to prepare the interim closure structure pumps for three more years' of service, let alone give any sense of confidence that the pumps furthest out from the stainless refits are not going to crap out this year. Further proof - in addition to the lack of activity at the site - is the fact the Corps' four year pump repair contract with Healtheon expires this year, and a solicitation to replace it has not been issued. [see update below] In fact, of the $6.75 million on that repair contract, over $3 million ($3,068,911.61 to be exact) remains unspent as of March 1, 2015. At the going rate of $170,000 per pump refit, that $3 million could have gotten 18 pumps back into fighting shape for a couple of more years. Instead, the Corps New Orleans District just sat on their asses and didn't do anything proactive for the last three years.

Update - 1/15/17

The Corps had actually extended the Healtheon contract in February of 2015, though it is unclear how long the extension was for. A new 3 year, $3.9 million pump repair contract was issued to J. Star Enterprises on October 8, 2015. To date, at least three pumps - W8 and W9 at 17th Street and E2 at London Avenue  - have been pulled for repairs.

This is beyond disappointing, since we now know the pumps remain ticking time bombs ready to fail at their first real test this year. I don't know how much more proof the Corps needs than at least three of their own pumps - refitted to their own specifications - failing within an average of three years after that refit. That doesn't even count the two remaining pumps at 17th Street (E5 and E7) which remain without the stainless refit entirely, though there exists the possibility the pump pulled out in December 2014 is one of them.

Mitigating this problem perhaps is the additional work performed on the London Avenue and 17th Street canals last year to Really Truly Be Able To Handle 8 Feet Of Water. The ability to have higher canal water levels means there is a small amount of breathing room between what the city sends the Corps' pumps and what the Corps' pumps must pump out. However, when one considers the large gap between what the Corps says (one day before Isaac in 2012: "Orleans Ave [canal] - Max. Water Level is 8.0.") and what they actually do (during Isaac in 2012: "we will discontinue one of the [city's] pumps [feeding the Corps' Orleans Avenue canal pumps] when we hit 3 to 4 foot"), their credibility isn't worth anything. The only thing you can trust is the dollars, and they haven't spent any money getting the outfall canal pumps ready, beyond fixing stuff that they found broken in the most obvious way - an oil slick on the water.

In fact, their lack of concern about the general end-of-life state of the interim closure structures is so huge, that the topmost Corps officials are even joking about it on camera. In WWL's "Eye on Hurricanes" special aired June 1st, Corps New Orleans District commander Richard Hansen compared the existing interim closure structures and pumps to the now-under-construction permanent pumping stations:
"Well I'd say the interim control structure is kinda like its backyard kitchen, okay? It's not fully protected from the weather. It can do its job out there, but the permanent features will be the Cadillac. That will be the feature designed to last for 100 years"

It's wildly inappropriate to joke about how inadequate your own storm protection structures are. In fact, the WWL reporter says in the very next breath:
"So the interim pump structures have been in since 2006 and have actually outlived their lifespan, but the Corps says it will not be a problem for them to remain in place and be working until the permanent pumps can be complete. And right now that's expected by the height of hurricane season in 2017"

"The Corps says" It's unbelievable this still is accepted as truth.

Based on the lack of any preventative maintenance activity, as well as the evidence of refitted pumps failing just two to three years after their refit, I would judge all but a handful of the the outfall hydraulic canal pumps in a stage of imminent failure. The drainage of the city during a major tropical event which includes large amounts of rainfall is now dependent on luck and hope that stuff won't rust through during a storm. That's very, very bad.


Sunday, July 06, 2014

Bricks, rebar, huge rocks inside the 17th Street canal levees (which is no surprise)

The photos showing the unearthing of massive amounts of unsuitable debris from the outfall canal levees this spring and summer bring into stark relief the true nature of flood protection in the greater New Orleans area. By the Corps' own standards, a deeply inadequate system remains in place - a system which under current design and construction guidelines would never get built today. We know this - not by having to plumb endless internal emails or looking through musty, decades-old records - but by simply comparing today's pictures to the records of the last 17th Street canal remediation project conducted just three years ago.

A new round of photos of the Corps of Engineers' 17th Street canal work site taken in late June detail more fully the debris that was piled into the 17th Street canal levees years ago:

Bricks and rebar:

...which we can see after moving closer to the area in red:

Bricks and large rocks:

...which are visible after zooming into the area in red:

None of this should come as a surprise, since the same junk was pulled out the levees a bit north of this year's work site three years ago during the first round of remediation (which I wrote about in 2012). That project offers insights applicable to this year's project.

Extensive "unsuitable" material found inside 17th Street canal levees in 2011

The 2011 17th Street canal remediation project consisted mostly of deep soil mixing of grout with existing soil to improve the strength of the soil along both east and west banks of the canal. Columns of dirt below the base of the levees were mixed with injected grout down to -20 to -40 feet. The contractor had to excavate a few feet off the top of the levee to give a flat platform on which to mount the drilling rigs and other work equipment. Here's the generic cross-sectional detail from the project issued-for-construction drawings showing this:

The excavated material is shown with the single hatching on top of the levee. It was expected to be reused as backfill for the same area at the completion of the project. That's not what happened.

Right from the start and throughout the entire 2011 project, the Corps' contractors continually found the levees on both sides of the canal were riddled with, shall we say, non-dirt objects. The greatest concentration appears to have been on the west - or Jefferson Parish - side. On the very first day of construction on the west bank (March 7, 2011, which was also Lundi Gras), the contractor dug into the levee near the intersection of Orpheum and Rosebud. The contractor's Quality Control report (all reports from the 2011 project are linked from this post) for that day said:
"Visually inspected the construction of the West Bank production pad. Material removed from the existing levee between stations 588+00 and 590+00 contained significant debris (rocks, concrete, bricks, shells, asphalt). Informed Bill and Derrick [Corps employees Bill Richardson and Derrick Parker] and they came over to look. We asked they to determine if this would be considered "unsuitable" to reuse in the construct of the levee"

Four days later, March 11th, a decision was made on this "significant debris:"
"Got a verbal answer from [Corps employee] Bill Richardson concerning the 'unsuitable material' encountered during the excavation of the West Bank platform. We are to excavate to the required depth to build the production platform only. All excavated material will be hauled offsite and replaced with suitable clay material from the Bonnet Carre borrow pit."

The 2011 project involved excavating the top of the west bank of the canal from Georgia Court to the Old Hammond Highway bridge, a distance of 3/4's of a mile. Thus, this statement implicitly said that the entire levee along that stretch  - at least as deep as the Corps wished to dig but no deeper - was made of dirt not suitable for levee construction:

The Corps' wink-and-a-nod instruction ("excavate to the required depth to build the production platform only") indicated they believed the rest of the levee was similarly riddled, but didn't want to bother dealing with it. That means there's likely still junk in the 17th Street canal levee today, just further down than where the 2011 project dug. Returning to the detail from the drawings, we can see this graphically:

Two weeks after the implicit acknowledgement of the unsuitability of the west side levees, the contractor made it explicit. In the March 24, 2011 Quality Control report they reported on excavation of the west bank levee along Orpheum between Chestnut and West Esplanade:
"Excavation (degrading/construction of production platform) on West side of canal today from station 581+80 to 579+00 to an elevation of +1.8 to +1.5'. All degraded material was hauled to laydown area on the East side of the canal. Material was considered unsuitable by USACE and will be stockpiled in laydown area on east side of the canal. The material will then be spread throughout the laydown yard at the end of the project per USACE direction."

As I noted when I wrote about this in 2012, at least the Corps didn't try to reuse this junk dirt. But they also didn't bother to tell the public and their government representatives in any meaningful fashion there were "rocks, concrete, bricks, shells, and asphalt" inside 3/4's of a mile of 17th Street canal levee that had been there since it was built decades ago. Or that such debris would remain within the levees even after the project was completed. They just proceeded to conduct business as usual, like it was a normal thing to discover - six years after the fatal failure of their levees - that the levees along the 17th Street canal were built out of trash. Day after day for months, they hauled dirt off the Jefferson Parish levee across the canal to their laydown yard on Bellaire Avenue along the east bank of the canal - in front of the 2005 New Orleans breach location - and told no one in the public the extent of the problem. Some of that bad dirt was spread out at the laydown yard during the summer, 2011 and literally buried under grass. The rest of it, including some stuff too big to be buried, was trucked offsite. The August 23, 2011 quality control report describes this disposition:
"East side of the canal - Spreading unsuitable material in laydown yard per USACE's direction. Large debris is being removed and stockpiled to be hauled off at later date."

Fortunately, three years later during another project involving digging into the Jefferson Parish side of the canal, we have pictures showing us just how poorly these levees were constructed - and indicating how much junk remains inside them today, so there's no way this information can be buried or hauled away again.

Debris so huge it couldn't be removed in 2011, or ever

Some of the junk was so huge in 2011 it couldn't even be moved out of the levee. An extreme example of this was documented by the contractor logs starting on March 25, 2011. That day, there came discovery of a very large concrete "object" just below the west bank levee near the intersection of Orpheum and West Esplanade. From the contractor's quality control report:
"During the construction of the production platform on the west side of the canal, a large concrete structure was exposed at approximately station 577+80. CKY excavated around object to try and see the size of the object and how to remove object. The size of the concrete object was found to be approximately 20'x13' with rebar protruding out of the concrete. Object was unable to be removed. USACE was contacted about the object and pictures were taken. GPS was used to mark the location of the object. The USACE will be asked in an RFI [request for information] if only elements ABC can be installed at this location due to the large concrete object."

The Corps quality assurance report has a little more detail:
"The concrete footing is quote [sic] thick, it could exceed 2 feet. The footing consists of spread footings that cover the area between stations 577+81.5 and 577+90.5. The depth, below levee grade, is near the elevation of the adjacent road."

That's right: a room sized chunk of concrete buried under the levee that didn't get discovered until the levee was dug out. This was so huge that even the Corps owned up to it, taking pictures two months later when they dealt with the thing:

This mass of concrete was in an area that had been extensively explored by Corps test borings. No fewer than 11 test bores and cone penetrometer tests had been drilled into the levee within 200 feet of this location, 10 of them since the 2005 levee failures, as seen on this detail:

One must ask how such a giant chunk of concrete could be unearthed as a surprise. It also raises the obvious question about the quality of the information gleaned from the test bores and other exploratory drilling the Corps does before a project.

In this particular case, the obstruction was so huge they actually left it in place. After covering it back up in March, they went back in late May. They uncovered the obstruction and then backfilled and compacted the affected area between May 24, 2011 and June 1, 2011. Presumably some work crew 20 years in the future will dig it out again.

Comparing the 2014 project to the 2011 project

Going back and reviewing all the records from the 2011 project has given me a little different perspective on what we can see in the 2014 project photos. During the 2011 project, all the construction on the levee was done with dirt from the levee. This included building of platforms and ramps - things occurring right now during the 2014 project. Clean dirt was not brought into the site in 2011 until all substantial work was completed. I have no reason to think the 2014 project is proceeding any differently.

So unlike what I wrote a couple of weeks back, I now believe that all the soil currently piled up on the levee came out the levee, including the stuff near the base that is acting as a "road" for the crane. This means giant boulders are also coming out of the levee, not just chunks of concrete, brick, and pieces of rebar. Here's a couple of the pictures from my prior entry, relabeled to include that conclusion:

In comparing the progress of the 2014 and 2011 projects, what do we make of the fact the Corps started removing bad dirt from the site within days of unearthing it in 2011, but the junk has remained on the levee for weeks in 2014? Some of the reason might simply be how this year's site is set up. The dirt at the bottom of the levee this year remains in use for access to the entire site, with only a single point of entry or exit for heavy equipment at the south end of the site. The crane using the platform at the levee bottom is needed for the entire duration of the project, which has not been completed yet (the contractor is moving from south to north). So the bad dirt making up the platform  probably can't be removed until all the sheet piling has been placed.

Debris throughout the entire 17th Street canal west side levees?

However, the logistics of this year's project are not nearly as important as the larger point: everywhere the inside of the 17th Street canal west side levee has been revealed - whether in 2011 or this year - massive amounts of debris have been found. And even more troubling, there are very long stretches which remain untouched:

Given the shallow depth it was found in 2011 and 2014, it makes sense some of the debris was part of the I-wall and levee construction in the early 1990's. That work did involve placement of dirt on top of the existing levees at the time. It extended the entire length of the canal (all the yellow and red lines shown above). Other, deeper debris has likely been there far longer. The records of nearly every soil boring taken in 1981 along the west bank - long before the I-walls were put in - have numerous mentions of shells, wood, bricks, and other "miscellaneous fill." Those records were used in the basis of design for the Corps' early 1990's work. This junk has been known to the Corps for a very, very long time. It also very likely extends the entire length of the canal on at least the Jefferson Parish side.

How it got there doesn't really matter now though, because all the junk is still there today, and the integrity of the outfall canals remains the responsibility of the Corps. As the decision recorded in the contractor quality control log of March 11, 2011 indicates, there is no way the 17th Street canal west bank levees would be built today with the debris-riddled dirt that is currently sitting in them. That dirt is - according to the Corps - "unsuitable." Yet there the levees sit.

Thus, this is what the Corps considers the recipe for sufficient levees along possibly the most well known stretch of levee in the nation:

These levees do not care what water presses against them - storm surge or rainwater. Thus it make no difference that the outlets of the outfall canals are "protected" by gates now. These levees still see pressure from stormwater during major rainfall events. One must wonder what local government officials and Corps personnel living behind these levees think seeing the true condition of the Corps' flood protection.

Labels: ,

Monday, June 23, 2014

Hidden in plain sight

[Update, 7/7/14: Based on photos taken after this post was published, and a review of the records of the 2011 17th Street canal remediation project, revisions have been made to this post. It is now believed the large rocks seen at the base of the 17th Street canal west side levee in photos taken June 15, 2014 did indeed come from within the levee. Text and photo captions have been adjusted accordingly. Further details and analysis are available at subsequent Fix the Pumps post published July 6, 2014.]

Unreported by anyone, the Corps has begun major work along two outfall canals in New Orleans in the last month. Very long sheet pile is being driven on the 17th Street and London Avenue canals, and rock is going to be placed at the base of the levees inside the canals starting in June or July. These are the same outfall canals that breached in three places in 2005 due to poor design. The ensuing floods of New Orleans devastated hundreds of thousands of lives throughout the greater New Orleans area and beyond.

The Corps' irrational need for secrecy about any work along the outfall canals is long known, but their actually beginning structural work without any public pronouncements, press releases, or whispers in the ears of the mainstream media is unprecedented, especially considering the scale of the work - which is extensive. I think they would rather nobody paid any more attention. But attention should be paid to work at the sites of the cause of the worst engineering disaster in American history.

The work underway already is for seepage control due to existing sheet piling being too short. Seepage happens when water works its way through sand below the bottoms of the I-walls along the canals. Should that water then percolate upward through the land-side (the side where people  live) soils, the levee and I-wall can fail. This was the primary mode of failure for the south breach of the London Avenue canal following the 2005 passage of Katrina dozens of miles to the east.

Problems have emerged just weeks into the project, as the soil being dug out of the levees appears to be packed with unacceptable debris like rocks and chunks of concrete. The Corps' own specifications forbid such debris in a levee since it decreases the strength of the soil. However, they have shown they are willing to ignore such specifications during many previous projects both before and since 2005, including a round of projects on all three outfall canals in 2011.

Those 2011 projects included similar seepage remediation work to this present project along the London Avenue canal and the Orleans Avenue canal. The 2011 17th Street project remediated against different modes of failure than seepage, and thus included different technologies than sheet piling.

For calculation purposes, the Corps and their consultants have broken each side of each canal into individual "reaches," based on differing soil and canal bottom conditions. The reaches are numbered sequentially, starting at the south end of the west side of each canal and finishing at the north end of the east side of each canal.

Here's the locations previously remediated along the London Avenue canal in 2011:

And here's the new locations in the 2014 project:

As you can see from the above map, after they are done with this round at London Avenue, nearly the entire canal north of Mirabeau will have had sheet piling placed outside the existing, too-short piling. It points up how poorly the I-walls were designed the first time around, and how inevitable their failure was.

The contract - which includes work at both the London Avenue and 17th Street sites - was awarded February 28, 2014 to Conquistador Dorado JV (aka Dorado Services) out of Sanford, Florida. There are indications Cycle Construction in Kenner is the local partner in the joint venture. The contract - actually a task order on a Service-Disabled Veteran-Owned Small Business (SDVOSB) contract awarded to a pool of four companies -  is worth $13,642,238.50 and is scheduled to be completed October 30, 2014, meaning construction will proceed throughout the 2014 hurricane season. Construction along the London Avenue canal started in late May with Reach 12, just north of Filmore Avenue on the west side of the canal.

I have had interested neighbors photographing the work as it proceeds. This photo is from what I believe was the first day piles were driven at the London Avenue canal site, May 29, 2014:

Sheet piling is being driven down to clay approximately 55 feet below sea level. The new sheet piling is being driven a few feet to the protected side of the existing I-wall. The existing sheet piling inside the I-walls goes down to sand about 15 feet below sea level. The Corps is using a pile pressing machine rather than a standard pile driving hammer. The pile presser supposedly creates fewer vibrations and is quieter. It's a machine they've used extensively in the dense urban environment of New Orleans and its surrounding metro area, especially where space is at a premium:

Here's piling being placed the following day:

To make room for the pile presser, the top couple of feet of levee soil next to the I-walls will be stripped off to make a level surface and the dirt stockpiled on the levee itself. Here's the excavators at the London Avenue site:

The sheet piling machine travels along the newly flat section and presses the piling in. This June 6, 2014 photo shows the first few linear feet of piling already driven:

Afterward, the soil will be placed back on top of the piling and grass will be planted. This was done at adjacent Reach 13 in 2011.

If we move closer to the stockpiled soil, details emerge. Worrisome details:

Here's enlargements of the photo above:

All the white bits are broken concrete. This next photo shows it best:

Here's the concrete debris highlighted:

The dirt coming out of the levee is absolutely shot through with the stuff. That's bad. Having debris like concrete in the clay decreases its strength because it cannot compact properly. Finding debris in these levees is not a surprise, since it was reported on either side of this stretch during the 2011 London Avenue project as well:

It is one thing to read about it months afterward, and another thing altogether to have pictures of it happening live.

After receiving the pictures of the London site, I asked for pictures of the 17th Street site. Based on those pictures - taken on June 15th, I believe the story is much the same.

The site location on the 17th Street canal is on the west side, just south of I-10:

It is reach 16, which I wrote about extensively 3 years ago. At that time, the Corps was considering placing dirt at the base of the levee on the protected side (outside) of the canal to prevent failure from underseepage. The idea was to provide enough weight on the landside to resist the upward motion of underseeping water. It's an indirect, cheaper way of addressing seepage, as opposed to the direct solution of cutting off the seepage with sheetpile driven into the clay below the sand. It's noteworthy nothing was done in the intervening three years. It's also noteworthy the Corps New Orleans District consultants' calculations in 2010-11 (as well as the Corps' District Engineering folks' marching orders to their consultants before any calculations were performed) claimed there were no seepage problems on reach 16 - or anywhere along the 17th Street canal. Because of that, this newest remediation project is the first one to address seepage concerns along the 17th Street canal. We'll get into what likely changed their mind in future postings.

Here's reach 16 in 2011 (photos taken from the June, 2011 Corps update on that year's remediation projects along the outfall canals):

Construction at the 17th Street canal site also started in late May. The contractor began at the southern end of the site, which is what is shown in the "before" pictures above.

Here's that same end of the site snapped June 15, 2014 (Update, 7/7/14: bottom caption used to read "Dirt placed to make 'road' for crane to travel. Dirt likely not from inside levee"):

I believe that the dirt toward the bottom of the levee is - I hope - fill brought from offsite and placed to allow the crane (which holds the piling vertically) to travel along the levee with the piling pressing machine. I do not think it, and the huge boulders in it, came out of the levee.

[Update, 7/7/14: Based on subsequent photos and a new review of the 2011 remediation project records, it is now believed the material at the bottom of the levee is from inside the levee. This includes the large boulders now visible.]

However, I believe the soil toward the top of the slope is from inside the levee. Even from a distance, we can see the chunks of debris. Getting a closer look shows a large amount of concrete and rocks (Update, 7/7/14: bottom caption used to read "Dirt placed to make 'road' for crane to travel. Dirt likely not from inside levee"):

Here's an enlargement of the above photo:

There's even a stump that appears to be about 2 feet long, just sitting on top of the levee. It seems very likely it came from within the levee (though I suppose it could have come from somewhere else, but then why would it be sitting on the top of the levee?):

Dirt taken out of the southern part of the site is also being stockpiled on the levee near the northern end of the site in front of the end of Lemon St. The copious debris - all the white bits - is obvious (Update, 7/7/14: bottom caption of first two photos used to read "Dirt placed to make 'road' for crane to travel. Dirt likely not from inside levee"):

By any standard, including the Corps', the dirt coming out of the levees at both these sites would be unsuitable for use in any levee. How do we know that? The Corps themselves said so just four years ago, during a different debris mess that made the news.

In late 2009 and early 2010, reports emerged about large amounts of concrete and other debris being found in the top layers of lakefront levee in Kenner during an enlargement project. That soil had been trucked there in the early 2000's and had not been systematically checked for debris. After a couple of Times-Picayune articles, the Corps held a public meeting on January 28, 2010. They showed the following slides as examples of unacceptable amounts of debris in the dirt:

Those photos look eerily similar to the pictures coming out of the London Avenue and 17th Street sites.

The 2010 slides explicitly called out concrete chunks as unacceptable:

In the 2009-10 lakefront case, the Corps' solution - after blaming the previous contractor and prior Corps employees and assuring the public these were isolated instances (which they weren't) - was to remove all the dirt with concrete in it. According to the Corps, the contaminated dirt went down 2.5 feet, so the Corps stripped it all out and replaced it with clean clay.

Here's the scary part: that might not be the plan for the outfall canals, for three reasons:

1) The bid specifications (as amended) for the outfall canal job call for the contractor to place most of the soil they remove from the levee during construction right back in the levee when the sheet piling work is done.

According to those amended specifications, 2350 cubic yards of soil is to be taken out of the levees and reused between the two canal sites. The Corps is only anticipating 650 additional cubic yards between the two sites to be "unsuitable." That is, they expect only one fifth of the total amount of dirt to come out of the levees to stay out, and the rest will go back in. That was the financial basis for awarding the job.

2) We know bad soil with debris just like what we're seeing this year was taken out of outfall canal levees and put back in during the 2011 projects. The Corps' own contractors' said so, doing so under orders from Corps employees.

3) Finally, we know the Corps has no compunction using or reusing debris-laden soil that violates their own specifications all over the greater New Orleans area. Besides the three outfall canal remediation projects in 2011, I counted 22 projects where it happened in the last few years:

For further information, you can consult my 14 (!) part series on debris in the levees. Part 1 is here, with the other parts linked from it. The Corps' use of unacceptable soil seems to be more the norm than the exception.

Why does this matter? After all, there's gates at the ends of the canals to protect them from hurricanes, right?

Well, the outfall canal walls and levees do not care where water comes from - whether it's the lake or the city's pump stations. Even if the lakefront gates are closed and secured during a high lake level event (a big "if," since during the last hurricane 5 of 11 gates at the London Avenue structure were not secured), the city still has to pump rainwater from within the city down the canals. That water level can get quite high - just as high as the water got from Katrina's surge. A levee with concrete in it instead of clay is simply not as strong, and could be prone to, um, difficulties.

In addition, there is the obvious question: if this is what the top couple of feet of the levees along both canals look like, what about the rest of the levee left untouched by these newest projects? Given that these stretches of levee have not been dug into since the failed I-walls were installed about 15 years before their 2005 failure, it is a reasonable concern that the entire levees - including previously remediated stretches - are just as bad as what we've seen here. Without exploratory trenching - as has been done on many other projects where debris was found - there exists a deep trust deficit between the Corps and the public who depend on these levees for their lives and livelihoods.

Labels: , ,

Go to older posts