50% contingency? Really?

After a long hunt, I received the missing link in the documents which led to the Corps' cost estimates for Options 1, 2 and 2a on the permanent pumping stations given to Congress in early 2009, months after their Congressionally imposed deadline.

The document is a cost estimate prepared by Corps personnel in the Walla Walla District. Here it is.

I will be posting in more detail about this soon, but I wanted to get this document out there as quicklly as possible. The findings are two-fold:

1) The costs the Corps gave to Congress in 2009 include a massive amount of slop. The Corps took base estimates prepared by Black and Veatch which did not include contingencies for each Option. The Corps figured out the contingencies in cost and schedule and added those to the base estimates to get the final numbers which were handed over to Congress. Those contingencies were all around 50% of the original base estimates. For example, the original Black and Veatch Option 2 estimate was $1.9 billion. The number reported to Congress was $3.4 billion. The difference - $1.5 billion - was made up of $850 million of contingency and $650 million of (I presume) escalation. That bit at the end - how much of the estimate reflected the massive uncertainty in the design - was never reported until now.

I wrote about this last fall, when I reached some improper conclusions because it was inconceivable to me to have an estimate that was nearly half contingency and escalation. But that is explicitly what the newest document reveals.

Imagine if that information had been conveyed to Congress and the public when the report was issued in 2009: "Our estimates include about 50% slop in the numbers." People would have had the reasonable reaction of, "Then your numbers are nothing more than guesses." That opportunity was foreclosed by the Corps.

2) This new Walla Walla report was issued just seven days (September 23, 2008) before the main cost report to Congress was due. The Walla Walla report included the stunning conclusion that the Corps' internal estimate for Option 1 (including contingency and escalation) ended up over $40 million more than the appropriated amount of $804 million. If this had been reported to Congress, all hell would have broken loose, because that would have meant that none of the options could get built for the money Congress had set aside.

So the Corps did what they do best - they made the Option 1 estimate fit the appropriation by just disappearing $40 million dollars and delayed the report five months, all without telling anyone. The problem is they didn't apply the same "methodology" (i.e. "just chop what you need to in order to make the numbers look good to Congress") to Options 2 and 2a - those estimates remained as high as possible, which was how they wanted it.

In effect, they lied to Congress. The internal Walla Walla estimates for Options 2 and 2a were based on a 90% confidence level (higher confidence translates into higher amounts of contingency), and those are the numbers reported to Congress. The original internal estimate for Option 1 - $845 million - was also created at the 90% confidence level, but it was more than the $804 million they had to spend. So alacazam! $845 million turned into $797 million with a footnote in the report to Congress that reads, "Within the authorized and appropriated programmatic cost of $804,000,000."

This way, the Corps got to keep the Option 1 train going without the uncomfortable question from Congress: "Why did your own cost estimate for your preferred option turn out to be more money than we gave you?"

Birth of a no bid contract

[This post includes links to hundreds of pages of internal Corps emails. See note at end of post.]

I'd like to drill into some of the behind the scenes activity (or inactivity, as we'll see) surrounding the corrosion of the Corps' lakefront hydraulic pumps, in an attempt to explain why a problem identified as affecting 54 pumps back in 2006 has taken over 6 years to be completely addressed. Let's go back to fall, 2009. Here's the facts regarding what was happening with the rusty pumps at that time:

Total hydraulic pumps installed: 40 large (60") and 14 smaller (42") across three sites
Pumps pulled for corrosion repairs to date at that time: 6 large and at least 2 smaller
Pumps remaining to be pulled at that time: 34 large and at most 12 smaller
Basic scope of 2009 repairs to pumps: replace rusting carbon steel parts with carbon steel and some stainless steel parts (wholly inadequate, which would be proven when some of the 2009 repairs would spring more leaks in 2011, forcing 2nd pulls)
Average cost per pump pull: Approximately $100,000 (based on work performed on four large pumps under task order 3, mods 0 and 1, of first Healtheon/Conhagen contract), which was also wholly inadequate because not all carbon steel parts were getting replaced with stainless steel
Information known to Corps regarding corrosion on the pumps: All pumps severely affected since installation in Lake Pontchartrain's salt water in 2006; numerous oil spills since installation in 2006; repair vendor said in October, 2009, "The severe corrosion seen on the piping internal of the pump should be cause for concern in the remaining pumps at this site. Leaks that were imminent on these pumps are sure to be imminent on the remaining."
Contracting methodology: fractured, with different contracts for repairs, crane work, spare parts.

A unified repair contract with enough funding to support the complete corrosion-related overhaul of all hydraulic pumps was needed. Ideally, it should have had provisions for multiple repair subcontractors to speed the work to ensure that pumps meant to drain the city during a major tropical event could do so without springing oil leaks through rusty holes. That's not the way things went.

Instead this is what happened (these are actual internal Corps emails obtained through FOIA, see note at end of post). First, they limited the number of pumps they were looking to repair:

November 12, 2009:
"From: Schneider, Donald C MVN
To: Marshall, Jim L MVN-Contractor
Cc: Accardo, Christopher J MVN; Newman, Raymond C MVN; Constantine, Donald A MVN; Robinson, Carl W MVN; Nguyen, Bac T MVN
Subject: Interim Canal Structures
Date: Thursday, November 12, 2009 1:52:20 PM
Importance: High

Jim,

We're scheduling preventative maintenance on 18 ICS pumps before the start of the 2010 tropical season. The frequency of corrosion-induced failures requires that we take this action.

We currently have approximately $3.2M available. Based upon our average annual expenditure ($1.8M) and the estimated cost of each pump rehab. (approx. $130,000), we require an additional $1M in the project to perform this necessary work.

Do we have the additional funds to complete this PM program?

Donald Schneider
Operations Manager, OD-G"

Already they were limited in their thinking, somehow believing only 18 pumps needed to be overhauled. Next they looked at a way to fund this too-limited work:

December 3, 2009:
"From: Cataldo, Ione M MVN
Sent: Thursday, December 03, 2009 12:45 PM
To: Nguyen, Bac T MVN; Newman, Raymond C MVN, Schneider, Donald C MVN
Subject: Overhaul of 16 Outfall Canal Pumps
Importance: High

Gentlemen,

I have done a little research into the MATOCs that I mentioned in our meeting yesterday that might be an alternative means of processing the subject requirement in the short term. After reviewing it appears that we will not be able to use the MATOCs because they are for construction type work; to do construction actions over $100,000 even under the MATOCs would require a BCOE certification, LD memo, legal sufficiency, and ROE if applicable. Historically the pump contracts seem to all be commercial service direct 8(a) contracts.

Which brings me to this suggestion:

It is possible we may be able to do the IDIQ contract as needed as a direct 8(a) Commercial Service Contract similar to W912P8-08-D-0090, give the 8(a) 15 days to submit a proposal, review for price reasonableness, and if all goes well award sometime in January-Feb 2010.

NOTE: The catch to whether I can go out direct 8(a) or not is 1) Actions $3 mil and under can go direct 8(a); anything over $3 mil would have to be competed, 2) I have to get SBA and OC's blessing for setting this aside as a commercial service under $3 mil (which shouldn't be hard to do since that's how the others have gone out), and 3) the service contract act forms would need to be prepared and submitted for approval in EX.

If you want to try putting the IDIQ in place now using the direct 8(a), I will need to know how much money you are planning on making the contract for ($3 mil; $5 mil; ...), how many years (3 years, 5 years,...), and if money is available to cover a minimum guarantee. I'll also need a brief description of the work to pass through SBA and OC to make sure they are not going to give me any roadblocks.

In reference to the two pots of funds to be used, I just wanted to clarify that the funds would need to be all of the same color money - can't mix monies.

Discuss among yourselves and let me know what you think. DONT WAIT TOO LONG TO GET BACK TO ME PLEASE!

Thanks,
Ione M. Cataldo
Contracting Officer/Team Leader
Project East Branch
Contracting Division
National Contracting Organization
USACE, New Orleans District"

"Direct 8(a)" refers to a no bid award to a SBA-certified small business. That is, Contracting told Operations: here's how to do this without putting the job out to public bid. (Note this is completely legal.) Theoretically that would make things go faster, but it still looks a little hinky to be given the choice to do half the job as a sole source rather than doing the whole job as a publicly bid job.

Operations jumped at the chance to sole source the contract, choosing to do half the job with a preferred vendor. Donald Schneider with the Operations group wrote back the same afternoon:

December 3, 2009:
"From Schneider, Donald C MVN
Sent: Thursday, December 03, 2009 4:27 PM
To: Cataldo, Ione M MVN
Cc: Nguyen, Bac T MVN; Newman, Raymond C MVN; Constantine Donald A MVN; Robinson, Carl W MVN; Brown, Jane L MVN
Subject: RE: Overhaul of 16+ Outfall Canal Pumps
Importance: High

Ione,
Let's move forward w/ the direct 8(a) Commercial Service Contract @ $2.875M for a 2-year duration. Bac will be the P.O.C. to provide you with any information you need.

Thanks for your help.

Don S."

The banality in the award of nearly 3 million taxpayer dollars via an afternoon exchange of emails is remarkable, though I doubt it is atypical with billions of dollars flowing through the New Orleans District since 2005.

Anyhow, as we can see, Operations decided to do a sole source, no-bid contract with their existing contractor - Healtheon/Conhagen - just under the threshold for public bidding, but much further under the funding required for overhauling ALL the pumps, something that same contractor told them was required to avoid imminent failures. The contract value should have been about $150,000 x 40 - or $6 million - assuming only the big pumps would be handled by the new contract, with the smaller pumps being repaired in the Corps' own shop. Once again, public safety lost out to making things easier on the Corps and their vendors.

On December 28, 2009, Direct 8(a) Service Contract solicitation W912P8-10-T-0048 was issued solely to Healtheon, Inc. Terms were a maximum value of $2.875 million and a maximum duration of 3 years. This was a year longer than initially asked for by Operations, meaning they were planning on slowing the already-too-few repairs down even further, giving the pumps even longer to rust out.

On March 8, 2010, Direct 8(a) Service Contract W912P8-10-D-0054 was awarded to Healtheon. The maximum award amount was $2.875 million and the maximum duration was 3 years. This was nearly four months after the initial heads up saying the repairs were necessary.

This strategy would unsurprisingly prove wrong. While 17 of the large pumps would ultimately be overhauled under contract 10-D-0054 from March, 2010 through late 2011 (a ridiculously slow rate) before they ran out of money on the contract, that would still leave 23 others unattended with five years of rust built up on them. Six of those 23 did receive partial repairs under earlier work, but that doesn't count because the repairs didn't replace all the critical carbon steel parts with stainless.

So just six months after the no-bid contract was awarded - in October, 2010 - the Corps ended up going out for public bid anyway on a third repair contract (bid documents here, original FedBizOpps webpage here). That contract, potentially worth $6.75 million and notably also awarded to Healtheon, actually had enough money set aside to do the rest of the pumps, work which is still underway as of this summer. They could have done that in the first place if they hadn't insisted on the stupid inadequately funded no bid second contract.

Now one might be willing to give the Corps a pass on some of this, a "at least they're trying" sentiment. But, again, here's the facts:

1) They knew as soon as they put the pumps in the water back in 2006 they would corrode very, very quickly. Within one month they were already seeing corrosion on the hose connections that were outside the brackish water (via SCPR Flickr):


Here's a detail of one of those rusty hose connections taken at the same time as the picture above (via SCPR Flickr):



What did they think was going on beneath the water's surface to their completely carbon steel equipment? Repairs did not even start until three years later, and a dedicated contract was not in place until March of 2010, nearly four years later.

2) One could also argue, "in going out as a no bid contract, they were trying to accelerate things. It would have taken too long to publicly bid it." From the bid issuance to contract issuance for the second contract was over two months, while the time from bid to contract award for the third, publicly bid repair contract a year later was about four months, so there was some time saved. However, the only reason they could go out as a sole source was because they decided to only fund repairs for fewer than half the pumps! If they had bothered to say to themselves, "the pumps at 17th Street are rusting like crazy, I bet all of them are doing the same thing, so we should look to repair all of them," they would have never gotten a sole source contract because even the most optimistic estimates would have been for more than the $3 million cutoff for public bidding. And then they wouldn't have lost the four months in the fall and winter of 2010-11 spent putting the third contract out for bid. This gang just can't shoot straight.

3) Maybe they were limited on their funding in the fall of 2009? Well, they were able to find nearly $7 million more just a few months later when they went out for bid on the third contract.

The bottom line is this: once again the Corps cheaped out and slowed down on vital life safety equipment meant to keep New Orleans dry. They went for a no bid contract with an existing, preferred vendor which was underfunded and not geared toward getting the job done as fast as possible. For goodness sakes, these pumps are meant to save people's lives, and the Corps couldn't be bothered to spend the money to fix their own stupid mistakes that had been festering for three years. Instead, they just puttered along in their same old insulated way, secure in the knowledge there will never be consequences for their decisions.

Special note regarding internal Corps emails
I placed a FOIA request a while back asking for all emails dealing with pumps from a select group of upper and mid-level Corps personnel. I confined the date ranges to a few days around storm events in late 2009 when the London Avenue canal gates were closed (in November) and when water got very high in the same canal and the gates were not closed (in December). I wanted to see how the Corps and their contractors - especially the controls repair contractor, Prime Controls - dealt with those events in real time. It took forever to get them, but it was worth it, because they included a lot of bonus material.

In asking for anything dealing with the pumps, I also got emails talking about contracts, repairs, and status updates. By pure luck, my net captured the emails I used above. If anyone wishes to see everything I got, all the emails are available here.

2011-12 rusty pump update

This post updated 2/14/17. Scroll down for the new content near the end.

When we last looked in on the rusty pumps at the Lake Pontchartrain lakefront, 15 of the 40 large hydraulic pumps installed across the three outfall canal closure sites - 17th Street, Orleans Avenue, and London Avenue - remained highly vulnerable to corrosion. Their major components such as pipes, housings, hydraulic motors and coolers remain immersed or very close to the brackish waters of the lake 24 hours a day. Those 15 pumps were vulnerable because all those components were made of carbon steel, which rusts like crazy in a salt environment.

Pumps continue to spring oil leaks during 2011 hurricane season

One of those 15 pumps - E3 at the Orleans Avenue canal - apparently sprung an oil leak on August 10, 2011, the third pump to do so during the 2011 hurricane season. I noted this at the time. The Corps' standby oil cleanup contractor - Quaternary Resources Investigations, or QRI - was summoned that afternoon according to the $6093.29 task order issued to QRI by the Corps. Six days later, that task order was increased in value by $3634.09 because the estimate of the spill size jumped from 25-50 gallons to 50-100 gallons. The final cost of of the spill cleanup was determined to be $9711.10, after a second smaller cost adjustment on September 21, 2011.

Repair of E3 was arranged over the next few days, with the official rebuilding task order (#10 on the second Healtheon contract) issued to Healtheon and their subcontractor Conhagen on August 18, 2011 in the amount of $170,067.56. Because of the Corps' tendency to file paperwork after the fact, it is possible the original August 10th leak actually occurred when the pump was being pulled out, since paperwork attached to Conhangen's repair report for Orleans E3 includes mention of a leak test of new oil coolers on August 18th, which could presumably only take place near the end of repairs. Further evidence of contractual hijinks comes from the fact the repair report mentions work starting on July 15, 2011, over a month before the task order was issued, and the same day the previously repaired pump (W9 at 17th Street) was reinstalled. Whatever the exact sequence of events, the result was yet another rusty pump dumping oil in the lake.

Looking at the repair report, we see the usual assortment of rusty innards, including the Rineer hydraulic motor at the heart of the pump...

marine growth-encrusted oil coolers...

and chipped inlet vanes...

The pump was put back in service on October 3, 2011. The somewhat long delay from the pull out in August (or possibly July?) until reinstallation in October was not explained.

Late 2011 (kinda) off-season work

Next up was what appears to be the planned repair of four pumps: W2 and W3 at 17th Street and W1 and W2 at Orleans Avenue in November, during hurricane season (the Corps has a history of ignoring the last month of hurricane season). We got a spill report for the 17th Street pumps on November 17, 2011, 26 hours after a spill the Corps claimed occurred on November 16th:

"Qty released: 60 gallons

DURING REPLACEMENT OF A PUMP HYDRAULIC OIL WAS NOTICED LEAKING WITHIN A BOOMED CONTAINMENT AREA. THE HYDRAULIC OIL WAS COMING FROM THE OLD PUMP THAT WAS BEING REPLACED.

CONTRACTOR WAS ON-SITE AND CONDUCTED CLEANUP"

The report to the state of Louisiana on this spill provides far more background, including evidence of lying in reporting. A chronology for the incident - part of internal correspondence between QRI and the Corps and attached to the state spill report - is titled "2011-11-16 - 17th Street Canal (Pump #2W) Spill Summary Report." However, the first line notes the spill was discovered on November 15th as part of QRI's weekly rounds, not as part of pump rebuild work. The date could be a typo, intended to be the 16th. Or it could actually be true. But the circumstances - discovery of the spill during the weekly maintenance - are totally different than what was described to the Coast Guard.

The state spill report - generated from information given the state by the Corps - claims the spill was discovered at 11:15 AM on the 16th. The chronology claims that cleanup started that afternoon after the Corps gave verbal approval at 1:55 PM. The chronology also places notification of the Coast Guard by the Corps in the two hours and 40 minutes between those two events:

"During QRI's weekly maintenance on November 15, 2011, the team spotted a hydraulic spill associated with pump #2W at the 17th Street pump station.
[...]
The crew alerted USACE representative Kim Branch of the spill.

Mr. Branch reported the spill to the appropriate authorities as per the SPCC plan. The DEQ incident number associated with this spill is 11-07-194.

QRI received a verbal NTP [Notice to Proceed] from Ione M. Cataldo, contracting officer/team leader at 1:55 PM on November 16, 2011"

Reporting of a spill per Corps' 17th Street Spill Prevention Contol and Containment - or "SPCC" - plan calls for notification of the Coast Guard's National Response Center by the facility owner - the Corps in this case - immediately upon discovery:

"All spills, as defined in 40 CFR, part 110.3 (sheen rule), will be reported to the National Response Center. LADEQ and EPA notification procedures and written report requirements for a reportable discharge are included in Appendix F.
[From Appendix F]
Any person in charge of a vessel or of an onshore or offshore facility shall, as soon as he or she has knowledge of any discharge of oil from such vessel or facility in violation of section 311(b)(3) of the Act, immediately notify the National Response Center (NRC)"

And from the Coast Guard's National Response Center spill report, here's the truth about when the report was actually received:

"Report taken at 13:36 on 17-NOV-11"

That's 26 hours after the Corps claimed the spill was discovered, and perhaps as much as 50 hours after it was actually discovered. By that point, according to the documents attached to the state report, the spill was already cleaned up. It's these little lies - along with the huge ones - that make the Corps so untrustworthy.

The chronology also notes the final quantity spilled was 77 gallons. That's over a quarter of the 300 gallon capacity of the hydraulic oil day tank on the engine skid for that pump. It is unlikely the pump would have run very well with that much oil missing.

A task order for QRI's cleanup work was written on November 16, 2011 for $9727.38. The final spill cost was $7129.03 after an April, 2012 modification.

Further confusion was introduced a week later, when on November 23rd, Healtheon got a $613,645.84 task order (#3 on the third Healtheon contract) for the removal, rebuilding, and reinstallation of the four pumps I listed above, which included 17th Street pump W2. It's very difficult to tell if this work was already underway on the 16th - which the spill reports claim - or if it didn't get going until the 23rd. I'm inclined to say it wasn't until the 23rd, since that's what the Conhagen repair report for that pump (and the report for pump W3) claims as the start date:

Admittedly, the repair reports for the other two pumps on this task order also claim the November 23rd date as the "job start" date, even though - as we'll see in a bit - they likely weren't pulled out until December However, if the work truly didn't start until November 23rd, that would mean the spill report to the Coast Guard was not only late, but also contained lies in the narrative.

Now to the contents of those 17th Street pump repair reports (W2 report, W3 report). They contain the usual pictures of rusty and marine growth-encrusted stuff. But what really attracted my attention was the condition of the splined drive shafts that come out of the bottom of the Rineer motors:

Rusty, corroding yuckiness! These shafts are enclosed in housings that are supposed to be sealed, but clearly were not:

It's not hard to imagine a pump refusing to work because of the poor condition of these shafts. In addition, if the shafts are this bad, all the internals on their motors would also be shot. And indeed, that is exactly what has happened. Repeatedly. For years.

Sidebar Rineer motors and salt water

On May 18, 2011, the Corps issued a $13,881.02 task order (#12 on the first Healtheon contract) which sent 8 Rineer motors for the main 60" pumps out for repair. At the time the scope only called for replacement of seals, shaft ball bearings, and fasteners (i.e. bolts) on the housings and seal plates. That work was supposed to take 6 weeks. Of course, if one is replacing seals, then that means the old seals are bad and stuff got into the motors.

So a $42,019.32 modification to the original Rineer repair order was sent out August 31, 2011, after the motors were taken apart and unsurprisingly found to have their internal moving parts utterly ruined. The delay between the original rebuild deadline (around July 1) and the task order finally getting issued (August 31) would correspond to the standard Corps dickering over whether they should spend the money to repair or replace the motors, as well as the usual delays in getting anyone to make an actual decision.

The modification to the original repairs was predictably wide ranging, given the fourfold expansion in costs. It reads:

"Provide labor and materials to replace the following additional components on the 8 Rineer hydraulic motors Model M125-A4-1H250-31-B1-0BB-213 for 60” hydraulic pumps, which were disassembled under the original Task Order. These components were found damaged upon disassembly; and were not included in the original scope of work.
The contractor will be given 24 weeks to complete the repairs to the motors upon award of this modification due to parts delivery lead times.

All Motors; Replace all motor housing plugs with new solid hex head plugs.

Motor #1, work order #9184;
Qty 1 complete vane cartridge kit (16 week del)

Motor #2, work order #9185;
Qty 2 complete vane cartridge kits (16 week del)

Motor #3, work order #9186;
Qty 1 complete vane cartridge kit (16 week del)

Motor #4, work order #9187;
Qty 1 Rear Housing (20 week del)

Motor #6, work order #9189;
Qty 1 complete vane cartridge kit (16 week del)

Motor #8, work order #9191;
Qty1 complete vane cartridge kit (16 week del)"

Replacement of the complete vane cartridge kit in these motors is the equivalent of having every cylinder in your car's engine rebuilt. Of the original 8 motors sent out in May, 2011, 5 (62.5%) were found to require this complete vane cartridge replacement. That is a shockingly high rate of failure, one that should make New Orleanians blanch.

But this major rebuild work on the Rineers is of a piece with what had been happening since mid-2010.

In August, 2010 a solicitation for 12 new Rineer motors was issued by the Corps. The solicitation remains up at the FedBizOpps webpage here. 8 of the Rineers were to be for the 60" pumps, with the other four going in the smaller 42" pumps installed on the structure at 17th Street. Rineer motors - especially special order ones like these - have a very long lead time. The winner of the $104,760 contract (W912P8-10-C-0116) was Hydraquip of Houston, TX, a Rineer rep with a local office near Armstrong International Airport in Kenner, LA. They had about 3 months to deliver the new motors, meaning the Corps would have had the new ones in hand around December, 2010.

And going back further, let's not forget that in July, 2010 the Corps issued a task order modification to rebuild 12 other Rineer motors, all for 60" pumps.

Finally, more recently the Corps sent an additional 6 Rineers for the 60" pumps out to Hydradyne, LLC for "overhauling" in a March 8, 2012 task order on the first Healtheon contract. The task order is not specific as to the scope of work; however, we can be assured it is the same as the more extensive work called for in August, 2011, because of this:

"Estimated completion date is 20 weeks from the award date (due to part availability from the manufacturer)"

Only replacement of vane cartridges could take 20 weeks based on parts availability.

So all this activity means that in a little over a year and a half, the Corps rebuilt 26 Rineer motors for the 40 big hydraulic pumps, and purchased 8 more. That's close to the number of pump pulls that occurred over that same span (41 by my count, including three pumps coming out twice).

However, the numbers really don't matter. The bottom line is that these Rineer motors are obviously ill suited to salt water exposure. Their seals are likely designed for fresh water. Leaving them where they are is simply asking for big trouble. Over the last three years I've published a slew of pictures showing Rineers rusting to bits. They've been getting salt water into them through the rusty pipes, the seals, and every other leak path. The Corps has tried replacing them and repairing them. Nothing works. They all get eaten up. These motors are the heart of the pumping system at all three Lakefront closure structures, and they are unreliable and rusting to this day.

The rest of the late 2011 work

The other two pumps on the November 23, 2011 task order - Orleans Avenue pumps W1 and W2 - didn't get pulled out until early December. This was the first time the Corps pulled out more than one Orleans Avenue pump at a time since the mass rebuilds in 2006-07, indicating they would finally be working on the pumps at that location in a coordinated fashion.

On December 8, 2011, a spill report was filed with the National Response Center for an incident at 2 PM that day:

"Qty Released: 10 GALLONS

CALLER STATED THAT THERE WAS A RELEASE OF 10 GALLONS OF HYDRAULIC OIL FROM A HYDRAULIC PUMP WHICH IS PART OF THE CANAL STRUCTURE, THE CAUSE WAS DUE TO OVERHAULING OF THE PUMP AND THE CAP ON THE PUMP WAS LEAKING.

CLEAN UP UNDERWAY, CONTRACTOR HAS BEEN HIRED"

The companion cleanup task order was issued to QRI that same day. The text of the $6,093.29 task order confirms the spill was connected to the pull-out of Orleans Avenue pumps W1 and W2:

"Verbal authorization for this work was given by Contracting Officer Cynthia A. Nicholas on 8 December 2011 at 3:18 p.m in a Not to Exceed amount of $6,093.29 for a 10 - 50 Gallon Spill at Orleans Avenue Canal Pump 1W and 2W."

A later modification dropped the actual cleanup price to $5,647.60.

As with the 17th Street pumps on this same task order, individual repair reports were compiled for each pump (Orleans W1 repair report, Orleans W2 repair report). As usual, there was rust, marine growth, and other scary stuff:

The 17th Street pumps went back in the water January 24, 2012. The Orleans Avenue pair followed on January 27, 2012.

London Avenue pump rebuild work is finished off

While those four were out, a remarkable decision to repair a fifth pump was made. That pump - W2 at London Avenue - was the last of the 12 London pumps not to be refitted over the past three years. A $142,459.56 task order (#11 on the second Healtheon contract) was issued December 20, 2011. There are no oil spill reports or task orders to QRI associated with this pump repair, which is odd. Single pump rebuilds like this usually come about because the pump sprung an oil leak. This could be yet another case of an unreported spill, or a more unlikely case of the Corps acting proactively to repair more than four pumps at a time.

The Conhagen repair report for London Avenue pump W2 is very spare on photos, almost as if they know their reports would be publicly disseminated. There is one photo of interest though, and it's an "after" shot:

This is probably a rebuilt Rineer motor. The color appears different than other motors which have come out, and part of the rebuilding task orders for Rineers has mentioned painting them. So that's one good thing, I suppose. But it will rust out again, long before the new pump stations come online in 2016, or whenever.

According to the repair report, London Avenue pump W2 went back in the water March 12, 2012.

First round of 2012 - focus on Orleans continues

That brings us to the next round - pumps W3 and W5 at the Orleans Avenue site. Healtheon received a $304,322.08 task order on January 30, 2012, the day after work wrapped up on the last set of 4 pumps. It's unusual for the Corps to only pull out 2 pumps rather than 4 in the off season, but perhaps London Avenue pump W2 was still in the shop at the time.

A spill report was filed with the National Response Center on February 22, 2012, for a 20 gallon spill supposedly discovered at the Orleans Avenue structure on February 21, 2012:

"Caller is reporting a discharge of hydraulic fluid from a leaking hydraulic pump. Spill contractor is on site [to] clean up material; material contained by boom in the drainage canal."

By this time, QRI had been awarded a shiny new spill cleanup contract, it's third. The four year agreement has a potential value of $581,376.24 (!) Task order #3 on it, worth $6360.39, was issued February 23, 2012, two days after the supposed discovery of the spill. The description in the task order seems to connect it to the Orleans Avenue pumps in the January repair order:

"This task order is [to] cover the clean up of hydraulic oil spill estimated to be in the 10 to 20 gallon range. The event happened from pump #3 at the Orleans Ave Canal."

While the task order doesn't specify east or west, just "pump #3," I'm going to guess it was referring to pump W3, meaning it and likely W5 came out on or around February 21, 2012. They probably went back in sometime in late March or early April.

Unfortunately, that's where my latest batch of documents ends, since I placed my latest FOIA request on March 9, 2012. However, there are ways to track what's going on even without FOIA requests.

The federal government's FPDS database posts all federal contract transactions. There's an immensely annoying 90 day hold on Pentagon transactions posting to the website, ostensibly for security purposes, but mostly to keep prying eyes away. However, since the transactions from February, March, and the beginning of April are now available, we know there was another task order on the third Healtheon pump repair contract issued. Task order #5 for $308,594.52 was issued March 27, 2012 for the repair of Orleans Avenue pumps E1 and E2, with a scheduled completion date of May 28, 2012. This would seem to indicate it was the last batch of pump repair work the Corps planned for the offseason before the 2012 hurricane season, which began on June 1, 2012.

The National Response Center online database is updated daily, so it's an excellent contemporaneous indicator of activity. Since the late February Orleans Avenue canal spill there have been three additional spills at the outfall canals:

May 1, 2012, Orleans Avenue structure
"Caller stated due to a faulty o-ring in a hydraulic line connector there was a spill of hydraulic fluid. Caller stated the sheen is burning off and absorbents will be used later as a boat will be placed in the water."
Estimated quantity spilled: 5 gallons

May 11, 2012, 17th Street structure
"Caller reported a spill of materials while repairing an engine on a hydraulic pump. Booms applied, cleanup underway."
Estimated quantity spilled: 14 gallons

July 2, 2012, Orleans Avenue structure
"Caller is reporting a discharge of hydraulic oil from a hydraulic pump due to unknown causes. Caller stated the release is estimated between 15 - 50 gallons. Double boom was in place, contractor is on scene cleaning up"
Estimate quantity spilled: 50 gallons

Knowing what we know about these spill reports - the data comes from the Corps, and it may or may not be true - I try not to put too much stock in the descriptions. The reports are really best for their timeliness. We'll find out later whether they're truly associated with pump repairs, scheduled or otherwise.

UPDATE 2/14/17

As noted above, at the beginning of 2012 the Corps was focused on completely refitting pumps at the Orleans Avenue site. Pumps W3 and W5 came out immediately after the return of pumps W1 and W2 in late January. W4 had been refit during the 2010 hurricane season. The task order for Orleans W3 and W5 is here, and the repair report is here.

The report is unremarkable, with only "after" photos. The pumps were accepted by the Corps March 29, 2012.

Next up were Orleans pumps E1 and E2. They came out the same week W3 and W5 went back in, the last week of March, 2012. The task order is here, and the repair report is here. The report has pictures of rusted out carbon steel piping cut off of the pumps (note: the labels of "1W" and "2W" are Conhagen's typos):

It's striking to see this piping discarded like so much junk.

These two pumps were accepted by the Corps June 19, 2012. While they were in the shop, two other pumps sprung oil leaks - 17th Street W3 and London Avenue W5. Both pumps had been in the water for less than a year following complete refits. Details on those failures can be found in an update at this subsequent post.

The two remaining Orleans Avenue pumps that were left unrefurbished at the beginning of the 2012 hurricane season - E4 and E5 - unsurprisingly leaked oil that summer and had to come out during the height of hurricane season. They were out of commission when Isaac came ashore in August, and their absence likely led to numerous pumping cutbacks while the storm moved through. Details on repairs to those two pumps  - and links to how their going missing hurt city drainage during Isaac - can also be found at the same subsequent post.

So this is what we can say about the pumps at each site, given the documents and other information I've sussed out:

These "reliability" estimates come with huge caveats. We know the critical Rineer motors are continuing to rust away, no matter how much they get replaced or repaired. We also know of numerous other problems back on the pumps' remotely placed engine skids unrelated to corrosion. In essence, all these graphics show is that some carbon steel piping has been switched out for stainless steel, and there's been other corrosion repairs and repainting. The mechanical reliability of the hydraulically driven pumps, especially due to the ongoing corrosion of the Rineer motors and the resulting saltwater ingress to the hydraulic oil loops, is highly questionable should they be called upon during a major tropical event.

The outfall canal remediation follies

Since my last three posts concentrated so closely on debris questions raised during the 2011 17th Street canal, London Avenue canal, and Orleans Avenue canal remediation projects, I didn't have room to mention other stories that emerged from the Quality Control (QC, compiled by the prime contractor) and Quality Assurance (QA, compiled by the Corps) reports on those projects. The contractor is responsible for actually putting out a quality product (QC), while the Corps' QA responsibilities are simply to make sure the specifications and drawings are followed. As such, the Corps' oversight abilities are not as strong as the general public might believe. However, they are not innocent bystanders either.

Which brings us to the first "other story" that comes out of the reports...

Copying and pasting alive and well at the Corps

This comes from the 17th Street project (prime contractor: Bailey-CKY Joint Venture). I read the QC and QA reports alongside each other to see if one report would mention something that another would not. While doing so, I started to notice remarkable similarities between the two groups of reports.

The February 26, 2011 QC report from the contractor reads:

"Started clearing and grubbing on the East Bank of canal between stations 567+00 and 581+00. Material is being hauled to the north end of the laydown area and stockpiled for subsequent hauling to offsite location. A small windrow of material was left along the silt fencing for additional jet grounting [sic] spoil protection. This material will be removed prior to levee re-construction."

And here's an excerpt from the Corps' February 26, 2011 report:

"Bailey-CKY crew started clearing and grubbing on the esat [sic] bank of the canal between stations 567+00 and 581+00. Material is being hauled to the north end of the laydown area and stockpiled for subsequent hauling to offsite location. A small windrow of material was left along the silt fencing for add jet grout spoil protection. This material will be removed prior to levee reconstruction."

When you see Corps personnel - that are supposed to be overseeing contractors - copying and pasting the contractor's reports into their own reports, it gives you a chilly feeling down your spine. And this was not an isolated incident. From the February 23, 2011 contractor QC report:

"Work performed today (loc. and descrip. of work performed by Contractor) - Layne Geo continued set up of equipment on site and assembly of the cement pumping systems. - Placed riprap on west side of canal between stations 558+70 and 590+15 (approx. 145 linear feet). Total linear footage of riprap placed to date is 400 linear feet."

And from the February 23, 2011 Corps QA report:

"Layne Geo continued to assemble B300 drill rig and assemble the concrete [sic] pumping systems. Crew place [sic] riprap on the westside of the canal between stations 558+70 and 590+15 (approx. 145 linear feet). Total Linear feet place [sic] is 400 linear feet."

There's lots more examples of this. There were also other QA curiosities.

The July 21, 2011 QA report reads:

"EASTBANK:
Contractor dressing/grading slopes along eastbank to final grades. Removing crane mats

WESTBANK:
Contractor grading slopes along eastbank to final grade. JC Creek on site for 1st day to start turf establishment. Removed large stones greater than 3" from levee. JC Creek placed DAP and Sulphur from stations 553+45 to 586+00.
J&R Concrete forming bike path.
Layne finished remedial coring at station 581+56B/C and 592+91.25B/C

OTHER
SSE performed survey work for grading.
Street sweeper maintaining streets at east and west banks.
Water truck in operation watering sreets [sic] for dust control around worksite"

When I read it, I thought it sounded familiar, so I paged back and found the following for the July 15, 2011 QA update:

"EASTBANK:
Contractor dressing/grading slopes along eastbank to final grades. Removing crane mats

WESTBANK:
Contractor grading slopes along eastbank to final grade. JC Creek on site for 1st day to start turf establishment. Removed large stones greater than 3" from levee. JC Creek placed DAP and Sulphur from stations 553+45 to 586+00.
J&R Concrete forming bike path.
Layne finished remedial coring at station 581+56B/C and 592+91.25B/C

OTHER
SSE performed survey work for grading.
Street sweeper maintaining streets at east and west banks.
Water truck in operation watering sreets [sic] for dust control around worksite"

Amazing! The exact same work performed the exact same way on two separate days!
Of course, that's not really what happened. Here's part of the contractor QC report from July 15, 2011:

"Work Performed Today (loc. and descrp. of work performed by Contractor)
JOB SITE:
- East side of canal embankment - No embankment work today due to rain. Unloaded 2 truk load[s] of round bales of hay.
- West side of canal embankment - No work today due to rain."

Interesting. They were actually rained out on the 15th. And here's the QC report from the 21st:

"Work Performed Today (loc. and descrp. of work performed by Contractor)

JOB SITE:
- East side of the canal embankment- Used Trimble GPS to layout levee for final grading. Loading crane mats from Bellaire laydown yard.
- West side of the canal embankment- Finalizing grading. JC Cheek on site to begin turf establishment. Bailey CKY JV had a crew of four employees to pick any debris 3" or larger as JC Cheek prepared the levee for turf establishment. Used street sweeper and shovels to clean Orpheum Drive."

Which matches the QA report from the 21st. So the content of the Corps QA report on the 21st was copied and pasted into the Corps QA report for the 15th, bringing into question when exactly the Corps QA inspectors were compiling their reports. It appears in this case, the July 15th QA report was compiled almost a week later.

All this copying and pasting makes one very suspicious of the oversight being performed by the Corps QA representatives.

Public Comment, Schmublic Comment

Last March, I was attempting to track the remediation projects in real time. One of the issues I came across was the Corps' use of publicly owned, mostly residential lots for access to the London Avenue canal.

To gain access to the canal with cranes and to stage building materials, in addition to areas they had previously announced the Corps proposed using 13 lots owned by the Louisiana Land Trust (aka "LLT," the repository for properties sold to the state of Louisiana through the Road Home program), as well as a 14th site owned by the City of New Orleans' Department of Parks and Parkways. Here's the location of the lots (the original staging locations are in yellow and the 14 new ones are in blue):




The Corps had originally wanted to do this staging and access from the bridges crossing the canal at Filmore and Mirabeau Avenues, but was not able to coordinate that work with the City of New Orleans, who was performing resurfacing work on and around the bridges at the same time.

To meet federal environmental requirements, the Corps - through the expedited Individual Environmental Reporting (IER) process set up post-Federal Flood - announced their intention to use the 14 lots in a supplemental to the IER which had authorized the canal remediation work. That supplemental - IERS 27.a - was publicly issued March 14, 2011 and was open for public comment for 30 days, until April 14, 2011. Until that public comment period closed, the extra lots were not to be used for staging.

Thanks to yeoman work by The Lens' Steve Beatty, I managed to catch the Corps using the 14 lots before the public comment period even opened:

Here's pictures of construction equipment on the empty lot at 5332 Warrington from February and March, 2011:





I posted about this on March 17, 2011, the day after I emailed the Corps asking what was going on (that email string is part of the final IER supplemental, found here). I received a response on March 22nd - included in the March 17th post - which seemed to admit their boo boo. At the time I noted they probably thought it was better to seek forgiveness rather than permission.

Now, with the contractor QC reports for the London Avenue project (prime contractor: Integrated Pro Services, or IPS), we know what was happening behind the scenes. The topic of the 14 lots was extensively discussed from the very beginning of the project. In fact, the very first QC report with text in it, from January 5, 2011, discusses the 13 LLT lots:

"Talked to Carmen Williams (USACE Project Engineer) about our Access Plan (submitted on 12-29-10) pertaining to the 'required site access' IPS required from LA Land Trust. I expressed to her would like to mob[ilize] our office trailers on Monday (1-10-11). Carmen stated since the 'required site access' is not environmental[ly] cleared we cannot proceed with mobilization on the 'required site access.' She has stated the Access Plan and environmental clearances have been routed to the properate [sic] offices to be reviewed and IPS should know something on these issues by 1-14-11.
[...]
Carmen stated she would talk to Greg Schultz (Contracting Officer) about our situation."

Another meeting took place two days later, on January 7, 2011:

"Met with Laura Lee and Lee (USACE Environmental) on site concerning the 'required site accesses.' We rode each individual site and explained that each site would only [be] temporary staging for concrete pump trucks, materials, and misc. items. We stated the site would be matted, so that the ground would not be disturbed during construction operations. They took pictures of each site and document[ed] the site conditions. They stated they would go back and talk to their group and decision will be made next week."

A third meeting took place the following Monday, January 10, 2011:

"Met with Carmen, Project Manager, and [USACE] Environmental to show and discuss the 'required access sites.' This was the third time meeting about these areas. The Environmental group has taken all the information back to their department to make a determination on these sites. They stated that a IER may have to take place to obtain approval for these site[s], this could take up to 60 days."

We know now an IER was required. The contractor didn't report on the extra sites again until February 1, 2011:

"Had weekly progress meeting. [Corps engineer] Chuck [Brannon] stated that the 'access properties' have been elevated to the Conlenal [sic, probably meant "Colonel," referring to the Colonel in charge of the Hurricane Protection Office] and should have an answer on the status by the end of the week. He also stated the supplemental IER package is complete and ready to be submitted. The submittal process should take about 45-50 days once submitted."

So all parties were aware at the beginning of February, 2011 that the 14 properties were on hold until the IER process had played out. That included allowing the public to be informed and comment on what was happening in their neighborhood and next door to some of their houses. Then came this mention on February 15, 2011:

"Greg Schultz (Contracting Officer) gave a verbal that the access properties are approved for access."

And that is how the public gets screwed. The Corps says, "forget all that public comment BS - just do what you want." Except in this case, they got caught.

The very same day I received my response from the Corps on the 14 lots admitting their error - March 22, 2011 - here's what was in the QC report:

"Greg Schultz (CO) stated that the access properties acquired [sic - should be "required"] cannot be used until the IER has been signed. The earliest these properties can be utilized is April 15, 2011 [the day after the public comment period for the IER closed]. A letter dated February 15, 2011, sent from Greg Schultz (CO) states that the "access properties" are approved to be utilized."

In other words, "We, the contractor, are throwing the Corps under the bus. They told us - in print - we could use the lots, and they were wrong."

I cannot believe this is an isolated case. Too many times public comment on Corps projects post-Federal flood has been just so much stagecraft because all the decisions have already been made. The most obvious case was the public's overwhelming desire for Option 2 (concrete-lined, ground level canals with a single pump station)along the outfall canals and the Corps' decision to use the cheaper Option 1 (remediated walls and multiple pump stations on each canal). I must say though it is remarkable seeing it in print.

17th Street wall movements - and leak!

I'm not exactly sure what to make of this one. It appears to fit a pattern of job requirements changing on Corps jobs once construction gets underway, from big stuff to small stuff, so I thought I would report on it.

The 17th Street remediation job consisted mostly of placement of deep soil mixing columns in groups of six along many sections of both sides of the canal. Here's a view from above of that arrangement from the project drawings:

In general, each grouping, called a "panel," was placed at 14.5' centers along the length of the wall (this was changed from the original 10' centers shown above based upon test columns that were placed at the beginning of the project). Note that while the drawings show nine columns for each panel, there were actually only six (presumably bigger diameter) columns in each panel. From the March 3, 2011 QC report:

"An element is the [sic] considered 3 columns and a panel is [a] complete 16.7-foot series of 2 elements."

The columns went down to somewhere between 20 and 40 feet below sea level, depending upon location along the canal. When you so radically alter the foundation of levees with sheet piling wall partially stuck in them, you can expect one thing: movement of the walls.

So the Corps had the contractor monitor the walls for movement, using Trimble GPS monitors and straightedges, something called a digital "smart" level and who knows what else. The point is they were watching the walls near where they were placing the columns. This started on February 28, 2011.

At first they were monitoring the walls the entire day in real time. The first numerical results from the wall monitoring appear in the March 5, 2011 QC report:

"After the first element installation it appeared that the wall moved slightly at the expansion joint near the top. Monitored the other three element installations by placing a tape measure across the expansion joints and monitoring the movement real time. The walls were observed to move between 5/16" and 1/2". Movement was a separation at the top of the wall at the expansion joint and slight movement toward the protected [i.e. land] side. We informed the onsite [Quality Assurance Representative] as soon as we confirmed wall movement."

Again, while this sounds bad, I'm not sure it shouldn't be expected.

Two days later, we got some more numbers. From the March 7, 2011 QC report:

"We were instructed to continue with jet grouting after we informed the USACE that we recorded wall movement up to 3/4" at the top of the wall expansion joint during the installation of one element."

So we got up to 3/4" two days later, and the Corps determined that was acceptable. Okay.

For the next three days, the QC reports said the following:

"Continued 'real time' monitoring of wall movement at expansion joints at the top of the wall as well as by use of a digital level and GPS system. Wall movement did not exceed 1/2" either vertically or horizontally during today's activities."

On March 11, 2011, "real time" measurement was discontinued. After that, measurements were only taken before and after column placement operations. The contractor did station a person to watch the wall during work to look for "significant" movement.

After March 11, the QC reports started reporting wall movement with nearly the same verbiage every time. A typical passage from the March 12, 2011 QC report:

"Continued to monitor movement on I-Wall. Measurements were recorded before and after drilling operations. We still had a person visually monitor the wall for any 'significant' movement. The was no wall movement in excess of 1/2" vertically or horizontally measured."

Note that it doesn't say there was zero movement, just that there wasn't any movement more than 1/2". This 1/2" criterion continued to be used through April 7. On April 8th, there was a significant change in the boilerplate (the verbiage had changed slightly by then):

"Continued to monitor movement on I-Wall. Two staff members of Bailey-CKY JV were on each side of the canal throughout grouting process. No wall movement in excess of 1.0" was physically measured during today's operations. Each monolithic panel where jetting has occurred is measured at the end of the day using static GPS and recorded."

So the limit/criterion/rule of thumb/whatever was doubled. There's no reason given for this in the QC or QA reports around that time. The 1.0" number remains in the QC reports all the way to the end of the column placement work on June 20, 2011.

Why does any of this matter, except as a matter of curiosity? Because when you move a segmented wall with water against it, sometimes stuff comes through the gaps. From the April 12, 2011 QA report:

"East bank coring:

I visually observed [deep soil mixing subcontractor] Layne Geo coring at EG58022 Element A. [W]hen I looked at the base of the I-wall a couple of away for [sic] where there [sic] coring saw some water leaking for [sic] the base of the I-wall. [T]hen started video recording on my camera the leak. I have save[d] the video on my computer. The water only leaked when they was [sic] coring, then it slow[ed] down to no more leaking after the coring stopped."

By way of explanation, "cores" are small diameter borings taken from completed columns for later testing to ensure the columns' integrity and conformance with specifications.

This leak from the canal - which is not mentioned in the April 12th QC report - combined with the apparent doubling in permitted permanent deformation of the walls four days earlier, raises some very interesting questions. The most obvious one is whether sections of the canal I-walls were permanently deformed by the deep soil mixing process? While walls may not have moved permanently more than 1.0", they could have moved 0.95" (horizontally or vertically) and not gotten a mention.

Another question is what the acceptable number is for permanent deformation of the walls during operations like this? The individual sections of wall (the stuff you can see above ground) have flexible waterstops between them. It was one of those waterstops that failed on the London Avenue canal just a month before the 17th Street canal leak reported above. Can those waterstops take up to 1.0" of deformation? That's unknown.

So the wall movement measurements during the 17th Street remediation project leave us with more questions than answers. What prompted the end of "real time" operations? Why was the limit for permanent deformation raised from 1/2" to 1" mid-project? Were there other leaks during the project that went unreported? Are the walls permanently deformed, and is that a concern?

In the next post post after next I'll wrap all this outfall canal stuff up and detail what's driving the future remediation work (not the permanent pumps, but more work on the levees and walls) still to come on all three canals.

Debris Part 14

Part 1
Part 2
Part 3
Part 4
Part 5
Part 6
Part 7
Part 8
Part 9
Part 10
Part 11
Part 12
Part 13

Let's move along to the Orleans Avenue canal. As before, a main source of information is the contractor's Quality Control (QC) and the Corps' Quality Assurance (QA) reports. There were two projects at the Orleans Avenue canal, OFC-4A (which is what most of the work was conducted under; prime contractor IPS, same as on London Avenue) and OFC-06 (which consisted only of installation of a stability berm along one reach on the east bank of the canal between Harrison and Filmore; prime contractor Progressive, with Cycle Construction performing all actual work):

As with other projects, the Corps didn't remediate everything that had a Safe Water Elevation below their target of 8 feet (we don't know why):

Here's the QC and QA reports for both projects:

OFC-4A QC reports March 1, 2011 - May 31, 2011
OFC-4A QC reports May 31, 2011 - August 21, 2011
OFC-4A QA reports March 1, 2011 - May 31, 2011
OFC-4A QA reports May 31, 2011 - August 11, 2011

OFC-06 QC reports January 6, 2011 - March 30, 2011
OFC-06 QC reports May 23, 2011 - July 28, 2011
OFC-06 QA reports January 6, 2011 - March 30, 2011
OFC-06 QA reports May 23, 2011 - July 28, 2011

All the quotes in this post, unless otherwise noted, are taken from the QC reports.

There don't seem to be any debris problems with much smaller project OFC-06. However, the very existence of OFC-06 provides a fascinating story I'll get into later.

At OFC-4A, there were substantial debris problems, both with the existing levee and with the new work. We'll start near the end of the project, in June, 2011.

Corps leaves jet grout spoils in levee

Part of the project involved installation of sheet piling on the west bank of the canal under the I-610 bridge. The contractor ran into problems on June 1, 2011 when they happened on existing sheet piling that could not be removed. The following day there was a meeting of the minds:

"USACE RE [Resident Engineer], PE [Project Engineer] and QC [contractor quality control rep] were on site in order to evaluate the situation of removing the existing sheet piles under the I-610 bridge. USACE PE instructed that due to the inability to remove the existing piles we will tie into them. USACE PE is looking into whether or not jet grouting will be required to ensure stabilization"

Here's a picture of the area of concern, taken from a June 8, 2011 update given by the Corps to the Southeast Louisiana Flood Protection Authority - East:

They kept trying to pull out the piles for four days after that, but they eventually stopped. Another meeting was held on site on June 9th:

"Meeting held on site with Hayward Baker PM, HB Super, HPO PM, USACE RE and QA, IPS QCM and Super [i.e. Hayward Baker (the jet grouting contractor), the Corps, and IPS (the prime contractor)]. Meeting was held at the site where the jet grouting will take place. Discussed the column location and the columns' width and how many would be installed. USACE RE state that the change order request was being issued to IPS in the next 24 hours and wanted Hayward Baker and IPS to start coordinating the work to be performed. He state that the submittals needed first would be equipment and the work plan. Hayward Baker will submit same mix design as used on previous Corps project."

So they would be using jet grouting to stabilize the tie-in between the new sheet piling and the old. Jet grouting is a process where a grouty slurry is injected into the soil and allowed to harden, forming a substance called "soilcrete." Sometimes the injected substance is just grout, sometimes there's air added, and sometimes there's air and water added. It's been used all over the greater New Orleans area since the 2005 levee failures. More detail about the process is available at Hayward Baker's website.

Preparation work for the jet grouting began on June 14th:

"Sealevel began installing backfill for sheet pile trench from STA 12+10 through STA 3+80. STA 3+80 through STA 2+00 left open intentionally to backfill trench with jet grout waste"

Now there's two ways to take that. From what I've learned about jet grouting, sometimes the spoils (the stuff that bubbles back to the surface after injecting the grout down into the soil) are routed to a trench so they don't spread all over the place. Then they are disposed of. Under Corps specifications for this project, and indeed every project, jet grouting spoils (a kind of cementy, dirty slurry) were "unsuitable material," and were to be disposed of from the project site within 48 hours. They can't be left inside the levee because they're not the right density. In other words, they're trash. So the above passage could be taken as routine preps to simply contain the spoils for later disposal. Or it could be read as a CYA by the contractor to let future readers know that jet grout spoils were to be left inside the levee. I believe it's the latter.

The prep work continued for a few days, and then on June 21st the jet grouting began:

"Hayward Baker began installing jet grout columns under the I-610 bridge. Baker began with the 1st 4' diameter column. The columns installed were 65' in length...

IPS was responsible for handling the jet grout spoils. IPS dug trenches along the jet grout area so the spoils could run into the sheet pile trench. The spoils were placed from STA 1+97 through 2+45. About 30 cubic yards of spoils were removed from the trench and designated for removal from site tomorrow. The remainder of the spoils is being used as backfill."

So they were trenching to collect the spoils. But more importantly, they also deliberately left spoils in the levee. This went on for two more days.

June 22:

"IPS is responsible for handling the jet grout spoils. IPS dug trenches along the jet grout area so the spoils could run into the sheet pile trench. The spoils were placed from STA 1+97 through 2+45. About 15 cubic yards of spoils were removed from the trench and designated for removal from site tomorrow. The remainder of the spoils is being used as backfill. IPS hauled out excess spoils from the site."

June 23:

"IPS is responsible for handling the jet grout spoils. IPS dug trenches along the jet grout area so the spoils could run into the sheet pile trench. The spoils were placed from STA 1+97 through 2+45 and STA 2+45 through STA 3+70. The spoils used for backfill were left 12" below the sheet pile top elevation and 18" below the top of final grade for STA 2+40 through STA 1+98."

A subsequent paragraph in the June 23rd report gives the impression they were trying to hide the fact spoils were being used as levee backfill:

"QC Manager inspected the installation of one 4' diameter 65' jet grout columns under the I-610 bridge. Inspected the placement of the spoils from jet grout columns into the PZ27 sheet pile trench. The spoils were placed at least 18" below the final grade where dirt is to be placed and grass is to be grown."

There is no doubt jet grout spoils are supposed to be disposed of. From the Corps' own specifications:

"3.7.4 Disposal of Spoil

The Contractor shall remove all waste and all spoil materials produced as a result of the ground improvement operation and remove the materials from the job site as part of the contract price."
- Section 31 32 13.00 12, 'Ground Improvement'"

Hayward Baker, the company doing the jet grouting, also has model specifications available on their website. Here's what they say about spoils:

"At completion of daily Jet Grouting operations, thoroughly clean site and dispose of all spoil debris, water, and spilled material. Spoil stockpiling overnight is permitted prior to transfer to a predetermined waste or fill location."

To put it simply, the Corps deliberately ignored their own specifications and their own contractor's published guidance and left unsuitable material inside the Orleans Avenue canal levees on June 21-23, 2011.

Stumps, light pole foundations, trash

All the following quotes are taken directly from the contractor QC reports. They speak for themselves.

March 24:

"While clearing and grubbing [subcontractor] Newline encountered chunks of concrete and 3.5" pipe that was in the soil. The debris will be removed prior to any embankment operations take [sic] place"

April 11:

[During initial excavation of area under I-610 bridge]
"[Sheet pile prep work subcontractor] Sealevel has found large obstructions in the excavated areas. Obstructions include concrete, rebar, trash, and other miscellaneous materials"

April 12

"Sealevel continued to excavate trench for sheet pile installation. Still, they are finding obstructions in the clay. Obstructions are being documented by photographs. RFI is being worked on for replacing [with] suitable material."

May 6

"[Dry soil mixing subcontractor] Hayward Baker installed test columns for ground improvements. Test columns were installed at STA 46+00. The first two test holes were abandoned due to Hayward Baker hitting concrete 4 meters into the hole."

May 12:

"[Prime contractor] IPS excavated 9' down and found a 4' in diameter cypress stump... the excavation was backfilled"

May 17:

"Sealevel began installing the 2nd PZ27 sheet pile but had to extract the sheet due to an obstruction hit 10' into the soil. Sealevel excavated the obstruction and uncovered a large cypress stump that was 5' in diameter and at least 4.5' deep. A RFI was inputted to USACE about the removal of the obstruction. Sealevel plans to mob[ilize] in a tiger tooth attachment to tear the stump up tomorrow."

May 24:

"When [sheet pile installation subcontractor] Blue Iron was driving the 10th pile for the day at STA 6+54 through STA 6+58 the encountered an obstruction 3' deep that they could not penetrate. It was too late in the day to begin excavation to investigate what the obstruction is. It seems to be concrete but IPS can not confirm until the obstruction is excavated. IPS will excavate the obstruction in the morning."

May 25:

"Obstruction at STA 6+54 through STA 6+58 was an old concrete light post foundation that was 5.5' in length and the girth was 8.5'. The concrete post was excavated and removed and the excavation was backfilled with suitable material.

At STA 6+62 through STA 6+66 Blue Iron hit another obstruction 4' into the soil. IPS uncovered the obstruction and uncovered another light post foundation the same size as the one found at STA 6+54. Total time lost due to the obstructions was 6 hours. IPS backfilled that excavation as well and compacted the fill."

June 1:

"At approximately 0930 hit an obstruction during installation of the PZC18's. Sealevel excavated and removed a 10'x12' cypress stump. It caused a delay of 4 hours.

At approximately 1700 HBI [Hayward Baker] encountered a large tree stump while installing Ground Improvement Column 62A. After several hours of trying to remove the stump the USACE RE [Resident Engineer] was contacted and we were told to move the columns 3' to the south. Due to the stump's size it broke several tools on the drill and the rest of the day was spent repairing the tools."

June 6:

"Sealevel large cypress tree stump at 2+65"

June 8:

"Blue Iron hit an obstruction at STA 10+90 while driving the PZC 18 sheet piles. The obstruction was 4' deep, IPS excavated the obstruction and found a cypress stump that was approximately 6' in diameter and 5' deep. The obstruction was removed and the excavation was backfilled and compacted."

These only represent what was found at the sections into which they dug. Are we seriously supposed to believe that the sections of levee untouched by this project are pristine after reading this stuff? 10'x12' stumps and concrete light pole foundations do not make for reliable levees.

Part of levee missing since 1997?

I recently obtained the latest iteration of the Corps' safe water elevation (SWE) reports. They've been rechristened "Mean Operating Water Level" (MOWL) reports and were issued in April, 2011:

17th Street
Orleans Avenue
London Avenue

The substance of the reports remains the same, but There have been significant revisions to calculations between the October, 2010 versions and these April, 2011 ones, plus they read much more smoothly now.

In the Orleans Avenue canal report, though, I came upon this new verbiage:

"The FOS calculated by the Spencer’s Method of analysis for Reaches 10B, 17, 18A, and 20B are slightly less than the required 1.4 with the water level in the canal at El 1.0 NAVD88. This water level corresponds with the normal Lake water level. This indicates an inadequate FOS without the influence of the canal water load. These low FOS were the result of the low shear strengths identified in the 2010 CPTs advanced at the toe of these levees. Reach 18A was designed with a protected side stabilization berm extending approximately 90 feet from the I-Wall. This berm is shown in DM-19 [6] and on the “as-built” drawings [11]. The topographic survey performed in 2010 [12] indicated that the berm width was about 30 feet in this reach."

Now it's bad enough there were at least were four reaches which could not meet the Corps' standards when the water was simply at normal tidal levels. In fact, reaches 10B and 20B were not included in the remediation project. But what I highlighted above is even worse.

What that's saying when you cut through the Corps-ese and check the drawings is that a part of the levee on the Orleans Avenue canal which was supposed to be built in the 1990's was not actually there, and the Corps didn't discover this until 2010.

Reach 18A is here, on the east bank of the canal between Harrison and Filmore Avenues, across from City Park:

It extends 1100 feet. It was supposed to have a stability berm - a bunch of dirt intended to keep the levee and I-wall in place - that extended from about 30 feet from the I-wall to about 90 feet from the I-wall. Indeed, the as-built drawings of the 1997 project - submitted in June, 1998 after the project was completed - do show such a berm on the cross sections:

What the Corps found in May, 2010 was that the berm was considerably short on width. Here's cross sections of Reaches 17 and 18A drawn from data collected during the 2010 survey. Like 18A, Reach 17 was also supposed to have a stability berm:

While the 2010 survey confirmed the existence of the Reach 17 berm extending the proper 60 feet from the levee toe (or about 90 feet from the I-wall), the Reach 18A berm extended only to about 30 feet from the levee toe (or about 60 feet from the I-wall). So a 30 foot by 1100 foot by 2 foot thick section of levee appears to have never been built, or was modified post-construction, or was modified during construction but the modification was left out of the as-built drawings. Then it went hidden in plain sight for possibly 13 years, five of which were after the 2005 levee failures. Additionally, in 1998 the contractor's Corps overseers submitted as-built drawings showing the stability berm as completed, and the Corps Engineering Division accepted them.

As confirmation, the December, 2010 construction drawings for OFC-06, a project which consisted solely of placement of a stability berm along Reach 18A in order to raise the safe water elevation from less than 1 foot to greater than 8 feet, reflect the May, 2010 survey for Reach 18A, showing no existing berm where the "new" berm was to be constructed in 2011:

The very existence of contract OFC-06 - a contract which appears to be designed solely to put in place work which was supposedly done in 1997 - raises all kinds of questions about what was going on in the Corps' offices before August 29, 2005, as well as since the storm. It also shows it may have been pure luck that none of the walls along the Orleans Avenue canal did not fail in 2005.

Summing up

So here's what we learned from the 2011 Orleans Avenue canal remediation QC reports and the new (April, 2011) "Mean Operating Water Level" (MOWL) reports:

1) The Corps went against their own specifications and industry standard - for unknown reasons - and deliberately left unsuitable material, known as "spoils," in the Orleans Avenue canal levee at the conclusion of three days of jet grouting in late June, 2011.

2) When the Corps' contractors dug into the Orleans Avenue levees during the remediation project, or when they tried to place sheet pile or deep soil mixing columns, they ran into obstruction after obstruction for months. Among the pieces of junk found in the levees were two concrete lightpost foundations, many, many cypress tree stumps (some as big as 12 feet), and assorted trash and junk.

3) Only in 2010 did the Corps discover that 1100 feet of the east bank levee between Harrison and Filmore Avenues was missing approximately 30 feet of stability berm which was supposed to have been there since the 1997 construction of the I-walls on that side of the canal. That stretch was remediated in 2011 with a dedicated contract. Nonetheless, it appears the levee along that stretch may have been unfinished for 14 years, including six years post-Federal Flood, despite as built drawings from 1998 showing the complete berm in place.

In the next entry the entry after the next entry, I'll tie up the last three posts in a bow and reveal that the Corps will be doing more remediation work along all three outfall canals in the near future.