Debris Part 8

Part 1
Part 2
Part 3
Part 4
Part 5
Part 6
Part 7
Part 9
Part 10
Part 11
Part 12
Part 13
Part 14

All of a sudden, the debris story grew legs over the last three days, with multiple reports in multiple outlets. In part it was due to concern over stuff like this:


We'll return to that a little later.

Media interest before SLFPA-W July board meeting

The spark for the sudden jump in interest was the Southeast Louisiana Flood Protection Authority - West's (SLFPA-W) July 25, 2011 regular board meeting, where the Corps was called to account for all the unresolved debris problems. The SLFPA-W board appears beyond fed up, as they released multiple documents to the media in an effort to overcome any Corps spin. By all accounts, they may have succeeded.

Before the meeting there were two reports, one from WWL-TV and one from Fox 8 (WVUE).

WWL's July 25th report included video from the SLFPAW of a large chunk of debris getting dug out of a stretch of extremely troubled project WBV-14c.2 (contractor: Phylway; borrow source: River Birch):


The video also includes a map of WBV-14c.2, seemingly taken from the SLFPA-W, showing the entire project is of concern:


The WWL report includes a number of quotes from a clearly fed up SLFPA-W President Susan Maclay, such as this one characterizing the debris:

"'... you have large pieces of bricks and concrete and rocks and steel rods and wood, almost tree trunks,' Maclay said. 'This is going to undermine the levee.'"

The WWL story also reveals (again through documents released by SLFPA-W) that some Corps Quality Assurance (QA) inspectors claimed they alerted Corps upper-ups about the problem:

"Even more disturbing, she said, is that the corps' quality assurance reports reveal that its own inspectors have also alerted corps management about the problem.

One entry says, '...again mentioned to CQC the importance of removing objectionable materials.'

'But management has not done anything about it,' Maclay said.

She said so far the corps' management has only responded with what she calls excuses, with the corps saying the amount of debris in the levees is within specifications.

'But if you read the specifications, clearly it's not,' Maclay said."

Based on the level of antipathy by the Corps toward the state inspectors we've seen recorded in the state inspector's reports, I think such excerpts from Corps QA folks need to be taken in perspective, but they are enlightening.

One other thing: while the WWL report indicates "CQC" in the quote above stands for "Corps Quality Control," that's wrong. It actually stands for "Contractor Quality Control." It is the contractor, not the Corps, which is responsible for quality control - like ensuring there's no debris in the material - on Corps jobs. Nevertheless, Corps upper management has long been aware of these problems.

Fox 8's first report aired a day earlier, on Sunday, July 24, 2011 (video can be found at link. I'm not embedding it because Fox 8's embedding doesn't seem to work right).

It highlights a July 15, 2011 letter from SLFPA-W President Maclay to Corps New Orleans District commander Ed Fleming (linked here), which says, among other things "the use of unsuitable material in the construction of this project has been pervasive and continues unabated." The letter concentrates only on three projects: WBV-14c.2, WBV-15a.2, and WBV-3b. The problems at WBV-14c.2 are well documented, as are those at WBV-15a.2, though notably the letter lists many other concerns at WBV-15a.2 besides debris.

WBV-3b seems to be a particular focus of the SLFPA-W. The inspection records available online do not give the full picture which undergirds that concern, but records recently released by the SLFPA-W do. How about this:



That is a huge log that was pulled out of the finished levee at WBV-3b on June 15th during part of the pre-final inspection of that project. But that's not the worst, because look what happens when the camera pulls back:


We can see giant chunks of concrete, what appear to be logs even bigger than the one in the detail above, and an assortment of junk that should make everyone at the Corps wonder what exactly their commanders were crowing about two weeks earlier on June 1st of this year, when they claimed the system was ready for a 100 year storm. And Maclay's July 15th letter indicates there was more:

"Because our inspectors reported finding large logs and other unsuitable material in this levee section [WBV-3b], the contractor [Shavers-Whittle] and USACE's construction division visted the site and removed utility trailers filled with unsuitable material."

That's "trailers," as in more than one. Yikes.

Interestingly, the Fox 8 report includes maps of the projects listed in the letter which appear to go beyond the projects highlighted. Specifically, this map:


The voiceover during the report says this shows the section of levee from the new Westwego pump station to Orleans Village. That description corresponds with the "official" description of only WBV-14c.2, which is located here, in the northern half of the highlighted section on the Fox 8 map:


The highlighted section on the Fox 8 map actually goes much further than the eastern end of WBV-14c.2, tracking all the way down to the end of another project with debris problems, WBV-14b.2:


I don't know what the source document for the Fox 8 graphic was. The July 15, 2011 letter from President Maclay does not explicitly reference WBV-14b.2, and the Fox 8 graphic misplaces the location of Orleans Village:


More than likely, this was simply a graphics error by Fox 8, even though we know WBV-14b.2 has also had debris problems.

Media reporting of the meeting

Anyhow, after the SLFPA-W board meeting, there was another flurry of stories. Again, WWL-TV and Fox 8 had reports (I'm only embedding WWL's video; Fox 8's can be found at the link because their embedding doesn't work too well):



The Times-Picayune also chimed in.

All three reports hit mostly the same notes. They all reported the same platitudes ("public safety is out number one priority" and such) and excuse-making by the Corps. From WWL's report:

"Monday, corps officials pointed out, the Westwego levee [specifically WBV-14c.2] is still under construction, and some areas analyzed could have been cleared of debris after inspectors left.

They said spot inspections often lack proper context."

This is pure BS. There are literally dozens of inspection reports, some on a weekly basis, with hundreds of photos taken along lengthy stretches of multiple projects showing debris all over the place. Indeed, we've seen the "we'll clean it up" excuse from the Corps' QA guys too many times to count. And the tone of such comments undermines the further BS about the Corps' work being a partnership with SLFPA-W and the state.

Perhaps it is then fitting that the New Orleans District is being shoved aside in part of their QA duties, as WWL goes on to report:

"[Fleming] said a new inspection team will be on the ground by next week, the number of inspections will be increasing and crews will trench the levees in some spots to determine how deep inside debris is located.

[...]

Corps officials said the new inspection team will come in from outside the New Orleans District."

Fox 8 says the outside QA team will be overseeing only the trenching:

"Also, a quality assurance team will arrive in town soon to oversee the trenching work."

And unfortunately, it doesn't appear that team will have much autonomy, as the T-P reports:

"Overseeing the efforts will be senior project manager Kevin Wagner."

Wagner is an old hand at the New Orleans District.

Of course, the trenching on the new levee is also a major development. We've already seen trenching occur on the existing levee at WBV-14c.2, when it was being used to build up the new levee:


It was found to contain a ton of debris, just like the new incoming dirt. The Corps tried to discount the findings of those excavations; fortunately the state did their own analysis which confirmed the excessive debris in the existing levee.

I personally think comments from SLFPA-W president Maclay in the first WWL story are the most on point:

"Maclay said it may well be necessary to tear down the levee and rebuild it from scratch."

Based on the reports, that sounds about right.

No surprise, Part 2

Over the last two years, we've learned many, many of the Corps' 54 hydraulic pumps have been leaking oil into Lake Pontchartrain as they rust away. Sometimes the Corps reports these spills - as they must under federal law - and sometimes they don't. And sometimes pumps are scheduled to come out for rebuilding, while other times pumps are yanked as a result of spilling oil.

The places oil can leak out on these pumps are numerous. They include a slew of different pipes on each pump through which hydraulic oil (which powers the pumps) flows; most of that piping sits in the brackish water of the lake 24 hours a day. When the pumps aren't running, the oil sits in the piping on the pumps and in the hundreds of feet of pipes leading away and from the pumps, all the way back to the reservoirs on the hydraulic power units (HPU's) located on the land sides of the canals.

The ongoing corrosion of these completely or partially immersed pump-mounted pipes, which were carbon steel when originally installed, leads directly to leaks. While the Corps has had their contractor yank out 25 of the 40 major 60" pumps located across all three sites (three of them twice) and replaced that carbon steel piping with stainless steel, the remaining 15 60" pumps still have the rust-friendly carbon steel piping on them. As we've seen in earlier posts, the salt water does a number on this piping:


That photo is of an oil cooler on pump W5 or W6 from the 17th Street site. It was taken a mere two and half years after these pumps were accepted for service by the Corps. In actuality, the pumps started spending time in the lake waters in mid-2006, but they were taken out and put back in so many times over the following year while dealing with deep-seated design issues (some of which remain unresolved to this day) that the corrosion clock only started started running consistently in the latter half of 2007.

There are many, many more pictures of rusty pipes like this from the Conhagen repair reports:





With all this rust causing all these oil leaks, it makes sense to ask, "If the oil is leaking out of the pumps, doesn't that mean water is leaking into them?"

The answer is "Yes." When the pumps are not running, the oil is not pressurized by the HPU's. So oil can seep out of leak points and water can come in. If the Corps turns on the pumps with those leaks and starts circulating all this oil mixed with water through the entire hydraulic oil loop, the water gets into the delicate parts on the HPU and can do serious damage.

That's why it should be a standard part of the repair to completely flush out the entire hydraulic loop for any pump that get's pulled out for repairs. That would include all the connecting piping from the pump back to the onshore HPU, as well as the HPU itself. Natually, though, the Corps doesn't do this. It's too expensive (for them, not for normal engineering organizations). When the pumps were put in back in 2006, there was pickling fluid remaining in the pipes when they were intially filled with oil. The resulting jello-like muck took two flushings to remove, at an expense of hundreds of thousands of dollars paid for by the U.S. taxpayer, despite the problem totally coming out the vendor's poor performance. I imagine that expense is what keeps the Corps from cleaning out the connecting pipes every time they yank out a pump for rebuilding.

So that just adds one more question about the performance of these things if they're ever needed in a major storm (earlier storms when they were turned on were absolute pea shooters). And it should come as no surprise that problems have cropped up directly related to contaminated oil.

On December 9, 2010, the Corps issued task order #7 on the second Healtheon/Conhagen pump repair contract (there have been three such contracts - more evidence of the Corps inability to plan ahead). The task order called for the rebuilding of London Avenue pumps E2 and E6.

I detailed the facts surrounding this set of repairs in my earlier post, "The latest on lakefront pump repairs." While there were some bumps along the way during these repairs, including apparently two oil spills totalling 125 gallons, the repairs themselves seemed to go somewhat smoothly (except for a distinct lack of information on the Rineer hydraulic motors). As expected, the pumps were in terrible condition when pulled out. In fact, one of the sample pictures above of really bad rust was taken from the Conhagen repair report for these two pumps:


The pumps were put back in the canal January 24, 2011.

However, that wasn't the end of the story. Just about a week after the pumps returned to service, the Corps issued $22,188.68 modification 2 to task order #7, increasing the cost for the repair of these two pumps to $304,898.07. The scope for the modification, knowing what we know about the oil leaks on all these pumps, should come as no surprise:

"- Take oil sample out of reservoir have analysis performed.
- Drain and pump oil out of reservoir, hoses, piping, etc. and flush to remove all contaminated oil.
- Remove the hydraulic pump, motor, coolers, and piping and flush to remove all contaminated oil.
- Dispose of contaminated oil
- Furnish new replacement hydraulic oil
- Refill reservoir on test unit
- Provide standby crane for this additional work.

Completion Date:
Work shall be completed within one week from the modification effective date."

The verbiage of the task order, as well as its price, lend itself to a couple of interpretations:

1) The Corps discovered the oil in either the E2 or E6 hydraulic loops at the London Avenue site was contaminated and had Conhagen remedy the situation. The fact that this modification was issued a week after the pumps were acceptance tested argues for this, although admittedly the Corps has a long history of post-dating their contract actions.

2) Conhagen, while they were pressure testing the pumps at their facility in Kenner, LA with one of the Corps' spare hydraulic power units, encountered a problem with contaminated oil and remedied it. The mention of the reservoir on the "test unit" argues for this. We have a picture of what a typical oil pressure test at Conhagen's site looks like, from the repair report for 17th Street pumps W4 and W7:


Whether the contaminated oil was encountered at Conhagen's shop, or out at the site, the larger point about likely widespread oil contamination still stands. and adds just one more question mark above the potential future performance of these pumps.

No surprise

This post is one in a series I've been running since May, 2010 about the corrosion-related repairs to the 54 hydraulic pumps at the Corps' three New Orleans outfall canal sites. I've only recently started to receive contract documents and reports on these repairs via the Frreedom of Information Act.

A post from earlier this week provides links to the earlier articles in the series. Since the documents released date back almost two years, many of those earlier articles have been heavily revised to reflect the newly gained knowledge from the documents.

But the released documents also included information on very recent events surrounding these pumps...

In the last post, I noted that two pumps at 17th Street (W8 and W9) had been pulled out in the last couple of months. Those two pumps were already pulled out in 2009 for repairs, yet here they are coming out again. Why is that happening?

Because the parts that weren't overhauled in the 2009 repairs - which make up most of the pumps - gave out.

Just as a reminder, the repairs to pumps in 2009 (covered in earlier posts here and here) only concerned themselves with replacing carbon steel piping on the inside of the pumps with stainless steel:



The piping on the outside of the pumps was replaced, but with the same carbon steel material that had rusted in the first place. Then it was repainted with the same paint as before as well:



Had there been new stainless steel piping on the outside of the pumps, it would appear the same as the silvery stainless steel piping shown in the picture of the inside of the pump above.

In addition, all the moving parts like the bearings and the Rineer motors were also put back.

So essentially, about a third of what needed to be fixed was fixed, and the pumps were put back in the brackish Lake Pontchartrain waters to start corroding again. Six 60" pumps - five at 17th Street and one at London Avenue - as well as an unknown number of 42" pumps at 17th Street received this suite of repairs. It is no surprise that these pumps would be subject to the same corrosion damage as any other, mostly-carbon steel piece of equipment sitting in salty water.

We saw the first evidence of this earlier this spring, when pump E1 at the London Avenue site was removed as part of a planned pump pull. I only recently figured out that this pump was repaired in 2009 in addition to the five 60" pumps at 17th Street.

The London E1 stainless steel interior pipes, installed in 2009 and seen in the photo above, did not appear severely rusted in spring, 2011...


...while the carbon steel external piping and coolers looked like they took some serious corrosion damage since June, 2009:


[More details on the 2011 repairs to this pump are in my earlier post, "The 2011 pump rebuild scramble."]

That this pump was taken out as part of a planned complete overhaul in the spring of 2011 is evidence the Corps recognized the 2009 repairs were inadequate. But with many other pumps completely untouched since 2007, they triaged and focused on the more damaged ones rather than the other five pumps repaired in 2009.

However, it was only a matter of time before the pumps partially fixed in 2009 also started breaking. That's what happened with 17th Street pumps W8 and W9 in the last two months.

The first failure appears to have occurred in May or June of this year. We learned this from $89,489.55 task order #8 on the second Healtheon contract, issued on June 7, 2011. Prior to this, the second Healtheon contract had lain fallow since the issuance of task order #7 and its immediate follow-up modification on December 9th and 10th, 2010. After that, the 60" hydraulic pump repair work had been moved over to the third Healtheon contract, which was inked in January, 2011. Issuance of a task order on a contract previously thought ended was a little odd, but not unprecedented. The first Healtheon contract has continued to get task orders issued on it as well.

Anyhow, the body of task order #8 reads,

"Remove pump # 8W at 17th St. Canal to inspect, recommend what need to be repaired and reinstall it.

Completion of this work will be six weeks after notice of award.

Note:
1. Pump # 8W was overhauled in Aug 2009 and is currently leaked [sic] outside of the pump.
2. After the contractor inspects and finds out what the pump problems are, we will provide Contracting Division a separate scope of work for repair."

So the Corps had discovered an oil leak on pump W8. Did they report it to the National Response Center, as required under federal law? Apparently not, because searches on the NRC website show no reports for such an incident prior to June 7, 2011. That's really no surprise - the Corps has little regard for the law on reporting oil spills.

Pump W8 was pulled out of the 17th Street site and sent to Conhagen's shop. A $30,178.32 modification to task order #8 was issued June 20, 2011:

"The following scope of work was found necessary upon removal and inspection of Pump #8W at the 17th St. ICS:
- Remove hydraulic motor and motor adapter. Replace motor with gov-furnished new one.
- Build up weld eroded and corroded areas on motor adapter and other pump surfaces.
- Replace oil coolers and connected piping. Spot prep minor corrosion on other exterior piping.
- Replace small bypass hose for motor case drain. Replace exterior pipe u-bolts.
- Provide a full coat of epoxy paint on pump assembly in addition to touch up of original scope.
- Have mechanical seal reconditioned at subcontractor’s shop (to be returned to gov). Install new gov-furnished seal."

I would have expected the scope to simply be a restatement of the normal scope of work, with perhaps the exception of replacement of the interior piping. Instead, we find this oddly worded scope, which is rather lacking in detail. Nevertheless, it appears to cover much of the work which would be expected.

Fortunately, Conhagen had a better grasp on what needed to be done. The Conhangen repair report for pump W8's second time in their shop gives a detailed scope which covers everything needed to bring this pump up to same status as other completely rebuilt units.

That report shows the areas which one would expect to be the most damaged - those not upgraded in the 2009 repairs - were indeed the most damaged. The carbon steel oil coolers installed in 2009 had unsurprisingly accumulated a great deal of marine growth and had rusted:


Admittedly, the damage wasn't as great as we have seen on other pumps, but it is evident. Here's a similar area on one of the pumps after it was repaired in 2009:


The Rineer motor and its mount were also rusty:


Here's the "before" picture of that area in 2009:


Conhagen fixed up the pump, leak tested it, and put it back in the water on June 6, 2011, according to the acceptance testing record included in Conhagen's report.

Wait, what?

The original task order for the removal of this pump was dated June 7, 2011, and the follow-up modification with the scope of repairs was dated June 20, 2011. So how did the pump go back in the water before there was any contract for repairs?

It appears this is once again the Corps' Contracting Division issuing after-the-fact task orders and modifications for work already completed. In fact, the cover page of the Conhagen repair report indicates their work on the pump actually started three weeks before they received their first contractual documents:


One has to wonder how exactly contracts are run at the New Orleans District, when we find stuff like this.

So apart from the contracting slight-of-hand, I think we can take the Conhagen photos of W8's repairs in May and June of 2011 as proof that it has been fully rebuilt, meaning the 17th Street pump status can be updated again. However, the fact that W8 after previously being partially repaired still sprung a leak means that the remaining 60" pumps fixed in 2009 are now not just questionable, but effectively unrepaired. So the revised 17th Street status is:


This now means there are sixteen 60" hydraulic pumps across all three sites which cannot be trusted to be free from corrosion-related failures in the future:

17th Street: E5, E7, W2, W3, W9, W10 (W2 and W3 never previously repaired)
Orleans Avenue: E1, E2, E3, E4, E5, W1, W2, W3, W5 (all never previously repaired)
London Avenue: W2 (not previously repaired)

Indeed, this assertion was confirmed just last week, as 17th Street pump W9, another of the pumps only partially repaired in 2009, also sprung a leak.

This time, there was an oil spill report to the National Response Center, on July 5, 2011:

"Official Material Name: HYDRAULIC OIL
Qty Released: 8 GALLON(S)
Qty in Water: 8 GALLON(S)
CALLER IS REPORTING A DISCHARGE OF HYDRAULIC OIL FROM A HYDRAULIC PUMP DUE TO UNKNOWN CAUSES.
REMEDIAL ACTIONS: CLEANING UP SPILL WITH ABSORBENT PADS WITHIN THE BOOM SYSTEM."

Two days later, on July 7, 2011, the Corps issued $89,489.55 task order #9 on the second Healtheon contract. That dollar amount is exactly the same as the initial task order (#8) on 17th Street pump W8 in June. This time, the task order reads:

"Remove pump # 9W at 17th St. Canal to inspect, recommend what need to be repaired and reinstall it.

Note:
1. Pump # 9W was overhauled in Aug 2009 and is currently leaked [sic] outside of the pump.
2. After the contractor inspects and finds out what the pump problems are, we will provide Contracting Division a separate scope of work for repair.
3. Estimated date to start is 7/8/ 2011 and overtime for the contractors to work on Saturday 7/9/11 and Sunday 7/10/11 should be authorized."

Like the dollar amount, this is a copy and paste from the initial task order for pump W8, down to the typo on the work "leaked."

Since my receipt of documents via FOIA only covered stuff through last week, I don't have the task order modification which covers the actual repairs of 17th Street pump W9, but I am certain it looks nearly identical to the corresponding modification for pump W8. Right now, pump W9 is sitting on Conhagen's shop floor, and it will probably go back in the water before July 31.

Update, 7/6/12

The modification that covered the additional scope of repairs on 17th Street W9 was issued July 19, 2011 for $29,803.42. Again, this appears to be an after-the-fact contract action, since documents attached to the Conhagen repair report indicate the pump went back in the water on July 15, 2011.

That report includes the usual sorts of pictures of excessive marine growth and corrosion on the pump:

Original post resumes

The pattern of pump repairs over the past two years has been for scheduled rebuild work to only be undertaken outside hurricane season, meaning from December 1 to May 31st. If individual pumps fail during or close to hurricane season, as 17th Street pumps W8 and W9 have in the past couple of months - or as three other pumps did during the 2010 hurricane season - they are yanked out and rebuilt. Everything else - including pumps that have been rusting for nearly five years now - is left in the canals. This is a remnant of a foolish Corps decision made after Katrina - and carried out for two years of engineering and construction in 2006 and 2007 - not to supply any spare pumping capacity at their closure structures.

At the pace demonstrated by the repair contractor over the past two years, it seems impossible all 16 remaining 60" pumps will be completely rebuilt by June 1, 2012. Only ten pumps each were rebuilt in the 2009-10 and 2010-11 hurricane off seasons. So it seems likely - barring further (admittedly very likely) oil spills from unrebuilt pumps - there will still be rusty pumps in the canals all the way through the 2012 hurricane season.

Finally

On July 8, 2011, the Corps finally started turning over the records on the rebuilding of the hydraulic pumps at all three outfall canal sites. I had requested these documents through the Freedom of Information Act over a year before. In the interim, there were a number of additional repairs, and many documents associated with those actions were also given to me.

As a result, I am currently heavily editing my previous entries about the pump fixes, adding in pictures and descriptions of the repairs taken from Corps contract actions and reports written by the pump repair shop, Conhagen, Inc, located in Kenner, LA.

Previously, I had only received a single Conhagen report which covered the repair of two pumps. I now have an additional nine reports documenting the repairs of 15 other pumps.

Here's links to the reports:

17th Street pump W10 and bridge pumps #1 and #6 (May-July, 2009) - no Conhagen report generated because the Corps did the work themselves

17th Street pumps E5, E7, W8 and W9 (July-September, 2009) (report received previously)

17th Street pumps W5 and W6 (February-March, 2010)

17th Street pumps E1 and E2 (March-April, 2010)

London Avenue pumps E3 and E4 (March-April, 2010)

17th Street pumps E6 and E8 (April-June, 2010)

London Avenue pumps W3 and W4 (April-June, 2010)

17th Street pump W1 (June-July, 2010)

London Avenue pump W1 (July-August, 2010)

Orleans Avenue pump W4 (October-November, 2010)

London Avenue pumps E2 and E6 (December, 2010 - January, 2011)

London Avenue pumps E1 and E5 (March-April, 2011)

London Avenue pumps W5 and W6 (April-May, 2011)

17th Street pumps W4 and W7 (March-May, 2011)

17th Street pumps E3 and E4 (May-June, 2011)

17th Street pump W8 (May-June, 2011)

17th Street pump W9 (July, 2011 - ?) - pump currently under repair.

Sharp eyed readers will notice the last two pumps, W8 and W9 at 17th Street, had supposedly already been rebuilt in 2009. However, those rebuilds only replaced a few of the carbon steel pipes, not all of them. They also didn't touch the bearings, Rineer motors or any moving parts. Unsurprisingly, these pumps have failed again, this time from the unrepaired parts.

The entries I'm updating or have already updated include:

Imminent, originally posted May 13, 2010

This year's scramble, originally posted June 3, 2010

Worse than previously known, originally posted June 11, 2010

The latest on lakefront pump repairs, originally posted March 23, 2011

The 2011 pump rebuild scramble, originally posted May 9, 2011

Brief update on London Avenue pumps, originally posted June 24, 2011

Other posts might also get updated or revised. And there will be coming posts on the current repairs, or "re-repairs" as it were.

In the meantime, here's some photos from the newly received reports...

A hole rusted straight through one of the oil coolers on pump W5 or W6 at 17th Street (report here):


Rust inside the supposedly sealed bearing housing on pump W5 or W6 at 17th Street (report here):


Extremely severe rust on London Avenue pump E4 oil cooler and attached piping (report here):


Identical corrosion on 17th Street pump W1 (report here):


Chunks missing from inlet suction bell on Orleans Avenue pump W4, due to corrosion (report here):


Other chunks missing from inlet suction bell on 17th Street pump E6 (report here):



More chunks missing from vanes inside suction bell on 17th Street pump E2 (report here):



Rusty Rineer hydraulic motor and attached piping on London Avenue pump E2 (report here):


Similarly rusty Rineer motor and piping on Orleans Avenue pump W4 (report here):



There's much more where that came from. But the important thing to remember is this: there are currently at least 16 pumps in the water which have had inadequate or no repairs to address these problems, which were evident after just two years in the canals. That includes 9 of the 10 pumps at the Orleans Avenue gates. Those 16 pumps have been in the canals for over four years now, and the only reason the Corps pulls them out is if they spring an oil leak.

Permanent pumps slip to Katrina 10th anniversary

[Updated July 12, 2011 - see bottom of post]

[Corps confirms 2015 permanent pump completion. See July 16, 2011 update at bottom of post.]

In my last post, I mentioned the outfall canal updates the Corps gives the Southeast Louisiana Flood Protection Authority - East. In the most recent update, from early June, there was also a slide about the permanent pump station project:


It indicates that the completion date for the permanent pumps will likely slip to 2015, due in part to the bid protests filed by losing bidders (but really due to the Corps' lassitude and intransigence in getting the project underway from 2006 through 2010). These stations were supposed to have been in service in 2010.

The permanent pump stations project was awarded April 13, 2011 to a coalition of companies calling themselves "CBY Builders." It includes CDM, Brasfield & Gorrie, and Yates Construction. It immediately garnered bid protests from two of the losing bidders. That's not a surprise, since the job value is $674,998,555.36. Notably, that's over $129 million less than the $804 million congressional appropriation for the job.

Who are the protestors? Who were the other bidders?

The Corps used a two stage bidding process. In the first stage, they accepted bids from anyone. They also made public the technical specification. Then, in the second and much more secret stage, they narrowed the list to the five top players (i.e. "the list that counts"). In this second stage the Corps didn't let anyone know who was bidding, they didn't release any specifications or drawings, and they didn't put out any press releases. It was remarkably hush-hush for a project worth over half a billion dollars.

We already know the winner. Here's the other four:

Bechtel team (protestor)

Bechtel partnered with New Orleans-area engineering firms Waldemar S. Nelson; Brown, Cunningham & Gannuch (aka "BCG"); N-Y Associates; and Eustis Engineering. During the bidding process, Bechtel had a website up at http://www.nops-pccp.com, but it went away after the contract was awarded.

PCCP Constructors (protestor)

PCCP Constructors is made up of Kiewit, Traylor Brothers, and the New Orleans-area based M.R. Pittman group. This is the same set of companies that make up the joint venture "Gulf Intracoastal Constructors," who has the prime contract for the West Closure Complex.

Louisiana Canal Constructors (no protest)

"Louisiana Canal Constructors" appears to be Alberici and Archer-Western (a subsidiary of Chicago-based Walsh Group), the same team that worked levee project LPV-111 along the Gulf Intracoastal Waterway in New Orleans East. Parsons was also involved. We know this because all three firms are listed on the Louisiana Secretary of State entry on LCC. Incidentally, that entry shows that LCC is actually not in good standing due to failure to file an annual report this year. That's not surprising, since their lack of bid protest activity says the LCC team has decided to call it a day on this project.

Weston group (no protest)

The Weston group also includes engineering heavyweight HNTB. Weston was so interested in this project last year that they actually lobbied congress on it. A lobbying disclosure form for Weston's chief lobbyist, Sean McGraw, notes the New Orleans Permanent Pump Stations as a target for their lobbying during debates over the FY2010 Energy & Water bill

Other companies floating around the bidding process included Odebrecht, Arcadis, Shaw (who had advertisements on job boards for work on this project two years ago) and Volkert, though I don't know who they were partnered with. All these names are rather heavy hitters in the engineering industry, which is no surprise considering the mammoth size of this prize.

Corps helps out finalists

Part of the reason we know who's who among the bidders is because of solicitations for small business subcontractors each team issued last fall. These sorts of subcontracting gatherings are common for bigger projects; they're useful for signing up small businesses to meet the Corps' small business goals. Sometimes the Corps puts them on, but in this case, each firm - each privately-owned firm - staged them. PCCP Constructors' event was on October 19th, 2010 at the University of New Orleans (UNO):


Bechtel's was on October 6th at the Best Western on Severn in Metairie:



Louisiana Canal Constructors' event was on September 23rd at Southern University:



Weston's team had an event on October 20th at UNO:



I can't find an announcement for an event hosted by the winning bidder, the CDM team.

Here's the twist: all these announcements for events put on by private companies were - and are - residing on the Corps' computers, specifically the subdirectory associated with the Corps' Small Business office. PCCP Constructors' is here, Bechtel's is here, LCC's is here, and Weston's is here.

At one point, the Corps provided links to the Weston and PCCP Constructors flyers on their Small Business Programs page:



[Note to Corps: I've already got Google screenshots showing each of these flyers on your servers, so don't bother taking them down.]

If you think having a government agency advertise for private companies looking to do business with that agency is unseemly, you're right. But it's par for the course for the New Orleans District, which enjoys cozy ties to many large firms (see the details on Kiewit's protest below). One of the flyers notes that two officials from the Small Business Administration (Jo Lawrence and actually-Corps-guy Randy Marchiafava) were confirmed attendees, and two Corps officials (Ned Foley and Michael H Bloom, both Corps small business guys) were invited, so the government was out there appearing at events held by private firms to help them improve their bids.

Details on the protests

It is noteworthy that there's not just one protest for each of the two protesting bidders. In fact, each of them have filed three protests since the contract award. According to the General Accountability Office's bid protest docket, Bechtel filed protests on April 26th, May 2nd, and June 6th. Kiewit's group filed protests on April 26th, June 6th, and June 24th.

I'm sure the April 26th protests by each firm were probably already prepared before the contract was even awarded. The June 6th protests appear to be protestors' responses to the Corps' initial report to the GAO, if the schedule above is to be believed.

[Update July 12, 2011

The July outfall canal update indicates the June protests by the bidders were "supplementals," whatever that means. See complete update at the bottom of this post.]

Back before all these protests started rolling in, the Corps' schedule (found on the last page of the April, 2011 Orleans Levee District status update) naively called for absolutely no impact by them, and a project completion date of October, 2014:


With Kiewit's protest filed on June 24th, the GAO's latest response date is now October 3, 2011 (the GAO has 100 days to decide bid protests). Note this is later than what early June schedule above shows, since the slide was produced before the latest protests were filed. Comparing the two schedules and the GAO protest dates, it is easy to conclude about six months of schedule slippage, pushing the project completion date to approximately March, 2015. And that assumes the project doesn't get rebid as a result of the protests, which is what one of the protestors is calling for.

[Update July 12, 2011

It is possible the protests are all part of one big file, with an August deadline still in effect. The Corps is playing the project delay card close to their vest. See the complete update at the bottom of this post.]

Kiewit goes public
Alleges Corps revolving door killed their bid

Kiewit went to the local media about their protest soon after filing in April. On May 4, 2011, WVUE-TV ran a story titled "Former employee accused of giving inside info to competing company."

The story reports:

"PCCP Constructors believes a senior level Corps employee provided inside information to the company that was ultimately chosen for the project, CBY Builders.

'It’s something that needs to be looked at and examined so that we know exactly what was said when and what kind of inside information may or may not have been disclosed that would benefit one party over another,' PCCP spokesman Kent Grisham explained. [note: Grisham actually works for Kiewit]

Grisham is talking about Richmond Kendrick. Kendrick was the chief of program execution for the Corps’ pump stations project. But in August 2010, Kendrick left the Corps and went to work for a company named CDM.

CDM is a partner of CBY Builders."

That's a pretty interesting charge. Of course, the revolving door between the Corps and contractors is well known within construction circles, but less so to the general public. Many of the most prominent, and some of the less prominent, figures involved in the Corps' post-Katrina work have left for contracting jobs which brought them right back into the Corps' offices working on that same post-Katrina work. In fact, here's a 2009 article in an industry trade organ featuring two of them: Dan Hitchings (former Task Force Hope leader and now working for Arcadis on Corps hurricane protection contracts) and Mike Rossi (who wasn't directly involved in the post-K work, but who now works for Vali Cooper International, the New Orleans-area company started by former Hurricane Protection Office leader Jeff Bedey and which claims to have Corps hurricane protection contracts, though they're more likely subcontracts). Ironically, the article also features an interview of Richmond Kendrick in his now-former capacity as deputy of program execution for the Corps’ Hurricane Protection Office.

Anyhow, it's unusual for the revolving door to get publicized in this fashion, but when there's hundreds of millions of dollars on the table...

The WVUE-TV report continues with an actual quote from one of Kiewit's protests. Given the May 4th date of the story, it could only be their first one, filed on April 26th:

"The protest filed by PCCP to the Government Accountability Office states, 'The most critical events in selecting the winning offeror for the PCCP (Permanent Canal Closure and Pumping) contract occurred after Mr. Kendrick took his inside knowledge concerning this procurement to CDM.'

Grisham says the company’s goal is to have the Army Corps re-open the bid process, now that Kendrick has left the government agency."

That would certainly be remarkable, though given the size of the contract, not unprecedented.

I'll stick close to this $674,998,555.36 story.

Update, July 12, 2011

The latest outfall canal remediation project update - issued in the first week of July - also contains a small update on the permanent pumps:


This update seemingly still calls for an August 4, 2011 response to all the protests. It calls the latest round of protests from June "supplemental protests." It also notably removes any reference to the start or completion date of the project.

I'm not sure what to make of the Corps insisting the GAO will come back August 4th with a decision. That date is based on the date of the original protests in April. The GAO's own docket indicates that each protest restarts the 100-day clock for them, so that while the earliest protest could be resolved in August, the later June ones might not get a decision until October.

Of course, we could just be seeing a limitation of the GAO's website, in that all three protests for each firm might indeed be part of a whole, but the website doesn't have a way to show that. Indeed, ther eis some support for that. The protest numbers given to PCCP Constructors' (nee Kiewit's) protest are B-405036.1, B-405036.4, and B-405036.6, while those assigned to Bechtel's protest are B-405036.2, B-405036.3, and B-405036.5. That would seem to indicate that they are simply part of an overall file numbered B-405036.

Two other shreds of information come out of this latest stab at a schedule by the Corps:

1) There was apparently some kind of hearing the last week of June in Washington.

2) That hearing was followed up by a briefing (presumably by GAO to the Corps) on July 7th, last Thursday.

Again, I'll try to stay on top of this.

Update, July 16, 2011

The Corps has confirmed the 2015 completion date for the permanent pumps.

In a July 12, 2011 presentation given to FEMA about the hurricane protection system by the Corps' Mississippi Valley Division (one step above the New Orleans District), the 2015 date was made explicit on this slide of completion dates for incomplete projects:


Incidentally, that is slide 1 of a pair of slides listing 30 incomplete 100 year protection projects:


The 16 projects with blue rows are those only at the 100 year level of protection if the Corps brings in Hesco baskets and sandbags immediately before a storm. The 13 projects with yellow rows actually have something built (like sheetpile or the like) to give interim 100 year protection. Remarkably, there's a still a project in St Charles Parish which hasn't even been awarded. So much for breaking their backs to meet the June 1st, 2011 deadline.

And finally, it turns out the Corps actually confirmed the 2015 date the same day this post first appeared. In a July 1, 2011 press release trumpeting their piecemeal, inadequate remediation work along the outfall canals, they included this:

"The 17th Street, Orleans Avenue and London Avenue outfall canals extend from interior pump stations to Lake Pontchartrain in metro New Orleans. Interim Closure Structures currently block 100-year storm surge from entering the canals; they will be replaced with permanent canal closures and pump stations that are scheduled to be complete in 2015. Because of the canal closures, the outfall canals’ primary purpose is to remove rainwater from the city. The floodwalls and levees lining the canals are no longer the primary storm surge defenses for New Orleans."