Rusty pumps keep rusting, Corps yawns

This post has been updated in January and February, 2017 with extra detail around pumps that were pulled and repaired in 2013 through 2016. Updates are in bold.

With the 2015 hurricane season upon us, and the continuously rusting pumps at the three interim closure structures (17th Street, Orleans Avenue, and London Avenue) well beyond their predicted life by any reasonable measure, I thought it would be time to check back in and see if the Corps has stepped up - or actually done any - preventative maintenance on them. For an overview of the rusty pumps saga from 2006 through 2012, see this wrap up post.

The pumps were put into the water in 2006, pulled out, and then put in for good in 2007. They were not expected to be in service more than a couple of years before they would get replaced by sturdier, bigger pumps. But planning for their replacement dragged on for years. The Corps, starting in 2009, stated the life of the entire structures - slyly without specifying the exact life of the pumps themselves - was "five to seven years." They said maintaining the pumps beyond 2013 would be difficult. They did this mostly to bring pressure to bear on locals to approve their permanent pumping scheme. But taking them at their word, if we started the lifespan clock in even 2007 - generous given construction of the structures began January of 2006 - we are now beyond even the worst case for the structures' and pumps' lives. One would have expected to see some effort toward life extension on these pumps.

And we did for a little while, with the replacement of many carbon steel components on the pumps with stainless steel between 2010 and 2012 (Note: Two pumps - E5 and E7 at the 17th Street site - remain untouched by those refit efforts). During that 2010-2012 period, the pumps were not expected to be needed past the 2014 storm season. However, in the intervening time the end date of the pumps has been extended even further into the future. Now they are expected to be in service until 2017, or 11 years after they first saw brackish lakewater. So what has been happening with the pumps since my last update during the 2012 hurricane season? Has the Corps done anything to make them last the additional three years?

In a word, no. Since 2012, the Corps has pulled six pumps across the three sites, almost all in reaction to oil leaks. There does not appear to have been any proactive moves to ensure the pumps can last until 2017. Here's what has happened:

2012-2013 off season: no pumps pulled.

May 20, 2013: Pump W10 at the 17th Street site sprung a hydraulic oil leak. The pump was pulled out a couple of weeks after the oil leak, just as the 2013 hurricane season was starting. It likely went back in the water some time in August, 2013 July 19, 2013 [see update below]. This failure was not a surprise, since W10 had not received the stainless steel upgrade in 2010-2012, but had been part of a group of six pumps given inferior carbon steel replacement repairs during a misbegotten effort in 2009.

Update, 2/5/17

The contract task order for this work is here, while the repair report is here. They confirm that this pump received stainless steel piping and coolers during this 2013 refit, which was essentially a complete rebuilding of the pump. Here's a picture of a severely corroded Rineer hydraulic motor, the device at the heart of pump, being pulled out:

July 2, 2013: Pump W9 at the 17th Street site sprung a leak. This failure is far, far more concerning. In fact, it's a canary in the coal mine. That's because W9 had been upgraded with the stainless steel [see update below] suite of repairs during the summer of 2011, just two years earlier. Now in 2013 it was leaking once again, causing a third round of repairs (W9 also got the 2009 carbon steel repairs).

This is troubling in the extreme. It means the stainless steel upgrades are all suspect, which makes the integrity of all the pumps suspect. This would be confirmed just a few months later on...

Update 2/5/17

The story on this pump is somewhat complicated. The pump had been part of the original set of half repairs in 2009, when only its internal carbon steel piping was replaced with stainless steel. The external oil coolers remained carbon steel. A picture of the new stainless internal piping appeared in the 2009 report:

Then in 2011 the pump had to be pulled out on an emergency basis when it started leaking during hurricane season, on July 5. Conhagen's work was very fast, from July 5th to July 15th that year. In fact, the task order for the work explicitly mentions overtime being required:

"Remove pump # 9W at 17th St. Canal to inspect, recommend what need to be repaired and reinstall it.

Note:
1. Pump # 9W was overhauled in Aug 2009 and is currently leaked [sic] outside of the pump.
2. After the contractor inspects and finds out what the pump problems are, we will provide Contracting Division a separate scope of work for repair.
3. Estimated date to start is 7/8/ 2011 and overtime for the contractors to work on Saturday 7/9/11 and Sunday 7/10/11 should be authorized."

Part of those repairs was replacement of rusted out oil coolers, seen here in a "before" picture from the 2011 report:

However, a picture of the internal piping at the completion of Conhagen's repairs from the same report appears to show the 2009-installed internal piping was reused. New stainless steel piping would have been shiny. Instead, the piping is black (a process known as passivation):

This is backed up by the scope of work in the 2011 report, which makes no mention of replacement of the internal piping, only the oil coolers. The reason for the piping reuse is likely time. Pump W9 was only out of the water for about a week in 2011, and the piping probably looked fine at that time. So rather than replace it preventively, it was left in place.
 
Then came the third removal in 2013. The contract task order for this work is here, while the repair report is here. Rust was again the cause for failure:

Note the spots of rust on the Rineer itself.

The pump was completely overhauled, including new stainless steel piping inside:

and outside:

The pump was accepted back into service during the height of hurricane season, on August 23, 2013.

So the removal in 2013 was an indication of the fragility of the repairs, but also one that demonstrated a lack of forethought. If the Corps had a complete spare pump assembly ready to go in 2011, they would not have had to rush those repairs. They could have simply replaced the entire pump and worked on it in a more reasonable timeframe, and then placed it back into service with a complete suite of repairs.

Not that any of that really matters, because the complete suite of repairs done in 2013 would prove inadequate three years later in July, 2016, when the pump would again get pulled. In fact, 17th Street pump W9 holds the record for most repairs to date since 2009, with four. It's neighbor, W8, is in a three way tie for second with three pulls. The other three time failures are London Avenue pumps E2 and W5. New Orleans is protected by phalanxes of Edsels and Pintos.

December 19, 2013: Pump E8 at the 17th Street site sprung a leak. It was pulled out for repairs in January of 2014 and went back in the next month. The canary started chirping louder, since E8 had also received the stainless steel upgrade, in this case three years earlier, during the spring of 2010. While we don't know the exact problem in 2013, there is some indication the pump did not need a complete overhaul. The original task order for the 2013 E8 repair was for $172,934.74, but a modification subtracted $130,693.20 of that. Full pump overhauls usually cost around $160,000 to $170,000. If there was only $42,000 of work done on the pump - and considering some of that was for crane rentals on both ends of the repair - it makes sense to think the problem was relatively small but critical. There's really only a few possibilities: a) a piping failure, b) a seal failure, or c) a failure of the Rineer hydraulic motor. [See update below] But it doesn't really matter, because there is no spare pumping capacity built into the interim closure structures, so a failure of any pump will hurt a storm response.

That was the only pump pulled in the 2013-2014 off season.

Update - 2/5/17

The task order covering the 2013 work on this pump is here, while the repair report showing the damage is here. Much of the damage was from corrosion, as can be seen in this picture of a very rusty Rineer motor:

In fact, the reduction in price for the rebuild of pump E8 at 17th Street was not due to a reduction in scope, but to a warranty claim related to the rusty Rineer. A rebuild of the bearings and seals was performed, along with weld repairs and presumably replacement of the Rineer. No piping was replaced. The Corps clawed back $130,693.20 because "the contractor installed steel plugs in lieu of stainless steel plugs in the previous overhaul which caused corrosion and damaged the seal/accessories." That appears to refer to carbon steel plugs installed on the Rineer motors and left in when the motors are supplied to Conhagen back in 2010. Conhagen apparently forgot to replace them with stainless steel plugs. 

The 2013 repair report doesn't clearly show the plugs on the Rineer motor when it came out. The picture above is the best one, and it doesn't clearly show the plugs. However, Conhagen actually took a picture of the motor being installed on this pump in 2010 and included it in their repair report:

It is impossible to tell if these plugs are stainless or carbon steel, or if they are indeed the final, as-built plugs (other documents and photos show that hex head plugs were used on other Rineer motors). Generally, stainless is shinier than carbon steel, and these appear to be stainless, but one cannot be absolutely sure.

And then late last year...

October 29, 2014: Two pumps at the London Avenue site sprung leaks. They were pulled out and repaired late in the year. The exact identification of the pumps is not known, but it is irrelevant, since all ten of the London Avenue hydraulic pumps underwent the stainless upgrade between 2010 and 2012. This means that the lifespan of the refitted pumps is about 3 years. Given that we are now in 2015, and the refits took place from 2010 to 2012, that means all the pumps - except the ones repaired since 2012 - are likely at the end of their lives due to corrosion. Again.

Update - 1/15/17

The London Avenue pumps that failed in late 2014 were E2 and W5. E2 received a stainless steel upgrade in late 2010, while W5 had gotten the same upgrade in spring 2011. The task order for the 2014 rework on both pumps is here, while the repair report is here. The work performed on both pumps was substantial, requiring repair or replacement of nearly every component of the pumps except for the external coolers. As noted above, this was the third time out of the water for both pumps.

Finally, a sixth pump was pulled for repairs last December. There are no details on which pump this was, not even a spill report. It likely went back in this past February. There do not appear to have been any other pumps pulled, but Corps contracting records on FPDS only extend through the beginning of March, 2015.

Update - 1/15/17

The pump pulled for repairs in December, 2014 was 17th Street pump E1. It got a full refit (except for piping replacement), as documented in the task order here and repair report here. It was back in the water for acceptance testing March 12, 2015. It had received the stainless steel overhaul in the spring of 2010. A notable line in the scope of the 2014 repairs is the explicit callout, "Change out carbon steel plugs with S/S [stainless steel - ed.] in customer furnished motor." This step was missed on the 17th Street pump E8 repair earlier in 2014 that resulted in the warranty reimbursement to the Corps.

Update - 1/28/17

Just three days after the original May 31, 2015 publication of this post, yet another pump sprung a leak - W7 at the 17th Street site. In this case, there was a spill report filed on June 3, 2015 and a cleanup costing $6706.53 was performed by QRI. The pump appears to have come out of the canal about a week and a half later.

It had been refitted in spring 2011, as documented here. The repair report for the 2015 repairs calls out the same cause of failure as the last pump that failed:

"This pump was pulled for hydraulic oil loss. After being pulled, it was found that one of the plugs in the motor had corroded away."

The pump was refit again at an expense of $163,546.47. Everything was repaired or replaced except the stainless piping and coolers, though one stainless line did have to be replaced. Presumably the Rineer motor got stainless plugs this time around. The pump was back in the water and accepted back for service July 23, 2015.

This would turn out to be the only pump pulled in 2015, though two of its neighboring pumps - W8 and W9 at 17th Street - would be pulled the following year. For W9, it would be the fourth time out of the water since 2009.

What's going on with these latest failures? Without the Conhagen repair reports, I can only venture a guess. There's the possibility of galvanic corrosion on the remaining carbon steel components, such as the deeply critical Rineer motors. Stainless steel plus carbon steel plus seawater sets up a battery, which can corrode things even faster than all-carbon steel units. There's also the possibility of inadequate amount of zinc anode protection placed during the refits. The anodes are supposed to corrode preferentially to the pumps, but who knows if that's taking place? The pumps aren't being pulled out preventatively to check such things, only when they are already corroded beyond use. Effectively, the Corps is driving their car with the engine light on.

Update 1/28/17

With the Conhagen reports and the Corps task orders, we know corrosion of the Rineer motors has been at the center of many of the failures, with Conhagen explicitly calling it out in their 2015 repair report of 17th Street pump W7 and the Corps calling it out in the 2014 contract action for repair of 17th Street pump E8. Additionally, the Corps has continued a constant rotation of Rineer motors, issuing a $9365.21 task order in August 2014 for repair of three of them, and then a $12,525.11 task order in February, 2015 for repairs to three more. I don't have the supporting documents for the 2014 repairs, but I did get the purchase order to the Harvey, LA office of Hydradyne for the 2015 repairs.While the repairs did not call for the replacement of the vane cartridges, which are the moving heart of the pumps, there were callouts for seal and bearing and cover replacements. There were also explicit callouts for stainless steel plugs. This points at water ingress and corrosion.

This is of a piece with the historical trend of heavy use of Rineer motors. As noted in this 2012 entry, the Corps rebuilt or purchased 34 Rineer motors between July, 2010 and March, 2012. Tellingly, one of the task orders then explicitly called out replacement of the housing plugs, though they were not called out to be stainless steel:

"All motors: Replace all motor housing plugs with new solid hex head plugs."

But that's really just details. The overall arc of Rineers failing due to corrosion and seal failures - either of the Rineers or the components connected to them - hasn't changed. What I wrote back then still applies:

"The bottom line is that these Rineer motors are obviously ill suited to salt water exposure ... Leaving them where they are is simply asking for big trouble ... I've published a slew of pictures showing Rineers rusting to bits. They've been getting salt water into them through the rusty pipes, the seals, and every other leak path. The Corps has tried replacing them and repairing them. Nothing works. They all get eaten up. These motors are the heart of the pumping system at all three Lakefront closure structures, and they are unreliable and rusting to this day."

So that's it. There does not seem to be any movement on the Corps' part to prepare the interim closure structure pumps for three more years' of service, let alone give any sense of confidence that the pumps furthest out from the stainless refits are not going to crap out this year. Further proof - in addition to the lack of activity at the site - is the fact the Corps' four year pump repair contract with Healtheon expires this year, and a solicitation to replace it has not been issued. [see update below] In fact, of the $6.75 million on that repair contract, over $3 million ($3,068,911.61 to be exact) remains unspent as of March 1, 2015. At the going rate of $170,000 per pump refit, that $3 million could have gotten 18 pumps back into fighting shape for a couple of more years. Instead, the Corps New Orleans District just sat on their asses and didn't do anything proactive for the last three years.

Update - 1/15/17

The Corps had actually extended the Healtheon contract in February of 2015, though it is unclear how long the extension was for. A new 3 year, $3.9 million pump repair contract was issued to J. Star Enterprises on October 8, 2015. To date, at least three pumps - W8 and W9 at 17th Street and E2 at London Avenue  - have been pulled for repairs.

This is beyond disappointing, since we now know the pumps remain ticking time bombs ready to fail at their first real test this year. I don't know how much more proof the Corps needs than at least three of their own pumps - refitted to their own specifications - failing within an average of three years after that refit. That doesn't even count the two remaining pumps at 17th Street (E5 and E7) which remain without the stainless refit entirely, though there exists the possibility the pump pulled out in December 2014 is one of them.

Mitigating this problem perhaps is the additional work performed on the London Avenue and 17th Street canals last year to Really Truly Be Able To Handle 8 Feet Of Water. The ability to have higher canal water levels means there is a small amount of breathing room between what the city sends the Corps' pumps and what the Corps' pumps must pump out. However, when one considers the large gap between what the Corps says (one day before Isaac in 2012: "Orleans Ave [canal] - Max. Water Level is 8.0.") and what they actually do (during Isaac in 2012: "we will discontinue one of the [city's] pumps [feeding the Corps' Orleans Avenue canal pumps] when we hit 3 to 4 foot"), their credibility isn't worth anything. The only thing you can trust is the dollars, and they haven't spent any money getting the outfall canal pumps ready, beyond fixing stuff that they found broken in the most obvious way - an oil slick on the water.

In fact, their lack of concern about the general end-of-life state of the interim closure structures is so huge, that the topmost Corps officials are even joking about it on camera. In WWL's "Eye on Hurricanes" special aired June 1st, Corps New Orleans District commander Richard Hansen compared the existing interim closure structures and pumps to the now-under-construction permanent pumping stations:

"Well I'd say the interim control structure is kinda like its backyard kitchen, okay? It's not fully protected from the weather. It can do its job out there, but the permanent features will be the Cadillac. That will be the feature designed to last for 100 years"

It's wildly inappropriate to joke about how inadequate your own storm protection structures are. In fact, the WWL reporter says in the very next breath:

"So the interim pump structures have been in since 2006 and have actually outlived their lifespan, but the Corps says it will not be a problem for them to remain in place and be working until the permanent pumps can be complete. And right now that's expected by the height of hurricane season in 2017"

"The Corps says" It's unbelievable this still is accepted as truth.

Based on the lack of any preventative maintenance activity, as well as the evidence of refitted pumps failing just two to three years after their refit, I would judge all but a handful of the the outfall hydraulic canal pumps in a stage of imminent failure. The drainage of the city during a major tropical event which includes large amounts of rainfall is now dependent on luck and hope that stuff won't rust through during a storm. That's very, very bad.

Rusty pumps - the summing up

4TH UPDATE, 1/29/17: Unknown at the time of the reporting on the 2012 Orleans Avenue pump failures was the fact that two other pumps had failed earlier in 2012 and had been out of the water during hurricane season: pump W5 at the London Avenue canal (repair work between April 24 and June 15) and pump W3 at the 17th Street canal (repair work between April 25 and July 17). What is remarkable about those failures is that both pumps had been refit less than a year earlier (entry about London Avenue pump W5 refit from April, 2011 to May, 2011 here, entry about 17th Street pump W3 refit from November, 2011 to January, 2012 here). In fact, the 17th Street pump failed after just three months in the water. Among the components replaced were the Rineer motors on both pumps.

The London Avenue W5 task order is here and the repair report is here. The 17th Street W3 task order is here and the repair report is here. There is a severe discrepancy in the dates that Conhagen started work and the dates the contract actions were issued for both these pumps. That is, the contract actions were issued 58 days (London W5) and 73 days (17th W3) after their respective starts of work. In both cases, the work was done before the contract actions were issued. This kind of delay is atypical but not unprecedented. In 2011, Conhagen started work over a month before issuance of two separate task orders for the unscheduled repair of pumps pulled out during hurricane season (17th Street W8 was pulled in May and Orleans Ave E3 was pulled in July). And earlier that year, a scheduled repair of four pumps in January and February (17th Street W4 and W7 and London Ave E1 and E5) had its task order issued 25 days after work commenced. The explanation for the discrepancies is unknown, but it is notable that four of the five occurred during hurricane season, when public attention on the pumps would be more focused. 

And while the explanation of the delays is unknown, the consequences are not. In combination with the 90 day hold on release of Defense Department contacting actions on the government's public contracting database, FPDS-NG, the public would have no indication that pumps were being pulled out during hurricane season until long after the pumps were back in the water.

For the record, all the rest of the pump repair actions from 2009 to 2015 had task orders issued less than 19 days after the start of Conhagen's work, and most were within a week.

3RD UPDATE, 1/21/17: Repair reports and task orders lend a little more clarity, but not much. The task order for Orleans E5 is here, and its' Conhagen repair report is here. The task order for Orleans E4 was missed in the Corps' FOIA response, but the repair report is here. The documents reveal Conhagen's work on the first failed pump - E5 - began July 12, 2012, almost three weeks before the Corps issued the task order for the work. It is unclear what that work would have been, since the contemporaneous Lens article mentions E5 remaining in the canal until August. A wait for parts is a possible explanation.

Both pumps were refit with all new stainless steel piping and oil coolers, along with new bearings, seals, Rineer motors, hose connections and everything else that suffered corrosion, which was pretty much everything but the impellers, as usual. They were accepted for service after being reinstalled on September 21, 2012. Their absence contributed to the Corps repeatedly refusing to allow the city to fully pump Isaac's rainfall into the Orleans Avenue canal on August 29th and 30th, as detailed in this 2012 post and this 2013 post.

2ND UPDATE, 8/28/12: Despite Corps protests earlier (from the Lens article: "Accardo said that the Orleans Canal has the highest-rated water level of the three. 'We’ve never had to close the (Orleans Avenue) gates during a storm. Not even during Ike, not even during Gustav,' he said. 'So the threat of these pumps being down and causing any problems is extremely remote. There is hardly any threat to City Park or Mid-City.'"), the gates at Orleans Avenue were dropped at 11:00 PM August 27th. So we will know exactly what the effect of missing pumps E4 and E5 is.

It is extremely troubling that the Orleans gates dropped (London and 17th were also closed). It says the Corps has zero confidence in the tens of millions of dollars they spend just last year on remediating all three canals to an 8 foot safe water elevation along their entire length. There's more details on those remediation efforts here, here, here, and here, including the copious amounts of debris found, left, or deliberately placed in their levees by the Corps.

UPDATE, 8/28/12 - Details regarding the two pumps at the Orleans Avenue site which were not pulled out before this hurricane season, but which unsurprisingly sprung oil leaks and had to be yanked, can be found in this August 27, 2012 article in The Lens. Apparently, the two pumps - E4 and E5 - also both suffered a hydraulic system failure which is extending their outage weeks beyond the normal time for corrosion repairs. The first pump - E5 - was leaking on July 2, 2012, but the Corps won't be putting it and E4 back in until mid-September at the earliest. E5 sat in the canal disconnected from its onshore engine skid for over a month before it got pulled out on August 7 August 6 (see update 3 below), the same day E4 came out. I can only assume E5 sat there for so long because the "hydraulic failure" was also identified during July, and there was no point doing the corrosion repairs if the hydraulic power unit to the pump wasn't going to work. The long delay points to the possibility of a long wait for parts, but that is merely speculation.

There's lots of matters the Corps of Engineers' New Orleans District would rather the public didn't know about, or at least pay attention to. Many times, it's the details of things. They say on June 1, 2011 they've met their goal of hundred year protection, but you find out there's dozens of unfinished projects. They point to how great the West Closure Complex is that same month, but then you learn there were only 8 of 11 pumps ready at the beginning of the 2011 hurricane season and an early season storm could have forced local pump stations to shut down. They say the levee system is better than ever, and then you find out it's filled with concrete, wood, steel, and other flotsam. They say the London Avenue canal has to be closed at the ridiculously low lake level of 2.5 feet until the walls get fixed, and then you find out the walls leaked at levels even lower than that four times in 2011.
So it goes with the 54 hydraulic pumps installed at the three outfall canal sites in New Orleans - 32 at 17th Street, 10 at Orleans Avenue and 12 at London Avenue. 40 of them are the 60" type, while the other 14 - all installed at 17th Street - are the smaller 42" version (this post focuses on the 60" pumps, due to a severe lack of contracting information for the repairs to the 42" pumps). The Corps would rather the public didn't bother with such details as whether the pumps work under storm conditions, or if they have crippling design flaws, or if the salt water in Lake Pontchartrain is turning them into rusty heaps. That's because they know if the public were privy to all those pesky details, they might freak out over how scary the story is.

Thank goodness for the Freedom of Information Act, though, because it allows us to push past the well financed Public Affairs curtain and see the entire rusty pump story for what it is: a fiasco six years in the making.

The first signs - summer, 2006

The problems started nearly instantly after the pumps first went in the canals in June, 2006, with corrosion appearing on the hose connections from the hard piping to the pumps. Hydraulic oil is driven from hydraulic power units onshore through the piping and hoses to the pumps which sit in the canals. Here's a picture of those rusty hose connections: (via SCPR Flickr):


And here's a closeup of one of those rusty hose connections (via SCPR Flickr):


This was a hint that the environment around the waters of Lake Pontchartrain was very salty, and materials susceptible to corrosion - such as the carbon steel making up the pumps - would perform quite poorly. This shouldn't have surprised anyone, since Lake Pontchartrain is the second largest salt water body in the country.

Missing the big picture - 2006 through 2008

But over the next few months, the Corps and its paid consultants (many of which were former Corps employees themselves) would not look at the systemwide problem augered by that early rust, but would instead focus narrowly on a tiny sliver of the corrosion problem. They concentrated on the hose connections to the pumps which sat beneath or near the waterline as the pumps rested in the canals (these hose connections were at the opposite end of the hoses from those shown in the June, 2006 photos above):


Remarkably, their "solution" to preventing the rusting of those connections in early 2007 was the addition of more carbon steel to the pumps, in the form of piping extensions:


This raised the lower hose connections above the waterline, but also had the effect of giving much more surface area for rust and marine organisms to attack. It was like discovering your car's engine was on fire, and deciding to check the spark plugs - while wearing a suit covered in gasoline!

With all that yummy carbon steel waiting like a buffet, the marine organisms in the Lake Pontchartrain water started chomping away at the coal tar epoxy paint, exposing the bare metal below in no time. Corrosion of the steel quickly followed. An oil spill from pump E5 at the 17th Street site on June 26, 2008 was a hint of things to come. The Corps paid over $10,000 to clean it up, but never bothered to report it to the federal or state authorities.

A month later a solicitation for comprehensive pump maintenance for pumps across all three sites was issued. It was a small business Section 8(a) set aside, which is a rather odd choice for such a critical function as maintenance of pumps upon which the residents of the City of New Orleans relied for protection of life and property. Nonetheless, the contract was awarded August 28, 2008 to small business Healtheon. However, pump repair task orders would not be issued against the contract for nearly a year.

Early self-repair efforts prove inadequate - Spring, 2009

By early 2009, the hydraulic oil which powered the pumps and which was contained within the pipes was seeping regularly from multiple pumps, and sometimes gushing into the canals. A halting repair effort began in March, at first concentrating only on pumps with known leaks.

The Corps attempted to repair them on their own, sometimes going to interesting lengths to conceal their efforts from the public. But after six pumps had to be pulled out in three months, accompanied by five oil spills (four of which went unreported) costing over $30,000 to clean up, it was clear professional help was needed.

Professionals are brought in - summer, 2009

Repair task orders started going out on the Healtheon contract in July, 2009. Healtheon hired Conhagen, Inc of Kenner, LA to perform the repairs. Conhagen had been performing similar services for the New Orleans Sewerage and Water Board for many years, and indeed it is unclear why the Corps simply didn't hire them directly.

The first task order, on July 21, 2009, called for the partial repair of four 60" pumps at the 17th Street site. New and more plentiful zinc anodes would be attached to the pumps to fight future corrosion, and bits of the pump housings eaten away by rust would be weld repaired.

Also, all the carbon steel piping on the pumps would be removed and replaced. However, only some of it - the piping inside the pump bore - would be upgraded to more corrosion resistant 316L stainless steel. The piping and oil coolers on the outside of the pumps, which sit in the same salty water as that inside the pumps, would remain carbon steel. Critical moving parts inside the pumps, like bearings and the Rineer hydraulic motors, would also not be replaced on a comprehensive basis. The pumps the Corps had pulled themselves earlier in 2009 also received this rather limited set of repairs, including two other 60" pumps. In sum, this work only addressed about a third of the corrosion damage with upgrades, replacing the rest of the rusted or damaged pieces in kind with exactly the same materials.

Fortunately, one of the contractual requirements was generation of reports by the contractor on what they found on these pumps when they came out of the water. The pictures in those reports were disturbing:


Here's a section of pipe removed from the pump sitting on the table at Conhagen (the fitting at the bottom indicates it may be from the inlet or outlet of one of the hydraulic coolers)


It appears that a large chunk of the pipe is completely gone in this photo. Also note the rusty hose fitting on the table:


Here's one of the oil coolers mounted to the sides of the pumps, with a generous coating of marine growth:


These coolers are meant to lower the temperature of still-hot hydraulic oil passing through after leaving the Rineer hydraulic motor but before returning to the reservoir on the engine skid back on the landside of the canal. The residual heat is supposed to be transferred to the canal water, but such severe fouling would undoubtedly hamper that from happening.

Here's some of the piping extensions installed over the winter of 2006-07, specifically the return piping (two larger lines) and the smaller case drain line:




It was clear there was an urgent problem facing the Corps - systemwide corrosion of all 54 pumps within two years of their acceptance for service. Indeed, Conhagen said so in their summary of the repairs, terming the corrosion-related failure of the Corps' other hydraulic pumps "imminent." However, the Corps continued on a rather slow pace of repairs, only contracting with a single repair shop and never sending more than four or five pumps out at any given time, even outside of hurricane season.

Repair scope widens - fall, 2009 through spring, 2010

Indeed, because of the Corps' slowness to respond - despite the dire warning from Conhagen - nearly seven months went by before any more pump repairs took place. Most of that time was spent by the Corps New Orleans District's Operations and Contracting groups preparing an underfunded, limited-scope, sole-source small business set-aside contract for Healtheon, the second such contract they received. This second contract would streamline the repair process, but it would also strangle its funding. Only enough money for 17 pump repairs was set aside, when there should have been enough for 40 such repairs. The contract was issued for $2.875 million, just below the $3 million threshold that would have triggered a requirement to place the contract out for public bids. So precious months outside of hurricane season were wasted in preparing a contract that seemingly had no real use but to reward a favored small business.

In February, 2010, a task order for further repairs was finally issued (still on the first Healtheon contract; the second contract would not be finalized until the following month), calling for two more pumps to be pulled out at the 17th Street site. They would receive a greatly enhanced set of repairs compared to the pumps fixed up in 2009. All the carbon steel piping - inside and out - would be replaced with stainless steel. Rotating parts like the Rineer motors and the bearings would also be replaced. Seals would be replaced, and new gland plates holding those seals would be manufactured of sturdier materials. All fasteners would also be upgraded to stainless steel. Essentially, the pumps would be completely rebuilt, inside and out.

It's a good thing the Corps finally expanded their focus to include the external piping, because here's one of the oil coolers on those first two pumps pulled in 2010:


It's also a good thing they started systematically replacing bearings, seals, and Rineer motors, because they were taking serious damage. Here's one of the Rineers from that same pair of pumps:


And here is the area around the shaft and bearings, showing obvious evidence of water infiltration:


From the amount of rust inside the bearing housing, as well as the significant discoloration along the shaft, it is clear the seals completely failed. This would have exposed the bearings and shaft to salt water, causing failure of the pump. And there is no doubt similar damage had been done to the seals on the Rineer motor, damaging it as well. Indeed, as we would discover later, the Rineers were in just as much jeopardy as the rest of the pump (see text below in section on 2011-2012 off season).

The pace of repairs would finally increase during the spring of 2010, with eight more 60" pumps going to Conhagen's shop for the upgraded rebuilds spread over two additional task orders, the first task orders issued on the second Healtheon contract. The pictures from the repair reports continued to show extensive corrosion of all pumps. Rust covered the outside of the pumps, such as 17th Street pump E2...


...as well as the interior, as seen inside London Avenue pump E4:


As with 17th Street pumps W5 and W6 earlier in 2010, there was also evidence of water infiltration into the parts of the mechanism that were supposed to be sealed. Here's part of the drive of 17th Street pump E6 showing corrosion on surfaces supposedly protected by seals:


This should not be showing any signs of rust, but there they are.

There was substantial corrosion damage to the main pump housings as well. At the very bottom of the pump is the suction bell, the tapered portion of the pump which funnels water up to the impeller. The funnelling effect is assisted by tapered vanes. The suction bell is always submerged, and the brackish water took its toll. Here's 17th Street pump E2's suction bell:



and 17th Street pump E6's suction bell:



Conhagen addressed all these areas of concern in their repairs that spring.

Work slows but corrosion continues - summer and fall 2010

As the 2010 hurricane season dawned on June 1, there were still 30 of the 60" pumps in the water with either partial (6 of them) or no repairs to arrest the ongoing corrosion damage, and the Corps had already spent half of the $2.875 million on the second contract repairing just 8 pumps. Fixing the other 30 would take at least $4.8 million, meaning a shortfall of at least $3 million. The effects of that lassitude would be seen immediately.

The next day, June 2, 2010, pump W1 at the 17th Street site would spring a leak, dumping 15 gallons of hydraulic fluid in the canal. It would be pulled out two weeks later and sent to Conhagen's shop for a rebuild. The leak should not have been a surprise in light of the extreme corrosion found on the pump's external piping:



The next month, pump W1 at London Avenue came out, and pump W4 at Orleans Avenue was sent to the shop in October, both likely after they leaked oil into Lake Pontchartrain. Both showed typical corrosion damage. For example, the Rineer motor in Orleans pump W4 looked particularly rusty:


While there were no spill reports for these two pumps, I bet the Corps pulled them out because they found oil in the water and didn't report it. There's no other reason for them to pull single pumps during hurricane season.

These three were the only 60" pump repairs during the 2010 hurricane season, since the Corps shut down their scheduled repair effort when the season started in June, another sign of a lack of urgency. In this case, it was brought about by another poor Corps decision, one made in 2006 and 2007 during the construction of the interim closure sturctures. The Corps did not install any spare pumping capacity at the structures, meaning they are unable to repair pumps while providing the required ability to drain the canals when the gates are closed against a storm surge. They duplicated this error at the massive West Closure Complex.

Repairs restart in earnest - fall, 2010 through spring, 2011

After the end of the 2010 hurricane season, more scheduled repairs with the enhanced scope were tasked to Healtheon under their second contract. Two pumps were pulled out at the London Avenue site in December, an event accompanied by not one, but two oil spills, totalling 125 gallons. Again, there was major corrosion found on the external piping:


With the Corps approaching the cost ceiling on their second pump repair contract, they solicited for a third one in October, 2010 and awarded it in February, 2011. This time, since the value was well above $3 million, it had to be put out for public bid. Yet again, Healtheon/Conhagen won. The cost ceiling was $6.75 million, over double the potential value of the prior contract. This should be more than enough to finish off the repairs (they average about $160,000 per pump) and have about $3 million left over until the permanent pumps come online in 2016. Why did it take them over two years to figure out they needed more cash for these repairs? Well, they knew exactly how much cash they needed - internal Corps emails show that conclusively. They held off because they were too interested in proceeding slowly and keeping the business solely with Healtheon and Conhagen, rather than putting a contract out for bid that would have repaired all the pumps in a single hurricane off-season, utilizing multiple repair shops around the country. In other words, they were more interested in doing business the Corps New Orleans way rather than protecting citizens from the failure of their rust heaps they claimed were pumps.

This new contract was put to use immediately in early 2011, with eight more pumps getting pulled out under two task orders on it in the spring of 2011. Again, the pictures in the Conhagen reports demonstrated these pumps should have been repaired years earlier:





And while these sorts of pictures and repairs were becoming depressingly typical, there was a twist in one of those spring 2011 task orders.

One of the pumps pulled out in March, 2011 was E1 at the London Avenue site. When the pictures from the repair report for that pump were examined, it was revealed that London Avenue E1 had already been out for repairs two years earlier, in the spring of 2009:


This showed the Corps did not have confidence in the 2009 partial repairs, and considered the pumps fixed that year as just as vulnerable to corrosion-related failure as those with no repairs. That would be proven just months later.

Earlier repairs prove inadequate - summer, 2011

As in 2010, the Corps shut down their scheduled pump repairs just before the beginning of the 2011 hurricane season. At that point 17 of the forty 60" pumps either had partial or no repairs. And just as in 2010, some of those pumps started spewing oil nearly immediately.

It is tough to nail down the exact date due to lack of spill reporting, but sometime in May or June, 2011, pump W8 at the 17th Street site was leaking oil. It got yanked and sent to Conhagen's shop for a rebuild. W8 was a pump that had been part of the incomplete 2009 repairs, proving those repairs were effectively money flushed down the drain. While the parts of the pumps which received upgrades in 2009, like the interior piping, appeared to do well...


...the other components which were replaced in kind in 2009, like the oil coolers made of carbon steel, didn't:


The total expense for both times this pump was pulled out would come to approximately $227,000. The average expense for these pump rebuildings has been around $160,000, meaning about $67,000 was wasted on this pump, though I doubt Healtheon and Conhagen saw it that way.

This story would be repeated just a month later, as pump W9 at 17th Street would also start leaking (the Corps reported that spill) and get sent to Conhagen for a rebuild. It too was a 2009 partial repair. Three more 60" pumps "fixed" in 2009 remain in service today at 17th Street, along with an unknown number of 42" pumps.

In August, a third pump would start seeping oil, this time at the Orleans Avenue canal site. Until that time, only a single pump had been pulled out of the Orleans Avenue site, as the Corps valued the pumps at the 17th Street and London Avenue sites more; according to their "safe water level" criteria, those two sites required gate closures at lower water levels, so were more likely to be forced to use their pumps. Apparently no one told the Corps' New Orleans Operations group that the Orleans Avenue walls and levees were just as flawed as those along the other two canals, and that the Orleans Avenue gate closure criteria should have been just as stringently low as that for the other two canals, a fact not truly confirmed until the 2010 release of the Orleans Avenue canal safe water elevation report. Additionally, the junk found in the levees along all three canals in 2011 makes one wonder why the Corps doesn't just keep the gates closed all the time.

Emphasis moves to Orleans Avenue - the 2011-12 off season

The 2011-12 off season would be the first since 2006-07 when the Corps would have contracts and funding in place to do as many repairs as they wished. 14 pumps, including 3 at 17th Street with prior repairs now proven to be useless, remained to be pulled and refitted. 8 of the pumps remaining were at the Orleans Avenue site, and that is indeed where the emphasis would be with the 2011-12 off season task orders. The first task order pulled the two pumps remaining at 17th Street which had never been touched, as well as two from the Orleans Avenue site.

All showed the now-standard signs of corrosion damage, like rusty pipes inside the pump bore:

external coolers encrusted with marine growth:

and deep corrosion of internal parts like shafts which were supposed to be sealed from the elements:

In addition to the four pump repairs in late 2011, the last unrepaired pump at the London Avenue site was refit during the first months of 2012, followed by four more pumps from the Orleans Avenue site. This left a total of 5 pumps very, very vulnerable to salt water on the eve of the 2012 hurricane season. Two were at the Orleans Avenue site, while the other 3 were at 17th Street. All three of those 17th Street pumps had partial, inadequate repairs performed on them in 2009. The Corps entered the 2012 hurricane season waiting for these 5 pumps to start leaking so they could get pulled out.

Both of the unrepaired Orleans Avenue pumps did in fact start leaking this summer. On July 2, 2012, E5 was found to be leaking and remained at the site in a disconnected state for over a month, out of service. The day it was finally pulled - August 7, 2012, E4 was also discovered to be leaking as well and is currently out of service. This leaves the entire Orleans Avenue system at a pumping level below what can be placed in the canal by the Sewerage and Water Board pumps at Drainage Pump Station #7, at the south end of the canal.

Why didn't they proactively refit pumps they know are more rust than steel? Because of their insistence on a single repair shop refit strategy that rewards favored businesses like Healtheon and Conhagen over the safety of citizens. If they were willing to entertain the work of other shops, all this work could have been completed over two years ago. That is, they chose to not keep their life safety equipment in working order.

Where we stand - hurricane season 2012

As of June 1, 2012, the Corps will have rebuilt 35 of the forty 60" hydraulic pumps over a span of about three years (three of them twice), leaving 5 of them effectively in jeopardy of failure during a storm due to leakage from piping of hydraulic oil, which is used to power the pumps. Here's the individual statuses at each site as of the most recent confirmed repairs:

Remember they were aware of this problem within a month of installing the pumps in 2006 but only started addressing it in 2009. And a very long three years after that, 12% of the large pumps (5 out of 40) remain in imminent danger of failure when called upon in a storm because of corrosion. Along the way there have been over a dozen oil spills totalling hundreds of gallons.

However, the news is even worse than it appears. The hydraulic pumps have at their center hydraulic motors made by Rineer, Inc. out of San Antonio, Texas. In parallel with the overall pump refits, the Corps has been switching out, repairing, and refitting the Rineers. However, due to the Rineers' materials of construction, they remain as vulnerable to corrosion as when the original ones went in the brackish water in mid-2006, as seen in these November, 2011 pictures of the drive shafts at the bottom of two Rineers from the 17th Street site:

The Corps can gussy up the pipes and fittings leading to and from the Rineers all they want, but unless they switch out the Rineers themselves for sturdier, more rust- and leak-resistant technology, the pumps are going to remain a total crapshoot for years to come. Put another way, each pump requires a heart transplant, and all the Corps has been doing is putting arterial grafts in. So one should not take any solace from those green numbers in the pictures above, since they only refer to ancillary parts of the pumps; the core of the pumps - the Rineers - remain unreliable due to saltwater corrosion.

Then there's the dollars. The Corps have spent over $5.9 million on these repairs, and that's just on the 60" pumps. There's likely another $900,000 to be spent rebuilding the remaining five 60" pumps. That's on top of the $37.8 million they spent originally on those pumps buying them and trying to make them work in 2006 and 2007, which included a $4.7 million delivery incentive.

So the total spent on each rebuilt 60" pump is $1.1 million on average. The original contract price for each pump - before all the modifications on the original contract and all the repairs over the last two years - was $750,000. Thus - on average - the Corps will have spent, once the repairs are done, nearly an additional 50% of the original purchase price since 2006 on each of the 60" pumps.

I mention this despite my earlier criticism of underfunding the repair effort. I mention it because dealing with saltwater corrosion is basic engineering that should have been taken care of at step zero in late 2005 and early 2006, when the Corps was copying and pasting the specifications for the pumps from the manufacturer's catalog. The material of construction is one of the first questions that gets answered on any project like this. Except in this case, it looks like it was just ignored, and now they're having to spend millions more dollars to get a pump that is kinda sorta corrosion resistant.

That being said, when they realized the problem (which would have been a couple of weeks after installation, as noted above), they should have gone ahead and spent the money quickly and lavishly to correct it completely. But, as with the original mistake, they failed to do that as well. The impression one gets is that no one at the Corps gives a damn about these pumps or whether they will work when needed.

In sum, this repair "effort" has seen zero urgency, inadequate funding, excessive deference to contractors at the expense of public safety, hundreds of gallons of oil spilled into Lake Pontchartrain, and months of wasted work in 2009 which must now be redone at extra expense. From beginning to end, the Corps has shown a deep reluctance to do the right thing: yank all the pumps out and send them to wherever they need to go to be repaired. Once they started fixing them in 2009, everything should have been done before June 1, 2010. Instead, at the lazy pace they've set, all the pumps will not be rebuilt until just before the 2013 hurricane season, about a year from now and seven years after the initial identification of corrosion as a serious problem. This is unacceptable, but disappointingly all too expected.

I will continue to follow this story.

Many but not all of the prior entries in this series are linked throughout the report above. For convenience, here's the complete set:

Imminent, originally posted May 13, 2010, covers the halting first repairs in 2009

How did the pumps get from..., originally posted May 27, 2010, covers the original 2006-07 decisions on corrosion

This year's scramble, originally posted June 3, 2010, covers the spring 2010 repairs

Worse than previously known, originally posted June 11, 2010, also covers the spring 2010 repairs

Corps of oil, Part 1, originally posted June 20, 2010, covers oil spills from 2006 and early 2007.

Corps of oil, Part 2, originally posted June 21, 2010, covers oil spills in later 2007

Corps of oil, Part 3, originally posted June 22, 2010, covers oil spills from 2008 through 2010

No urgency, originally posted August 18, 2010, reports on repairs during summer, 2010

Quick update (with pretty new trees!), originally posted August 30, 2010, gives a summary of corrosion repair events to date at the fifth anniversary of Katrina, as well as revealing the Corps' wasteful spending on tree planting around the closure sites

The latest on lakefront pump repairs, originally posted March 23, 2011, looks at the repairs from summer 2010 through early 2011

The 2011 pump rebuild scramble, originally posted May 9, 2011, covers the spring, 2011 repairs

Brief update on London Avenue pumps, originally posted June 24, 2011, also covers the spring, 2011 repairs

Finally, originally posted July 13, 2011, provides an update on receipt of FOIA-requested documents, all of which were eventually transmitted by the Corps

No surprise, originally posted July 15, 2011, looks at the summer, 2011 re-repairs of pumps also pulled out in 2009

No surprise, Part 2, originally posted July 20, 2011, covers possible contamination of the pumps' hydraulic fluid with water

2011-12 rusty pump update, originally posted July 9, 2012, chronicles repairs to ten 60" pumps between August, 2011 and March, 2012

Birth of a no bid contract, originally posted July 28, 2012, uses internal Corps emails from November and December, 2009 to tell the story of how Healtheon's second contract was created in the 2009-10 hurricane off season