Rusty pumps keep rusting, Corps yawns

This post has been updated in January and February, 2017 with extra detail around pumps that were pulled and repaired in 2013 through 2016. Updates are in bold.

With the 2015 hurricane season upon us, and the continuously rusting pumps at the three interim closure structures (17th Street, Orleans Avenue, and London Avenue) well beyond their predicted life by any reasonable measure, I thought it would be time to check back in and see if the Corps has stepped up - or actually done any - preventative maintenance on them. For an overview of the rusty pumps saga from 2006 through 2012, see this wrap up post.

The pumps were put into the water in 2006, pulled out, and then put in for good in 2007. They were not expected to be in service more than a couple of years before they would get replaced by sturdier, bigger pumps. But planning for their replacement dragged on for years. The Corps, starting in 2009, stated the life of the entire structures - slyly without specifying the exact life of the pumps themselves - was "five to seven years." They said maintaining the pumps beyond 2013 would be difficult. They did this mostly to bring pressure to bear on locals to approve their permanent pumping scheme. But taking them at their word, if we started the lifespan clock in even 2007 - generous given construction of the structures began January of 2006 - we are now beyond even the worst case for the structures' and pumps' lives. One would have expected to see some effort toward life extension on these pumps.

And we did for a little while, with the replacement of many carbon steel components on the pumps with stainless steel between 2010 and 2012 (Note: Two pumps - E5 and E7 at the 17th Street site - remain untouched by those refit efforts). During that 2010-2012 period, the pumps were not expected to be needed past the 2014 storm season. However, in the intervening time the end date of the pumps has been extended even further into the future. Now they are expected to be in service until 2017, or 11 years after they first saw brackish lakewater. So what has been happening with the pumps since my last update during the 2012 hurricane season? Has the Corps done anything to make them last the additional three years?

In a word, no. Since 2012, the Corps has pulled six pumps across the three sites, almost all in reaction to oil leaks. There does not appear to have been any proactive moves to ensure the pumps can last until 2017. Here's what has happened:

2012-2013 off season: no pumps pulled.

May 20, 2013: Pump W10 at the 17th Street site sprung a hydraulic oil leak. The pump was pulled out a couple of weeks after the oil leak, just as the 2013 hurricane season was starting. It likely went back in the water some time in August, 2013 July 19, 2013 [see update below]. This failure was not a surprise, since W10 had not received the stainless steel upgrade in 2010-2012, but had been part of a group of six pumps given inferior carbon steel replacement repairs during a misbegotten effort in 2009.

Update, 2/5/17

The contract task order for this work is here, while the repair report is here. They confirm that this pump received stainless steel piping and coolers during this 2013 refit, which was essentially a complete rebuilding of the pump. Here's a picture of a severely corroded Rineer hydraulic motor, the device at the heart of pump, being pulled out:

July 2, 2013: Pump W9 at the 17th Street site sprung a leak. This failure is far, far more concerning. In fact, it's a canary in the coal mine. That's because W9 had been upgraded with the stainless steel [see update below] suite of repairs during the summer of 2011, just two years earlier. Now in 2013 it was leaking once again, causing a third round of repairs (W9 also got the 2009 carbon steel repairs).

This is troubling in the extreme. It means the stainless steel upgrades are all suspect, which makes the integrity of all the pumps suspect. This would be confirmed just a few months later on...

Update 2/5/17

The story on this pump is somewhat complicated. The pump had been part of the original set of half repairs in 2009, when only its internal carbon steel piping was replaced with stainless steel. The external oil coolers remained carbon steel. A picture of the new stainless internal piping appeared in the 2009 report:

Then in 2011 the pump had to be pulled out on an emergency basis when it started leaking during hurricane season, on July 5. Conhagen's work was very fast, from July 5th to July 15th that year. In fact, the task order for the work explicitly mentions overtime being required:

"Remove pump # 9W at 17th St. Canal to inspect, recommend what need to be repaired and reinstall it.

Note:
1. Pump # 9W was overhauled in Aug 2009 and is currently leaked [sic] outside of the pump.
2. After the contractor inspects and finds out what the pump problems are, we will provide Contracting Division a separate scope of work for repair.
3. Estimated date to start is 7/8/ 2011 and overtime for the contractors to work on Saturday 7/9/11 and Sunday 7/10/11 should be authorized."

Part of those repairs was replacement of rusted out oil coolers, seen here in a "before" picture from the 2011 report:

However, a picture of the internal piping at the completion of Conhagen's repairs from the same report appears to show the 2009-installed internal piping was reused. New stainless steel piping would have been shiny. Instead, the piping is black (a process known as passivation):

This is backed up by the scope of work in the 2011 report, which makes no mention of replacement of the internal piping, only the oil coolers. The reason for the piping reuse is likely time. Pump W9 was only out of the water for about a week in 2011, and the piping probably looked fine at that time. So rather than replace it preventively, it was left in place.
 
Then came the third removal in 2013. The contract task order for this work is here, while the repair report is here. Rust was again the cause for failure:

Note the spots of rust on the Rineer itself.

The pump was completely overhauled, including new stainless steel piping inside:

and outside:

The pump was accepted back into service during the height of hurricane season, on August 23, 2013.

So the removal in 2013 was an indication of the fragility of the repairs, but also one that demonstrated a lack of forethought. If the Corps had a complete spare pump assembly ready to go in 2011, they would not have had to rush those repairs. They could have simply replaced the entire pump and worked on it in a more reasonable timeframe, and then placed it back into service with a complete suite of repairs.

Not that any of that really matters, because the complete suite of repairs done in 2013 would prove inadequate three years later in July, 2016, when the pump would again get pulled. In fact, 17th Street pump W9 holds the record for most repairs to date since 2009, with four. It's neighbor, W8, is in a three way tie for second with three pulls. The other three time failures are London Avenue pumps E2 and W5. New Orleans is protected by phalanxes of Edsels and Pintos.

December 19, 2013: Pump E8 at the 17th Street site sprung a leak. It was pulled out for repairs in January of 2014 and went back in the next month. The canary started chirping louder, since E8 had also received the stainless steel upgrade, in this case three years earlier, during the spring of 2010. While we don't know the exact problem in 2013, there is some indication the pump did not need a complete overhaul. The original task order for the 2013 E8 repair was for $172,934.74, but a modification subtracted $130,693.20 of that. Full pump overhauls usually cost around $160,000 to $170,000. If there was only $42,000 of work done on the pump - and considering some of that was for crane rentals on both ends of the repair - it makes sense to think the problem was relatively small but critical. There's really only a few possibilities: a) a piping failure, b) a seal failure, or c) a failure of the Rineer hydraulic motor. [See update below] But it doesn't really matter, because there is no spare pumping capacity built into the interim closure structures, so a failure of any pump will hurt a storm response.

That was the only pump pulled in the 2013-2014 off season.

Update - 2/5/17

The task order covering the 2013 work on this pump is here, while the repair report showing the damage is here. Much of the damage was from corrosion, as can be seen in this picture of a very rusty Rineer motor:

In fact, the reduction in price for the rebuild of pump E8 at 17th Street was not due to a reduction in scope, but to a warranty claim related to the rusty Rineer. A rebuild of the bearings and seals was performed, along with weld repairs and presumably replacement of the Rineer. No piping was replaced. The Corps clawed back $130,693.20 because "the contractor installed steel plugs in lieu of stainless steel plugs in the previous overhaul which caused corrosion and damaged the seal/accessories." That appears to refer to carbon steel plugs installed on the Rineer motors and left in when the motors are supplied to Conhagen back in 2010. Conhagen apparently forgot to replace them with stainless steel plugs. 

The 2013 repair report doesn't clearly show the plugs on the Rineer motor when it came out. The picture above is the best one, and it doesn't clearly show the plugs. However, Conhagen actually took a picture of the motor being installed on this pump in 2010 and included it in their repair report:

It is impossible to tell if these plugs are stainless or carbon steel, or if they are indeed the final, as-built plugs (other documents and photos show that hex head plugs were used on other Rineer motors). Generally, stainless is shinier than carbon steel, and these appear to be stainless, but one cannot be absolutely sure.

And then late last year...

October 29, 2014: Two pumps at the London Avenue site sprung leaks. They were pulled out and repaired late in the year. The exact identification of the pumps is not known, but it is irrelevant, since all ten of the London Avenue hydraulic pumps underwent the stainless upgrade between 2010 and 2012. This means that the lifespan of the refitted pumps is about 3 years. Given that we are now in 2015, and the refits took place from 2010 to 2012, that means all the pumps - except the ones repaired since 2012 - are likely at the end of their lives due to corrosion. Again.

Update - 1/15/17

The London Avenue pumps that failed in late 2014 were E2 and W5. E2 received a stainless steel upgrade in late 2010, while W5 had gotten the same upgrade in spring 2011. The task order for the 2014 rework on both pumps is here, while the repair report is here. The work performed on both pumps was substantial, requiring repair or replacement of nearly every component of the pumps except for the external coolers. As noted above, this was the third time out of the water for both pumps.

Finally, a sixth pump was pulled for repairs last December. There are no details on which pump this was, not even a spill report. It likely went back in this past February. There do not appear to have been any other pumps pulled, but Corps contracting records on FPDS only extend through the beginning of March, 2015.

Update - 1/15/17

The pump pulled for repairs in December, 2014 was 17th Street pump E1. It got a full refit (except for piping replacement), as documented in the task order here and repair report here. It was back in the water for acceptance testing March 12, 2015. It had received the stainless steel overhaul in the spring of 2010. A notable line in the scope of the 2014 repairs is the explicit callout, "Change out carbon steel plugs with S/S [stainless steel - ed.] in customer furnished motor." This step was missed on the 17th Street pump E8 repair earlier in 2014 that resulted in the warranty reimbursement to the Corps.

Update - 1/28/17

Just three days after the original May 31, 2015 publication of this post, yet another pump sprung a leak - W7 at the 17th Street site. In this case, there was a spill report filed on June 3, 2015 and a cleanup costing $6706.53 was performed by QRI. The pump appears to have come out of the canal about a week and a half later.

It had been refitted in spring 2011, as documented here. The repair report for the 2015 repairs calls out the same cause of failure as the last pump that failed:

"This pump was pulled for hydraulic oil loss. After being pulled, it was found that one of the plugs in the motor had corroded away."

The pump was refit again at an expense of $163,546.47. Everything was repaired or replaced except the stainless piping and coolers, though one stainless line did have to be replaced. Presumably the Rineer motor got stainless plugs this time around. The pump was back in the water and accepted back for service July 23, 2015.

This would turn out to be the only pump pulled in 2015, though two of its neighboring pumps - W8 and W9 at 17th Street - would be pulled the following year. For W9, it would be the fourth time out of the water since 2009.

What's going on with these latest failures? Without the Conhagen repair reports, I can only venture a guess. There's the possibility of galvanic corrosion on the remaining carbon steel components, such as the deeply critical Rineer motors. Stainless steel plus carbon steel plus seawater sets up a battery, which can corrode things even faster than all-carbon steel units. There's also the possibility of inadequate amount of zinc anode protection placed during the refits. The anodes are supposed to corrode preferentially to the pumps, but who knows if that's taking place? The pumps aren't being pulled out preventatively to check such things, only when they are already corroded beyond use. Effectively, the Corps is driving their car with the engine light on.

Update 1/28/17

With the Conhagen reports and the Corps task orders, we know corrosion of the Rineer motors has been at the center of many of the failures, with Conhagen explicitly calling it out in their 2015 repair report of 17th Street pump W7 and the Corps calling it out in the 2014 contract action for repair of 17th Street pump E8. Additionally, the Corps has continued a constant rotation of Rineer motors, issuing a $9365.21 task order in August 2014 for repair of three of them, and then a $12,525.11 task order in February, 2015 for repairs to three more. I don't have the supporting documents for the 2014 repairs, but I did get the purchase order to the Harvey, LA office of Hydradyne for the 2015 repairs.While the repairs did not call for the replacement of the vane cartridges, which are the moving heart of the pumps, there were callouts for seal and bearing and cover replacements. There were also explicit callouts for stainless steel plugs. This points at water ingress and corrosion.

This is of a piece with the historical trend of heavy use of Rineer motors. As noted in this 2012 entry, the Corps rebuilt or purchased 34 Rineer motors between July, 2010 and March, 2012. Tellingly, one of the task orders then explicitly called out replacement of the housing plugs, though they were not called out to be stainless steel:

"All motors: Replace all motor housing plugs with new solid hex head plugs."

But that's really just details. The overall arc of Rineers failing due to corrosion and seal failures - either of the Rineers or the components connected to them - hasn't changed. What I wrote back then still applies:

"The bottom line is that these Rineer motors are obviously ill suited to salt water exposure ... Leaving them where they are is simply asking for big trouble ... I've published a slew of pictures showing Rineers rusting to bits. They've been getting salt water into them through the rusty pipes, the seals, and every other leak path. The Corps has tried replacing them and repairing them. Nothing works. They all get eaten up. These motors are the heart of the pumping system at all three Lakefront closure structures, and they are unreliable and rusting to this day."

So that's it. There does not seem to be any movement on the Corps' part to prepare the interim closure structure pumps for three more years' of service, let alone give any sense of confidence that the pumps furthest out from the stainless refits are not going to crap out this year. Further proof - in addition to the lack of activity at the site - is the fact the Corps' four year pump repair contract with Healtheon expires this year, and a solicitation to replace it has not been issued. [see update below] In fact, of the $6.75 million on that repair contract, over $3 million ($3,068,911.61 to be exact) remains unspent as of March 1, 2015. At the going rate of $170,000 per pump refit, that $3 million could have gotten 18 pumps back into fighting shape for a couple of more years. Instead, the Corps New Orleans District just sat on their asses and didn't do anything proactive for the last three years.

Update - 1/15/17

The Corps had actually extended the Healtheon contract in February of 2015, though it is unclear how long the extension was for. A new 3 year, $3.9 million pump repair contract was issued to J. Star Enterprises on October 8, 2015. To date, at least three pumps - W8 and W9 at 17th Street and E2 at London Avenue  - have been pulled for repairs.

This is beyond disappointing, since we now know the pumps remain ticking time bombs ready to fail at their first real test this year. I don't know how much more proof the Corps needs than at least three of their own pumps - refitted to their own specifications - failing within an average of three years after that refit. That doesn't even count the two remaining pumps at 17th Street (E5 and E7) which remain without the stainless refit entirely, though there exists the possibility the pump pulled out in December 2014 is one of them.

Mitigating this problem perhaps is the additional work performed on the London Avenue and 17th Street canals last year to Really Truly Be Able To Handle 8 Feet Of Water. The ability to have higher canal water levels means there is a small amount of breathing room between what the city sends the Corps' pumps and what the Corps' pumps must pump out. However, when one considers the large gap between what the Corps says (one day before Isaac in 2012: "Orleans Ave [canal] - Max. Water Level is 8.0.") and what they actually do (during Isaac in 2012: "we will discontinue one of the [city's] pumps [feeding the Corps' Orleans Avenue canal pumps] when we hit 3 to 4 foot"), their credibility isn't worth anything. The only thing you can trust is the dollars, and they haven't spent any money getting the outfall canal pumps ready, beyond fixing stuff that they found broken in the most obvious way - an oil slick on the water.

In fact, their lack of concern about the general end-of-life state of the interim closure structures is so huge, that the topmost Corps officials are even joking about it on camera. In WWL's "Eye on Hurricanes" special aired June 1st, Corps New Orleans District commander Richard Hansen compared the existing interim closure structures and pumps to the now-under-construction permanent pumping stations:

"Well I'd say the interim control structure is kinda like its backyard kitchen, okay? It's not fully protected from the weather. It can do its job out there, but the permanent features will be the Cadillac. That will be the feature designed to last for 100 years"

It's wildly inappropriate to joke about how inadequate your own storm protection structures are. In fact, the WWL reporter says in the very next breath:

"So the interim pump structures have been in since 2006 and have actually outlived their lifespan, but the Corps says it will not be a problem for them to remain in place and be working until the permanent pumps can be complete. And right now that's expected by the height of hurricane season in 2017"

"The Corps says" It's unbelievable this still is accepted as truth.

Based on the lack of any preventative maintenance activity, as well as the evidence of refitted pumps failing just two to three years after their refit, I would judge all but a handful of the the outfall hydraulic canal pumps in a stage of imminent failure. The drainage of the city during a major tropical event which includes large amounts of rainfall is now dependent on luck and hope that stuff won't rust through during a storm. That's very, very bad.