How did the pumps get from...

This...


to this?


More pictures of the rusty pumps at the previous entry here.

How could the pumps have corroded so completely in just two years? We have to go back to the beginning, to decisions made in 2006 and 2007 during the installation and subsequent near-constant repair of the hydraulic pumps.

This post will look at those early decisions, the consequences of which are playing out right now. Let's start out looking at the relevant features as they were specified and initially installed.

The background (January 2006 - September 2006)
In the case of these steel water pumps, there were two main methods used to prevent corrosion: painting and electrolytic protection via zinc anodes. The painting system was a coal tar epoxy system. It was specified initially with a single sentence in the contract. The vast majority of the pump specifications in that Corps contract (also found in the bid solicitation) were copied and pasted from MWI's standard specifications without any modification. The sentence on painting read,

"The complete pump assembly shall be sand blasted to near white and painted inside and out with black bitumastic enamel equal to Zophar triple A or with Porter Tarset epoxy or equal."

Discovering the exact details of the painting system is difficult, since they don't appear to exist in any of the official contract documents that the Corps has released under the Freedom of Information Act.

Each hydraulic pump came with one sacrificial zinc anode (actually, a pair of plate anodes, but for ease of discussion we'll refer to it as one) meant to take the corrosion burden off the pumps themselves. Zinc anodes on a submerged steel structure serve the same function that aluminum or magnesium anodes do in a water heater; they corrode before the equipment they protect. Zinc anodes are a proven anti-corrosion technology; they have been in use to prevent marine corrosion since the early 1800's. It's a good thing they come as standard on MWI's pumps because they are not actually called out in the specifications at all.

Here's a couple of pictures of the original zinc anodes:





Originally, the pumps had their zinc-clad non-stainless steel threaded hose connections below the waterline:



It's worth noting that while the original hose connections spent much of their time above the normal lake level (except for the Phase 2 17th Street pumps, which were installed lower than the rest of the pumps and did have their hose connections completely submerged), when the lake went up during a particularly high tide, they would likley be submerged. Also, they were in what is known as the "splash zone," just above the water surface. Add in the salty environment just above the surface of the water, and those fittings were going to see a substantially corrosive atmosphere even if they weren't dunked all the time.

That was the setup in the late summer of 2006.


First attempts to deal with corrosion (September 2006 - early November 2006)
Throughout the last half of 2006, the Corps and MWI were messing with the pumps, fumbling around in an effort to fix the vibration problem that stemmed from the Rineer motors being undersized. Pumps were going in and out of the water constantly. Because of the vibrations, none of the pumps appeared able to run more than about 75% of their design capacity. The Corps knew the eventual solution would likely involve removing all the pumps and sending them to a shop, probably for repairs to the Rineer motors.

Against the background of that possibility, they were tackling a separate problem: corrosion. Perhaps they could deal with the corrosion problem at the same time the pumps were out of the water for the vibration problem.

While the Corps was pulling pumps in September and October to mess with the vibration problem, they also noticed the threaded hose connections on the pumps were rotting away. Also, the anodes were disappearing at an alarming pace - most pumps had only been in the water for about three months at that point. Finally, there was marine growth (barnacles) all over the pumps.

While some will debate it, there appears to be a strong correlation between marine growth and corrosion. I believe the barnacles can cause the paint to debond from the substrate, leaving that substrate vulnerable to rusting. There were hints of it in the pictures I took of the pumps at Orleans Avenue in March, 2007. I didn't recognize it at the time, but I'm not one of the people in charge of designing or supplying storm protection pumping equipment meant to preserve life and property in New Orleans.

So, with the anodes disintegrating and barnacles growing all over the place - indicating a potentially deep systemic problem with corrosion - the Corps instead zeroed in on the rusty hose connections.

Through September and October, the Corps and MWI went back and forth on how to deal with the corrosion of the hose connections. At first MWI sandblasted off the rust (and zinc coating) on the hose connections and painted them. I managed to catch a photo of that on October 21, 2006:



From what I can surmise, this type of repair was only performed on the five pumps from the east side of the Orleans Avenue site that spent the fall and winter out of the water. They had been removed at some point in late September or early October due to internal and external oil leaks.

The idea of leaving the threaded hose connections below the water with only paint on them didn't last long, and throughout October, 2006 discussions on a more substantial fix continued. There was talk of just making the hose connections stainless steel but still leaving them below the waterline. There was also talk of moving the hose connections above the waterline by placing them at the end of vertical lengths of pipe. Throughout October and the first week of November, they seemed to be moving toward simply replacing the hose connections with stainless. But after Norman E Kramer, an employee at PBS&J, the Corps contractor that was preparing official government estimates, sent an estimate for the hard piping choice that was cheaper than the stainless hose fittings ($211,000 vs $275,000), a modification to the contract to was issued in early November calling for the pipe extension idea. MWI was told to start work on it pending negotiation of a final price.

During those October 2006 discussions, MWI claimed they had never been informed that the pumps would sit in salty water until discovering it when pulling pumps out in September, 2006. This claim strains credulity. To the Corps' credit, Cynthia Nicholas - the contracting officer on the MWI contract - is quoted by MWI saying they "should have known." MWI personnel were on the ground within days of Katrina striking, and were present in New Orleans for weeks afterward as their pumps spit a minor amount of lake water back into the outfall canals while the city's pumps and personnel drained the city far ahead of projections. Here they are with the then-military-head of the Corps, Lt. General Carl Strock and the then-head of the Mississippi Valley Division, Brig. General Robert Crear:



That photo is from an article in the November 2005 issue of "the voice of the pump and rotating equipment industry," "Pumps & Systems" (the article is pretty inaccurate, but one cannot argue with the pictures). In the article, the photo above appears right next to another MWI-credited photo of the 17th Street canal breach:



Note how grey and dead everything looks. The whole city looked that way when the waters receded. Everything living the floodwaters touched, they killed. That's because Lake Pontchartrain is a brackish lake. It is the second largest saltwater lake in the United States, behind only Utah's Great Salt Lake. I cannot imagine spending more than five minutes on the ground in immediately post-Katrina New Orleans after the waters were receding and not noticing that everything on the ground - all the grass, plants, and shrubs - was dead. Reinforcing that was the fact that there was no ambient noise from any nature; all the insects, the lizards, the toads, and all the other ground-dwelling fauna were dead; most of the birds were gone as a result. It is inconceivable that personnel from a company headquartered in the even more tropical climate of south Florida - a place just as green as New Orleans had been pre-storm - were on the ground in post-K New Orleans for weeks and never once had the discussion that everyone else had about how the salty water in the lake had killed all the plants and grass.

Negotiations (early November 2006 - January 2007)
Anyhow, after the November contract modification, the Corps and MWI went back and forth over price for another two and a half months. That's not really a surprise, since MWI's initial quote for the work came in $784,000 and a December PBS&J re-estimate was at $633,000, both far above the original $211,000 PBS&J estimate. In the end, they settled on a price of $584,120. The November modification was cancelled January 17, 2007 and a new modification for essentially the same scope of work was issued the next day.

As I mentioned earlier, lost in all the discussion over the hose connections was the depletion of the zinc anodes back in September, 2006. That depletion signaled there could be a serious corrosion problem with the entire pump, not just the hose connections. It certainly appeared the original paint system wasn't working, and a single set of anodes per pump wasn't enough.

So let's look at both those topics as they relate to the 2006-07 timeframe.

Paint

During the negotiations, the Corps discussed an option to coat the pumps with an anti-fouling paint which might have resisted future marine growth on the pumps. We know the marine growth was bad. Here's the Orleans Avenue - West pumps on the deck in March 2007 just before they went to Associated Pump & Supply in Houma for the new piping (the Rineer motors would be pulled out and sent to Rineer's facility in San Antonio for refits):


The anti-fouling paint was not necessarily a cure-all. The manufacturer's specification notes that it works best when water is flowing past it, while most of the time water would be static around the pumps. Also, the specifications note that brackish water could reduce the paint's effectiveness. But with an effective preventive maintenance program, it may have helped.

But that was all moot, because the Corps (represented on the contracting side by Corps employee Cynthia Nicholas and on the technical side by former-Corps-employees-turned-contractors Gordon Hebert and Dennis Strecker) decided in January, 2007 - because of cost - to not go with the anti-fouling paint. Instead, the pumps and the new piping would be simply recoated with exact same paint system as before.

Zinc anodes


This one is easier to track through the documents. No extra anodes were added in 2007, despite evidence they were getting depleted after just a few months in the water. In fact, the Corps and MWI made the situation worse by adding more carbon steel in the form of the piping.


New piping installed (late 2006 - early 2007)
Here's a drawing of the piping added to the pumps, done by MWI in November, 2006 and received via FOIA from the Corps:

Note this view does not show the piping internal to the pump. Photos of those are below.

Here's how the piping turned out, in a picture of one the the pumps on the west side of the Orleans Avenue:



And you can see the piping in the post-repair pictures from the Conhagen report on the 2009 repairs to pumps E5 and E7:




Corrosion (early 2007 - present)
Here's what happened after two years (from the Conhagen report):





The marine growth was eating through the standard coal tar epoxy paint job and allowing the carbon steel to corrode severely, sometimes completely.

Now this spring, as detailed in the next post and the post after that, the Corps is spending over a million taxpayer dollars (dollars appropriated by Congress for construction of the permanent pumping stations) on:

a) Installing more anodes on each pump
b) Replacing the completely corroded carbon steel pipe with stainless steel pipe
c) Replacing all the moving parts inside some of the pumps, including the bearings and the Rineer motors

There's pictures of the first two aspects of this work in the Conhagen report (there are two other anodes on the side of the bell not shown in the left photo):






Analysis (or "This is not difficult stuff")

This latest repair work, and its cost, makes some of those decisions between November 2006 and January 2007 when they obsessed over the hose fittings while ignoring the entire rest of the pumps look pennywise and poundfoolish. They had the chance to do the right thing back then. The extra anodes would have been cheap. And while stainless steel piping would have been more expensive then, that would have been cheap compared to the ridiculous expense of taking these things out just two years later and trucking them to another shop to do essentially the same piping work, just with stainless steel instead of plain carbon steel. Not to mention the expense of cleaning up the oil spilled through the corroded pipe. Instead, they may have had to lift them out a little more often, but the only expense involved would likely have been anode replacements, which is standard for any marine system. Well, anodes are standard for marine systems that don't have an impressed current cathodic protection system, which is what Black & Veatch recommended to the Corps for this system in a 2007 report that wasn't "finalized"/released until 2009 (available here and here).

This is not simply second guessing. The idea of making the piping - which carries 3000 psi hydraulic oil through brackish canal waters - as resistant to corrosion as possible by installing stainless steel is not radical. It's not like stainless steel was invented in 2007. And zinc anodes have been around for almost two hundred years in marine applications; they should have been sticking those suckers all over those pumps. This is corrosion protection 101.

And even the most generous interprettion of Corps ignorance doesn't really fly. The idea of replacing the carbon steel piping on the pumps with stainless steel is mentioned in the Corps' own federally mandated Spill Prevention, Control, and Countermeasure (SPCC) plans for all three sites: 17th Street, Orleans Avenue, and London Avenue. All three plans have the same verbiage:

"Hydraulic Pump Platform Piping. Three-inch diameter pipes (hoses) transport hydraulic oil to the pumps. It is estimated that the maximum flow volume that may be released due to rupture of one of these pipes is about 50 gallons. More likely is leakage from the quick disconnect connection to the hydraulic motors (5 to 10 gallons). Calculations are based on this scenario. These are located over water and oil discharge will need to be contained. Currently, there is an oil boom surrounding the structure. This should remain in place to capture any leakage. The hydraulic piping (hoses) to the pumps could be changed out to stainless steel pipe to reduce the likelihood of rupture. All piping in this area has been pressure tested to 4,500 psi."

Also, the Corps can't use their usual excuse for their corrosion-related actions/inactions on these pumps of "we were rushing to get them in the water by June 1, 2006." All these discussions - which went on for over 3 months - were taking place at the end of 2006 and into January, 2007, in the comparitive calm of the hurricane off-season. They had plenty of time to fully analyze the problem and come up with the right solution the first time, rather than doing it wrong and having to come back two years later.

There's also the apparent truth that there was no comprehensive preventive maintenance program. Much of the damage seen on these pumps - especially pipes with near-zero wall thickness - would have been picked up and repaired if they had been getting yanked out on a regular, rotating basis in 2008. That obviously didn't get done, or there wouldn't have been an oil spill of "hundreds of gallons."


Conclusion


So to bottom line it, these pumps failed due to inadequate corrosion protection, poor preventive maintenance, and really cruddy Corps decision-making, and there will likely be more failures this summer. The Corps caused the corrosion problems they are now hurredly attempting to fix.

Will there be more failures? Yes, because the corrosion continues every minute you're reading this, and the Corps can't possibly add anodes and stainless steel piping to all the hydraulic pumps before the 2010 hurricane season. In my next post, I'll detail what they have been scrambling to get done this spring. Here's a preview: it's not enough. And then I'll provide more detail on this spring's work, including a breakdown on the dollars. Again, it's not enough.

Preview of post after that: The Corps has spilled thousands of gallons of oil in the canals and on the land around it (through pump leaks and other means), and much of it does not appear to have been reported to the federal or state authorities.

Molly Peterson and Karen Gadbois contributed to this report