Fix the pumps

Thursday, May 27, 2010

How did the pumps get from...


to this?

More pictures of the rusty pumps at the previous entry here.

How could the pumps have corroded so completely in just two years? We have to go back to the beginning, to decisions made in 2006 and 2007 during the installation and subsequent near-constant repair of the hydraulic pumps.

This post will look at those early decisions, the consequences of which are playing out right now. Let's start out looking at the relevant features as they were specified and initially installed.

The background (January 2006 - September 2006)
In the case of these steel water pumps, there were two main methods used to prevent corrosion: painting and electrolytic protection via zinc anodes. The painting system was a coal tar epoxy system. It was specified initially with a single sentence in the contract. The vast majority of the pump specifications in that Corps contract (also found in the bid solicitation) were copied and pasted from MWI's standard specifications without any modification. The sentence on painting read,
"The complete pump assembly shall be sand blasted to near white and painted inside and out with black bitumastic enamel equal to Zophar triple A or with Porter Tarset epoxy or equal."

Discovering the exact details of the painting system is difficult, since they don't appear to exist in any of the official contract documents that the Corps has released under the Freedom of Information Act.

Each hydraulic pump came with one sacrificial zinc anode (actually, a pair of plate anodes, but for ease of discussion we'll refer to it as one) meant to take the corrosion burden off the pumps themselves. Zinc anodes on a submerged steel structure serve the same function that aluminum or magnesium anodes do in a water heater; they corrode before the equipment they protect. Zinc anodes are a proven anti-corrosion technology; they have been in use to prevent marine corrosion since the early 1800's. It's a good thing they come as standard on MWI's pumps because they are not actually called out in the specifications at all.

Here's a couple of pictures of the original zinc anodes:

Originally, the pumps had their zinc-clad non-stainless steel threaded hose connections below the waterline:

It's worth noting that while the original hose connections spent much of their time above the normal lake level (except for the Phase 2 17th Street pumps, which were installed lower than the rest of the pumps and did have their hose connections completely submerged), when the lake went up during a particularly high tide, they would likley be submerged. Also, they were in what is known as the "splash zone," just above the water surface. Add in the salty environment just above the surface of the water, and those fittings were going to see a substantially corrosive atmosphere even if they weren't dunked all the time.

That was the setup in the late summer of 2006.

First attempts to deal with corrosion (September 2006 - early November 2006)
Throughout the last half of 2006, the Corps and MWI were messing with the pumps, fumbling around in an effort to fix the vibration problem that stemmed from the Rineer motors being undersized. Pumps were going in and out of the water constantly. Because of the vibrations, none of the pumps appeared able to run more than about 75% of their design capacity. The Corps knew the eventual solution would likely involve removing all the pumps and sending them to a shop, probably for repairs to the Rineer motors.

Against the background of that possibility, they were tackling a separate problem: corrosion. Perhaps they could deal with the corrosion problem at the same time the pumps were out of the water for the vibration problem.

While the Corps was pulling pumps in September and October to mess with the vibration problem, they also noticed the threaded hose connections on the pumps were rotting away. Also, the anodes were disappearing at an alarming pace - most pumps had only been in the water for about three months at that point. Finally, there was marine growth (barnacles) all over the pumps.

While some will debate it, there appears to be a strong correlation between marine growth and corrosion. I believe the barnacles can cause the paint to debond from the substrate, leaving that substrate vulnerable to rusting. There were hints of it in the pictures I took of the pumps at Orleans Avenue in March, 2007. I didn't recognize it at the time, but I'm not one of the people in charge of designing or supplying storm protection pumping equipment meant to preserve life and property in New Orleans.

So, with the anodes disintegrating and barnacles growing all over the place - indicating a potentially deep systemic problem with corrosion - the Corps instead zeroed in on the rusty hose connections.

Through September and October, the Corps and MWI went back and forth on how to deal with the corrosion of the hose connections. At first MWI sandblasted off the rust (and zinc coating) on the hose connections and painted them. I managed to catch a photo of that on October 21, 2006:

From what I can surmise, this type of repair was only performed on the five pumps from the east side of the Orleans Avenue site that spent the fall and winter out of the water. They had been removed at some point in late September or early October due to internal and external oil leaks.

The idea of leaving the threaded hose connections below the water with only paint on them didn't last long, and throughout October, 2006 discussions on a more substantial fix continued. There was talk of just making the hose connections stainless steel but still leaving them below the waterline. There was also talk of moving the hose connections above the waterline by placing them at the end of vertical lengths of pipe. Throughout October and the first week of November, they seemed to be moving toward simply replacing the hose connections with stainless. But after Norman E Kramer, an employee at PBS&J, the Corps contractor that was preparing official government estimates, sent an estimate for the hard piping choice that was cheaper than the stainless hose fittings ($211,000 vs $275,000), a modification to the contract to was issued in early November calling for the pipe extension idea. MWI was told to start work on it pending negotiation of a final price.

During those October 2006 discussions, MWI claimed they had never been informed that the pumps would sit in salty water until discovering it when pulling pumps out in September, 2006. This claim strains credulity. To the Corps' credit, Cynthia Nicholas - the contracting officer on the MWI contract - is quoted by MWI saying they "should have known." MWI personnel were on the ground within days of Katrina striking, and were present in New Orleans for weeks afterward as their pumps spit a minor amount of lake water back into the outfall canals while the city's pumps and personnel drained the city far ahead of projections. Here they are with the then-military-head of the Corps, Lt. General Carl Strock and the then-head of the Mississippi Valley Division, Brig. General Robert Crear:

That photo is from an article in the November 2005 issue of "the voice of the pump and rotating equipment industry," "Pumps & Systems" (the article is pretty inaccurate, but one cannot argue with the pictures). In the article, the photo above appears right next to another MWI-credited photo of the 17th Street canal breach:

Note how grey and dead everything looks. The whole city looked that way when the waters receded. Everything living the floodwaters touched, they killed. That's because Lake Pontchartrain is a brackish lake. It is the second largest saltwater lake in the United States, behind only Utah's Great Salt Lake. I cannot imagine spending more than five minutes on the ground in immediately post-Katrina New Orleans after the waters were receding and not noticing that everything on the ground - all the grass, plants, and shrubs - was dead. Reinforcing that was the fact that there was no ambient noise from any nature; all the insects, the lizards, the toads, and all the other ground-dwelling fauna were dead; most of the birds were gone as a result. It is inconceivable that personnel from a company headquartered in the even more tropical climate of south Florida - a place just as green as New Orleans had been pre-storm - were on the ground in post-K New Orleans for weeks and never once had the discussion that everyone else had about how the salty water in the lake had killed all the plants and grass.

Negotiations (early November 2006 - January 2007)
Anyhow, after the November contract modification, the Corps and MWI went back and forth over price for another two and a half months. That's not really a surprise, since MWI's initial quote for the work came in $784,000 and a December PBS&J re-estimate was at $633,000, both far above the original $211,000 PBS&J estimate. In the end, they settled on a price of $584,120. The November modification was cancelled January 17, 2007 and a new modification for essentially the same scope of work was issued the next day.

As I mentioned earlier, lost in all the discussion over the hose connections was the depletion of the zinc anodes back in September, 2006. That depletion signaled there could be a serious corrosion problem with the entire pump, not just the hose connections. It certainly appeared the original paint system wasn't working, and a single set of anodes per pump wasn't enough.

So let's look at both those topics as they relate to the 2006-07 timeframe.


During the negotiations, the Corps discussed an option to coat the pumps with an anti-fouling paint which might have resisted future marine growth on the pumps. We know the marine growth was bad. Here's the Orleans Avenue - West pumps on the deck in March 2007 just before they went to Associated Pump & Supply in Houma for the new piping (the Rineer motors would be pulled out and sent to Rineer's facility in San Antonio for refits):

The anti-fouling paint was not necessarily a cure-all. The manufacturer's specification notes that it works best when water is flowing past it, while most of the time water would be static around the pumps. Also, the specifications note that brackish water could reduce the paint's effectiveness. But with an effective preventive maintenance program, it may have helped.

But that was all moot, because the Corps (represented on the contracting side by Corps employee Cynthia Nicholas and on the technical side by former-Corps-employees-turned-contractors Gordon Hebert and Dennis Strecker) decided in January, 2007 - because of cost - to not go with the anti-fouling paint. Instead, the pumps and the new piping would be simply recoated with exact same paint system as before.

Zinc anodes

This one is easier to track through the documents. No extra anodes were added in 2007, despite evidence they were getting depleted after just a few months in the water. In fact, the Corps and MWI made the situation worse by adding more carbon steel in the form of the piping.

New piping installed (late 2006 - early 2007)
Here's a drawing of the piping added to the pumps, done by MWI in November, 2006 and received via FOIA from the Corps:

Note this view does not show the piping internal to the pump. Photos of those are below.

Here's how the piping turned out, in a picture of one the the pumps on the west side of the Orleans Avenue:

And you can see the piping in the post-repair pictures from the Conhagen report on the 2009 repairs to pumps E5 and E7:

Corrosion (early 2007 - present)
Here's what happened after two years (from the Conhagen report):

The marine growth was eating through the standard coal tar epoxy paint job and allowing the carbon steel to corrode severely, sometimes completely.

Now this spring, as detailed in the next post and the post after that, the Corps is spending over a million taxpayer dollars (dollars appropriated by Congress for construction of the permanent pumping stations) on:

a) Installing more anodes on each pump
b) Replacing the completely corroded carbon steel pipe with stainless steel pipe
c) Replacing all the moving parts inside some of the pumps, including the bearings and the Rineer motors

There's pictures of the first two aspects of this work in the Conhagen report (there are two other anodes on the side of the bell not shown in the left photo):

Analysis (or "This is not difficult stuff")

This latest repair work, and its cost, makes some of those decisions between November 2006 and January 2007 when they obsessed over the hose fittings while ignoring the entire rest of the pumps look pennywise and poundfoolish. They had the chance to do the right thing back then. The extra anodes would have been cheap. And while stainless steel piping would have been more expensive then, that would have been cheap compared to the ridiculous expense of taking these things out just two years later and trucking them to another shop to do essentially the same piping work, just with stainless steel instead of plain carbon steel. Not to mention the expense of cleaning up the oil spilled through the corroded pipe. Instead, they may have had to lift them out a little more often, but the only expense involved would likely have been anode replacements, which is standard for any marine system. Well, anodes are standard for marine systems that don't have an impressed current cathodic protection system, which is what Black & Veatch recommended to the Corps for this system in a 2007 report that wasn't "finalized"/released until 2009 (available here and here).

This is not simply second guessing. The idea of making the piping - which carries 3000 psi hydraulic oil through brackish canal waters - as resistant to corrosion as possible by installing stainless steel is not radical. It's not like stainless steel was invented in 2007. And zinc anodes have been around for almost two hundred years in marine applications; they should have been sticking those suckers all over those pumps. This is corrosion protection 101.

And even the most generous interprettion of Corps ignorance doesn't really fly. The idea of replacing the carbon steel piping on the pumps with stainless steel is mentioned in the Corps' own federally mandated Spill Prevention, Control, and Countermeasure (SPCC) plans for all three sites: 17th Street, Orleans Avenue, and London Avenue. All three plans have the same verbiage:
"Hydraulic Pump Platform Piping. Three-inch diameter pipes (hoses) transport hydraulic oil to the pumps. It is estimated that the maximum flow volume that may be released due to rupture of one of these pipes is about 50 gallons. More likely is leakage from the quick disconnect connection to the hydraulic motors (5 to 10 gallons). Calculations are based on this scenario. These are located over water and oil discharge will need to be contained. Currently, there is an oil boom surrounding the structure. This should remain in place to capture any leakage. The hydraulic piping (hoses) to the pumps could be changed out to stainless steel pipe to reduce the likelihood of rupture. All piping in this area has been pressure tested to 4,500 psi."

Also, the Corps can't use their usual excuse for their corrosion-related actions/inactions on these pumps of "we were rushing to get them in the water by June 1, 2006." All these discussions - which went on for over 3 months - were taking place at the end of 2006 and into January, 2007, in the comparitive calm of the hurricane off-season. They had plenty of time to fully analyze the problem and come up with the right solution the first time, rather than doing it wrong and having to come back two years later.

There's also the apparent truth that there was no comprehensive preventive maintenance program. Much of the damage seen on these pumps - especially pipes with near-zero wall thickness - would have been picked up and repaired if they had been getting yanked out on a regular, rotating basis in 2008. That obviously didn't get done, or there wouldn't have been an oil spill of "hundreds of gallons."


So to bottom line it, these pumps failed due to inadequate corrosion protection, poor preventive maintenance, and really cruddy Corps decision-making, and there will likely be more failures this summer. The Corps caused the corrosion problems they are now hurredly attempting to fix.

Will there be more failures? Yes, because the corrosion continues every minute you're reading this, and the Corps can't possibly add anodes and stainless steel piping to all the hydraulic pumps before the 2010 hurricane season. In my next post, I'll detail what they have been scrambling to get done this spring. Here's a preview: it's not enough. And then I'll provide more detail on this spring's work, including a breakdown on the dollars. Again, it's not enough.

Preview of post after that: The Corps has spilled thousands of gallons of oil in the canals and on the land around it (through pump leaks and other means), and much of it does not appear to have been reported to the federal or state authorities.

Molly Peterson and Karen Gadbois contributed to this report



  • Nice post. This is right up my alley.

    Cheapening out on corrosion protection is stupid, but it's sadly common, even in South Louisiana.

    A couple of notes from my experience:
    * For brackish water, corrosion protection can be extra difficult. Induced current and sacrificial anodes may not work because it's more corrosive than freshwater, but not electrically conductive enough to get proper coverage from anodes. The best way to be sure is metallurgy (316L SS or better, Cu/Ni or Ni/Al/Bronze)
    * Paint is its own specialty. A good paint job that's well maintained will carry you a lot further than people give it credit for. Sandblast the base material white, put a couple coats of zinc-rich primer, and put on a good epoxy topcoat and that will work wonders.
    * Splashzones are always major corrosion headaches. There are only two effective countermeasures: extra steel (say 1/4" or 1/2" corrosion allowance is COMMON) or Splashtron (a proprietary coating system that, while expensive, works wonders).
    * I don't like threaded connections with 3000# working pressures.
    * They're STILL going to have corrosion with the SS piping because of dissimilar metal galvanic action. The SS piping will be the cathode and the carbon steel will be the anode and the corrosion will pile up fast, unless the put in an isolation kit (which, from you're description, I'm betting didn't happen).
    * In my experience, anti-fouling paint is next to worthless. Even Cu/Ni, which has a bad "taste" to marine life, can suffer serious fouling problems. I've seen an 8" 90/10 Cu/Ni line reduced to a 2" effective bore by Zebra mussels. The only truly effective anti-fouling method is injection of trace amounts of sodium hypochlorite (bleach). 1-2 PPM continuous or 5-6 PPM batch.

    In the end, carbon steel shouldn't be used in a marine environment for mechanical equipment if you can get away with something else. Problem is, you're looking at an exponential increase in the time you're waiting for your better-metallurgy castings from the foundry.

    By Blogger Clay, at May 27, 2010 11:07 PM  

  • Clay,

    As far as using 316L, that is what has been done on the repairs, though to differing degrees.

    On the repairs last summer to 60" pumps E5, E7, W8, and W9 at 17th Street, only the piping inside the pumps was replaced with 316L stainless. For some reason, the piping outside the pump but still below the waterline was replaced with carbon steel again.

    This time around, they have gotten better. On the repairs to pumps W5 and W6 at 17th Street that started in February, ALL the piping is being replaced with 361L stainless. Even the hydraulic coolers, which are original equipment, are being replaced with 316L. That is a major difference, but one unfortunately that leaves the pumps fixed last summer vulnerable to corrosion once again. I'll be highlighting this difference in my next post.

    As far as dealing with brackish water by going with stainless, while 316L SS can still be galvanicaly corroded, its a lot less likely to do so compared to carbon steel. With a preventive maintenance program (yeah - like that's gonna happen), at least the pumps will have a fighting chance until 2014 or 2015 when the permanent ones go in.

    It occurs to me - do the residents of Jefferson Parish have annual concerns about their lakefront pumps corroding away? No they don't.

    By Blogger mcbrid35, at May 29, 2010 6:16 AM  

  • As far as dealing with brackish water by going with stainless, while 316L SS can still be galvanicaly corroded, its a lot less likely to do so compared to carbon steel. With a preventive maintenance program (yeah - like that's gonna happen), at least the pumps will have a fighting chance until 2014 or 2015 when the permanent ones go in.

    Galvanic series chart:

    If they're putting that much 316L in, the galvanic corrosion on the remaining carbon steel members (unchanged piping, impeller housings, etc.) will be substantial.

    The 316 would only build up significant amounts of corrosion if there's a large whetted surface of something more noble than it (Alloy 20, Titanium, etc.).

    What are they really going to do about the marine growth? Doesn't look like anything. It's possible to blast and paint stainless with anti-fouling paint, but that paint doesn't work as well as advertised, in my experience, and sandblasting stainless is VERY tricky. The outer layer of the steel is very hard, so you have to take your time ($, lots of sand) to do a thorough job.

    By Blogger Clay, at May 29, 2010 11:10 AM  

  • If they're putting that much 316L in, the galvanic corrosion on the remaining carbon steel members (unchanged piping, impeller housings, etc.) will be substantial.

    To clarify, on the latest repairs there is no more carbon steel piping; it is all 316L (at least according to the task order). Even the coolers are now 316L. However, there are other pieces inside the pump (bearing housing, motor mount, Rineer motor housing(!), vanes), as well as the housing itself, which remain carbon steel. The only thing preventing corrosion on those pieces is paint, the zinc anodes, and hope.

    As far as getting the marine growth off, that's simply not part of the maintenance. I've got every task order for the Corps' diving company back to Katrina. They usually come out once a year to clean out the trenches that the gate sections fit into. They'll sometimes come out ahead of a storm (even thought the Corps' own manual says they're supposed to be there during every storm) and some stuff, but it is very inconsistent. For example, there's no diving task order last year for Hurricane Ida, even though the gates were closed at London Ave. Same for the September gate closure at London, and for the December rain event when the gates weren't closed. I seriously think there is no plan to do anything about marine growth. The correspondence attached to the relevant contract modifications on the MWI contract indicates the Corps' engineeringcontractors then in charge were only worried about marine growth impeding the flow of water, not causing corrosion.

    Also, the 316L remains unpainted on the latest repairs. This is explicitly mentioned in task orders 3 and 4.

    By Blogger mcbrid35, at May 29, 2010 4:06 PM  

  • The biggest danger for this installation would be marine growth impeding the heat exchange of the hydraulic fluid cooling units. That would cause degradation of the seals in the hydraulic pump and degraded performance.

    By Blogger Clay, at May 29, 2010 6:04 PM  

  • There are much bigger dangers than inadequate heat transfer.

    Among the dangers:

    1) No flushing of the oil out of the system after pulling pumps. If oil's spilling out of the rusted pipes, water's getting in. That means water contamination thru the entire system, including the Denison hydraulic pumps back on the drive skids, Hundreds of thousands of dollars were spent on TWO rounds of flushing when the pumps went in because the oil got contaminated with pickling fluid and water.

    2) No checking of the rest of the hydraulic system for contamintion. Denisons need to be torn down and checked with a microscope; they're not.

    3) Inadequate post-repair operational testing - only raising the hydraulic oil to 2500 psi for likely little more than 1/2 hour.

    By Blogger mcbrid35, at May 29, 2010 8:18 PM  

  • 3.- Isn't the MAWP of the piping supposed to be 3,000 or 3,200 psig? Are they derating the system (and pumping capacity) by ~20%?

    By Blogger Clay, at May 29, 2010 11:14 PM  

  • MAWP is 3200 psi. There is no explanation given in Conhagen report or the task orders for why the pump units are specified to be tested at 2500 psi.

    In addition, the testing records sent in response to my FOIA request only said the pumps were tested "at full speed", pumping water until the Corps rep (called as "Kim from the Corps" and likely Kim L Branch) said everything was okay. There were no pressure, flow, temperature, or duration numbers, though they may exist on a another document which was not sent. This kind of stuff - having to go through multiple requests and months-long waits - happens a lot, and the slowness of the Corps' response to FOIA requests is a serious impediment to discovering exactly what is going on.

    By Blogger mcbrid35, at May 30, 2010 6:06 AM  

Post a Comment

<< Home

Go to older posts Go to newer posts