How did the pumps get from...

This...


to this?


More pictures of the rusty pumps at the previous entry here.

How could the pumps have corroded so completely in just two years? We have to go back to the beginning, to decisions made in 2006 and 2007 during the installation and subsequent near-constant repair of the hydraulic pumps.

This post will look at those early decisions, the consequences of which are playing out right now. Let's start out looking at the relevant features as they were specified and initially installed.

The background (January 2006 - September 2006)
In the case of these steel water pumps, there were two main methods used to prevent corrosion: painting and electrolytic protection via zinc anodes. The painting system was a coal tar epoxy system. It was specified initially with a single sentence in the contract. The vast majority of the pump specifications in that Corps contract (also found in the bid solicitation) were copied and pasted from MWI's standard specifications without any modification. The sentence on painting read,

"The complete pump assembly shall be sand blasted to near white and painted inside and out with black bitumastic enamel equal to Zophar triple A or with Porter Tarset epoxy or equal."

Discovering the exact details of the painting system is difficult, since they don't appear to exist in any of the official contract documents that the Corps has released under the Freedom of Information Act.

Each hydraulic pump came with one sacrificial zinc anode (actually, a pair of plate anodes, but for ease of discussion we'll refer to it as one) meant to take the corrosion burden off the pumps themselves. Zinc anodes on a submerged steel structure serve the same function that aluminum or magnesium anodes do in a water heater; they corrode before the equipment they protect. Zinc anodes are a proven anti-corrosion technology; they have been in use to prevent marine corrosion since the early 1800's. It's a good thing they come as standard on MWI's pumps because they are not actually called out in the specifications at all.

Here's a couple of pictures of the original zinc anodes:





Originally, the pumps had their zinc-clad non-stainless steel threaded hose connections below the waterline:



It's worth noting that while the original hose connections spent much of their time above the normal lake level (except for the Phase 2 17th Street pumps, which were installed lower than the rest of the pumps and did have their hose connections completely submerged), when the lake went up during a particularly high tide, they would likley be submerged. Also, they were in what is known as the "splash zone," just above the water surface. Add in the salty environment just above the surface of the water, and those fittings were going to see a substantially corrosive atmosphere even if they weren't dunked all the time.

That was the setup in the late summer of 2006.


First attempts to deal with corrosion (September 2006 - early November 2006)
Throughout the last half of 2006, the Corps and MWI were messing with the pumps, fumbling around in an effort to fix the vibration problem that stemmed from the Rineer motors being undersized. Pumps were going in and out of the water constantly. Because of the vibrations, none of the pumps appeared able to run more than about 75% of their design capacity. The Corps knew the eventual solution would likely involve removing all the pumps and sending them to a shop, probably for repairs to the Rineer motors.

Against the background of that possibility, they were tackling a separate problem: corrosion. Perhaps they could deal with the corrosion problem at the same time the pumps were out of the water for the vibration problem.

While the Corps was pulling pumps in September and October to mess with the vibration problem, they also noticed the threaded hose connections on the pumps were rotting away. Also, the anodes were disappearing at an alarming pace - most pumps had only been in the water for about three months at that point. Finally, there was marine growth (barnacles) all over the pumps.

While some will debate it, there appears to be a strong correlation between marine growth and corrosion. I believe the barnacles can cause the paint to debond from the substrate, leaving that substrate vulnerable to rusting. There were hints of it in the pictures I took of the pumps at Orleans Avenue in March, 2007. I didn't recognize it at the time, but I'm not one of the people in charge of designing or supplying storm protection pumping equipment meant to preserve life and property in New Orleans.

So, with the anodes disintegrating and barnacles growing all over the place - indicating a potentially deep systemic problem with corrosion - the Corps instead zeroed in on the rusty hose connections.

Through September and October, the Corps and MWI went back and forth on how to deal with the corrosion of the hose connections. At first MWI sandblasted off the rust (and zinc coating) on the hose connections and painted them. I managed to catch a photo of that on October 21, 2006:



From what I can surmise, this type of repair was only performed on the five pumps from the east side of the Orleans Avenue site that spent the fall and winter out of the water. They had been removed at some point in late September or early October due to internal and external oil leaks.

The idea of leaving the threaded hose connections below the water with only paint on them didn't last long, and throughout October, 2006 discussions on a more substantial fix continued. There was talk of just making the hose connections stainless steel but still leaving them below the waterline. There was also talk of moving the hose connections above the waterline by placing them at the end of vertical lengths of pipe. Throughout October and the first week of November, they seemed to be moving toward simply replacing the hose connections with stainless. But after Norman E Kramer, an employee at PBS&J, the Corps contractor that was preparing official government estimates, sent an estimate for the hard piping choice that was cheaper than the stainless hose fittings ($211,000 vs $275,000), a modification to the contract to was issued in early November calling for the pipe extension idea. MWI was told to start work on it pending negotiation of a final price.

During those October 2006 discussions, MWI claimed they had never been informed that the pumps would sit in salty water until discovering it when pulling pumps out in September, 2006. This claim strains credulity. To the Corps' credit, Cynthia Nicholas - the contracting officer on the MWI contract - is quoted by MWI saying they "should have known." MWI personnel were on the ground within days of Katrina striking, and were present in New Orleans for weeks afterward as their pumps spit a minor amount of lake water back into the outfall canals while the city's pumps and personnel drained the city far ahead of projections. Here they are with the then-military-head of the Corps, Lt. General Carl Strock and the then-head of the Mississippi Valley Division, Brig. General Robert Crear:



That photo is from an article in the November 2005 issue of "the voice of the pump and rotating equipment industry," "Pumps & Systems" (the article is pretty inaccurate, but one cannot argue with the pictures). In the article, the photo above appears right next to another MWI-credited photo of the 17th Street canal breach:



Note how grey and dead everything looks. The whole city looked that way when the waters receded. Everything living the floodwaters touched, they killed. That's because Lake Pontchartrain is a brackish lake. It is the second largest saltwater lake in the United States, behind only Utah's Great Salt Lake. I cannot imagine spending more than five minutes on the ground in immediately post-Katrina New Orleans after the waters were receding and not noticing that everything on the ground - all the grass, plants, and shrubs - was dead. Reinforcing that was the fact that there was no ambient noise from any nature; all the insects, the lizards, the toads, and all the other ground-dwelling fauna were dead; most of the birds were gone as a result. It is inconceivable that personnel from a company headquartered in the even more tropical climate of south Florida - a place just as green as New Orleans had been pre-storm - were on the ground in post-K New Orleans for weeks and never once had the discussion that everyone else had about how the salty water in the lake had killed all the plants and grass.

Negotiations (early November 2006 - January 2007)
Anyhow, after the November contract modification, the Corps and MWI went back and forth over price for another two and a half months. That's not really a surprise, since MWI's initial quote for the work came in $784,000 and a December PBS&J re-estimate was at $633,000, both far above the original $211,000 PBS&J estimate. In the end, they settled on a price of $584,120. The November modification was cancelled January 17, 2007 and a new modification for essentially the same scope of work was issued the next day.

As I mentioned earlier, lost in all the discussion over the hose connections was the depletion of the zinc anodes back in September, 2006. That depletion signaled there could be a serious corrosion problem with the entire pump, not just the hose connections. It certainly appeared the original paint system wasn't working, and a single set of anodes per pump wasn't enough.

So let's look at both those topics as they relate to the 2006-07 timeframe.

Paint

During the negotiations, the Corps discussed an option to coat the pumps with an anti-fouling paint which might have resisted future marine growth on the pumps. We know the marine growth was bad. Here's the Orleans Avenue - West pumps on the deck in March 2007 just before they went to Associated Pump & Supply in Houma for the new piping (the Rineer motors would be pulled out and sent to Rineer's facility in San Antonio for refits):


The anti-fouling paint was not necessarily a cure-all. The manufacturer's specification notes that it works best when water is flowing past it, while most of the time water would be static around the pumps. Also, the specifications note that brackish water could reduce the paint's effectiveness. But with an effective preventive maintenance program, it may have helped.

But that was all moot, because the Corps (represented on the contracting side by Corps employee Cynthia Nicholas and on the technical side by former-Corps-employees-turned-contractors Gordon Hebert and Dennis Strecker) decided in January, 2007 - because of cost - to not go with the anti-fouling paint. Instead, the pumps and the new piping would be simply recoated with exact same paint system as before.

Zinc anodes


This one is easier to track through the documents. No extra anodes were added in 2007, despite evidence they were getting depleted after just a few months in the water. In fact, the Corps and MWI made the situation worse by adding more carbon steel in the form of the piping.


New piping installed (late 2006 - early 2007)
Here's a drawing of the piping added to the pumps, done by MWI in November, 2006 and received via FOIA from the Corps:

Note this view does not show the piping internal to the pump. Photos of those are below.

Here's how the piping turned out, in a picture of one the the pumps on the west side of the Orleans Avenue:



And you can see the piping in the post-repair pictures from the Conhagen report on the 2009 repairs to pumps E5 and E7:




Corrosion (early 2007 - present)
Here's what happened after two years (from the Conhagen report):





The marine growth was eating through the standard coal tar epoxy paint job and allowing the carbon steel to corrode severely, sometimes completely.

Now this spring, as detailed in the next post and the post after that, the Corps is spending over a million taxpayer dollars (dollars appropriated by Congress for construction of the permanent pumping stations) on:

a) Installing more anodes on each pump
b) Replacing the completely corroded carbon steel pipe with stainless steel pipe
c) Replacing all the moving parts inside some of the pumps, including the bearings and the Rineer motors

There's pictures of the first two aspects of this work in the Conhagen report (there are two other anodes on the side of the bell not shown in the left photo):






Analysis (or "This is not difficult stuff")

This latest repair work, and its cost, makes some of those decisions between November 2006 and January 2007 when they obsessed over the hose fittings while ignoring the entire rest of the pumps look pennywise and poundfoolish. They had the chance to do the right thing back then. The extra anodes would have been cheap. And while stainless steel piping would have been more expensive then, that would have been cheap compared to the ridiculous expense of taking these things out just two years later and trucking them to another shop to do essentially the same piping work, just with stainless steel instead of plain carbon steel. Not to mention the expense of cleaning up the oil spilled through the corroded pipe. Instead, they may have had to lift them out a little more often, but the only expense involved would likely have been anode replacements, which is standard for any marine system. Well, anodes are standard for marine systems that don't have an impressed current cathodic protection system, which is what Black & Veatch recommended to the Corps for this system in a 2007 report that wasn't "finalized"/released until 2009 (available here and here).

This is not simply second guessing. The idea of making the piping - which carries 3000 psi hydraulic oil through brackish canal waters - as resistant to corrosion as possible by installing stainless steel is not radical. It's not like stainless steel was invented in 2007. And zinc anodes have been around for almost two hundred years in marine applications; they should have been sticking those suckers all over those pumps. This is corrosion protection 101.

And even the most generous interprettion of Corps ignorance doesn't really fly. The idea of replacing the carbon steel piping on the pumps with stainless steel is mentioned in the Corps' own federally mandated Spill Prevention, Control, and Countermeasure (SPCC) plans for all three sites: 17th Street, Orleans Avenue, and London Avenue. All three plans have the same verbiage:

"Hydraulic Pump Platform Piping. Three-inch diameter pipes (hoses) transport hydraulic oil to the pumps. It is estimated that the maximum flow volume that may be released due to rupture of one of these pipes is about 50 gallons. More likely is leakage from the quick disconnect connection to the hydraulic motors (5 to 10 gallons). Calculations are based on this scenario. These are located over water and oil discharge will need to be contained. Currently, there is an oil boom surrounding the structure. This should remain in place to capture any leakage. The hydraulic piping (hoses) to the pumps could be changed out to stainless steel pipe to reduce the likelihood of rupture. All piping in this area has been pressure tested to 4,500 psi."

Also, the Corps can't use their usual excuse for their corrosion-related actions/inactions on these pumps of "we were rushing to get them in the water by June 1, 2006." All these discussions - which went on for over 3 months - were taking place at the end of 2006 and into January, 2007, in the comparitive calm of the hurricane off-season. They had plenty of time to fully analyze the problem and come up with the right solution the first time, rather than doing it wrong and having to come back two years later.

There's also the apparent truth that there was no comprehensive preventive maintenance program. Much of the damage seen on these pumps - especially pipes with near-zero wall thickness - would have been picked up and repaired if they had been getting yanked out on a regular, rotating basis in 2008. That obviously didn't get done, or there wouldn't have been an oil spill of "hundreds of gallons."


Conclusion


So to bottom line it, these pumps failed due to inadequate corrosion protection, poor preventive maintenance, and really cruddy Corps decision-making, and there will likely be more failures this summer. The Corps caused the corrosion problems they are now hurredly attempting to fix.

Will there be more failures? Yes, because the corrosion continues every minute you're reading this, and the Corps can't possibly add anodes and stainless steel piping to all the hydraulic pumps before the 2010 hurricane season. In my next post, I'll detail what they have been scrambling to get done this spring. Here's a preview: it's not enough. And then I'll provide more detail on this spring's work, including a breakdown on the dollars. Again, it's not enough.

Preview of post after that: The Corps has spilled thousands of gallons of oil in the canals and on the land around it (through pump leaks and other means), and much of it does not appear to have been reported to the federal or state authorities.

Molly Peterson and Karen Gadbois contributed to this report

Imminent

The lakefront hydraulic pumps are rusting to bits, and have been since they were installed. Some have already fallen apart. The Corps was warned all of them would do the same thing "imminently" in mid-2009, and did next to nothing for nearly the entire 2009 hurricane season. For those that don't want to read the whole story, evidence is in this report from the Corps New Orleans District's pump repair contractor.

Only in recent weeks has the Corps' New Orleans District embarked on a crash repair initiative hidden from the public in plain sight, in an effort to make up for years of deferred - but not unfunded - preventive maintenance on pumps which they knew from the very beginning would fall apart within a matter of a couple of years, if not quicker. That lack of maintenance likely left New Orleans' front line flood defenses in a precarious state of decay during the entire length of last year's hurricane season, and may have actually played a part in a year-end flooding event. [UPDATE: It is more likely that the December 8, 2009 systemwide failure of the Corps SCADA control system for the pumps and gates played a larger part in the decision not to lower the gates and run the pumps during widespread flooding and elevated lake levels on December 12-13, 2009. Details can be found in my subsequent post, "SCADA."] It also may have been a pawn in the Corps recent negotiations with the state to impose their inadequate Option 1 scheme. I'll get to the bits about the current work, the year-end flooding event, and Option 1 in future posts. For now, I'm just going to concentrate on the 2009 repairs.

In spring, 2009, the Corps issued their first work order on contract W912P8-08-D-0090, a potentially five-year-long pump maintenance they inked with a company named Healtheon in 2008.

That $28,138 work order, whose value was later increased by $1487.43 made reference to pulling three pumps at 17th Street because of "hydraulic oil leaking problems:"

"Repair hydraulic oil leaking problems of one 60" hydraulic pump #10W and two 40"[sic - should be 42"] bridge hydraulic pumps #1 and #6 at 17th Street Canal. One MWI technician is needed for two days on May 28 and May 29 and three days during the week of June 1-5. The exact date of these three days will be determined later due to the availability of the crane. Healtheon will coordinate closely with Corps employees, Ms Lynn Tinto and Jack Rouquette, during pump pullup, repair, and reinstallation of the pumps."

As a quick reminder, most of the pumps at all three interim closure sites on the lakefront are hydraulically powered. Hydraulic oil is raised to approximately 3000 psi by hydraulic pumps manufactured by Denison Hydraulics at engine skids placed in sheds remote from the pumps (though at 17th Street, some of the pumps have their engine skids adjacent to them). That pressurized oil flows along hundreds of feet of pipe (or a few dozen feet of rubber hose, in the case of the 17th Street skid-adjacent pumps) to the water pumps in the canals. Inside the water pumps, the energy of the pressurized hydraulic oil is translated to rotational energy by a hydraulic motor manufactured by a Texas outfit known as Rineer.

The Rineer motor (the original was four port model 125, though it has since been modified - see below) is attached to a shaft which spins an impeller, pushing stormwater out of the canal, through the pump, and down 9-foot-diameter pipes which travel past the gates to the lake. The depressurized hydraulic oil travels through coolers mounted on the water pump, and then back through more piping to a reservoir at the drive unit. From there, the cycle repeats.

There are 18 60" hydraulic pumps at 17th Street, and 14 42" models. The size (60" or 42") refers to diameter of the pump housing. The original order to MWI for the hydraulic pumps in January, 2006 was for 34 60" pumps, with 12 each going in at 17th Street and London Avenue, and the remaining 10 going in at Orleans Avenue. Those were the "Phase 1" pumps. Another six 60" pumps were installed at 17th Street later in 2006 ("Phase 2"), and the 14 42" pumps went in during the spring of 2007 ("Phase 3").

There are also 11 direct drive units at 17th Street and 8 direct drive units at London Avenue, installed during the spring and summer of 2007. Those pumps have their drives directly coupled to their shafts, eliminating all the problems inherent in a hydraulically powered system. Here's the layout of the pumps at 17th Street:


Here's the numbering of each pump at 17th Street:


Here's the London Ave layout, with the original 12 hydraulic pumps noted as "Phase 1:"


And the numbering for the pumps at London Avenue:



The numbering at Orleans Avenue is similar to at London Avenue, with hydraulic pumps E1 through E5 on the east side and W1 through W5 on the west side.

The piping which conveys the hydraulic oil is coupled to the water pumps via hoses. To repair the water pumps, the hoses must be decoupled. There is no way to blow the hydraulic oil back to the drive units, so the oil spills in the canal when the hoses are detached. As it turns out, these "planned" oil spills are excellent indicators of work (or trouble) happening on the hydraulic pumps.

That's because the Corps also has a contract for oil spill cleanups. It is with Quaternary Resource Investigations (QRI) of Baton Rouge. Every time a pump is pulled, oil is spilled, so QRI should be on site to clean it up. QRI also provides and maintains the permanent oil protection booms that are at each site.

Thus, if a pump (or pumps) is pulled, QRI gets a task order on their contract. According to federal law and the Corps own oil spill response plans (linked: 17th Street SPCC; Orleans Avenue SPCC; London Avenue SPCC), the oil spill should also be reported to the National Response Center (NRC), and it should show up on the NRC report query website. Depending on the size of the spill (the state only requires spill reports for spills over 42 gallons, or 1 barrel), the state should also have a record of the spill on the LDEQ Electronic Document Management Service website. Reporting to the state is also a requirement under the law and the Corps' written plans.

So back to the May, 2009 pump repairs...

The work was - according to the task order - to take place in late May and early June. Every year in late May, the Corps holds a hurricane preparedness exercise, part of which they put on in front of the cameras of the local news crews. It is mostly a dog-and-pony show. In 2009, the Corps had the media out to the London Avenue canal gates on May 27, 2009 (Fox 8 story here), which was a departure from their usual M.O. Whenever the New Orleans District wants to emphasize how wonderful the gates and pumps are, they have the press out to the much larger 17th Street site. No one in the local press seemed to question why they were at London instead.

However, the Corps themselves obliquely acknowledged the 17th Street work in their post-exercise press release on May 27, 2009, which was the same day the repair work was due to commence:

"Because of previously scheduled maintenance on some of the pumps at the 17th St. Canal (due to be complete next week), the pumps could not be turned on immediately following the gate closure for the safety of the contractors working in the area. However, each of the pumps, excluding those affected by the ongoing maintenance, were run through the start up cycle at some point during the exercise and effectively pumped. Those affected by the ongoing maintenance will be run when the work is complete."

The "previously scheduled" thing is a canard. The work appears to have begun in a hurry, since the Healtheon task order #1 has an effective date of June 17, 2009, or three weeks after the work began (likely with some kind of unofficial Notice to Proceed like an email or a phone call). Other contracts issued for the work all had dates after the work began, except for the crane rental contract, which went out less than a week before (see below). Also note how vague the press release is, giving the impression of "nothing to see here, move along." That was made easier by keeping the prying eyes of the local TV crews miles away from the 17th Street site.

A more detailed account of what was going on May 27, 2009 at 17th Street did get created: the QRI-written spill report for the pulling of 60" hydraulic pump W10. It was generated as part of QRI's work under task order #5. It gives an extensive narrative of that day's events:

"At 0630 hours on May 27, 2009, QRI's Field Supervisor and Field Technician arrived at the Orleans Canal storage (connex) box to retrieve staged spill response materials. At 0645 hours QRI's Field Manager and Site Safety Officer made contact with Corps employee Ms. Lynn Tinto. Following an 0715 Safety Meeting, QRI personnel launched from the 17th Street Canal boat launch and proceeded to the vicinity of the W 10 pump.

Pump W10 was isolated by installing both hard and soft boom[s] - to form a double boom around the perimeter of pump W10. Following placement of the boom, the QRI crew and boat staged on the lake side of the canal to allow USACE personnel to access pump W10.

At 1240 hours the pump was lifted out and approximately 40 gallons of hydraulic fluid was spilled. QRI's response crew placed napkins within the double boom area to absorb the spilled fluid. An additional 150 foot section of soft boom was placed outside the isolated area to absorb any oil escaping from the double-boomed section. QRI personnel used 12 1/2 additional bags of napkins during the cleanup process.

QRI's safety officer and USACE personnel discussed regulations outlined in the 17th Street Canal SPCC Plan and determined that notification to LDEQ was necessary. Incident number T115306 was assigned by LDEQ."

Note that notification of the National Response Center is oddly omitted. That's strange, because the state reportable quantity (RQ) for oil is 1 barrel (42 gallons), while the federal government RQ is any amount, no matter how tiny. Indeed, there is no report of this spill on the NRC website.

Anyway, the report describes a lot of activity. QRI took date-stamped pictures too:





It would have aroused questions. Keeping the cameras away sidestepped that possibility.

So they lifted out W10 and then what? It's not easy to tell. There's a number of other contract actions related to the work which went out before the time of the Healtheon task order, with its effective date of June 17, 2009 but its text requiring work before that date.

There's the $16,370 (note: FPDS-NG reports the value as $31,492) contract for rental of the crane seen in the photos above that pulled pump W10, W912P8-09-P-0210, issued to RPL Oil Distributor and Supply, LLC of Gretna, LA, a frequent go-to company for the New Orleans District for all manner of things. That contract was issued with an effective date of May 21, 2009:

"Furnish one (1) 240-ton capacity hydraulic truck crane, with operator and maintenance to work as directed by the Government to remove Pump No. 10 for repairs and replace same. Based on 5 days at 10 hours. Also includes Rental of Crane Mats for 5 days."

The Corps used their own, rented crane to pull the bridge pumps.

Speaking of those bridge pumps, there's also the $25,850 contract for scaffolding around those smaller 42" bridge pumps nos. 1 and 6 that were also pulled. That was contract W912P8-09-P-0219, also issued to RPL Oil Distributor and Supply with an effective date of June 1, 2009. The contract text reads:

"Rent and install scaffolding for bridge pumps #1 and # 6 at 17th Street Canal interim pumping station. Work will involve providing and installing 1 level of hanging scaffolding under the bridge at the interim structure of approximately 10 feet by 10 feet by 13 feet. The scaffolding and installation will meet OSHA codes. The government will provide crane service to load and unload material. Work will start on 1 Jun and will be completed by 31 Jul or sooner."

Also on the topic of the bridge pumps, there's the $9000 task order 56 on the Corps contract with their diving company, Independent Divers, dated June 3, 2009:

"Provide Dive Team to inspect bridge pumps 1 and 6, and its foundations, and remove any existing obstructions. All work shall be performed in accordance with the contract terms and conditions, this task order, and the attached Scope of Work. Diving shall be performed on 4 June 2009."

The scope of work reiterates the same basic text.

Finally, there was the $15,551.19 contract for all the stainless steel hardware, as well as hose assemblies, needed to replace all the carbon steel fittings and hose on all three pumps. That contract, W921P8-09-P-0222 was issued to QPL, Inc of Thibodaux, LA with an effective date of June 4, 2009. That contract text is a long listing of model numbers for hydraulic piping fittings and other bits of hardware, so I won't reproduce it here. Suffice to say that I figured out it is the exact requirement for replacement of three pumps' worth of piping, which matches with the fact that three pumps were pulled.

That's what this is about - all the carbon steel piping on the water pumps which conveys the hydraulic fluid had to be replaced because of severe corrosion. That's where the oil in the water was coming from - holes in the pipes. There's more on this below when I talk about the second round of 2009 repairs.

Based on all those individual pieces-and-parts contracts, I believe the Corps did the repairs on these pumps themselves and did not send them out to a shop (as they would later in the summer, see below). Indeed, with all that other contract activity, it's not clear exactly what Healtheon brought to the table. A report about W10 and the other two pumps pulled under this work order (QRI was also tasked with the cleanup on those two bridge pumps' removal under task order #6 of their contract) was required under the Healtheon contract, but it is unknown if was really written. If it was, the Corps hasn't released it yet - despite a pending FOIA request for the information.

There's scant information on the repairs to these three pumps beyond what I've gotten, but there is one more data point. A Flickr user named "billread" visited the 17th Street site at the time, apparently on July 12, 2009. He snapped a couple of shots (on Flickr here and here):





The tipoff that something is up in these shots is the presence of the Corps' rental crane up on the main structure. Unless it is needed, it is parked off the structure on the Jefferson Parish side of the site.



Unfortunately one cannot see pump W10 in these shots, but we have a good sight line at many of the bridge pumps, including what I believe to be then-recently repaired bridge pump number 6.

Zooming in, take a closer look:



Here's the same area as viewed from the other photo:



Given that bridge pumps #1 and #6 were pulled, it appears the pump I've marked as "Clean" is pump #6 in the water immediately after its refit. The cleanliness is an indication of a fresh paint job, while the rust or marine growth on the adjacent units indicates they've been in the water a while.

Combined with 1) the position of the crane toward the west end of the structure, and 2) the lack of scaffolding which was called for in the RPL contract (though that's not conclusive because we don't have definitive evidence that scaffolding was built; but it most likely was), and I believe these pictures were taken very soon after replacement of one or both of the bridge pumps. That would make sense, because it shouldn't take more than about a month to refit those pumps, and they were apparently pulled around the first week of June. The inspection dive by independent Divers was on June 4, and the QRI oil spill cleanup for the pump pull was likely on June 8 (according to QRI task order number 6)

But more important is that W10 and bridge pumps #1 and #6 were not the last pumps with corrosion-related hydraulic oil leaks - not by far.

In early July, 2009, there was another flurry of activity at 17th Street. Another task order (number 3, linked here) was issued to Healtheon for hydraulic pump repairs, this time for four more 60" pumps at 17th Street. The value was $349,357.35, and was later increased to $405,596.95 (in modification 1, to include "additional expense that could not have been anticipated until commencement of the work.") On this new work order, a lot more detail was provided.

The four pumps were 5E, 7E, 8W, and 9W. The problems with the original order of 34 pumps - of which pump 5E is a part - are well documented and legion.

As mentioned above, the main Healtheon contract called for creation of reports documenting the work they and their subs did on each task order. For task order #3, a report was generated in (I'm guessing) early September, 2009 after the work was complete. However, since it summarizes all the work happening on pumps 5E and 7E from July to September, including the initial evaluations, it's fair to assume that key Corps personnel were informally made aware of Conhagen's findings all throughout July. Though, considering they were already in the middle of replacing the piping on three other pumps because of corrosion, it's doubtful those findings came as a surprise.

You WANT to read this report and look at the pictures in it. Trust me.

The report includes passages such as the following:

"The first thing we did was to remove the inspection covers from the pumps and check the inside of the pumps for oil. Of the 4 units overhauled, only pump 5E had a significant amount of oil inside the pump (several hundred gallons). The oil was pump[ed] out and disposed of."

"Several hundred gallons:" verbiage similar to the Healtheon task order #3. However, in the task order, it was pump 7E that had the spill of approximately 700 gallons. Note that quantity is unlikely because the reservoirs on the engine skids hold less than half that volume of hydraulic oil (unless they were continually filling the oil reservoirs after test runs - yikes!). It probably represents an estimate of internal volume of the pump. In any case, while it is unclear if it was 7E, or 5E, or both that had major oil spills, it is crystal clear there was a bunch of oil in the water.

A spill of this size would certainly need to be reported to the state, and a spill of any size must reported to the National Response Center. Searches of both databases (including through public records requests) do not turn up reports for the spill, though QRI was dispatched - and presumably paid - under task order #7 of their spill cleanup contract. Curiously, the QRI task order claims the spill was small - between 10 and 50 gallons. The Healtheon task order for the repairs and the Conhagen repair report do not seem to describe a small spill. Given what Conhagen found, I'm strongly inclined to believe their narrative over the QRI task order.

And just to clarify (or confuse): it's not clear whether the pictures in the report below are of pump 5E or pump 7E, but the point is really moot. All the pumps sit in the same water, see the same weather conditions, and are exposed to the same salinity. Any problem that strikes one of them in such a catastrophic fashion would strike all of them similarly. And when I say, "all," I mean ALL the pumps across all three sites.

Other excerpts from Conhagen report:

"The motors and motor stands were removed from the diffuser housing. These were initially supposed to remain mounted, but the corrosion on the piping connection on the motor stand required that they be removed to make the repairs. One of the motors (5E) also had to be replaced due to being contaminated with water and having internal corrosion."

The Rineer motors were blamed for the severe vibration problems that struck these pumps upon their startup during the summer and fall of 2006. All of the motors were supposedly fixed by installing sturdier springs. It is extremely unclear if the Conhagen replacement of the Rineer motor on pump 5E included the sturdier springs, or if they went with the standard Rineer "Code 61" motors with flimsier springs that caused the original problems. Note that a beefier, high-pressure "Code 62" Rineer would work as well, and original installation of Code 62's in 2006 would have likely avoided the vibration problems altogether. That's because the standard Code 61 model has a maximum pressure of 3000 psi, which is also the operating pressure of the system. The maximum pressure of the Code 62 is 4500-5000 psi, well above the operating pressure and a much wiser choice (that's why they had the Code 62 spring package installed eventually). The Corps has not responded to a FOIA request for the purchasing details on the replacement Rineer motor. The motor (and other later replacements this year) were likely pulled from the Corps' stock of spare motors, but the lack of response to the FOIA request leaves that as speculation.

[Update, 7/23/12: The Corps eventually did respond to this FOIA request in July, 2011, and confirmed that the Rineer motor placed in pump 5E was from their stock of spares, and that all their spares had previously been upgraded with the Code 62 spring package.]

Then there's the final two paragraphs of the report:

"All of the piping was removed from the pump and motor. All of the piping showed corrosion. The piping and flanges internal of the pump had the most severe corrosion. Wall thickness in some of the areas was down to zero. The flanges were severely corroded with the heads of some of the flange bolts completely corroded away.

The severe corrosion seen on the piping internal of the pump should be cause for concern in the remaining pumps at this site. Leaks that were imminent on these pumps are sure to be imminent on the remaining."

One does not have to be an engineer to grasp the meaning of what Conhagen wrote there. In fact, they are even understating the matter, since they note that hundreds of gallons of oil had already leaked, so the corrosion failures were no longer imminent - they were existent. Hydraulic oil that was supposed to be going to the motor to turn the pump impeller was going straight in the water.

Pump 5E was part of the original order of 34 pumps. Pump 7E went in shortly after the original order as part of the Phase 2 installation. After periodic dunkings in 2006, both had been in the water continuously since about March or April of 2007. They, along with the rest of the original 34 and the later six at 17th St, spent most of the fall and winter of 2006-07 in the shop getting new Rineer springs and having piping extensions installed.

So under the most generous interpretation, the corrosion clock started ticking in about April of 2007, with the pumps having the same paint as originally specified. By June of 2009, one or both of the pumps' internals had completely corroded, causing catastrophic failure and an inability to drain water from the city during a storm event when the gates were closed.

Reading about it is one thing. Seeing the pictures is another. For comparison purposes, let's look at the pumps before they were installed. These pictures were taken in the spring of 2006:


and then just to the left of those:


Let's zoom into the pump on the left:


And then zoom in further for a closer look at the Rineer motor:


"HP" means high pressure, "LP" means low pressure. All the piping is carbon steel. The case drain line transports small amounts of oil away from the motor. Keep in mind these photos were in 2006. All the hydraulic pumps went into the shop in late 2006 and early 2007 for refits on the Rineer motors and addition of piping extensions to the points where the oil lines go through the pump housing. The paint system used for those refits was identical to the original system applied during manufacture in early 2006.

Here's pictures of either 5E or 7E at 17th Street when it was pulled in July 2009:


To make comparisons with the pictures above easier, I've labeled the pipes:


Here's a section of pipe removed from the pump sitting on the table at Conhagen (the fitting at the bottom indicates it may be from the inlet or outlet of one of the hydraulic coolers)


It appears that a large chunk of the pipe is completely gone in this photo. Also note the rusty hose fitting on the table:


Damage like that is the result of two things: 1) poor initial design related to corrosion protection and painting and 2) an utter lack of preventative maintenance. It is inconceivable that a properly specified and maintained piece of equipment - especially one required for life and safety like these pumps - could have a hand-sized hole corrode away before someone responsible did something. Inconceivable for anyone but the Corps New Orleans District.

The report mentions rust inside the Rineer motor, requiring its replacement. Besides the rust on the pipes causing holes, it's also likely that a failed, rusty seal on the bottom of the motor might have also contributed:


There was marine growth all over the things. Here's one of the oil coolers mounted to the sides of the pumps.


The still-hot hydraulic oil passes through these after leaving the Rineer motor but before returning to the reservoir on the engine skid. The residual heat is supposed to be transferred to the canal water. How much heat could really be transferred with all that fouling?

Here's some of the piping installed over the winter of 2006-07, specifically the return piping (two larger lines) and the smaller case drain line:


Is it any wonder the lines started corroding?

The same lines, with a bit more of the pump housing:


It's a wonder there's not giant holes in the sides of the pump, rather than just the pipes!

One doesn't need to be a genius to look at these pictures - and others I haven't highlighted - in the Conhagen report and wonder, "If this is what just one or two of the pumps looked like after just two years, what about the rest of those pumps that went in the water at exactly the same time and saw exactly the same conditions? Shouldn't they have gotten yanked out too, and right quick?" After all, much of the damage is simply not visible by removing an inspection hatch and looking down into the pump for oil leaks. For example, the Rineer motor corrosion on pump 5E was internal to the motor. The only way to see it was to pull the entire water pump, cart the pump to a shop on a flatbed truck, and cut the piping leading to the Rineer because the bolts holding the piping to the motor had rusted to nothing. Then the motor itself had to be cracked open and inspected.

Well, the Corps did none of that. They did the repair work on water pumps 5E, 7E and the other two pumps - 8W and 9W - laid out in Healtheon task order #3, and then stopped. The hopelessly corroded carbon steel pipe sections were removed and replaced with stainless steel (what should have gone in originally). 5E and 7E went back in the water in early September and underwent some cursory on-site performance testing. The other two pumps went through the same repair process during September (oddly there's no QRI task order for spill cleanup during the removal of those two pumps), finally getting back in the canal in early October.

According to publicly available contracting data for 2009, the work on those four 17th Street pumps was the last repair work done on any of the hydraulic pump systems for the 2009 hurricane season. Despite having a report from their own contractor warning them that all the other water pumps were imminently close to failure, the Corps New Orleans District (yes, the local Corps office) apparently did nothing additional for the entire 2009 hurricane season to address the situation. And of course, they never informed the public about any of this, despite a series of public meetings and presentations about the outfall canals.

Take a look at those pictures above, which show the damage after just over two years immersed in the canals. Imagine what another year in the water has done to the pumps. How does that make you feel? How does that inaction reflect on the local leadership and decisionmakers of the New Orleans District?

So what do we have here? We've got the Corps knowing that hydraulic water pumps are rusting to bits and doing practically nothing about it for the whole 2009 storm season. We've got oil pouring into the canals through those rusty bits, and no evidence of anyone reporting it to the federal government. We've got the Corps refusing to turn the rusty pumps on during a big storm in December, despite water depths in not one but two of the canals exceeding the Corps' own criteria for activation (more on that one later).

And finally, we've got the Corps embarking on what can only be called a crash repair program to belatedly fix the problem, but not telling anyone about it despite numerous contemporaneous public meetings and interviews.

As I said at the beginning of this post, for the last couple of months the Corps has been yanking nearly every hydraulic pump out of the 17th Street and London Avenue sites and sending them out for corrosion-related overhauls. The first task order of the year - number 4 on the Healtheon contract - was issued February 1, 2010 and is valued at $238,846.95. It is for the repair of 17th Street Phase 1 pumps W5 and W6. The scope of repairs for those two pumps is more extensive than for anything done during 2009. Not only was all the interior and exterior carbon steel piping to be replaced with stainless steel, but also both external oil coolers on each pump were to be completely swapped out, switching out carbon steel for stainless steel. When one considers that other replacements include seals, bearings, Rineer motors, gland packing plates, hoses and fittings, as well as an estimated 24 man-hours of weld repairs on each pump, it's easy to conclude that there is almost nothing usable left on these pumps after a little over 2 years in the water.

The Corps is in such a hurry that they are pulling out more than two at a time at 17th Street. I've had a FOIA request in for that information for nearly two months, but it has not been fulfilled in full yet. I got task order number 4, but they still have not transmitted task orders covering the numerous other pumps that have gotten yanked this spring. Despite that, I'll have more on that work in an upcoming post.

[Update, 6/12/10: I wrote about the rest of this spring's work in "This year's scramble" on June 3. The Corps transmitted contract information for the other pumps pulled out this spring on June 11. I posted about it in my June 11 post, "Worse than previously known". Despite the work this spring, the vast majority of the large hydraulic pumps remain corroding and unrepaired today.]

But that's enough for now. I'm going to have much more on this story over the coming days, because it has a lot of tentacles. Next up is how the Corps got into this mess. Stay tuned...

[Postscript: The Corps closed out task order #3 on the first Healtheon contract in September, 2010 with modification 2. They reconciled the Healtheon/Conhagen invoices and subtracted $20,005.63, meaning the total cost of rebuild for the four 17th Street 60" pumps was $385,591.32]