Worse than previously known

This entry was updated in July, 2011 to bring it in line with revisions made elsewhere on this blog. These revisions came about after a large, long-delayed release of pump repair documents in June and July, 2011 in response to FOIA requests.

At the time this post was originally written - June, 2010 - the title referred to the discovery through receipt of contract documents that same month that not only were the carbon steel piping and oil coolers being replaced on the pumps under repair, but so were critical mechanical components including bearings, seals, and the Rineer hydraulic motors. This meant the pumps were essentially being completely rebuilt a very brief time after their placement in the canals in 2006/07.

At that time, I believed the first pumps to get new bearings, seals, and hydraulic motors were the ones removed in mid-March, 2010 under task order #2 of the second Healtheon contract. But the documents received in July, 2011 show that pump repairs before that also included bearing, seal, and Rineer replacements, specifically the pair of pumps taken out of 17th Street in February, 2010. In addition, it is possible some of the 2009 repairs also included similar work, but that cannot be confirmed until receipt of additional documents.

I have included the information on bearing, seal, and Rineer replacement in the rewritten version of the prior post, "This year's scramble." As such, the tone of "discovery" found in the post below is somewhat muted when you read both entries in their original chronological order. At some point, I'll probably come back and clean this up.

This complement of repairs would eventually become the standard set, finally getting completely written into the specifications for the third Healtheon/Conhagen pump repair contract, signed in January, 2011.

The Corps issued a second pump repair contract to Healtheon/Conhagen on March 8, 2010. This new contract, W912P8-10-D-0054, was a small business set-aside to Healtheon again. It's got a maximum value of $2.875 million and a maximum term of three years. Task orders started going out on it shortly afterward; I reported on that work in my last entry, "This year's scramble." The contract documents fill in some gaps and give important new information on how messed up these pumps really are presently as we head into the height of the 2010 hurricane season.

The second contract anticipated the eventual overhaul of only some of the 60-inch hydraulic water pumps that sit in the canals at all three sites - 17th Street, Orleans Avenue, and London Avenue. $2.8 million is about enough for overhauling 16 pumps - far fewer than needed overhauling when the contract was signed in March. As we will see, they have already burned through almost half the expected contract value of the three year deal in just three months.

The basic scope of work, now standardized, matches that of the last task order (no. 4) on the first Healtheon/Conhagen contract. That is, all the carbon steel piping and piping hardware on the pumps is to be replaced with 316L stainless steel piping and hardware. That includes the piping inside the pumps, the piping outside the pumps, and the pairs of oil coolers on each pump. They've all seriously corroded. I laid out the details on what is involved in that work in my recent posts "Imminent," "How did the pumps get from...," and "This year's scramble."

Task order #1 - Minimum guarantee
Task order #1, issued two days after the main contract, was merely the contractually-required payment of a minimum guarantee of 2% of the overall potential value of the deal, or $57,500. Because the value of the repair work since the start of the contract on March 8th easily dwarfed that amount, this was credited back to Healtheon with a modification on October 15, 2010.

Task order #2 - four pumps, with lots more work
That brings us to the actual work on the pumps. Remember that 17th Street pumps W5 and W6 were already overhauled in February and March under task order #4 on the old Healtheon contract (details and pictures here). Under the new contract, the first work was $632,308 task order #2, effective on March 19th, but likely including work already underway days before that. This task order has stuff we already knew, but with a wrinkle.

We already knew which pumps were pulled out under this task order. They were:

17th Street: E1 and E2




London Avenue: E3 and E4



What we didn't know was how extensive the work was on these four pumps.

Not only was all the piping switched per the "standard" overhaul (that's kind of crazy when you think about it - just three years after going in the water for good there's a standard overhaul to make these things barely functional), but pretty much the entire guts of the pumps were also completely ripped out and replaced. We know this because a modification to task order #2 went out on April 26, 2010.

That $65,910 modification was likely after-the-fact and accounted for work already performed at Conhagen's shop by that point. The scope of extra work is scary:

For Pumps – 17th St. 1E and 2E, London Ave. 3E and 4E:
1. Repair/Change Out Bearings Hydraulic Pumps 4 Ea. As follows:
- Remove Pump Shaft Bearings
- Replace Radial bearings Impeller End (3 Each)
- Replace Radial Bearing Impeller End (1 Each Customer
Furnished)
- Replace Radial Bearings Motor End (3 Each)
- Replace Radial bearing Motor End (1 Each Customer Furnished)
- Replace Thrust Bearings (3 Sets)
- Replace Thrust Bearings (1 Set Customer Furnished)
- Replace bearing Spacers (3 Each)
- Replace Hydraulic Motors (4 Each Customer Furnished)
- Inspect Shaft And Housing
- Mount New Bearings
- Clean All Parts For Assembly
- Reinstall Pump Shafts

[Pictures and details of these repairs can be found at the previous entry, "This year's scramble."]

The list of repairs to the moving parts above covers every moving item inside the pumps, including the motor that turns the impeller. And since this happened on pumps from both 17th Street and London Avenue, we know the problem (failure-level corrosion of all internal moving parts) is systemic to all the pumps, not some site-specific phenomenon.

Add these replacements to the replacements of all the piping and the oil coolers, and one can say with confidence that there is nothing of substance left on these pumps except for the shafts, the impellers, and the housings. And as we know from the repairs Conhagen made in 2009, much of the housings were probably gone too.

Think about that for a moment: every critical component of these pumps had to be replaced just four years after they were ordered. There's no reason to doubt that most of the rest of the pumps (excepting the few most recently overhauled) are in the exact same rotted condition.

In future work, this suite of repairs (all piping replaced, weld repairs to housing, all bearings and seals replaced, and replacement or reconditioning of Rineer motors) would become the standard set of repairs. Any pumps that didn't receive it - including the pumps repaired in 2009 - would be suspect and would likely fail.

Task order #3 - the last round this spring
That brings us to the final round of pump rebuilding this spring, task order #3 on the new contract. Task order #3 had an initial value of $619,642 and an effective date of April 12.

The scope was for the repair of four more pumps:

17th Street: E6 and E8

London Avenue: W3 and W4

This squares with the chronology we saw play out at the gates, as The Lens' Steve Beatty captured pictures on April 15th at 17th Street of pumps E6 and E8 coming out:




Unfortunately, I don't have any photos of removal of pumps W3 and W4 on the west side of the London Avenue site, but I do have witnesses who tell me they saw the elbows on the ground in late April. Additional documentary evidence received via FOIA requests confirmed the work on them.

As with the work under task order #2, a modification to this task order was also issued in June, 2010 to account for bearing, seal, and Rineer motor replacements. Modification 1 to task order #3, for $52,975.34, was issued on June 11, 2010 with a projected completion date of June 16, 2010 (a two week extension of the original completion date of May 31, 2010. The scope from the task order #3 modification mimics that of the task order #2 modification. It reads,

"5. Line item 1003 – Growth work for pump repair, replacement shaft seal components listed below:
- Repair Seals 4 Each
- Remove Pump Shaft Bearings
- Replace Radial bearings Impeller End (4 Each)
- Replace Radial Bearings Motor End (4 Each)
- Replace Thrust Bearings (4 Sets)
- Replace bearing Spacers (4 Each)
- Replace Hydraulic Motors (4 Each Customer Furnished)
- Inspect Shaft And Housing
- Mount New Bearings
- Clean All Parts For Assembly
- Reinstall Pump Shafts"

The Conhagen repair reports (report for 17th Street pumps E6 and E8 and report for London Avenue pumps W3 and W4) for the work on these four pumps shows the extent of damage before the repairs.

Let's start with the suction bell and the liner on 17th Street pump E6:


The marine growth and rust are fairly obvious. However, it is the inside of the suction bell that truly shows the effects of corrosion.

Here's a detail picture taken through the top of the liner:


And here's two more details of the suction bell with the liner section removed:



It looks like something took a bite out of the carbon steel. This amount of corrosion is unaccepatable for a pump in such critical service, especially one that was only in the water three years.

Like the previously rebuilt pumps, the piping connected to the Rineer motors on these pumps showed signs of severe corrosion:



The damage to the internal drive assemblies indicated water got into places where seals had failed. For example:


And while I cannot be sure, this photo of removal of the shaft assembly from the bearing housing appears to show discolored liquid - possibly an oil/water mix - flowing out of the supposedly sealed housing:


There's also evidence there of rust on components that should definitely not be rusty, including some ugly discoloration on the shaft itself. It's clearer on a set of before and after pictures of the same area on one of the London Avenue pumps:




Note how shiny the shaft is in the lower "after" photo compared to the upper "before" one.

Pictures like these show why it was important to replace all the components like bearings and seals. Those parts could not do their job with this level of contamination by water.

Task order #3, modification #2 - a sidebar about Rineer motors


Modification 2 to task order #3, for $53,241.07, was issued retroactively on July 8, 2010 with an effective date of July 1, 2010 and a projected completion date of August 6, 2010.

The text of modification 2 to task order #3 says:

"For Pumps – 17th St. 6E and 7E, London Ave. 3W and 4W:
Provide the services of the hydraulic motor manufacturer, Rineer Hydraulics, Inc. to rebuild the following motors to be used for repair of the pump assemblies of this task order, with the remaining motors to be returned as spares.
Description: Hydraulic Motor, Rineer 125 Series, Quantity: 12"

The verbiage on this task order is somewhat odd. It claims four of the motors are meant for the four identified pumps. But those pumps were already back in the water by the time this modification was issued three weeks later. More than likely, the motors for the four pumps pulled under this task order were sent to Rineer in San Antonio in April and were rebuilt, and this task order is an after-the-fact modification to account for them and 8 others.

It would be helpful at this point to have some background on the Rineer hydraulic motors. Here's a quick reminder of where the motors sit inside the pump:


The motor spins the shaft through a series of gears. The shaft is connected to the impeller and rides on bearings inside a (theoretically) sealed chamber. We've seen above those seals failed. This photo is taken through the outlet of the pump while it sits on the ground horizontally. When in service, the pump is mounted vertically and water would flow up through the pump. Here's a more detailed shot of the motor and the hydraulic lines going in and out of it:


The motor is powered by high pressure (HP) hydraulic oil, which flows from the remotely positioned engine through the HP lines. The hydraulic oil loses its presurized energy to the motor and flows back through the low pressure (LP) lines to the engine.

In addition to the twelve motors rebuilt under the July, 2010 modification to task order #3, just a month later the Corps put out solicitation W912P8-10-R-0077 for purchase of 12 more hydraulic motors. 8 of the motors were for the 60" pumps, the other 4 motors were for the 42" bridge pumps found only at 17th Street. The contract for the 12 Rineers - number W912P8-10-C-0116 worth $104,760.00 - was awarded September 14, 2010 to Hydraquip Distribution, a Rineer representative with headquarters in Houston and an office located in St Rose, LA, just west of Armstrong International Airport in the New Orleans metro. The completion date of the contract was December 15, 2010.

The solicitation included model numbers for the new Rineers. Remember that back in 2006 the Rineers were undersized "type 61" models. MWI specified them on the edge of their operating envelope, which led to the entire pump units vibrating madly. The pumps were nearly immediately withdrawn from service after being put in the water in the summer of 2006. They didn't return until spring, 2007, after undergoing major repairs to the Rineers and the addition of piping extensions. Other necessary repairs didn't take place until later in 2007.

The vibration fix to the Rineers was to effectively turn them into more robust "type 62's" with stiffer springs (allowing them to handle pressures up to 4500 psi rather than the type 61's 3000 psi), though the seals were possibly not as sturdy, which likely explains a number of the oil spills in 2006, 2007, and continuing to the present (though the extreme corrosion over the years has also played a large part). The vibration problem was the only one the Corps acknowledged publicly with MWI's pumps through 2006 and into 2007, until Maria Garzino's May, 2006 memo was released in March, 2007. After that, all hell broke loose, and it was abundantly clear how troubled these pumps were - and are.

It's worth remembering that in January, 2008 - almost a year after their supposed "fix" to the vibration problem - the Corps admitted that the pumps actually still vibrate. Corps engineer Ray Newman said it in January, 2008 to an interviewer from PentagonTV (click here for Molly Peterson's KPCC "Pumps Under Pressure" special report. Scroll to the right in the timeline until you come to January, 2008 and click on note, "Engineer to PentagonTV: 'Pumps vibrate'"):

"The pumps were not designed to run at low canal level, and they do tend to vibrate when you run that, for instance, a zero canal level."

With all that in mind, it is germane to wonder whether the new Rineers the Corps bought in 2010 are the "upgraded" type 61's or full blown sturdier type 62's. From the solicitation (and confirmed in FPDS-NG), here's the model number for the 8 motors meant for the 60" pumps:

M125-A4-1H-250-31-B1-TBB-213

Rineer provides product documentation on their website. And we know which document to look at, because the Corps included it in their solicitation: "Mounting dimensions, shaft dimensions, port dimensions and configuration shall match the envelope dimensions given in Rineer Hydraulics, Inc. publication DS1251009 4/04."

So, looking at that publication, we find that the "1" in the "1H" disappointingly refers to type 61. This kind of makes sense, since "type 61" in hydraulics-speak refers to not only a presure rating (~3000 psi), but also a standard dimensional configuration for connections. Given some of these motors would be going in existing pumps which have pipes with type 61 end connections, it makes sense to keep the bolt holes aligned. However, given that they are being ordered for pumps that are having their guts torn out anyway, one wonders why the Corps simply didn't upgrade the motors and the piping to type 62, which has a 4500 psi rating, far above what the engines powering these motors can put out.

Anyhow, I believe "213" in the last part of the model number refers to whether the springs are upgraded or not (note: the 213 also appears at the end of the model numbers of the 4 Rineers purchased for the bridge pumps). Rineer's nomenclature refers to this last section of their model number as a "special code designator." That is, it's a catch-all where they describe any special modifications done to the motor. It could refer to anything from a different paint color to a highly technical customization. Without knowing exactly what "213" refers to, we can't know anything for sure, other than these new motors are not stock items. That is reason to hope the springs in them are upgraded. But the continuing use of type 61 motor housings still makes me wonder whether the seals will hold up over the long term.

Hopefully the "213" covers all the modifications needed to make these things work, including the stiffer springs, because the Corps definitely wanted the upgraded models. The solicitation calls for,

Specifications -
Displacement: 250 cubic inches per revolution
Pressure Rating: 3000 psi (or higher)
Rated Speed: 350 RPM (or higher) continuous
Horsepower Rating: 600 HP (or higher) continuous

While the Rineer literature says,

"HORSEPOWER LIMITATION -
Maximum horsepower limitation may vary with different applications. When using the 125 Series standard motor above 300HP, consult a Rineer Application Engineer."

This was a significant bone of contention during the Corps' 2006-07 internal investigation of the pumps. We can only hope they didn't make the same mistakes this time that they made in 2006.

Finally, it's unclear exactly how many Rineer motors the Corps had on hand after all this contracting activity. Presumably the new motors purchased in September, 2010 went into pumps as they got yanked out for rebuilds, perhaps implying that the July, 2010 modification to repair 12 Rineers didn't work out, or maybe implying the Corps wanted as many Rineers as possible in anticipation of many future pump repairs. Purchases and rebuilds of Rineers show up on other task orders as well. Whatever the specifics, it's just more confirmation of my statement above: every critical component of these pumps had to be replaced just four years after they were ordered.

Burning through cash

Speaking of money, remember that the expected value of the new Healtheon contract was $2.875 million. By early June, 2010, the Corps had already issued task orders worth $1.32 million on 8 pumps and they were only three months into the three year contract. Jumping ahead here, there were still 25 sixty-inch hydraulic pumps which had not been touched, including all 10 pumps at the Orleans Avenue canal. Another 5 had incomplete repairs in 2009 and would likely need to come out soon as well. Doing the math - they only had cash for complete overhhauls of about 10 of those 30 pumps. It's obvious the New Orleans District either hasn't set aside enough cash (from the permanent pump station funds - quite a pickle they've put themselves in!) for this work, or they plan on just rolling the dice on most of the unrepaired pumps for the next three years. And that doesn't even take into account the fact that the permanent pump stations meant to replace these pumps won't be ready until the 2015 hurricane season.

[Note: a third contract to Healtheon/Conhagen would be issued in 2011, with a potential value of $6.75 million. See this March, 2011 post and this May, 2011 post for details.]

Corps sleight of hand...
Originally the Corps wanted the 17th Street and London Avneue pumps of task order #3 back in their respective canals May 31st ahead of the June 1st start of the 2010 hurricane season. With the additional work to replace bearings, seals, and hydraulic motors, there was a slight delay in the completion date. As we noted above, it was extended to June 16th in modification #1.

This presented a potential PR problem for the Corps. Every year since 2007, the Corps has run a hurricane preparedness exercise which involves turning on pumps for the press at one of the closure sites. With two sites missing pumps, and the Corps likely nervous about getting questions on the condition of the remaining unrepaired pumps, they moved the press availability to the only site without pumps out for repair: Orleans Avenue. They used a similar maneuver in 2009 when, in order to hide the repair work happening at 17th Street (pump W10 was getting pulled at 17th Street the same day as the drill), they had the press out to London Avenue.

In the case of 2010 event, they didn't put out a press release to the public until the day of the event, after it had already taken place.

And with all the BP oil spill coverage, it's no surprise hardly any press came. Fox 8 sent a reporter and they got some video, but it is long gone from their website. Industry cheerleader Engineering News Record sent a reporter as well, who dutifully reported what the Corps told her. Notable about the drill - at least the part when the press was there - was the fact that the Corps didn't even lower the Orleans Avenue gates when they turned the pumps on for the media:


Seems like they're not even trying anymore, because exactly what kind of drill is it when the gates don't go down?

Wrapping up

Conhagen eventually didn't need to work all the way until June 16th. The 17th Street pumps went back in on Wednesday, June 9th. Karen Gadbois of The Lens, photographed this work on the east side of the site:


London Avenue pumps W3 and W4 went back in a couple of days later.

After all the invoices were received, modification 3 to task order #3 was issued May 18, 2011. It subtracted $12,807.35, bringing the total for all the task order #3 work (including rebuilding of 8 Rineer motors not directly related to the 4 pumps repaired under this task order) to $713,051.23.

Summary and conclusions


I'll went back and updated my last post to reflect what I learned from the task order #2 and #3 contract documents. My basic conclusions from that post remain in effect. That is, the vast majority of the hydraulic pumps across all three sites remain in a dangerous state of decay and some will very likely fail or underperform during a storm event. This will occur because of extensive corrosion of all the internal and external parts on the partially or completely unrepaired pumps.

Here's the numbers of unreliable corroding 60" pumps:

17th Street: 12 (7 pumps unrepaired and 5 pumps repaired in 2009 with carbon steel piping)
Orleans Avenue: all 10
London Avenue: 8
(note: the 11 direct drive pumps at 17th Street and 8 direct drive pumps at London Avenue are likely reliable. Also, these numbers do not include the 14 smaller 42" bridge pumps at 17th Street, which the Corps is repairing themselves and for which I have not received any documentation. Those bridge pumps put out an estimated 1600 cubic feet per second, about 18% of the total output at 17th Street, and should be considered unreliable pending further information.)

That's 30 out of 40 major hydraulic pumps, or 75% of the major hydraulically-powered drainage capacity which the Corps is crossing their fingers on this summer. Thank goodness for the direct drive pumps.

Here's the effect of those un-overhauled or partially overhauled pumps in terms of cubic feet per second of total drainage capacity at each site:

17th Street


Orleans Avenue


London Avenue


These numbers are actually the best case, because they assume that the bearings and the Rineer motors are functioning well in all the pumps in which they have not been replaced, which - as of June, 2010, was all but the eight pumps repaired under task orders #2 and #3 described above.

I do not enourage citizens to depend upon these pumps during a storm event of any magnitude.

Karen Gadbois and Molly Peterson contributed to this report.

This year's scramble

After I received a large number of documents related to the New Orleans Corps' corrosion rebuilds of the hydraulic pumps in July, 2011, I went back and rewrote all the prior blog entries I posted on the effort since May, 2010, including the one below. I incorporated photos and text from these new documents in order to give a fuller picture of what was happening.

At the time these entries were written, I was mainly relying on photography of the sites by friends to track the movements of pumps in and out of each canal. The Corps had turned over a few documents to me, but the bulk of the knowledge was coming from studying the site pictures.

With the receipt of all my requested documents, we now have the complete picture of the rebuild effort. During the time covered by the post (February through early June, 2010), 10 of the 40 large 60" hydraulic pumps were pulled out in three batches for complete rebuilds. At the time I wrote this entry (May, 2010), I wasn't aware of the complete extent of the work to fix corrosion damage. I knew more carbon steel piping was replaced with stainless steel in the 2010 work than in 2009, but I didn't know the critical internal components including bearings, seals, and Rineer hydraulic motors were also getting changed out. When I discovered in June, 2010 that was the case for the later spring, 2010 repairs, I wrote the entry after this one, "Worse than previously known." But even that knowledge was incomplete.

The complete set of contract documents and reports now shows that bearings, seals, and Rineers were being changed out in the first set of 2010 repairs as well. I include that information below, as well as information on the bearing, seal, and Rineer replacements during the other spring, 2010 repairs. This somewhat blunts the original tone of "discovery" in "Worse than previously known."

More important than changing tones, though, is the finding of such repairs on the first set of pumps pulled out in 2010. That moves that part of the rebuilding chronology back further than previously known, and shows how quickly these parts were rendered nonfunctional by the brackish water in Lake Pontchartrain, an important consideration with the installation of the permanent pumps intended to replace these hydraulic units now delayed until 2015.

Remarkably, many of the pumps at the three sites as of this writing - late July, 2011 - have still not been pulled out, over two years after these first extensive replacements. They have been sitting in the salty water of the canals for five years now, corroding into uselessness.

So we've detailed the repair work done on the rusty pumps last year, and the Corps decisions in 2006-07 which led to that work. The repair work is being funded with dollars that were supposed to go toward placement of a permanent pumping system at the lakefront. Instead, they're fixing their three-year-old screwups.

March 16, 2010
The first (and only public) inkling there was repair work happening in 2010 was from a TV news story about those permanent pumps: Fox 8's March 16, 2010 piece about the signing ceremony for the permanent pump station Project Partnership Agreement between the state and the Corps, held at the 17th Street site. Fox 8 was one of three media outlets there, along with WWL-TV (story here) and the Times-Picayune (story here). Unmentioned by any of the stories was the fact the document had actually been signed four days previous to the ceremony.

But the Project Partnership Agreement is beside the issue. What came out of the Fox story was the first real-time footage showing the repair work to the pumps. Fox 8 got two shots of 60" 17th Street pump W5 being placed back in the platform after its earlier removal for repairs. Here's a screencap from the first shot:



You can see the "W5" label on the elbow, clearly identifying this pump.

Here's another shot of W5 going back into the structure. The new zinc anodes on the pump's suction bell installed as part of the repairs are clear in this shot.



This March 16, 2010 replacement was the end of the work on $238,846.95 task order #4 of the first Healtheon contract, which involved 17th Street pumps W5 and W6. This task order, issued February 1, 2010, was a watershed in the pump rebuild effort. Whereas the pumps pulled out in 2009 only had piping on their insides replaced with stainless steel...



...while the piping outside the pumps and the oil coolers were left as carbon steel and painted black, as shown in these photos from the Conhagen report on the repairs to 17th Street pumps E5 and E7 under task order #3 to the first Healtheon contract...


...this time around, not only was the internal piping changed out for stainless, so were the oil coolers and the external piping. That new external stainless steel piping can be seen in the Fox 8 footage:



That is, for the 2010 repairs to 17th Street pumps W5 and W6 under task order #4, every bit of piping touching the water or in the splash zone was upgraded to 316L stainless, even the oil coolers.

Here's a summary of the differences between the 2009 and 2010 repairs:

Piping internal to pump:
2009: 316L stainless steel
2010: 316L stainless steel

Piping external to pump:
2009: Carbon steel
2010: 316L stainless steel

All pipe fitting fasteners:
2009: Grade 8 (a medium carbon alloy steel)
2010: 316L stainless steel

And, the oil coolers:
2009: sandblasted and weld repaired, but original carbon steel remained
2010: completely replaced with new 316L stainless steel

Here's one of the pumps with the new stainless oil coolers and external piping on the Conhagen shop floor, immediately before its March 16, 2010 reinstallation:


Per standard practice, the stainless steel is left unpainted, appearing a silvery gray. That is why the new piping and oil coolers are lighter in color than the painted portions of the pump, as also shown in the Fox 8 pictures of W5 above.

And while the Fox 8 video tipped me off to the expanded work on the piping, it wasn't until the July, 2011 receipt of other contracting documents that we could know just how extensive the work was on W5 and W6.

One of those documents allows us to see the condition of the pumps when they went to Conhagen's shop in early February, 2010. The Conhagen repair report for 17th Street pumps W5 and W6 shows pumps corroded to the point of failure just two and a half years after being accepted by the Corps.

This picture explicitly tells the story of why the external piping had to be changed out:


That hole is on one of the oil coolers, which is on the exterior of the pump:


We already knew about the external piping changes. The repair report shows that stuff inside the pump, subject to the same corrosion from brackish water, had also rotted. Here's one of the Rineer hydraulic motors:


And even further into the innards, sections of the pump mechanism which had multiple seals protecting them from water showed obvious evidence of water infiltration. Here is the area around the shaft and bearings:


From the amount of rust inside the bearing housing, as well as the significant discoloration along the shaft, it is clear the seals completely failed. This would have exposed the bearings and shaft to salt water, causing failure of the pump. And there is no doubt similar damage had been done to the seals on the Rineer motor, damaging it as well.

So it makes sense that $23,511.72 modification to task order #4 on the first Healtheon contract was issued to address this issue. In addition to the original expansion of work to include the replacement of the external piping and oil coolers, this modification further expanded the rework to include bearing, seal, and Rineer motor repairs and replacements:

"Scope of Work in addition to original Task Order.

For Pumps – 17th St. 5W and 6W

TAKE DELIVERY OF EQUIPMENT AS FOLLOWS:
- Pick up Equipment at Job Site With our Truck
- Develop Complete Job Work Scope

MANUFACTURE/FURNISH THE FOLLOWING:
- Replace Radial Bearing
- Replace Hydraulic Motor (Customer Furnished)
- Replace Thrust Bearings (Customer Furnished)
- Provide replacement parts as required to recondition Mechanical Seals (2 Each)

REPAIR THE FOLLOWING:
- Remove All Cooling Lines And Hoses
- Remove Hydraulic Motor
- Disassemble And Inspect Motor
- Remove Pump Shaft And Bearings And Inspect
- Change Out Pump Shaft Bearings
- Clean All Parts For Assembly
- Reinstall Pump Shaft And Bearings
- Reinstall Hydraulic Motor
- Reinstall Lube Lines And Hoses
- Reinstall Seals And Impeller
- Retest System For Leaks
- Retouch Up Tar set Disturb During Disassembly And Reassembly
- Prep For Shipping

PREPARE FOR SHIPPING AS FOLLOWS:
- Bring Pump Back To 17th Street
- Retest Pump"

This modification was issued April 27, 2010, over a month after the pumps went back in the water. Considering the relatively small dollar value, this modification appears to be backdated. That is, if it represented new work - which would have involved crane work and taking the pumps back to Conhagen's shop for a second visit - it would have been valued much higher. A final September, 2010 cost reconciling brought the final cost of the work on this task order down by $10,263.75 to $252,094.92, indicating the only crane work was that paid for in the intial task order, and that the first modification was indeed only accounting for extra work done in the shop.

So when W5 and W6 went back in the water in March, 2010, they had essentially been completely remanufactured, with only the cast stainless steel impellers and the main carbon steel housings (which themselves had to have significant weld repairs) remaining from when the pumps were originally accepted by the Corps in September, 2007. That is an extraordinarily short service life, and it pointed toward the extensive threat posed by corrosion to all the other hydraulic pumps still sitting in the salty water, corroding away. That's why pumps started getting yanked in bulk for the rest of the 2010 spring.

Last week of March, 2010
Through 2009 and most of 2010, it was easy to tell when any of the 60" pumps got pulled. The Corps' had a crane contractor lift the pump and the elbow atop it in one piece. They unbolted the elbow and left it at the site, while the pump was trucked offsite to a shop. The hole left in the piping on the structure was obvious.

Here's an example from October, 2006, when I took pictures at the Orleans Avenue gates. All five of the pumps from the east side of the site had been removed the previous month due to internal and external oil leaks. They had been placed up on the deck, and the elbows were on the levee next to them:



It was pretty easy to see that the pumps were out:



So to see if pumps are out at any of the sites, all we had to look for are elbows on the ground and missing elbows on the structure. Soon after the March 16, 2010 replacement of 17th Street pumps W5 and W6, those telltale signs appeared elsewhere.

At 17th Street, pumps E1 and E2 were pulled in mid-March:







At London Avenue, pumps E3 and E4 were pulled at the same time:





This work was carried out under $632,308.77 task order #2 of the second Healtheon repair contract, issued March 19, 2010. These repairs followed the same pattern as that of the prior work on 17th Street pumps W5 and W6. The complete replacement of all carbon steel piping and hardware, along with pump housing weld repairs, was covered under the original task order, and a modification to task order #2 (worth $65,910.82 and again postdated, this time to April 26, 2010) called for replacement of all the bearings, seals, Rineer motors on all four pumps.

We can see why such extensive work was necessary in the pages of the repair reports from Conhagen for these four pumps. Conhagen wrote a report for 17th Street pumps E1 and E2 and a separate report for London Avenue pumps E3 and E4. According to the photos in those reports, the pumps were in bad shape when they came out of the water.

The oil coolers and external piping were covered with rust and marine growth. Here's 17th Street pump E1:


And here's 17th Street pump E2:


The London Avenue pumps were just as bad. Here's London Avenue pump E3:


And London Avenue pump E4:


This particular view shows the severe corrosion buildup on the piping which was normally submerged when compared to the piping which stayed above the waterline.

The interior guts didn't fare any better. Here's the inside of 17th Street pump E2:


These are vertical pumps. The picture above is taken looking down toward the pump's inlet. So oil would have leaked out of the Rineer toward the impeller, away from the viewer.

The damage inside London Avenue pump E4 is nearly identical, with rust everywhere:


As with 17th Street pumps W5 and W6 earlier in 2010, there was also evidence of water infiltration into the parts of the mechanism that were supposed to be sealed. Here's part of the drive of London Avenue pump E3 with its cover removed:


This should not be showing any signs of rust, but there they are.

There was substantial corrosion damage to the main pump housings as well. At the very bottom of the pump is the suction bell, the tapered portion of the pump which funnels water up to the impeller. The funnelling effect is assisted by tapered vanes. The suction bell is always submerged, and the brackish water took its toll. Here's 17th Street pump E2's suction bell:




The single pair of anodes on that same pump had long stopped working by the time it came out:


So had the 17th Street pump E1 anodes:


All these problems were addressed in the repairs. The carbon steel coolers and piping were replaced with stainless steel. Here's London pump E4, showing a typical cooler and piping replacement:


They also got new Rineer hydraulic motors, as shown on London Avenue pump E4:


Remember those rusty bits on the internal drive assembly? They were completely replaced (photo shows 17th Street pump E1):


And here's that same shaft assembly - upside down on a table, with a cleaned up shaft - just before it was placed in the bearing housing shown above:


Note the shiny new bearings.

So all four of these pumps got the full rebuild treatment during their stay in Conhagen's shop in March and April. According to acceptance testing records attached to each report, 17th Street pumps E1 and E2 went back in the water April 13, 2010. and London Avenue pumps E3 and E4 went back in April 16, 2010. Here's pictures of both sets back in their places:




The main 60" pumps were not the only ones getting attention in early 2010. Some of the fourteen 42" bridge pumps were coming out at 17th Street too. The March 16, 2010 Fox 8 report had captured evidence of that. The Corps had taped off the area around the bridge pump drives and there were hydraulic oil hoses up on the deck; they had been disconnected from one of the bridge pumps, possibly #2 or #3:


Also, the Corps' rental crane - normally kept off the deck unless needed - was parked looming over the gathered guests. From its position with its boom above the midpoint of the structure, it appeared there was work happening on bridge pump #6, #7, or #8:


Additional evidence of work on the bridge pumps comes from the Corps putting up scaffolding around to disconnect the hoses and unbolt the pumps from their elbows. As of mid-March, it looked like at least bridge pumps #7 and #8 had been pulled:


Here's a clearer view of the scaffolding:


Bridge pump #3 would come out later in March, but I don't have a good contemporaneous picture of that.

April 15, 2010
By mid-April, more pumps were getting rebuilt. At 17th Street, pumps E6 and E8 came out:



And at London Avenue, pumps W3 and W4 were pulled. This work was carried out under $619,642.17 task order #3 to the second Healtheon repair contract, issued April 12, 2010. The work under task order #3 was similar to that performed on the other six 60" hydraulic pumps earlier in 2010, with the major effort in replacing all carbon steel piping, oil coolers, and piping hardware with 316L stainless steel, as well as replacement of bearings, seals, and Rineer hydraulic motors. The details on the repairs to 17th Street pump E6 and E8 and London Avenue pumps W3 and W4 can be found in the subsequent entry, "Worse than previously known."

April 28, 2010
By late April, E6 and E8 were still out at 17th Street, and were due back in by mid-to-late-May (though that would be delayed until early June).

Back at 17th Street, it looked like they were still working on the same bridge pumps as back in mid-March, nos. 7 and 8:


While we can only see that #7 is definitely out, the scaffolding is still there for #8.

There's evidence that #3 was also still out (scaffolding), but it's not conclusive due to the angle of the photo.

The repairs so far
So the final tally that we know of:

2009 repairs (with only some stainless steel replacement):
17th Street: 60 inch pumps E5, E7, W8, W9, and W10, Bridge pumps #1 and #6
London Avenue: none
Orleans Avenue: none

2010 repairs (possibly definitely with total stainless steel replacement):
17th Street: 60 inch pumps W5, W6, E1, E2, E6, E8, Bridge pumps #3, #7, #8 (unknown scope of repairs), possibly other bridge pumps
London Avenue: 60 inch pumps E3, E4, W3, W4 possibly two other 60 inch pumps
Orleans Avenue: none

Or, in graphic form...

17th Street:


London Avenue:


Orleans Avenue:


Let me explain these graphics. The green pumps are the ones I'm pretty positive have gotten as much corrosion repair as possible, i.e. those fixed this year. I'm coming to that conclusion through FOIA results and deduction from which pumps were pulled since February.

The yellow pumps are either those that were repaired in 2009 (and thus still have carbon steel piping) or those pumps for which I don't have adequate documentation (see my comments about FOIA frustrations below). The red pumps are ones that haven't yet been pulled and had their corrosion damage repaired or replaced. All this is based on the best information I could get, which hasn't been a lot, thanks to the Corps New Orleans District.

These number and pictures mean that - by my best estimates - 75% 80% (32 30 of 40) of the 60" hydraulic pumps are unrepaired or partially repaired and continue to corrode far faster than they should:
17th Street: 12 (7 pumps unrepaired and 5 pumps repaired in 2009 with carbon steel piping)
Orleans Avenue: all 10
London Avenue: 10 (possibly definitely 8
(note: the 11 direct drive pumps at 17th Street and 8 direct drive pumps at London Avenue are likely reliable.)

And just a reminder here, these are what one of those unrepaired pumps likely looked like one year ago, after just two years in the water:







Imagine how they look now, after yet another year in the same brackish water without repairs.

As mentioned above, the totals of unrepaired and questionable pumps do not include an unknown number of 17th Street bridge pumps, which represent a total of 1600 cfs at that site. They are harder to get a handle on. The Corps is doing the repairs themselves, and they are not as visible to the public.

FOIA difficulties
And speaking of visibility, that leads me to how difficult getting this information has been. I have been forced to rely on reporter friends and acquaintances as primary sources when tracking the repairs, because the Corps failed at the time to reveal almost anything about the work. Fortunately, those friends and acquaintances are wonderful folks. Without the yeoman work of my good friend and award-winning reporter Karen Gadbois and her editor Steve Beatty of The Lens, as well as my good friend (and also award-winning reporter) Molly Peterson of KPCC-FM, this post would have been impossible, and this information would have been lost completely. It is grass-roots, on-the-ground reporting like this that truly makes good journalism, and I'm in their debt for their assistance.

However, it should not take sustained investigative reporting efforts and FOIA requests to find out the extent of damage and the status of repairs to critical, taxpayer-funded flood protection works - the Corps should be publicizing it. How can people judge their level of risk (hint - it's quite high) if they don't know the Corps' pumps are rusting to bits?

And while the fact-gathering on the ground was great, the FOIA process was terrible and grueling. My initial request for the 2009 repair information was filed in September, 2009. I had to wait until September because the government's central public contracting database, FPDS-NG, has a 90 day delay on displaying all Defense Department contract actions. So stuff that happens in June - like the 2009 repairs - isn't made public until September. I had to wait until I had the exact contract numbers and dates to make sure the Corps wouldn't worm out of my requests; if you don't craft them extremely specificly, you don't get anything. The Corps New Orleans District General Counsel's office does not volunteer anything for a FOIA request - you have to already know what you're asking for before you ask for it.

In response to that September, 2009 request, I did not receive contract documents until January, 2010. I then filed a second round of requests to get Conhagen's repair reports associated with that contract work, and did not receive a response to that (the Conhagen report on the repairs to 17th Street E5 and E7) until March. I didn't receive any other repair reports on the rest of the 2009 work until July, 2011, and never got emails associated with the first Conhagen report I got.

It didn't get any better when it came to 2010's round of repairs. After the Fox 8 report in March, 2010, when I repeatedly asked for everything related to the 2010 pump work, I had to wait another two months just to get task order #4 to the first Healtheon contract, covering the repairs to just two pumps - 17th Street pumps W5 and W6. Thanks only to all the photography and fact-gathering displayed above, we knew there was MUCH more contracting and repair activity in 2010 than just the work on those two pumps. I received a few additional documents on June 11, 2010. The remainder of my requested documents arrived over a year later, in July, 2011.

With all these delays, I concluded the Corps withheld as much information as possible on these repairs, because they knew how embarrassing they are. They didn't issued a single press release after May, 2009 (in which they only obliquely referred to the repair work, never using the word "corrosion"), they didn't mentioned the repairs once at any public meeting related to the gates and pumps, and they refused my own repeated requests for comment in connection with this latest series of posts.

Coming up next
At some point, someone in the mainstream media will pick up on this story, and the Corps will be forced to say something (because, you know, no one reads blogs). It'll probably be total spin. I suspect that may come about after my next post, in which I plan to detail the approximately 2000 gallons of oil (at least) spilled at the gates sites over the last four years, and how much of it in the last two years has simply gone unreported by the Corps to the state and federal authorities. That's another set of FOIA requests unfulfilled, at least by the Corps, which I will also detail. Fortunately, other agencies have been far more forthcoming in responding to my document requests. For now, I'll give you the list of 20 spills I've compiled [Update, 6/9/10: It turns out this list is very incomplete. I've come across 17 more incidents]. Stay tuned.

Karen Gadbois, Steve Beatty, and Molly Peterson contributed to this report