Worse than previously known

This entry was updated in July, 2011 to bring it in line with revisions made elsewhere on this blog. These revisions came about after a large, long-delayed release of pump repair documents in June and July, 2011 in response to FOIA requests.

At the time this post was originally written - June, 2010 - the title referred to the discovery through receipt of contract documents that same month that not only were the carbon steel piping and oil coolers being replaced on the pumps under repair, but so were critical mechanical components including bearings, seals, and the Rineer hydraulic motors. This meant the pumps were essentially being completely rebuilt a very brief time after their placement in the canals in 2006/07.

At that time, I believed the first pumps to get new bearings, seals, and hydraulic motors were the ones removed in mid-March, 2010 under task order #2 of the second Healtheon contract. But the documents received in July, 2011 show that pump repairs before that also included bearing, seal, and Rineer replacements, specifically the pair of pumps taken out of 17th Street in February, 2010. In addition, it is possible some of the 2009 repairs also included similar work, but that cannot be confirmed until receipt of additional documents.

I have included the information on bearing, seal, and Rineer replacement in the rewritten version of the prior post, "This year's scramble." As such, the tone of "discovery" found in the post below is somewhat muted when you read both entries in their original chronological order. At some point, I'll probably come back and clean this up.

This complement of repairs would eventually become the standard set, finally getting completely written into the specifications for the third Healtheon/Conhagen pump repair contract, signed in January, 2011.

The Corps issued a second pump repair contract to Healtheon/Conhagen on March 8, 2010. This new contract, W912P8-10-D-0054, was a small business set-aside to Healtheon again. It's got a maximum value of $2.875 million and a maximum term of three years. Task orders started going out on it shortly afterward; I reported on that work in my last entry, "This year's scramble." The contract documents fill in some gaps and give important new information on how messed up these pumps really are presently as we head into the height of the 2010 hurricane season.

The second contract anticipated the eventual overhaul of only some of the 60-inch hydraulic water pumps that sit in the canals at all three sites - 17th Street, Orleans Avenue, and London Avenue. $2.8 million is about enough for overhauling 16 pumps - far fewer than needed overhauling when the contract was signed in March. As we will see, they have already burned through almost half the expected contract value of the three year deal in just three months.

The basic scope of work, now standardized, matches that of the last task order (no. 4) on the first Healtheon/Conhagen contract. That is, all the carbon steel piping and piping hardware on the pumps is to be replaced with 316L stainless steel piping and hardware. That includes the piping inside the pumps, the piping outside the pumps, and the pairs of oil coolers on each pump. They've all seriously corroded. I laid out the details on what is involved in that work in my recent posts "Imminent," "How did the pumps get from...," and "This year's scramble."

Task order #1 - Minimum guarantee
Task order #1, issued two days after the main contract, was merely the contractually-required payment of a minimum guarantee of 2% of the overall potential value of the deal, or $57,500. Because the value of the repair work since the start of the contract on March 8th easily dwarfed that amount, this was credited back to Healtheon with a modification on October 15, 2010.

Task order #2 - four pumps, with lots more work
That brings us to the actual work on the pumps. Remember that 17th Street pumps W5 and W6 were already overhauled in February and March under task order #4 on the old Healtheon contract (details and pictures here). Under the new contract, the first work was $632,308 task order #2, effective on March 19th, but likely including work already underway days before that. This task order has stuff we already knew, but with a wrinkle.

We already knew which pumps were pulled out under this task order. They were:

17th Street: E1 and E2




London Avenue: E3 and E4



What we didn't know was how extensive the work was on these four pumps.

Not only was all the piping switched per the "standard" overhaul (that's kind of crazy when you think about it - just three years after going in the water for good there's a standard overhaul to make these things barely functional), but pretty much the entire guts of the pumps were also completely ripped out and replaced. We know this because a modification to task order #2 went out on April 26, 2010.

That $65,910 modification was likely after-the-fact and accounted for work already performed at Conhagen's shop by that point. The scope of extra work is scary:

For Pumps – 17th St. 1E and 2E, London Ave. 3E and 4E:
1. Repair/Change Out Bearings Hydraulic Pumps 4 Ea. As follows:
- Remove Pump Shaft Bearings
- Replace Radial bearings Impeller End (3 Each)
- Replace Radial Bearing Impeller End (1 Each Customer
Furnished)
- Replace Radial Bearings Motor End (3 Each)
- Replace Radial bearing Motor End (1 Each Customer Furnished)
- Replace Thrust Bearings (3 Sets)
- Replace Thrust Bearings (1 Set Customer Furnished)
- Replace bearing Spacers (3 Each)
- Replace Hydraulic Motors (4 Each Customer Furnished)
- Inspect Shaft And Housing
- Mount New Bearings
- Clean All Parts For Assembly
- Reinstall Pump Shafts

[Pictures and details of these repairs can be found at the previous entry, "This year's scramble."]

The list of repairs to the moving parts above covers every moving item inside the pumps, including the motor that turns the impeller. And since this happened on pumps from both 17th Street and London Avenue, we know the problem (failure-level corrosion of all internal moving parts) is systemic to all the pumps, not some site-specific phenomenon.

Add these replacements to the replacements of all the piping and the oil coolers, and one can say with confidence that there is nothing of substance left on these pumps except for the shafts, the impellers, and the housings. And as we know from the repairs Conhagen made in 2009, much of the housings were probably gone too.

Think about that for a moment: every critical component of these pumps had to be replaced just four years after they were ordered. There's no reason to doubt that most of the rest of the pumps (excepting the few most recently overhauled) are in the exact same rotted condition.

In future work, this suite of repairs (all piping replaced, weld repairs to housing, all bearings and seals replaced, and replacement or reconditioning of Rineer motors) would become the standard set of repairs. Any pumps that didn't receive it - including the pumps repaired in 2009 - would be suspect and would likely fail.

Task order #3 - the last round this spring
That brings us to the final round of pump rebuilding this spring, task order #3 on the new contract. Task order #3 had an initial value of $619,642 and an effective date of April 12.

The scope was for the repair of four more pumps:

17th Street: E6 and E8

London Avenue: W3 and W4

This squares with the chronology we saw play out at the gates, as The Lens' Steve Beatty captured pictures on April 15th at 17th Street of pumps E6 and E8 coming out:




Unfortunately, I don't have any photos of removal of pumps W3 and W4 on the west side of the London Avenue site, but I do have witnesses who tell me they saw the elbows on the ground in late April. Additional documentary evidence received via FOIA requests confirmed the work on them.

As with the work under task order #2, a modification to this task order was also issued in June, 2010 to account for bearing, seal, and Rineer motor replacements. Modification 1 to task order #3, for $52,975.34, was issued on June 11, 2010 with a projected completion date of June 16, 2010 (a two week extension of the original completion date of May 31, 2010. The scope from the task order #3 modification mimics that of the task order #2 modification. It reads,

"5. Line item 1003 – Growth work for pump repair, replacement shaft seal components listed below:
- Repair Seals 4 Each
- Remove Pump Shaft Bearings
- Replace Radial bearings Impeller End (4 Each)
- Replace Radial Bearings Motor End (4 Each)
- Replace Thrust Bearings (4 Sets)
- Replace bearing Spacers (4 Each)
- Replace Hydraulic Motors (4 Each Customer Furnished)
- Inspect Shaft And Housing
- Mount New Bearings
- Clean All Parts For Assembly
- Reinstall Pump Shafts"

The Conhagen repair reports (report for 17th Street pumps E6 and E8 and report for London Avenue pumps W3 and W4) for the work on these four pumps shows the extent of damage before the repairs.

Let's start with the suction bell and the liner on 17th Street pump E6:


The marine growth and rust are fairly obvious. However, it is the inside of the suction bell that truly shows the effects of corrosion.

Here's a detail picture taken through the top of the liner:


And here's two more details of the suction bell with the liner section removed:



It looks like something took a bite out of the carbon steel. This amount of corrosion is unaccepatable for a pump in such critical service, especially one that was only in the water three years.

Like the previously rebuilt pumps, the piping connected to the Rineer motors on these pumps showed signs of severe corrosion:



The damage to the internal drive assemblies indicated water got into places where seals had failed. For example:


And while I cannot be sure, this photo of removal of the shaft assembly from the bearing housing appears to show discolored liquid - possibly an oil/water mix - flowing out of the supposedly sealed housing:


There's also evidence there of rust on components that should definitely not be rusty, including some ugly discoloration on the shaft itself. It's clearer on a set of before and after pictures of the same area on one of the London Avenue pumps:




Note how shiny the shaft is in the lower "after" photo compared to the upper "before" one.

Pictures like these show why it was important to replace all the components like bearings and seals. Those parts could not do their job with this level of contamination by water.

Task order #3, modification #2 - a sidebar about Rineer motors


Modification 2 to task order #3, for $53,241.07, was issued retroactively on July 8, 2010 with an effective date of July 1, 2010 and a projected completion date of August 6, 2010.

The text of modification 2 to task order #3 says:

"For Pumps – 17th St. 6E and 7E, London Ave. 3W and 4W:
Provide the services of the hydraulic motor manufacturer, Rineer Hydraulics, Inc. to rebuild the following motors to be used for repair of the pump assemblies of this task order, with the remaining motors to be returned as spares.
Description: Hydraulic Motor, Rineer 125 Series, Quantity: 12"

The verbiage on this task order is somewhat odd. It claims four of the motors are meant for the four identified pumps. But those pumps were already back in the water by the time this modification was issued three weeks later. More than likely, the motors for the four pumps pulled under this task order were sent to Rineer in San Antonio in April and were rebuilt, and this task order is an after-the-fact modification to account for them and 8 others.

It would be helpful at this point to have some background on the Rineer hydraulic motors. Here's a quick reminder of where the motors sit inside the pump:


The motor spins the shaft through a series of gears. The shaft is connected to the impeller and rides on bearings inside a (theoretically) sealed chamber. We've seen above those seals failed. This photo is taken through the outlet of the pump while it sits on the ground horizontally. When in service, the pump is mounted vertically and water would flow up through the pump. Here's a more detailed shot of the motor and the hydraulic lines going in and out of it:


The motor is powered by high pressure (HP) hydraulic oil, which flows from the remotely positioned engine through the HP lines. The hydraulic oil loses its presurized energy to the motor and flows back through the low pressure (LP) lines to the engine.

In addition to the twelve motors rebuilt under the July, 2010 modification to task order #3, just a month later the Corps put out solicitation W912P8-10-R-0077 for purchase of 12 more hydraulic motors. 8 of the motors were for the 60" pumps, the other 4 motors were for the 42" bridge pumps found only at 17th Street. The contract for the 12 Rineers - number W912P8-10-C-0116 worth $104,760.00 - was awarded September 14, 2010 to Hydraquip Distribution, a Rineer representative with headquarters in Houston and an office located in St Rose, LA, just west of Armstrong International Airport in the New Orleans metro. The completion date of the contract was December 15, 2010.

The solicitation included model numbers for the new Rineers. Remember that back in 2006 the Rineers were undersized "type 61" models. MWI specified them on the edge of their operating envelope, which led to the entire pump units vibrating madly. The pumps were nearly immediately withdrawn from service after being put in the water in the summer of 2006. They didn't return until spring, 2007, after undergoing major repairs to the Rineers and the addition of piping extensions. Other necessary repairs didn't take place until later in 2007.

The vibration fix to the Rineers was to effectively turn them into more robust "type 62's" with stiffer springs (allowing them to handle pressures up to 4500 psi rather than the type 61's 3000 psi), though the seals were possibly not as sturdy, which likely explains a number of the oil spills in 2006, 2007, and continuing to the present (though the extreme corrosion over the years has also played a large part). The vibration problem was the only one the Corps acknowledged publicly with MWI's pumps through 2006 and into 2007, until Maria Garzino's May, 2006 memo was released in March, 2007. After that, all hell broke loose, and it was abundantly clear how troubled these pumps were - and are.

It's worth remembering that in January, 2008 - almost a year after their supposed "fix" to the vibration problem - the Corps admitted that the pumps actually still vibrate. Corps engineer Ray Newman said it in January, 2008 to an interviewer from PentagonTV (click here for Molly Peterson's KPCC "Pumps Under Pressure" special report. Scroll to the right in the timeline until you come to January, 2008 and click on note, "Engineer to PentagonTV: 'Pumps vibrate'"):

"The pumps were not designed to run at low canal level, and they do tend to vibrate when you run that, for instance, a zero canal level."

With all that in mind, it is germane to wonder whether the new Rineers the Corps bought in 2010 are the "upgraded" type 61's or full blown sturdier type 62's. From the solicitation (and confirmed in FPDS-NG), here's the model number for the 8 motors meant for the 60" pumps:

M125-A4-1H-250-31-B1-TBB-213

Rineer provides product documentation on their website. And we know which document to look at, because the Corps included it in their solicitation: "Mounting dimensions, shaft dimensions, port dimensions and configuration shall match the envelope dimensions given in Rineer Hydraulics, Inc. publication DS1251009 4/04."

So, looking at that publication, we find that the "1" in the "1H" disappointingly refers to type 61. This kind of makes sense, since "type 61" in hydraulics-speak refers to not only a presure rating (~3000 psi), but also a standard dimensional configuration for connections. Given some of these motors would be going in existing pumps which have pipes with type 61 end connections, it makes sense to keep the bolt holes aligned. However, given that they are being ordered for pumps that are having their guts torn out anyway, one wonders why the Corps simply didn't upgrade the motors and the piping to type 62, which has a 4500 psi rating, far above what the engines powering these motors can put out.

Anyhow, I believe "213" in the last part of the model number refers to whether the springs are upgraded or not (note: the 213 also appears at the end of the model numbers of the 4 Rineers purchased for the bridge pumps). Rineer's nomenclature refers to this last section of their model number as a "special code designator." That is, it's a catch-all where they describe any special modifications done to the motor. It could refer to anything from a different paint color to a highly technical customization. Without knowing exactly what "213" refers to, we can't know anything for sure, other than these new motors are not stock items. That is reason to hope the springs in them are upgraded. But the continuing use of type 61 motor housings still makes me wonder whether the seals will hold up over the long term.

Hopefully the "213" covers all the modifications needed to make these things work, including the stiffer springs, because the Corps definitely wanted the upgraded models. The solicitation calls for,

Specifications -
Displacement: 250 cubic inches per revolution
Pressure Rating: 3000 psi (or higher)
Rated Speed: 350 RPM (or higher) continuous
Horsepower Rating: 600 HP (or higher) continuous

While the Rineer literature says,

"HORSEPOWER LIMITATION -
Maximum horsepower limitation may vary with different applications. When using the 125 Series standard motor above 300HP, consult a Rineer Application Engineer."

This was a significant bone of contention during the Corps' 2006-07 internal investigation of the pumps. We can only hope they didn't make the same mistakes this time that they made in 2006.

Finally, it's unclear exactly how many Rineer motors the Corps had on hand after all this contracting activity. Presumably the new motors purchased in September, 2010 went into pumps as they got yanked out for rebuilds, perhaps implying that the July, 2010 modification to repair 12 Rineers didn't work out, or maybe implying the Corps wanted as many Rineers as possible in anticipation of many future pump repairs. Purchases and rebuilds of Rineers show up on other task orders as well. Whatever the specifics, it's just more confirmation of my statement above: every critical component of these pumps had to be replaced just four years after they were ordered.

Burning through cash

Speaking of money, remember that the expected value of the new Healtheon contract was $2.875 million. By early June, 2010, the Corps had already issued task orders worth $1.32 million on 8 pumps and they were only three months into the three year contract. Jumping ahead here, there were still 25 sixty-inch hydraulic pumps which had not been touched, including all 10 pumps at the Orleans Avenue canal. Another 5 had incomplete repairs in 2009 and would likely need to come out soon as well. Doing the math - they only had cash for complete overhhauls of about 10 of those 30 pumps. It's obvious the New Orleans District either hasn't set aside enough cash (from the permanent pump station funds - quite a pickle they've put themselves in!) for this work, or they plan on just rolling the dice on most of the unrepaired pumps for the next three years. And that doesn't even take into account the fact that the permanent pump stations meant to replace these pumps won't be ready until the 2015 hurricane season.

[Note: a third contract to Healtheon/Conhagen would be issued in 2011, with a potential value of $6.75 million. See this March, 2011 post and this May, 2011 post for details.]

Corps sleight of hand...
Originally the Corps wanted the 17th Street and London Avneue pumps of task order #3 back in their respective canals May 31st ahead of the June 1st start of the 2010 hurricane season. With the additional work to replace bearings, seals, and hydraulic motors, there was a slight delay in the completion date. As we noted above, it was extended to June 16th in modification #1.

This presented a potential PR problem for the Corps. Every year since 2007, the Corps has run a hurricane preparedness exercise which involves turning on pumps for the press at one of the closure sites. With two sites missing pumps, and the Corps likely nervous about getting questions on the condition of the remaining unrepaired pumps, they moved the press availability to the only site without pumps out for repair: Orleans Avenue. They used a similar maneuver in 2009 when, in order to hide the repair work happening at 17th Street (pump W10 was getting pulled at 17th Street the same day as the drill), they had the press out to London Avenue.

In the case of 2010 event, they didn't put out a press release to the public until the day of the event, after it had already taken place.

And with all the BP oil spill coverage, it's no surprise hardly any press came. Fox 8 sent a reporter and they got some video, but it is long gone from their website. Industry cheerleader Engineering News Record sent a reporter as well, who dutifully reported what the Corps told her. Notable about the drill - at least the part when the press was there - was the fact that the Corps didn't even lower the Orleans Avenue gates when they turned the pumps on for the media:


Seems like they're not even trying anymore, because exactly what kind of drill is it when the gates don't go down?

Wrapping up

Conhagen eventually didn't need to work all the way until June 16th. The 17th Street pumps went back in on Wednesday, June 9th. Karen Gadbois of The Lens, photographed this work on the east side of the site:


London Avenue pumps W3 and W4 went back in a couple of days later.

After all the invoices were received, modification 3 to task order #3 was issued May 18, 2011. It subtracted $12,807.35, bringing the total for all the task order #3 work (including rebuilding of 8 Rineer motors not directly related to the 4 pumps repaired under this task order) to $713,051.23.

Summary and conclusions


I'll went back and updated my last post to reflect what I learned from the task order #2 and #3 contract documents. My basic conclusions from that post remain in effect. That is, the vast majority of the hydraulic pumps across all three sites remain in a dangerous state of decay and some will very likely fail or underperform during a storm event. This will occur because of extensive corrosion of all the internal and external parts on the partially or completely unrepaired pumps.

Here's the numbers of unreliable corroding 60" pumps:

17th Street: 12 (7 pumps unrepaired and 5 pumps repaired in 2009 with carbon steel piping)
Orleans Avenue: all 10
London Avenue: 8
(note: the 11 direct drive pumps at 17th Street and 8 direct drive pumps at London Avenue are likely reliable. Also, these numbers do not include the 14 smaller 42" bridge pumps at 17th Street, which the Corps is repairing themselves and for which I have not received any documentation. Those bridge pumps put out an estimated 1600 cubic feet per second, about 18% of the total output at 17th Street, and should be considered unreliable pending further information.)

That's 30 out of 40 major hydraulic pumps, or 75% of the major hydraulically-powered drainage capacity which the Corps is crossing their fingers on this summer. Thank goodness for the direct drive pumps.

Here's the effect of those un-overhauled or partially overhauled pumps in terms of cubic feet per second of total drainage capacity at each site:

17th Street


Orleans Avenue


London Avenue


These numbers are actually the best case, because they assume that the bearings and the Rineer motors are functioning well in all the pumps in which they have not been replaced, which - as of June, 2010, was all but the eight pumps repaired under task orders #2 and #3 described above.

I do not enourage citizens to depend upon these pumps during a storm event of any magnitude.

Karen Gadbois and Molly Peterson contributed to this report.