2011-12 rusty pump update

This post updated 2/14/17. Scroll down for the new content near the end.

When we last looked in on the rusty pumps at the Lake Pontchartrain lakefront, 15 of the 40 large hydraulic pumps installed across the three outfall canal closure sites - 17th Street, Orleans Avenue, and London Avenue - remained highly vulnerable to corrosion. Their major components such as pipes, housings, hydraulic motors and coolers remain immersed or very close to the brackish waters of the lake 24 hours a day. Those 15 pumps were vulnerable because all those components were made of carbon steel, which rusts like crazy in a salt environment.

Pumps continue to spring oil leaks during 2011 hurricane season

One of those 15 pumps - E3 at the Orleans Avenue canal - apparently sprung an oil leak on August 10, 2011, the third pump to do so during the 2011 hurricane season. I noted this at the time. The Corps' standby oil cleanup contractor - Quaternary Resources Investigations, or QRI - was summoned that afternoon according to the $6093.29 task order issued to QRI by the Corps. Six days later, that task order was increased in value by $3634.09 because the estimate of the spill size jumped from 25-50 gallons to 50-100 gallons. The final cost of of the spill cleanup was determined to be $9711.10, after a second smaller cost adjustment on September 21, 2011.

Repair of E3 was arranged over the next few days, with the official rebuilding task order (#10 on the second Healtheon contract) issued to Healtheon and their subcontractor Conhagen on August 18, 2011 in the amount of $170,067.56. Because of the Corps' tendency to file paperwork after the fact, it is possible the original August 10th leak actually occurred when the pump was being pulled out, since paperwork attached to Conhangen's repair report for Orleans E3 includes mention of a leak test of new oil coolers on August 18th, which could presumably only take place near the end of repairs. Further evidence of contractual hijinks comes from the fact the repair report mentions work starting on July 15, 2011, over a month before the task order was issued, and the same day the previously repaired pump (W9 at 17th Street) was reinstalled. Whatever the exact sequence of events, the result was yet another rusty pump dumping oil in the lake.

Looking at the repair report, we see the usual assortment of rusty innards, including the Rineer hydraulic motor at the heart of the pump...

marine growth-encrusted oil coolers...

and chipped inlet vanes...

The pump was put back in service on October 3, 2011. The somewhat long delay from the pull out in August (or possibly July?) until reinstallation in October was not explained.

Late 2011 (kinda) off-season work

Next up was what appears to be the planned repair of four pumps: W2 and W3 at 17th Street and W1 and W2 at Orleans Avenue in November, during hurricane season (the Corps has a history of ignoring the last month of hurricane season). We got a spill report for the 17th Street pumps on November 17, 2011, 26 hours after a spill the Corps claimed occurred on November 16th:

"Qty released: 60 gallons

DURING REPLACEMENT OF A PUMP HYDRAULIC OIL WAS NOTICED LEAKING WITHIN A BOOMED CONTAINMENT AREA. THE HYDRAULIC OIL WAS COMING FROM THE OLD PUMP THAT WAS BEING REPLACED.

CONTRACTOR WAS ON-SITE AND CONDUCTED CLEANUP"

The report to the state of Louisiana on this spill provides far more background, including evidence of lying in reporting. A chronology for the incident - part of internal correspondence between QRI and the Corps and attached to the state spill report - is titled "2011-11-16 - 17th Street Canal (Pump #2W) Spill Summary Report." However, the first line notes the spill was discovered on November 15th as part of QRI's weekly rounds, not as part of pump rebuild work. The date could be a typo, intended to be the 16th. Or it could actually be true. But the circumstances - discovery of the spill during the weekly maintenance - are totally different than what was described to the Coast Guard.

The state spill report - generated from information given the state by the Corps - claims the spill was discovered at 11:15 AM on the 16th. The chronology claims that cleanup started that afternoon after the Corps gave verbal approval at 1:55 PM. The chronology also places notification of the Coast Guard by the Corps in the two hours and 40 minutes between those two events:

"During QRI's weekly maintenance on November 15, 2011, the team spotted a hydraulic spill associated with pump #2W at the 17th Street pump station.
[...]
The crew alerted USACE representative Kim Branch of the spill.

Mr. Branch reported the spill to the appropriate authorities as per the SPCC plan. The DEQ incident number associated with this spill is 11-07-194.

QRI received a verbal NTP [Notice to Proceed] from Ione M. Cataldo, contracting officer/team leader at 1:55 PM on November 16, 2011"

Reporting of a spill per Corps' 17th Street Spill Prevention Contol and Containment - or "SPCC" - plan calls for notification of the Coast Guard's National Response Center by the facility owner - the Corps in this case - immediately upon discovery:

"All spills, as defined in 40 CFR, part 110.3 (sheen rule), will be reported to the National Response Center. LADEQ and EPA notification procedures and written report requirements for a reportable discharge are included in Appendix F.
[From Appendix F]
Any person in charge of a vessel or of an onshore or offshore facility shall, as soon as he or she has knowledge of any discharge of oil from such vessel or facility in violation of section 311(b)(3) of the Act, immediately notify the National Response Center (NRC)"

And from the Coast Guard's National Response Center spill report, here's the truth about when the report was actually received:

"Report taken at 13:36 on 17-NOV-11"

That's 26 hours after the Corps claimed the spill was discovered, and perhaps as much as 50 hours after it was actually discovered. By that point, according to the documents attached to the state report, the spill was already cleaned up. It's these little lies - along with the huge ones - that make the Corps so untrustworthy.

The chronology also notes the final quantity spilled was 77 gallons. That's over a quarter of the 300 gallon capacity of the hydraulic oil day tank on the engine skid for that pump. It is unlikely the pump would have run very well with that much oil missing.

A task order for QRI's cleanup work was written on November 16, 2011 for $9727.38. The final spill cost was $7129.03 after an April, 2012 modification.

Further confusion was introduced a week later, when on November 23rd, Healtheon got a $613,645.84 task order (#3 on the third Healtheon contract) for the removal, rebuilding, and reinstallation of the four pumps I listed above, which included 17th Street pump W2. It's very difficult to tell if this work was already underway on the 16th - which the spill reports claim - or if it didn't get going until the 23rd. I'm inclined to say it wasn't until the 23rd, since that's what the Conhagen repair report for that pump (and the report for pump W3) claims as the start date:

Admittedly, the repair reports for the other two pumps on this task order also claim the November 23rd date as the "job start" date, even though - as we'll see in a bit - they likely weren't pulled out until December However, if the work truly didn't start until November 23rd, that would mean the spill report to the Coast Guard was not only late, but also contained lies in the narrative.

Now to the contents of those 17th Street pump repair reports (W2 report, W3 report). They contain the usual pictures of rusty and marine growth-encrusted stuff. But what really attracted my attention was the condition of the splined drive shafts that come out of the bottom of the Rineer motors:

Rusty, corroding yuckiness! These shafts are enclosed in housings that are supposed to be sealed, but clearly were not:

It's not hard to imagine a pump refusing to work because of the poor condition of these shafts. In addition, if the shafts are this bad, all the internals on their motors would also be shot. And indeed, that is exactly what has happened. Repeatedly. For years.

Sidebar Rineer motors and salt water

On May 18, 2011, the Corps issued a $13,881.02 task order (#12 on the first Healtheon contract) which sent 8 Rineer motors for the main 60" pumps out for repair. At the time the scope only called for replacement of seals, shaft ball bearings, and fasteners (i.e. bolts) on the housings and seal plates. That work was supposed to take 6 weeks. Of course, if one is replacing seals, then that means the old seals are bad and stuff got into the motors.

So a $42,019.32 modification to the original Rineer repair order was sent out August 31, 2011, after the motors were taken apart and unsurprisingly found to have their internal moving parts utterly ruined. The delay between the original rebuild deadline (around July 1) and the task order finally getting issued (August 31) would correspond to the standard Corps dickering over whether they should spend the money to repair or replace the motors, as well as the usual delays in getting anyone to make an actual decision.

The modification to the original repairs was predictably wide ranging, given the fourfold expansion in costs. It reads:

"Provide labor and materials to replace the following additional components on the 8 Rineer hydraulic motors Model M125-A4-1H250-31-B1-0BB-213 for 60” hydraulic pumps, which were disassembled under the original Task Order. These components were found damaged upon disassembly; and were not included in the original scope of work.
The contractor will be given 24 weeks to complete the repairs to the motors upon award of this modification due to parts delivery lead times.

All Motors; Replace all motor housing plugs with new solid hex head plugs.

Motor #1, work order #9184;
Qty 1 complete vane cartridge kit (16 week del)

Motor #2, work order #9185;
Qty 2 complete vane cartridge kits (16 week del)

Motor #3, work order #9186;
Qty 1 complete vane cartridge kit (16 week del)

Motor #4, work order #9187;
Qty 1 Rear Housing (20 week del)

Motor #6, work order #9189;
Qty 1 complete vane cartridge kit (16 week del)

Motor #8, work order #9191;
Qty1 complete vane cartridge kit (16 week del)"

Replacement of the complete vane cartridge kit in these motors is the equivalent of having every cylinder in your car's engine rebuilt. Of the original 8 motors sent out in May, 2011, 5 (62.5%) were found to require this complete vane cartridge replacement. That is a shockingly high rate of failure, one that should make New Orleanians blanch.

But this major rebuild work on the Rineers is of a piece with what had been happening since mid-2010.

In August, 2010 a solicitation for 12 new Rineer motors was issued by the Corps. The solicitation remains up at the FedBizOpps webpage here. 8 of the Rineers were to be for the 60" pumps, with the other four going in the smaller 42" pumps installed on the structure at 17th Street. Rineer motors - especially special order ones like these - have a very long lead time. The winner of the $104,760 contract (W912P8-10-C-0116) was Hydraquip of Houston, TX, a Rineer rep with a local office near Armstrong International Airport in Kenner, LA. They had about 3 months to deliver the new motors, meaning the Corps would have had the new ones in hand around December, 2010.

And going back further, let's not forget that in July, 2010 the Corps issued a task order modification to rebuild 12 other Rineer motors, all for 60" pumps.

Finally, more recently the Corps sent an additional 6 Rineers for the 60" pumps out to Hydradyne, LLC for "overhauling" in a March 8, 2012 task order on the first Healtheon contract. The task order is not specific as to the scope of work; however, we can be assured it is the same as the more extensive work called for in August, 2011, because of this:

"Estimated completion date is 20 weeks from the award date (due to part availability from the manufacturer)"

Only replacement of vane cartridges could take 20 weeks based on parts availability.

So all this activity means that in a little over a year and a half, the Corps rebuilt 26 Rineer motors for the 40 big hydraulic pumps, and purchased 8 more. That's close to the number of pump pulls that occurred over that same span (41 by my count, including three pumps coming out twice).

However, the numbers really don't matter. The bottom line is that these Rineer motors are obviously ill suited to salt water exposure. Their seals are likely designed for fresh water. Leaving them where they are is simply asking for big trouble. Over the last three years I've published a slew of pictures showing Rineers rusting to bits. They've been getting salt water into them through the rusty pipes, the seals, and every other leak path. The Corps has tried replacing them and repairing them. Nothing works. They all get eaten up. These motors are the heart of the pumping system at all three Lakefront closure structures, and they are unreliable and rusting to this day.

The rest of the late 2011 work

The other two pumps on the November 23, 2011 task order - Orleans Avenue pumps W1 and W2 - didn't get pulled out until early December. This was the first time the Corps pulled out more than one Orleans Avenue pump at a time since the mass rebuilds in 2006-07, indicating they would finally be working on the pumps at that location in a coordinated fashion.

On December 8, 2011, a spill report was filed with the National Response Center for an incident at 2 PM that day:

"Qty Released: 10 GALLONS

CALLER STATED THAT THERE WAS A RELEASE OF 10 GALLONS OF HYDRAULIC OIL FROM A HYDRAULIC PUMP WHICH IS PART OF THE CANAL STRUCTURE, THE CAUSE WAS DUE TO OVERHAULING OF THE PUMP AND THE CAP ON THE PUMP WAS LEAKING.

CLEAN UP UNDERWAY, CONTRACTOR HAS BEEN HIRED"

The companion cleanup task order was issued to QRI that same day. The text of the $6,093.29 task order confirms the spill was connected to the pull-out of Orleans Avenue pumps W1 and W2:

"Verbal authorization for this work was given by Contracting Officer Cynthia A. Nicholas on 8 December 2011 at 3:18 p.m in a Not to Exceed amount of $6,093.29 for a 10 - 50 Gallon Spill at Orleans Avenue Canal Pump 1W and 2W."

A later modification dropped the actual cleanup price to $5,647.60.

As with the 17th Street pumps on this same task order, individual repair reports were compiled for each pump (Orleans W1 repair report, Orleans W2 repair report). As usual, there was rust, marine growth, and other scary stuff:

The 17th Street pumps went back in the water January 24, 2012. The Orleans Avenue pair followed on January 27, 2012.

London Avenue pump rebuild work is finished off

While those four were out, a remarkable decision to repair a fifth pump was made. That pump - W2 at London Avenue - was the last of the 12 London pumps not to be refitted over the past three years. A $142,459.56 task order (#11 on the second Healtheon contract) was issued December 20, 2011. There are no oil spill reports or task orders to QRI associated with this pump repair, which is odd. Single pump rebuilds like this usually come about because the pump sprung an oil leak. This could be yet another case of an unreported spill, or a more unlikely case of the Corps acting proactively to repair more than four pumps at a time.

The Conhagen repair report for London Avenue pump W2 is very spare on photos, almost as if they know their reports would be publicly disseminated. There is one photo of interest though, and it's an "after" shot:

This is probably a rebuilt Rineer motor. The color appears different than other motors which have come out, and part of the rebuilding task orders for Rineers has mentioned painting them. So that's one good thing, I suppose. But it will rust out again, long before the new pump stations come online in 2016, or whenever.

According to the repair report, London Avenue pump W2 went back in the water March 12, 2012.

First round of 2012 - focus on Orleans continues

That brings us to the next round - pumps W3 and W5 at the Orleans Avenue site. Healtheon received a $304,322.08 task order on January 30, 2012, the day after work wrapped up on the last set of 4 pumps. It's unusual for the Corps to only pull out 2 pumps rather than 4 in the off season, but perhaps London Avenue pump W2 was still in the shop at the time.

A spill report was filed with the National Response Center on February 22, 2012, for a 20 gallon spill supposedly discovered at the Orleans Avenue structure on February 21, 2012:

"Caller is reporting a discharge of hydraulic fluid from a leaking hydraulic pump. Spill contractor is on site [to] clean up material; material contained by boom in the drainage canal."

By this time, QRI had been awarded a shiny new spill cleanup contract, it's third. The four year agreement has a potential value of $581,376.24 (!) Task order #3 on it, worth $6360.39, was issued February 23, 2012, two days after the supposed discovery of the spill. The description in the task order seems to connect it to the Orleans Avenue pumps in the January repair order:

"This task order is [to] cover the clean up of hydraulic oil spill estimated to be in the 10 to 20 gallon range. The event happened from pump #3 at the Orleans Ave Canal."

While the task order doesn't specify east or west, just "pump #3," I'm going to guess it was referring to pump W3, meaning it and likely W5 came out on or around February 21, 2012. They probably went back in sometime in late March or early April.

Unfortunately, that's where my latest batch of documents ends, since I placed my latest FOIA request on March 9, 2012. However, there are ways to track what's going on even without FOIA requests.

The federal government's FPDS database posts all federal contract transactions. There's an immensely annoying 90 day hold on Pentagon transactions posting to the website, ostensibly for security purposes, but mostly to keep prying eyes away. However, since the transactions from February, March, and the beginning of April are now available, we know there was another task order on the third Healtheon pump repair contract issued. Task order #5 for $308,594.52 was issued March 27, 2012 for the repair of Orleans Avenue pumps E1 and E2, with a scheduled completion date of May 28, 2012. This would seem to indicate it was the last batch of pump repair work the Corps planned for the offseason before the 2012 hurricane season, which began on June 1, 2012.

The National Response Center online database is updated daily, so it's an excellent contemporaneous indicator of activity. Since the late February Orleans Avenue canal spill there have been three additional spills at the outfall canals:

May 1, 2012, Orleans Avenue structure
"Caller stated due to a faulty o-ring in a hydraulic line connector there was a spill of hydraulic fluid. Caller stated the sheen is burning off and absorbents will be used later as a boat will be placed in the water."
Estimated quantity spilled: 5 gallons

May 11, 2012, 17th Street structure
"Caller reported a spill of materials while repairing an engine on a hydraulic pump. Booms applied, cleanup underway."
Estimated quantity spilled: 14 gallons

July 2, 2012, Orleans Avenue structure
"Caller is reporting a discharge of hydraulic oil from a hydraulic pump due to unknown causes. Caller stated the release is estimated between 15 - 50 gallons. Double boom was in place, contractor is on scene cleaning up"
Estimate quantity spilled: 50 gallons

Knowing what we know about these spill reports - the data comes from the Corps, and it may or may not be true - I try not to put too much stock in the descriptions. The reports are really best for their timeliness. We'll find out later whether they're truly associated with pump repairs, scheduled or otherwise.

UPDATE 2/14/17

As noted above, at the beginning of 2012 the Corps was focused on completely refitting pumps at the Orleans Avenue site. Pumps W3 and W5 came out immediately after the return of pumps W1 and W2 in late January. W4 had been refit during the 2010 hurricane season. The task order for Orleans W3 and W5 is here, and the repair report is here.

The report is unremarkable, with only "after" photos. The pumps were accepted by the Corps March 29, 2012.

Next up were Orleans pumps E1 and E2. They came out the same week W3 and W5 went back in, the last week of March, 2012. The task order is here, and the repair report is here. The report has pictures of rusted out carbon steel piping cut off of the pumps (note: the labels of "1W" and "2W" are Conhagen's typos):

It's striking to see this piping discarded like so much junk.

These two pumps were accepted by the Corps June 19, 2012. While they were in the shop, two other pumps sprung oil leaks - 17th Street W3 and London Avenue W5. Both pumps had been in the water for less than a year following complete refits. Details on those failures can be found in an update at this subsequent post.

The two remaining Orleans Avenue pumps that were left unrefurbished at the beginning of the 2012 hurricane season - E4 and E5 - unsurprisingly leaked oil that summer and had to come out during the height of hurricane season. They were out of commission when Isaac came ashore in August, and their absence likely led to numerous pumping cutbacks while the storm moved through. Details on repairs to those two pumps  - and links to how their going missing hurt city drainage during Isaac - can also be found at the same subsequent post.

So this is what we can say about the pumps at each site, given the documents and other information I've sussed out:

These "reliability" estimates come with huge caveats. We know the critical Rineer motors are continuing to rust away, no matter how much they get replaced or repaired. We also know of numerous other problems back on the pumps' remotely placed engine skids unrelated to corrosion. In essence, all these graphics show is that some carbon steel piping has been switched out for stainless steel, and there's been other corrosion repairs and repainting. The mechanical reliability of the hydraulically driven pumps, especially due to the ongoing corrosion of the Rineer motors and the resulting saltwater ingress to the hydraulic oil loops, is highly questionable should they be called upon during a major tropical event.