Birth of a no bid contract

[This post includes links to hundreds of pages of internal Corps emails. See note at end of post.]

I'd like to drill into some of the behind the scenes activity (or inactivity, as we'll see) surrounding the corrosion of the Corps' lakefront hydraulic pumps, in an attempt to explain why a problem identified as affecting 54 pumps back in 2006 has taken over 6 years to be completely addressed. Let's go back to fall, 2009. Here's the facts regarding what was happening with the rusty pumps at that time:

Total hydraulic pumps installed: 40 large (60") and 14 smaller (42") across three sites
Pumps pulled for corrosion repairs to date at that time: 6 large and at least 2 smaller
Pumps remaining to be pulled at that time: 34 large and at most 12 smaller
Basic scope of 2009 repairs to pumps: replace rusting carbon steel parts with carbon steel and some stainless steel parts (wholly inadequate, which would be proven when some of the 2009 repairs would spring more leaks in 2011, forcing 2nd pulls)
Average cost per pump pull: Approximately $100,000 (based on work performed on four large pumps under task order 3, mods 0 and 1, of first Healtheon/Conhagen contract), which was also wholly inadequate because not all carbon steel parts were getting replaced with stainless steel
Information known to Corps regarding corrosion on the pumps: All pumps severely affected since installation in Lake Pontchartrain's salt water in 2006; numerous oil spills since installation in 2006; repair vendor said in October, 2009, "The severe corrosion seen on the piping internal of the pump should be cause for concern in the remaining pumps at this site. Leaks that were imminent on these pumps are sure to be imminent on the remaining."
Contracting methodology: fractured, with different contracts for repairs, crane work, spare parts.

A unified repair contract with enough funding to support the complete corrosion-related overhaul of all hydraulic pumps was needed. Ideally, it should have had provisions for multiple repair subcontractors to speed the work to ensure that pumps meant to drain the city during a major tropical event could do so without springing oil leaks through rusty holes. That's not the way things went.

Instead this is what happened (these are actual internal Corps emails obtained through FOIA, see note at end of post). First, they limited the number of pumps they were looking to repair:

November 12, 2009:
"From: Schneider, Donald C MVN
To: Marshall, Jim L MVN-Contractor
Cc: Accardo, Christopher J MVN; Newman, Raymond C MVN; Constantine, Donald A MVN; Robinson, Carl W MVN; Nguyen, Bac T MVN
Subject: Interim Canal Structures
Date: Thursday, November 12, 2009 1:52:20 PM
Importance: High

Jim,

We're scheduling preventative maintenance on 18 ICS pumps before the start of the 2010 tropical season. The frequency of corrosion-induced failures requires that we take this action.

We currently have approximately $3.2M available. Based upon our average annual expenditure ($1.8M) and the estimated cost of each pump rehab. (approx. $130,000), we require an additional $1M in the project to perform this necessary work.

Do we have the additional funds to complete this PM program?

Donald Schneider
Operations Manager, OD-G"

Already they were limited in their thinking, somehow believing only 18 pumps needed to be overhauled. Next they looked at a way to fund this too-limited work:

December 3, 2009:
"From: Cataldo, Ione M MVN
Sent: Thursday, December 03, 2009 12:45 PM
To: Nguyen, Bac T MVN; Newman, Raymond C MVN, Schneider, Donald C MVN
Subject: Overhaul of 16 Outfall Canal Pumps
Importance: High

Gentlemen,

I have done a little research into the MATOCs that I mentioned in our meeting yesterday that might be an alternative means of processing the subject requirement in the short term. After reviewing it appears that we will not be able to use the MATOCs because they are for construction type work; to do construction actions over $100,000 even under the MATOCs would require a BCOE certification, LD memo, legal sufficiency, and ROE if applicable. Historically the pump contracts seem to all be commercial service direct 8(a) contracts.

Which brings me to this suggestion:

It is possible we may be able to do the IDIQ contract as needed as a direct 8(a) Commercial Service Contract similar to W912P8-08-D-0090, give the 8(a) 15 days to submit a proposal, review for price reasonableness, and if all goes well award sometime in January-Feb 2010.

NOTE: The catch to whether I can go out direct 8(a) or not is 1) Actions $3 mil and under can go direct 8(a); anything over $3 mil would have to be competed, 2) I have to get SBA and OC's blessing for setting this aside as a commercial service under $3 mil (which shouldn't be hard to do since that's how the others have gone out), and 3) the service contract act forms would need to be prepared and submitted for approval in EX.

If you want to try putting the IDIQ in place now using the direct 8(a), I will need to know how much money you are planning on making the contract for ($3 mil; $5 mil; ...), how many years (3 years, 5 years,...), and if money is available to cover a minimum guarantee. I'll also need a brief description of the work to pass through SBA and OC to make sure they are not going to give me any roadblocks.

In reference to the two pots of funds to be used, I just wanted to clarify that the funds would need to be all of the same color money - can't mix monies.

Discuss among yourselves and let me know what you think. DONT WAIT TOO LONG TO GET BACK TO ME PLEASE!

Thanks,
Ione M. Cataldo
Contracting Officer/Team Leader
Project East Branch
Contracting Division
National Contracting Organization
USACE, New Orleans District"

"Direct 8(a)" refers to a no bid award to a SBA-certified small business. That is, Contracting told Operations: here's how to do this without putting the job out to public bid. (Note this is completely legal.) Theoretically that would make things go faster, but it still looks a little hinky to be given the choice to do half the job as a sole source rather than doing the whole job as a publicly bid job.

Operations jumped at the chance to sole source the contract, choosing to do half the job with a preferred vendor. Donald Schneider with the Operations group wrote back the same afternoon:

December 3, 2009:
"From Schneider, Donald C MVN
Sent: Thursday, December 03, 2009 4:27 PM
To: Cataldo, Ione M MVN
Cc: Nguyen, Bac T MVN; Newman, Raymond C MVN; Constantine Donald A MVN; Robinson, Carl W MVN; Brown, Jane L MVN
Subject: RE: Overhaul of 16+ Outfall Canal Pumps
Importance: High

Ione,
Let's move forward w/ the direct 8(a) Commercial Service Contract @ $2.875M for a 2-year duration. Bac will be the P.O.C. to provide you with any information you need.

Thanks for your help.

Don S."

The banality in the award of nearly 3 million taxpayer dollars via an afternoon exchange of emails is remarkable, though I doubt it is atypical with billions of dollars flowing through the New Orleans District since 2005.

Anyhow, as we can see, Operations decided to do a sole source, no-bid contract with their existing contractor - Healtheon/Conhagen - just under the threshold for public bidding, but much further under the funding required for overhauling ALL the pumps, something that same contractor told them was required to avoid imminent failures. The contract value should have been about $150,000 x 40 - or $6 million - assuming only the big pumps would be handled by the new contract, with the smaller pumps being repaired in the Corps' own shop. Once again, public safety lost out to making things easier on the Corps and their vendors.

On December 28, 2009, Direct 8(a) Service Contract solicitation W912P8-10-T-0048 was issued solely to Healtheon, Inc. Terms were a maximum value of $2.875 million and a maximum duration of 3 years. This was a year longer than initially asked for by Operations, meaning they were planning on slowing the already-too-few repairs down even further, giving the pumps even longer to rust out.

On March 8, 2010, Direct 8(a) Service Contract W912P8-10-D-0054 was awarded to Healtheon. The maximum award amount was $2.875 million and the maximum duration was 3 years. This was nearly four months after the initial heads up saying the repairs were necessary.

This strategy would unsurprisingly prove wrong. While 17 of the large pumps would ultimately be overhauled under contract 10-D-0054 from March, 2010 through late 2011 (a ridiculously slow rate) before they ran out of money on the contract, that would still leave 23 others unattended with five years of rust built up on them. Six of those 23 did receive partial repairs under earlier work, but that doesn't count because the repairs didn't replace all the critical carbon steel parts with stainless.

So just six months after the no-bid contract was awarded - in October, 2010 - the Corps ended up going out for public bid anyway on a third repair contract (bid documents here, original FedBizOpps webpage here). That contract, potentially worth $6.75 million and notably also awarded to Healtheon, actually had enough money set aside to do the rest of the pumps, work which is still underway as of this summer. They could have done that in the first place if they hadn't insisted on the stupid inadequately funded no bid second contract.

Now one might be willing to give the Corps a pass on some of this, a "at least they're trying" sentiment. But, again, here's the facts:

1) They knew as soon as they put the pumps in the water back in 2006 they would corrode very, very quickly. Within one month they were already seeing corrosion on the hose connections that were outside the brackish water (via SCPR Flickr):


Here's a detail of one of those rusty hose connections taken at the same time as the picture above (via SCPR Flickr):



What did they think was going on beneath the water's surface to their completely carbon steel equipment? Repairs did not even start until three years later, and a dedicated contract was not in place until March of 2010, nearly four years later.

2) One could also argue, "in going out as a no bid contract, they were trying to accelerate things. It would have taken too long to publicly bid it." From the bid issuance to contract issuance for the second contract was over two months, while the time from bid to contract award for the third, publicly bid repair contract a year later was about four months, so there was some time saved. However, the only reason they could go out as a sole source was because they decided to only fund repairs for fewer than half the pumps! If they had bothered to say to themselves, "the pumps at 17th Street are rusting like crazy, I bet all of them are doing the same thing, so we should look to repair all of them," they would have never gotten a sole source contract because even the most optimistic estimates would have been for more than the $3 million cutoff for public bidding. And then they wouldn't have lost the four months in the fall and winter of 2010-11 spent putting the third contract out for bid. This gang just can't shoot straight.

3) Maybe they were limited on their funding in the fall of 2009? Well, they were able to find nearly $7 million more just a few months later when they went out for bid on the third contract.

The bottom line is this: once again the Corps cheaped out and slowed down on vital life safety equipment meant to keep New Orleans dry. They went for a no bid contract with an existing, preferred vendor which was underfunded and not geared toward getting the job done as fast as possible. For goodness sakes, these pumps are meant to save people's lives, and the Corps couldn't be bothered to spend the money to fix their own stupid mistakes that had been festering for three years. Instead, they just puttered along in their same old insulated way, secure in the knowledge there will never be consequences for their decisions.

Special note regarding internal Corps emails
I placed a FOIA request a while back asking for all emails dealing with pumps from a select group of upper and mid-level Corps personnel. I confined the date ranges to a few days around storm events in late 2009 when the London Avenue canal gates were closed (in November) and when water got very high in the same canal and the gates were not closed (in December). I wanted to see how the Corps and their contractors - especially the controls repair contractor, Prime Controls - dealt with those events in real time. It took forever to get them, but it was worth it, because they included a lot of bonus material.

In asking for anything dealing with the pumps, I also got emails talking about contracts, repairs, and status updates. By pure luck, my net captured the emails I used above. If anyone wishes to see everything I got, all the emails are available here.

2011-12 rusty pump update

This post updated 2/14/17. Scroll down for the new content near the end.

When we last looked in on the rusty pumps at the Lake Pontchartrain lakefront, 15 of the 40 large hydraulic pumps installed across the three outfall canal closure sites - 17th Street, Orleans Avenue, and London Avenue - remained highly vulnerable to corrosion. Their major components such as pipes, housings, hydraulic motors and coolers remain immersed or very close to the brackish waters of the lake 24 hours a day. Those 15 pumps were vulnerable because all those components were made of carbon steel, which rusts like crazy in a salt environment.

Pumps continue to spring oil leaks during 2011 hurricane season

One of those 15 pumps - E3 at the Orleans Avenue canal - apparently sprung an oil leak on August 10, 2011, the third pump to do so during the 2011 hurricane season. I noted this at the time. The Corps' standby oil cleanup contractor - Quaternary Resources Investigations, or QRI - was summoned that afternoon according to the $6093.29 task order issued to QRI by the Corps. Six days later, that task order was increased in value by $3634.09 because the estimate of the spill size jumped from 25-50 gallons to 50-100 gallons. The final cost of of the spill cleanup was determined to be $9711.10, after a second smaller cost adjustment on September 21, 2011.

Repair of E3 was arranged over the next few days, with the official rebuilding task order (#10 on the second Healtheon contract) issued to Healtheon and their subcontractor Conhagen on August 18, 2011 in the amount of $170,067.56. Because of the Corps' tendency to file paperwork after the fact, it is possible the original August 10th leak actually occurred when the pump was being pulled out, since paperwork attached to Conhangen's repair report for Orleans E3 includes mention of a leak test of new oil coolers on August 18th, which could presumably only take place near the end of repairs. Further evidence of contractual hijinks comes from the fact the repair report mentions work starting on July 15, 2011, over a month before the task order was issued, and the same day the previously repaired pump (W9 at 17th Street) was reinstalled. Whatever the exact sequence of events, the result was yet another rusty pump dumping oil in the lake.

Looking at the repair report, we see the usual assortment of rusty innards, including the Rineer hydraulic motor at the heart of the pump...

marine growth-encrusted oil coolers...

and chipped inlet vanes...

The pump was put back in service on October 3, 2011. The somewhat long delay from the pull out in August (or possibly July?) until reinstallation in October was not explained.

Late 2011 (kinda) off-season work

Next up was what appears to be the planned repair of four pumps: W2 and W3 at 17th Street and W1 and W2 at Orleans Avenue in November, during hurricane season (the Corps has a history of ignoring the last month of hurricane season). We got a spill report for the 17th Street pumps on November 17, 2011, 26 hours after a spill the Corps claimed occurred on November 16th:

"Qty released: 60 gallons

DURING REPLACEMENT OF A PUMP HYDRAULIC OIL WAS NOTICED LEAKING WITHIN A BOOMED CONTAINMENT AREA. THE HYDRAULIC OIL WAS COMING FROM THE OLD PUMP THAT WAS BEING REPLACED.

CONTRACTOR WAS ON-SITE AND CONDUCTED CLEANUP"

The report to the state of Louisiana on this spill provides far more background, including evidence of lying in reporting. A chronology for the incident - part of internal correspondence between QRI and the Corps and attached to the state spill report - is titled "2011-11-16 - 17th Street Canal (Pump #2W) Spill Summary Report." However, the first line notes the spill was discovered on November 15th as part of QRI's weekly rounds, not as part of pump rebuild work. The date could be a typo, intended to be the 16th. Or it could actually be true. But the circumstances - discovery of the spill during the weekly maintenance - are totally different than what was described to the Coast Guard.

The state spill report - generated from information given the state by the Corps - claims the spill was discovered at 11:15 AM on the 16th. The chronology claims that cleanup started that afternoon after the Corps gave verbal approval at 1:55 PM. The chronology also places notification of the Coast Guard by the Corps in the two hours and 40 minutes between those two events:

"During QRI's weekly maintenance on November 15, 2011, the team spotted a hydraulic spill associated with pump #2W at the 17th Street pump station.
[...]
The crew alerted USACE representative Kim Branch of the spill.

Mr. Branch reported the spill to the appropriate authorities as per the SPCC plan. The DEQ incident number associated with this spill is 11-07-194.

QRI received a verbal NTP [Notice to Proceed] from Ione M. Cataldo, contracting officer/team leader at 1:55 PM on November 16, 2011"

Reporting of a spill per Corps' 17th Street Spill Prevention Contol and Containment - or "SPCC" - plan calls for notification of the Coast Guard's National Response Center by the facility owner - the Corps in this case - immediately upon discovery:

"All spills, as defined in 40 CFR, part 110.3 (sheen rule), will be reported to the National Response Center. LADEQ and EPA notification procedures and written report requirements for a reportable discharge are included in Appendix F.
[From Appendix F]
Any person in charge of a vessel or of an onshore or offshore facility shall, as soon as he or she has knowledge of any discharge of oil from such vessel or facility in violation of section 311(b)(3) of the Act, immediately notify the National Response Center (NRC)"

And from the Coast Guard's National Response Center spill report, here's the truth about when the report was actually received:

"Report taken at 13:36 on 17-NOV-11"

That's 26 hours after the Corps claimed the spill was discovered, and perhaps as much as 50 hours after it was actually discovered. By that point, according to the documents attached to the state report, the spill was already cleaned up. It's these little lies - along with the huge ones - that make the Corps so untrustworthy.

The chronology also notes the final quantity spilled was 77 gallons. That's over a quarter of the 300 gallon capacity of the hydraulic oil day tank on the engine skid for that pump. It is unlikely the pump would have run very well with that much oil missing.

A task order for QRI's cleanup work was written on November 16, 2011 for $9727.38. The final spill cost was $7129.03 after an April, 2012 modification.

Further confusion was introduced a week later, when on November 23rd, Healtheon got a $613,645.84 task order (#3 on the third Healtheon contract) for the removal, rebuilding, and reinstallation of the four pumps I listed above, which included 17th Street pump W2. It's very difficult to tell if this work was already underway on the 16th - which the spill reports claim - or if it didn't get going until the 23rd. I'm inclined to say it wasn't until the 23rd, since that's what the Conhagen repair report for that pump (and the report for pump W3) claims as the start date:

Admittedly, the repair reports for the other two pumps on this task order also claim the November 23rd date as the "job start" date, even though - as we'll see in a bit - they likely weren't pulled out until December However, if the work truly didn't start until November 23rd, that would mean the spill report to the Coast Guard was not only late, but also contained lies in the narrative.

Now to the contents of those 17th Street pump repair reports (W2 report, W3 report). They contain the usual pictures of rusty and marine growth-encrusted stuff. But what really attracted my attention was the condition of the splined drive shafts that come out of the bottom of the Rineer motors:

Rusty, corroding yuckiness! These shafts are enclosed in housings that are supposed to be sealed, but clearly were not:

It's not hard to imagine a pump refusing to work because of the poor condition of these shafts. In addition, if the shafts are this bad, all the internals on their motors would also be shot. And indeed, that is exactly what has happened. Repeatedly. For years.

Sidebar Rineer motors and salt water

On May 18, 2011, the Corps issued a $13,881.02 task order (#12 on the first Healtheon contract) which sent 8 Rineer motors for the main 60" pumps out for repair. At the time the scope only called for replacement of seals, shaft ball bearings, and fasteners (i.e. bolts) on the housings and seal plates. That work was supposed to take 6 weeks. Of course, if one is replacing seals, then that means the old seals are bad and stuff got into the motors.

So a $42,019.32 modification to the original Rineer repair order was sent out August 31, 2011, after the motors were taken apart and unsurprisingly found to have their internal moving parts utterly ruined. The delay between the original rebuild deadline (around July 1) and the task order finally getting issued (August 31) would correspond to the standard Corps dickering over whether they should spend the money to repair or replace the motors, as well as the usual delays in getting anyone to make an actual decision.

The modification to the original repairs was predictably wide ranging, given the fourfold expansion in costs. It reads:

"Provide labor and materials to replace the following additional components on the 8 Rineer hydraulic motors Model M125-A4-1H250-31-B1-0BB-213 for 60” hydraulic pumps, which were disassembled under the original Task Order. These components were found damaged upon disassembly; and were not included in the original scope of work.
The contractor will be given 24 weeks to complete the repairs to the motors upon award of this modification due to parts delivery lead times.

All Motors; Replace all motor housing plugs with new solid hex head plugs.

Motor #1, work order #9184;
Qty 1 complete vane cartridge kit (16 week del)

Motor #2, work order #9185;
Qty 2 complete vane cartridge kits (16 week del)

Motor #3, work order #9186;
Qty 1 complete vane cartridge kit (16 week del)

Motor #4, work order #9187;
Qty 1 Rear Housing (20 week del)

Motor #6, work order #9189;
Qty 1 complete vane cartridge kit (16 week del)

Motor #8, work order #9191;
Qty1 complete vane cartridge kit (16 week del)"

Replacement of the complete vane cartridge kit in these motors is the equivalent of having every cylinder in your car's engine rebuilt. Of the original 8 motors sent out in May, 2011, 5 (62.5%) were found to require this complete vane cartridge replacement. That is a shockingly high rate of failure, one that should make New Orleanians blanch.

But this major rebuild work on the Rineers is of a piece with what had been happening since mid-2010.

In August, 2010 a solicitation for 12 new Rineer motors was issued by the Corps. The solicitation remains up at the FedBizOpps webpage here. 8 of the Rineers were to be for the 60" pumps, with the other four going in the smaller 42" pumps installed on the structure at 17th Street. Rineer motors - especially special order ones like these - have a very long lead time. The winner of the $104,760 contract (W912P8-10-C-0116) was Hydraquip of Houston, TX, a Rineer rep with a local office near Armstrong International Airport in Kenner, LA. They had about 3 months to deliver the new motors, meaning the Corps would have had the new ones in hand around December, 2010.

And going back further, let's not forget that in July, 2010 the Corps issued a task order modification to rebuild 12 other Rineer motors, all for 60" pumps.

Finally, more recently the Corps sent an additional 6 Rineers for the 60" pumps out to Hydradyne, LLC for "overhauling" in a March 8, 2012 task order on the first Healtheon contract. The task order is not specific as to the scope of work; however, we can be assured it is the same as the more extensive work called for in August, 2011, because of this:

"Estimated completion date is 20 weeks from the award date (due to part availability from the manufacturer)"

Only replacement of vane cartridges could take 20 weeks based on parts availability.

So all this activity means that in a little over a year and a half, the Corps rebuilt 26 Rineer motors for the 40 big hydraulic pumps, and purchased 8 more. That's close to the number of pump pulls that occurred over that same span (41 by my count, including three pumps coming out twice).

However, the numbers really don't matter. The bottom line is that these Rineer motors are obviously ill suited to salt water exposure. Their seals are likely designed for fresh water. Leaving them where they are is simply asking for big trouble. Over the last three years I've published a slew of pictures showing Rineers rusting to bits. They've been getting salt water into them through the rusty pipes, the seals, and every other leak path. The Corps has tried replacing them and repairing them. Nothing works. They all get eaten up. These motors are the heart of the pumping system at all three Lakefront closure structures, and they are unreliable and rusting to this day.

The rest of the late 2011 work

The other two pumps on the November 23, 2011 task order - Orleans Avenue pumps W1 and W2 - didn't get pulled out until early December. This was the first time the Corps pulled out more than one Orleans Avenue pump at a time since the mass rebuilds in 2006-07, indicating they would finally be working on the pumps at that location in a coordinated fashion.

On December 8, 2011, a spill report was filed with the National Response Center for an incident at 2 PM that day:

"Qty Released: 10 GALLONS

CALLER STATED THAT THERE WAS A RELEASE OF 10 GALLONS OF HYDRAULIC OIL FROM A HYDRAULIC PUMP WHICH IS PART OF THE CANAL STRUCTURE, THE CAUSE WAS DUE TO OVERHAULING OF THE PUMP AND THE CAP ON THE PUMP WAS LEAKING.

CLEAN UP UNDERWAY, CONTRACTOR HAS BEEN HIRED"

The companion cleanup task order was issued to QRI that same day. The text of the $6,093.29 task order confirms the spill was connected to the pull-out of Orleans Avenue pumps W1 and W2:

"Verbal authorization for this work was given by Contracting Officer Cynthia A. Nicholas on 8 December 2011 at 3:18 p.m in a Not to Exceed amount of $6,093.29 for a 10 - 50 Gallon Spill at Orleans Avenue Canal Pump 1W and 2W."

A later modification dropped the actual cleanup price to $5,647.60.

As with the 17th Street pumps on this same task order, individual repair reports were compiled for each pump (Orleans W1 repair report, Orleans W2 repair report). As usual, there was rust, marine growth, and other scary stuff:

The 17th Street pumps went back in the water January 24, 2012. The Orleans Avenue pair followed on January 27, 2012.

London Avenue pump rebuild work is finished off

While those four were out, a remarkable decision to repair a fifth pump was made. That pump - W2 at London Avenue - was the last of the 12 London pumps not to be refitted over the past three years. A $142,459.56 task order (#11 on the second Healtheon contract) was issued December 20, 2011. There are no oil spill reports or task orders to QRI associated with this pump repair, which is odd. Single pump rebuilds like this usually come about because the pump sprung an oil leak. This could be yet another case of an unreported spill, or a more unlikely case of the Corps acting proactively to repair more than four pumps at a time.

The Conhagen repair report for London Avenue pump W2 is very spare on photos, almost as if they know their reports would be publicly disseminated. There is one photo of interest though, and it's an "after" shot:

This is probably a rebuilt Rineer motor. The color appears different than other motors which have come out, and part of the rebuilding task orders for Rineers has mentioned painting them. So that's one good thing, I suppose. But it will rust out again, long before the new pump stations come online in 2016, or whenever.

According to the repair report, London Avenue pump W2 went back in the water March 12, 2012.

First round of 2012 - focus on Orleans continues

That brings us to the next round - pumps W3 and W5 at the Orleans Avenue site. Healtheon received a $304,322.08 task order on January 30, 2012, the day after work wrapped up on the last set of 4 pumps. It's unusual for the Corps to only pull out 2 pumps rather than 4 in the off season, but perhaps London Avenue pump W2 was still in the shop at the time.

A spill report was filed with the National Response Center on February 22, 2012, for a 20 gallon spill supposedly discovered at the Orleans Avenue structure on February 21, 2012:

"Caller is reporting a discharge of hydraulic fluid from a leaking hydraulic pump. Spill contractor is on site [to] clean up material; material contained by boom in the drainage canal."

By this time, QRI had been awarded a shiny new spill cleanup contract, it's third. The four year agreement has a potential value of $581,376.24 (!) Task order #3 on it, worth $6360.39, was issued February 23, 2012, two days after the supposed discovery of the spill. The description in the task order seems to connect it to the Orleans Avenue pumps in the January repair order:

"This task order is [to] cover the clean up of hydraulic oil spill estimated to be in the 10 to 20 gallon range. The event happened from pump #3 at the Orleans Ave Canal."

While the task order doesn't specify east or west, just "pump #3," I'm going to guess it was referring to pump W3, meaning it and likely W5 came out on or around February 21, 2012. They probably went back in sometime in late March or early April.

Unfortunately, that's where my latest batch of documents ends, since I placed my latest FOIA request on March 9, 2012. However, there are ways to track what's going on even without FOIA requests.

The federal government's FPDS database posts all federal contract transactions. There's an immensely annoying 90 day hold on Pentagon transactions posting to the website, ostensibly for security purposes, but mostly to keep prying eyes away. However, since the transactions from February, March, and the beginning of April are now available, we know there was another task order on the third Healtheon pump repair contract issued. Task order #5 for $308,594.52 was issued March 27, 2012 for the repair of Orleans Avenue pumps E1 and E2, with a scheduled completion date of May 28, 2012. This would seem to indicate it was the last batch of pump repair work the Corps planned for the offseason before the 2012 hurricane season, which began on June 1, 2012.

The National Response Center online database is updated daily, so it's an excellent contemporaneous indicator of activity. Since the late February Orleans Avenue canal spill there have been three additional spills at the outfall canals:

May 1, 2012, Orleans Avenue structure
"Caller stated due to a faulty o-ring in a hydraulic line connector there was a spill of hydraulic fluid. Caller stated the sheen is burning off and absorbents will be used later as a boat will be placed in the water."
Estimated quantity spilled: 5 gallons

May 11, 2012, 17th Street structure
"Caller reported a spill of materials while repairing an engine on a hydraulic pump. Booms applied, cleanup underway."
Estimated quantity spilled: 14 gallons

July 2, 2012, Orleans Avenue structure
"Caller is reporting a discharge of hydraulic oil from a hydraulic pump due to unknown causes. Caller stated the release is estimated between 15 - 50 gallons. Double boom was in place, contractor is on scene cleaning up"
Estimate quantity spilled: 50 gallons

Knowing what we know about these spill reports - the data comes from the Corps, and it may or may not be true - I try not to put too much stock in the descriptions. The reports are really best for their timeliness. We'll find out later whether they're truly associated with pump repairs, scheduled or otherwise.

UPDATE 2/14/17

As noted above, at the beginning of 2012 the Corps was focused on completely refitting pumps at the Orleans Avenue site. Pumps W3 and W5 came out immediately after the return of pumps W1 and W2 in late January. W4 had been refit during the 2010 hurricane season. The task order for Orleans W3 and W5 is here, and the repair report is here.

The report is unremarkable, with only "after" photos. The pumps were accepted by the Corps March 29, 2012.

Next up were Orleans pumps E1 and E2. They came out the same week W3 and W5 went back in, the last week of March, 2012. The task order is here, and the repair report is here. The report has pictures of rusted out carbon steel piping cut off of the pumps (note: the labels of "1W" and "2W" are Conhagen's typos):

It's striking to see this piping discarded like so much junk.

These two pumps were accepted by the Corps June 19, 2012. While they were in the shop, two other pumps sprung oil leaks - 17th Street W3 and London Avenue W5. Both pumps had been in the water for less than a year following complete refits. Details on those failures can be found in an update at this subsequent post.

The two remaining Orleans Avenue pumps that were left unrefurbished at the beginning of the 2012 hurricane season - E4 and E5 - unsurprisingly leaked oil that summer and had to come out during the height of hurricane season. They were out of commission when Isaac came ashore in August, and their absence likely led to numerous pumping cutbacks while the storm moved through. Details on repairs to those two pumps  - and links to how their going missing hurt city drainage during Isaac - can also be found at the same subsequent post.

So this is what we can say about the pumps at each site, given the documents and other information I've sussed out:

These "reliability" estimates come with huge caveats. We know the critical Rineer motors are continuing to rust away, no matter how much they get replaced or repaired. We also know of numerous other problems back on the pumps' remotely placed engine skids unrelated to corrosion. In essence, all these graphics show is that some carbon steel piping has been switched out for stainless steel, and there's been other corrosion repairs and repainting. The mechanical reliability of the hydraulically driven pumps, especially due to the ongoing corrosion of the Rineer motors and the resulting saltwater ingress to the hydraulic oil loops, is highly questionable should they be called upon during a major tropical event.