The last portion of story of the repairs to the rusty 60" hydraulic pumps at the three New Orleans lakefront gate complexes left off in June of 2010. The Corps continued to pull pumps since then. A brief overview of the rusty pump story can be found at this earlier post. One of the more detailed posts, "Worse than previously known," traces the first few task orders of the contract described below, which is the second of three repair contracts given to small business Healtheon, with pump rebuild shop Conhagen doing the actual repairs.
It took over a year of badgering the Corps of Engineers to release contract documents and reports on these repairs, but they finally did so July 8, 2011. Thankfully, we didn't have to wait all that time, because there was plentiful evidence on the ground of what was happening.
In sum, there were six more pump pulls between June, 2010 and March, 2011 involving 7 pumps across all three lakefront sites. Two of those pumps - E1 and E5 on the east side of the London Avenue canal site - came out the week of March 23, 2011.
However, that still leaves about 18 pumps remaining totally untouched, with another 5 incompletely repaired in 2009 and requiring the full overhaul treatment. The lack of urgency continues unabated. Meanwhile, pumps continue to rust in the brackish waters of Lake Pontchartrain.
June 14, 2010
Healtheon contract W912P8-10-D-0054, Task Order #4, $186,793.42
The FPDS-NG description on this task order reads, "Removal, reconditioning, and reinstallation of Pumps at London Avenue Canal, 17th Street Canal Interim Closure Structures per scope of work dtd 09 June 2010."
The dollar amount of this task order indicated the repair of a single pump, but that appeared to be contradicted by the plain text of the description, which mentions two sites. Upon receipt of the contract documentation, the confusion was cleared up.
There was a spill of oil from 17th Street pump W1 on June 2, 2010. A Corps contracting officer contacted QRI that day and told them to clean it up. Remarkably, the Corps also reported the spill to the National Response Center, as required under federal law.
A task order covering the cleanup work - task order #9 - was drafted the next day. The text of the task reads:
"Clean up of hydraulic oil spill at Pump #12 at 17th Street Canal. The spill is estimated at 15 gallons and is within scope of a small spill on water. Verbal authorization to begin work was given on 6/2/2010 by a Contracting Officer."
The description of "Pump #12" in the QRI cleanup task order - which got propagated to FPDS-NG - is a typo. The normal nomenclature used by the Corps to label the pumps is [Number][Direction]. The number - "1" - was correct, but someone's finger slipped and typed "2" instead of "W" for the direction. We know this from subsequent documentation described below.
As with so many of the other pump repairs, the Corps was reactive rather than proactive, and only pulled this pump out after it sprung an oil leak. They issued task order #4 on the second Healtheon contract officially on June 15, 2010, though the enclosed scope of work in the task order is dated June 9, 2010.
The pump was pulled out sometime in the first half of June and sent to Conhagen's shop. Courtesy of the Conhagen repair report for the 17th Street pump W1 work, we have pictures of the pump in the shop.
The leak could have come from anywhere that was corroded. Maybe it was at this nearly unrecognizable corroded pipe fitting at the discharge of one of the oil coolers:
or the Rineer motor, or through the steel seal flanges:
or anywhere else, because the single pair of zinc anodes originally installed on the pump to prevent corrosion certainly hadn't been doing its job for a very long time:
Wherever the leak occurred, it doesn't really matter, because this pump got nearly the full rebuilding treatment. All the internal and external carbon steel pipes, including the oil coolers, were changed out for stainless steel. All the seals, bearings, and associated hardware were completely replaced. The pump housing was hydroblasted, and any chunks missing from corrosion were weld repaired. Effectively, the pump was completely rebuilt, except perhaps for the Rineer hydraulic motor, which isn't mentioned as getting replaced.
Based on the dates on the Conhagen report, this pump probably went back in the water around July 10, 2010. A reconciling of of the rebuild costs was recorded as modification 1 to task order #4, issued on April 6, 2011. $2,619.84 was subtracted, bringing the final cost of the 17th Street pump W1 rebuild to $184,173.58.
July 14, 2010
Healtheon contract W912P8-10-D-0054, Task Order #5, $153,560.66
The FPDS-NG description of this task order reads, "Remove, recondition & reinstall Pump #1W at London Ave. Canal."
It is unclear why pump W1 at the London Avenue site was pulled. There's no record of an oil spill preceding its removal, but that doesn't necessarily mean anything, since the Corps is absolutley awful about following the federal law requiring immediate reporting of all oil spills. It is unusual for the Corps to pull single pumps instead of pairs, as noted in the scope of work below, so it seems likely there was an unreported spill.
Anyhow, this pump got the expanded rebuild treatment that the previous few pumps got, including new bearings and seals, except for one aspect: like 17th Street pump W1, apparently its Rineer motor was not replaced. Here's the scope of work from the task order:
"The Scope of Work for this task order is for a single pump in lieu of a pair of pumps as noted in Schedule B of the contract.
The Scope of Work for this task order shall also include the following additional work:
Repair Mechanical Seals at subcontractor shop 1 Each
Remove Pump Shaft Bearings
Replace Radial bearings Impeller End (1 Each)
Replace Radial Bearings Motor End (1 Each)
Replace Thrust Bearings (1 Set)
Replace bearing Spacers (1 Each)
Inspect Shaft And Housing
Mount New Bearings
Clean All Parts For Assembly
Reinstall Pump Shafts"
It's unclear why the Corps wouldn't have the Rineer motor replaced on this unit and 17th Street pump W1.
The pictures in the Conhagen repair report for London Avenue pump W1 don't show as much as in other reports. But they do show one part that got pretty beat up:
The seal flange is a piece of metal meant to help keep oil from leaking out and water from leaking in to the enclosed section of the pump which contains the shaft bearings. Looking at these pictures, it seems unlikely it was performing its task. So, as with all the other pumps, a new seal flange was remanufactured by Conhagen.
Based on the dates on the Conhagen report, this pump probably went back in the water around August 15, 2010.
October 8, 2010
Healtheon contract W912P8-10-D-0054, Task Order #6, $169,746.50
The task order description in FPDS-NG reads, "Removal, Reconditioning, and Reinstallation of Pump 4W at Orleans Avenue Canal, Interim Closure Structure."
Again, like the last pump, we don't know exactly why this pump was pulled out. There's no spill report, and it remains unusual for the Corps to pull a single pump rather than a pair (a fact again noted within the scope of work in the task order for repair of this pump). Like pump W1 at the London Avenue site, this is probably also a case of an unreported oil spill.
Steve Beatty of The Lens got a number of photos of this pump removal on the day it happened, October 12, 2010. This is my personal favorite:
This is a Corps employee making sure to document who's taking pictures. That's pretty chilling. I wonder what a FOIA request for the picture this guy took - along with all other pictures taken by Corps employees and contractors on the pump repair job - would turn up?
Here's some more shots. The pump was already pulled out of the water when Mr. Beatty arrived. However, there was plenty of other work caught on camera
Here the elbow from atop the pump is replaced above the now empty pump slot:
The inspection hatch is what the Corps opens up to see if there's oil leaking, which is a primary criterion for pump removal. Whether such a spill gets properly reported or not is a separate question (answer: mostly "not").
Here's a nice clear shot of the elbow without the pump while the pump was in Conhagen's shop:
I believe pump elbows are now being replaced when pumps are removed for only one reason: to camoflauge pump removals from the prying eyes of the public, especially after I noted the easy way to determine if pumps were out. That is, the Corps used to leave the elbows on the levee next to the gates:
Now they put the elbows back after they pull pumps. It is money being spent for no good reason other than effectively stifling public information.
The next step after the elbow was replaced was to prepare to lift the pump from the deck...
[The pump is still connected to its suction bell and vortex suppressor here. They would be transported to Conhagen's shop separately.]
...And place it on temporary braces for transport:
Then it is lifted to a flatbed truck from the prime repair contractor, Conhagen:
The rusting and buildup of marine growth is obvious in these photos.
The scope of work for this pump, like the prior two, included the extended rebuild package (new bearings and seals, in addition to new stainless piping and stainless oil coolers and weld repairs). And also like the prior two, seemingly did not include work on the Rineer motor. Here's the scope of work from the Orleans Ave pump W4 rebuild task order:
"1. Line Item 1001CC (Revised - See Note): Removal, reconditioning and reinstallation of pump #4W at Orleans Ave. Canal Interim Closure Structure according to the Scope of Work in Section C-1-Summary of Work in the subject contract.
Note: This scope is for a single pump in lieu of the scope for a quantity of two pumps in line item 1001CC, schedule B of the contract. The scope of work for the single pump shall be revised to include the following additional work:
Repair Mechanical Seals at subcontractor shop 1 Each
Remove Pump Shaft Bearings
Replace Radial bearings Impeller End (1 Each)
Replace Radial Bearings Motor End (1 Each)
Replace Thrust Bearings (1 Set)
Replace bearing Spacers (1 Each)
Inspect Shaft and Housing
Mount New Bearings
Clean All Parts For Assembly
Reinstall Pump Shafts"
There's no mention of the Rineer motor, which is strange, especially in light of modification #2 to task order 3 on this contract, which called for the rebuilding of 12 Rineer motors, 8 which were to be kept as spares. That modificiation was issued in July, 2010, and was scheduled to be completed in August, two months before this pump was pulled out.
Anyhow, the Conhagen report of the rebuilding of Orleans Ave. pump W4 includes pictures of its particularly rusty innards. Here's the Rineer motor and all the pipes leading into and out of it, with a considerable coating of rust:
And here is the suction bell of the pump (the suction bell is the flared section at the bottom of the pump where the water enters):
Note the substantial chunks missing due to corrosion.
After rebuilding, this pump was placed back in service November 20, 2010.
Upon review of the contractors' invoices, this task order was closed out with modification no. 1 on June 15, 2011. $4,150.46 was subtracted, bringing the final cost for the Orleans Avenue pump W4 rebuild to $165,596.04.
October 15, 2010
Solicitation W912P8-10-R-0066 issued
The Corps opened the pump repairs up for public bid on this day, with a solicitation for "Removal, Reconditioning, and Reinsitallation" of the pumps at all three canal sites. The solicitation has since disappeared from the FedBizOpps site, but it and all five of its amendments can be found here.
The solicitation follows the same general scope as the repairs to date. That is, pumps are to be pulled, they're to be hosed down, all their carbon steel guts are to be replaced with stainless steel, zinc anodes are to be added, and any other bits and pieces that are messed up are to be repaired before they're put back in.
The difference on this solicitation from the second Healtheon contract, issued in March, 2010, is that the Rineer hydraulic motors at the heart of each pump unit are paid more attention. In the scope of work, this paragraph has been added:
"Remove the Rineer hydraulic motors from the pump housing and transport to a hydraulic facility approved by the [Contracting Officer's Representative (COR)]. Disassemble, clean and inspect. Provide written report of conditions found to the COR. Provide a written recommendation for any necessary repairs to be performed under the “Reimbursable Expenses” contract line item."
This likely ties in to the purchase of 12 Rineer motors in September from a Rineer distributor in San Antonio. There's much more detail about those Rineers in an update I embedded in this earlier post. More than likely, the Rineer purchases were in anticipation of replacement in case repairs didn't work out. That's probably wise, considering the Corps had already replaced a bunch earlier in the repair effort (also noted in that earlier post).
This solicitation led a later contract, number W912P8-11-D-0015 issued February 7, 2011, again to Healtheon. Details on that contract can be found below in the section titled "March 22, 2011."
December 9, 2010
Healtheon contract W912P8-10-D-0054, Task Order #7, $282,709.39
The FPDS-NG description reads "ITEM 1001BC - Pumps 2E and 6E - Remove, Recondition, and Reinstall Pumps at London Avenue Canal."
Unlike the earlier 2010 removals and repairs of London Avenue pump W1 and Orleans Avenue pump W4, this pair of pump pulls was accompanied by sudden uptick in oil spill cleanup operations. There was a flurry of task orders and modifications to QRI in December, 2010 in conjunction with this Healtheon task order. The first was issuance of a task order for a large oil spill on December, 8, 2010. It was task order #10 on the QRI oil spill response contract, with a value of $944.06. The description within the task order reads:
"Clean up of hydraulic oil spill at London Avenue Canal. The spill was from pump #2E and #6E, spread from pump booms over within the south and north booms of the canal due to the high water level. The spill is estimated at 100 gallons."
"Verbal authorization was given by the Contracting Officer on 12/08/2010 at 1:50 p.m."
The jumbled verbiage ("spread from pump booms over within the south and north booms of the canal due to the high water level") refers to the fact that the oil spread from the primary containment which immediately surrounds the pumps, and was out in the canal, between the booms to the south and the north of the site.
A few days later, the Corps added more funds:
December 15, 2010
QRI contract W912P8-09-D-0022, Task Order #10, modification 1, added $3,439.64
The FPDS-NG description reads, "Mod to Adjust Funds." The description on the task order is more detailed:
"The work under this task order w as completed on 8 December 2010. Actual totals were recieved and funding for this task order is increased by $3,439.64. Total task order amount is now $4,383.70. Task order completion date has been changed to 14 December 2010 so that the modification could be processed."
This makes sense. If the spill was 100 gallons, it would cost a lot more than $944 to clean it up, based on previous QRI task orders for spills of similar magnitude.
A couple of days later, they added even more funds, seemingly in response to a second spill:
December 21, 2010
QRI contract W912P8-09-D-0022, Task Order #10, modification 2, added $3,979.89
The FPDS-NG description reads, "Mod for an addition oil spill of approximately 25 gallons."
The task order reads,
"The above numbered task order for a hydraulic oil spill at London Avenue Canal is hereby modified to provide for additional oil clean up at pump 2E. The oil spill is approximately 25 gallons and will use contract CLIN 1002AA. The modification will increase funds in the amount of $3,979.89. Total task order amount is now $8,363.59. The completion date has been extended until 21 December 2010 as a result of this modification."
The effective date of the task order is December 17th. I'm not sure what to make of this mention of a second spill when I would think the pump would be in the shop.
Making it difficult to suss out is the fact none of this oil spill activity appears at the website of the National Response Center. These apparent multiple Clean Water Act violations are of a pattern with the Corps' activity around these pump repairs, as documented above and in Corps of Oil, Part 3.
The task order for the repair of these pumps was issued on December 9, 2010, the day after the oil spill. At first blush, this would appear to be typical of the Corps' reactive - rather than proactive - approach to maintenance. However, there's something hinky about the dates of work on these pumps.
The Conhagen report on the rebuild of London Ave. pumps E2 and E6 provides the oddity. Its cover page mentions that their work on these two pumps began November 29, 2010, over a week before the issuance of the task order for that work:
That's somewhat unusual for these types of task orders.
In addition, the date of completion on the task order is December, 24, 2010. That would theoretically give only 16 days for Conhagen to rebuild two pumps, when they've averaged 4 to 8 weeks on previous pairs of pumps. A December 24th completion date argues for an earlier start date, like the November 29th date shown on the front of the Conhagen report.
Putting two and two together indicates that the first 100 gallon spill occurred during the pull of these pumps, not before it. For some reason, that pull - along with the issuance of the task order to Healtheon for it - had been delayed from the last week of November until December 8th. And then, when Conhagen went to pull the pumps, they dumped 100 gallons of hydraulic oil in the canal. If National Response Center records are to be believed, the Corps didn't bother to report that spill, or another spill that occurred a few days later. If it is true that oil was spilled during the pumps being pulled (as we know it has in many, many other instances), it will be quite important, as I'll describe in a future entry.
But enough about the oil spills. As to the rebuild work on the pumps themselves, they got what had become - at that point - a standard scope of work for rebuilding. See the above entries on repairs to other pumps over the summer and fall of 2010 for details on that scope. The salient part is that neither the scope of work in task order #7, nor that in the Conhagen repair report, mentions rebuilding or repalcement of the Rineer motor. This makes five pumps in 2010 where the repair of the Rineer motor was not explicitly addressed, which is troubling, because the motor is the very heart of the pump unit. If all the other parts of the pumps were rusted to bits and required replacement, it's very likely the Rineers required it as well.
The Conhagen report on the rebuilding of London Avenue pumps E2 and E6 contains a number of pictures of the pumps. They did not look good when received at Conhagen's shop in Kenner, LA.
Here's the rusty Rineer hydraulic motor from pump E2, along with rusty pipes:
Pump E6's Rineer motor and attached pipes looked just as bad:
This is the bottom section of pump E2, which was removed for transport:
It's made up of three parts: a vortex suppressor, the suction bell, and the liner.
The vortex suppressors were added to the pumps at London and Orleans Avenues after installation. They are supposed to cut down on the formation of small whirlpools called vortexes (or vortices), brought on by what boils down to poor design of the pumps and their facilities. In essence, the pumps don't see enough water when the lake is at elevation 0. This is part of the reason the Corps only operates the pumps for extended periods when the canal elevation at the closure structures is above 2 feet.
The suction bells and liners are original parts of the pumps. They guide the incoming water to the impeller.
As one can see from the photo above, there was significant corrosion happening on this pump, as evidenced by the severe depletion of the single pair of zinc anodes, as well as the considerable marine growth.
That marine growth appears to have caused the paint on the liner to delaminate, leading to corrosion of the base carbon steel underneath:
The carbon steel piping of the oil coolers and pipe extensions on E2 never stood a chance in Lake Pontchartrain's brackish water:
Conhagen replaced the piping above, and repainted and weld repaired the suction bell, vortex suppressor, liner, and whatever else was missing metal due to rust and marine growth. They also put in new bearings, seals, gland plates, and other stuff to completely rebuild the pumps.
Since the start of the work appears to have been delayed, it was no surprise that modification 1 to the task order for this work extended the delivery date to January 10, 2011. Even so, the work took the usual six weeks for a pair of pumps, and they went back in on January 24, 2011.
However, the story was not complete with these pumps. You can read the unsurprising surprise conclusion in a subsequent post.
March 22, 2011
In the first spill report to the National Response Center since 17th Street pump W1 leaked oil on June 2, 2010 (despite the copious pump removal work in the interim documented above), the Corps reported this:
"CALLER IS REPORTING THAT DURING TWO PREVENTATIVE PUMP MAINTENANCE PULLS OVER THE PAST TWO DAYS THERE WAS A RELEASE OF ABOUT 1 CUP OF HYDRAULIC OIL INTO THE BOOM THAT WAS ALREADY IN PLACE. THE MATERIAL NEVER LEFT THE CONFINES OF THE CANAL STRUCTURE OR THE BOOM."
Regulations stipulate that these spills are supposed to be reported when they happen, not two days later. Also, the quantity released is unbelievable in the extreme, especially given the well documented history of tens of gallons per spill when pumps are pulled. In fact one only has to go back to the above section on the repairs last December to find spills (unreported of course) of "Over 50 to 100 Gallons" and "approximately 25 gallons."
The London Avenue pumps removed during this activity were E1 and E5, the last two hydraulic pumps on the east side of the site not to be repaired. They went back in sometime around the third or fourth week of April, as seen in this subsequent post.
This work is part of a third contract to Healtheon for pump repairs. On February 7, 2011, Healtheon was awarded contract W912P8-11-D-0015, in response to solicitation W912P8-10-R-0066 (original FBO page, complete solicitation), which was circulated in late 2010. It is a four year contract (a base year and three option years) which has a maximum value of $6.75 million.
That third option year is interesting, because that places the possible work for this contract in 2015. Yet the Corps has said in its recent announcement of an award for the permanent pump stations that the new stations are due to be finished in 2014. Could this be a hint that the schedule is actually slipping once again?
[Note: The schedule for the permanent pumps did slip, as a result of multiple bid protests after the contract award in April, 2011. Details are available at the subsequent post, "Permanent pumps slip to Katrina 10th anniversary."]
Anyhow, on March 14, 2011, the new Healtheon contract had task order #1 issued on it for a value of $598,101.68, calling for pulling of four pumps: E1 and E5 at London Avenue and W4 and W7 at 17th Street. We'll concentrate on the London Avenue pumps for now.
I have pictures of the empty slots where London's E1 and E5 usually reside. First, E1 (with the elbow back in, as noted above):
I know this picture's a little busy. You can tell that E1 is out because you can't see its piping extensions, like you can on E2:
Noteworthy is the yellow boom near the E1 slot. It's been left there by oil spill contractor QRI after the pump removal, and got tossed onto the supporting steel by the tides or bad weather:
Here's a picture showing E5 out of its slot:
And again, more yellow temporary boom placed by QRI when the pump was removed:
The rest of the work on London pumps E1 and E5, as well as other work under the the third Healtheon/Conhagen contract, is covered in subsequent posts, starting with "The 2011 pump rebuild scramble." There you can find links to more contract documents and Conhagen repair reports, as well as Conhagen pictures of the pumps pulled out under that new contract.
Summary
At the time of this post, details on many 17th Street pumps were unknown. All that was known for certain for certain was that pump W1 came out for repairs between June, 2010 and late March, 2011. So the 17th Street pump status looked like this:
while the Orleans and London statuses were:
Since E1 and E5 were still out at London Avenue, they were not counted as "repaired" in the tallies. See this subsequent post for confirmation of their replacement, and further details on their repair.
Karen Gadbois and Steve Beatty of The Lens contributed to this report.
No comments:
Post a Comment