Imminent
The lakefront hydraulic pumps are rusting to bits, and have been since they were installed. Some have already fallen apart. The Corps was warned all of them would do the same thing "imminently" in mid-2009, and did next to nothing for nearly the entire 2009 hurricane season. For those that don't want to read the whole story, evidence is in this report from the Corps New Orleans District's pump repair contractor.
Only in recent weeks has the Corps' New Orleans District embarked on a crash repair initiative hidden from the public in plain sight, in an effort to make up for years of deferred - but not unfunded - preventive maintenance on pumps which they knew from the very beginning would fall apart within a matter of a couple of years, if not quicker. That lack of maintenance likely left New Orleans' front line flood defenses in a precarious state of decay during the entire length of last year's hurricane season, and may have actually played a part in a year-end flooding event. [UPDATE: It is more likely that the December 8, 2009 systemwide failure of the Corps SCADA control system for the pumps and gates played a larger part in the decision not to lower the gates and run the pumps during widespread flooding and elevated lake levels on December 12-13, 2009. Details can be found in my subsequent post, "SCADA."] It also may have been a pawn in the Corps recent negotiations with the state to impose their inadequate Option 1 scheme. I'll get to the bits about the current work, the year-end flooding event, and Option 1 in future posts. For now, I'm just going to concentrate on the 2009 repairs.
In spring, 2009, the Corps issued their first work order on contract W912P8-08-D-0090, a potentially five-year-long pump maintenance they inked with a company named Healtheon in 2008.
That $28,138 work order, whose value was later increased by $1487.43 made reference to pulling three pumps at 17th Street because of "hydraulic oil leaking problems:"
As a quick reminder, most of the pumps at all three interim closure sites on the lakefront are hydraulically powered. Hydraulic oil is raised to approximately 3000 psi by hydraulic pumps manufactured by Denison Hydraulics at engine skids placed in sheds remote from the pumps (though at 17th Street, some of the pumps have their engine skids adjacent to them). That pressurized oil flows along hundreds of feet of pipe (or a few dozen feet of rubber hose, in the case of the 17th Street skid-adjacent pumps) to the water pumps in the canals. Inside the water pumps, the energy of the pressurized hydraulic oil is translated to rotational energy by a hydraulic motor manufactured by a Texas outfit known as Rineer.
The Rineer motor (the original was four port model 125, though it has since been modified - see below) is attached to a shaft which spins an impeller, pushing stormwater out of the canal, through the pump, and down 9-foot-diameter pipes which travel past the gates to the lake. The depressurized hydraulic oil travels through coolers mounted on the water pump, and then back through more piping to a reservoir at the drive unit. From there, the cycle repeats.
There are 18 60" hydraulic pumps at 17th Street, and 14 42" models. The size (60" or 42") refers to diameter of the pump housing. The original order to MWI for the hydraulic pumps in January, 2006 was for 34 60" pumps, with 12 each going in at 17th Street and London Avenue, and the remaining 10 going in at Orleans Avenue. Those were the "Phase 1" pumps. Another six 60" pumps were installed at 17th Street later in 2006 ("Phase 2"), and the 14 42" pumps went in during the spring of 2007 ("Phase 3").
There are also 11 direct drive units at 17th Street and 8 direct drive units at London Avenue, installed during the spring and summer of 2007. Those pumps have their drives directly coupled to their shafts, eliminating all the problems inherent in a hydraulically powered system. Here's the layout of the pumps at 17th Street:
Here's the numbering of each pump at 17th Street:
Here's the London Ave layout, with the original 12 hydraulic pumps noted as "Phase 1:"
And the numbering for the pumps at London Avenue:
The numbering at Orleans Avenue is similar to at London Avenue, with hydraulic pumps E1 through E5 on the east side and W1 through W5 on the west side.
The piping which conveys the hydraulic oil is coupled to the water pumps via hoses. To repair the water pumps, the hoses must be decoupled. There is no way to blow the hydraulic oil back to the drive units, so the oil spills in the canal when the hoses are detached. As it turns out, these "planned" oil spills are excellent indicators of work (or trouble) happening on the hydraulic pumps.
That's because the Corps also has a contract for oil spill cleanups. It is with Quaternary Resource Investigations (QRI) of Baton Rouge. Every time a pump is pulled, oil is spilled, so QRI should be on site to clean it up. QRI also provides and maintains the permanent oil protection booms that are at each site.
Thus, if a pump (or pumps) is pulled, QRI gets a task order on their contract. According to federal law and the Corps own oil spill response plans (linked: 17th Street SPCC; Orleans Avenue SPCC; London Avenue SPCC), the oil spill should also be reported to the National Response Center (NRC), and it should show up on the NRC report query website. Depending on the size of the spill (the state only requires spill reports for spills over 42 gallons, or 1 barrel), the state should also have a record of the spill on the LDEQ Electronic Document Management Service website. Reporting to the state is also a requirement under the law and the Corps' written plans.
So back to the May, 2009 pump repairs...
The work was - according to the task order - to take place in late May and early June. Every year in late May, the Corps holds a hurricane preparedness exercise, part of which they put on in front of the cameras of the local news crews. It is mostly a dog-and-pony show. In 2009, the Corps had the media out to the London Avenue canal gates on May 27, 2009 (Fox 8 story here), which was a departure from their usual M.O. Whenever the New Orleans District wants to emphasize how wonderful the gates and pumps are, they have the press out to the much larger 17th Street site. No one in the local press seemed to question why they were at London instead.
However, the Corps themselves obliquely acknowledged the 17th Street work in their post-exercise press release on May 27, 2009, which was the same day the repair work was due to commence:
The "previously scheduled" thing is a canard. The work appears to have begun in a hurry, since the Healtheon task order #1 has an effective date of June 17, 2009, or three weeks after the work began (likely with some kind of unofficial Notice to Proceed like an email or a phone call). Other contracts issued for the work all had dates after the work began, except for the crane rental contract, which went out less than a week before (see below). Also note how vague the press release is, giving the impression of "nothing to see here, move along." That was made easier by keeping the prying eyes of the local TV crews miles away from the 17th Street site.
A more detailed account of what was going on May 27, 2009 at 17th Street did get created: the QRI-written spill report for the pulling of 60" hydraulic pump W10. It was generated as part of QRI's work under task order #5. It gives an extensive narrative of that day's events:
Note that notification of the National Response Center is oddly omitted. That's strange, because the state reportable quantity (RQ) for oil is 1 barrel (42 gallons), while the federal government RQ is any amount, no matter how tiny. Indeed, there is no report of this spill on the NRC website.
Anyway, the report describes a lot of activity. QRI took date-stamped pictures too:
It would have aroused questions. Keeping the cameras away sidestepped that possibility.
So they lifted out W10 and then what? It's not easy to tell. There's a number of other contract actions related to the work which went out before the time of the Healtheon task order, with its effective date of June 17, 2009 but its text requiring work before that date.
There's the $16,370 (note: FPDS-NG reports the value as $31,492) contract for rental of the crane seen in the photos above that pulled pump W10, W912P8-09-P-0210, issued to RPL Oil Distributor and Supply, LLC of Gretna, LA, a frequent go-to company for the New Orleans District for all manner of things. That contract was issued with an effective date of May 21, 2009:
The Corps used their own, rented crane to pull the bridge pumps.
Speaking of those bridge pumps, there's also the $25,850 contract for scaffolding around those smaller 42" bridge pumps nos. 1 and 6 that were also pulled. That was contract W912P8-09-P-0219, also issued to RPL Oil Distributor and Supply with an effective date of June 1, 2009. The contract text reads:
Also on the topic of the bridge pumps, there's the $9000 task order 56 on the Corps contract with their diving company, Independent Divers, dated June 3, 2009:
Finally, there was the $15,551.19 contract for all the stainless steel hardware, as well as hose assemblies, needed to replace all the carbon steel fittings and hose on all three pumps. That contract, W921P8-09-P-0222 was issued to QPL, Inc of Thibodaux, LA with an effective date of June 4, 2009. That contract text is a long listing of model numbers for hydraulic piping fittings and other bits of hardware, so I won't reproduce it here. Suffice to say that I figured out it is the exact requirement for replacement of three pumps' worth of piping, which matches with the fact that three pumps were pulled.
That's what this is about - all the carbon steel piping on the water pumps which conveys the hydraulic fluid had to be replaced because of severe corrosion. That's where the oil in the water was coming from - holes in the pipes. There's more on this below when I talk about the second round of 2009 repairs.
Based on all those individual pieces-and-parts contracts, I believe the Corps did the repairs on these pumps themselves and did not send them out to a shop (as they would later in the summer, see below). Indeed, with all that other contract activity, it's not clear exactly what Healtheon brought to the table. A report about W10 and the other two pumps pulled under this work order (QRI was also tasked with the cleanup on those two bridge pumps' removal under task order #6 of their contract) was required under the Healtheon contract, but it is unknown if was really written. If it was, the Corps hasn't released it yet - despite a pending FOIA request for the information.
There's scant information on the repairs to these three pumps beyond what I've gotten, but there is one more data point. A Flickr user named "billread" visited the 17th Street site at the time, apparently on July 12, 2009. He snapped a couple of shots (on Flickr here and here):
The tipoff that something is up in these shots is the presence of the Corps' rental crane up on the main structure. Unless it is needed, it is parked off the structure on the Jefferson Parish side of the site.
Unfortunately one cannot see pump W10 in these shots, but we have a good sight line at many of the bridge pumps, including what I believe to be then-recently repaired bridge pump number 6.
Zooming in, take a closer look:
Here's the same area as viewed from the other photo:
Given that bridge pumps #1 and #6 were pulled, it appears the pump I've marked as "Clean" is pump #6 in the water immediately after its refit. The cleanliness is an indication of a fresh paint job, while the rust or marine growth on the adjacent units indicates they've been in the water a while.
Combined with 1) the position of the crane toward the west end of the structure, and 2) the lack of scaffolding which was called for in the RPL contract (though that's not conclusive because we don't have definitive evidence that scaffolding was built; but it most likely was), and I believe these pictures were taken very soon after replacement of one or both of the bridge pumps. That would make sense, because it shouldn't take more than about a month to refit those pumps, and they were apparently pulled around the first week of June. The inspection dive by independent Divers was on June 4, and the QRI oil spill cleanup for the pump pull was likely on June 8 (according to QRI task order number 6)
But more important is that W10 and bridge pumps #1 and #6 were not the last pumps with corrosion-related hydraulic oil leaks - not by far.
In early July, 2009, there was another flurry of activity at 17th Street. Another task order (number 3, linked here) was issued to Healtheon for hydraulic pump repairs, this time for four more 60" pumps at 17th Street. The value was $349,357.35, and was later increased to $405,596.95 (in modification 1, to include "additional expense that could not have been anticipated until commencement of the work.") On this new work order, a lot more detail was provided.
The four pumps were 5E, 7E, 8W, and 9W. The problems with the original order of 34 pumps - of which pump 5E is a part - are well documented and legion.
As mentioned above, the main Healtheon contract called for creation of reports documenting the work they and their subs did on each task order. For task order #3, a report was generated in (I'm guessing) early September, 2009 after the work was complete. However, since it summarizes all the work happening on pumps 5E and 7E from July to September, including the initial evaluations, it's fair to assume that key Corps personnel were informally made aware of Conhagen's findings all throughout July. Though, considering they were already in the middle of replacing the piping on three other pumps because of corrosion, it's doubtful those findings came as a surprise.
You WANT to read this report and look at the pictures in it. Trust me.
The report includes passages such as the following:
"Several hundred gallons:" verbiage similar to the Healtheon task order #3. However, in the task order, it was pump 7E that had the spill of approximately 700 gallons. Note that quantity is unlikely because the reservoirs on the engine skids hold less than half that volume of hydraulic oil (unless they were continually filling the oil reservoirs after test runs - yikes!). It probably represents an estimate of internal volume of the pump. In any case, while it is unclear if it was 7E, or 5E, or both that had major oil spills, it is crystal clear there was a bunch of oil in the water.
A spill of this size would certainly need to be reported to the state, and a spill of any size must reported to the National Response Center. Searches of both databases (including through public records requests) do not turn up reports for the spill, though QRI was dispatched - and presumably paid - under task order #7 of their spill cleanup contract. Curiously, the QRI task order claims the spill was small - between 10 and 50 gallons. The Healtheon task order for the repairs and the Conhagen repair report do not seem to describe a small spill. Given what Conhagen found, I'm strongly inclined to believe their narrative over the QRI task order.
And just to clarify (or confuse): it's not clear whether the pictures in the report below are of pump 5E or pump 7E, but the point is really moot. All the pumps sit in the same water, see the same weather conditions, and are exposed to the same salinity. Any problem that strikes one of them in such a catastrophic fashion would strike all of them similarly. And when I say, "all," I mean ALL the pumps across all three sites.
Other excerpts from Conhagen report:
The Rineer motors were blamed for the severe vibration problems that struck these pumps upon their startup during the summer and fall of 2006. All of the motors were supposedly fixed by installing sturdier springs. It is extremely unclear if the Conhagen replacement of the Rineer motor on pump 5E included the sturdier springs, or if they went with the standard Rineer "Code 61" motors with flimsier springs that caused the original problems. Note that a beefier, high-pressure "Code 62" Rineer would work as well, and original installation of Code 62's in 2006 would have likely avoided the vibration problems altogether. That's because the standard Code 61 model has a maximum pressure of 3000 psi, which is also the operating pressure of the system. The maximum pressure of the Code 62 is 4500-5000 psi, well above the operating pressure and a much wiser choice (that's why they had the Code 62 spring package installed eventually). The Corps has not responded to a FOIA request for the purchasing details on the replacement Rineer motor. The motor (and other later replacements this year) were likely pulled from the Corps' stock of spare motors, but the lack of response to the FOIA request leaves that as speculation.
[Update, 7/23/12: The Corps eventually did respond to this FOIA request in July, 2011, and confirmed that the Rineer motor placed in pump 5E was from their stock of spares, and that all their spares had previously been upgraded with the Code 62 spring package.]
Then there's the final two paragraphs of the report:
One does not have to be an engineer to grasp the meaning of what Conhagen wrote there. In fact, they are even understating the matter, since they note that hundreds of gallons of oil had already leaked, so the corrosion failures were no longer imminent - they were existent. Hydraulic oil that was supposed to be going to the motor to turn the pump impeller was going straight in the water.
Pump 5E was part of the original order of 34 pumps. Pump 7E went in shortly after the original order as part of the Phase 2 installation. After periodic dunkings in 2006, both had been in the water continuously since about March or April of 2007. They, along with the rest of the original 34 and the later six at 17th St, spent most of the fall and winter of 2006-07 in the shop getting new Rineer springs and having piping extensions installed.
So under the most generous interpretation, the corrosion clock started ticking in about April of 2007, with the pumps having the same paint as originally specified. By June of 2009, one or both of the pumps' internals had completely corroded, causing catastrophic failure and an inability to drain water from the city during a storm event when the gates were closed.
Reading about it is one thing. Seeing the pictures is another. For comparison purposes, let's look at the pumps before they were installed. These pictures were taken in the spring of 2006:
and then just to the left of those:
Let's zoom into the pump on the left:
And then zoom in further for a closer look at the Rineer motor:
"HP" means high pressure, "LP" means low pressure. All the piping is carbon steel. The case drain line transports small amounts of oil away from the motor. Keep in mind these photos were in 2006. All the hydraulic pumps went into the shop in late 2006 and early 2007 for refits on the Rineer motors and addition of piping extensions to the points where the oil lines go through the pump housing. The paint system used for those refits was identical to the original system applied during manufacture in early 2006.
Here's pictures of either 5E or 7E at 17th Street when it was pulled in July 2009:
To make comparisons with the pictures above easier, I've labeled the pipes:
Here's a section of pipe removed from the pump sitting on the table at Conhagen (the fitting at the bottom indicates it may be from the inlet or outlet of one of the hydraulic coolers)
It appears that a large chunk of the pipe is completely gone in this photo. Also note the rusty hose fitting on the table:
Damage like that is the result of two things: 1) poor initial design related to corrosion protection and painting and 2) an utter lack of preventative maintenance. It is inconceivable that a properly specified and maintained piece of equipment - especially one required for life and safety like these pumps - could have a hand-sized hole corrode away before someone responsible did something. Inconceivable for anyone but the Corps New Orleans District.
The report mentions rust inside the Rineer motor, requiring its replacement. Besides the rust on the pipes causing holes, it's also likely that a failed, rusty seal on the bottom of the motor might have also contributed:
There was marine growth all over the things. Here's one of the oil coolers mounted to the sides of the pumps.
The still-hot hydraulic oil passes through these after leaving the Rineer motor but before returning to the reservoir on the engine skid. The residual heat is supposed to be transferred to the canal water. How much heat could really be transferred with all that fouling?
Here's some of the piping installed over the winter of 2006-07, specifically the return piping (two larger lines) and the smaller case drain line:
Is it any wonder the lines started corroding?
The same lines, with a bit more of the pump housing:
It's a wonder there's not giant holes in the sides of the pump, rather than just the pipes!
One doesn't need to be a genius to look at these pictures - and others I haven't highlighted - in the Conhagen report and wonder, "If this is what just one or two of the pumps looked like after just two years, what about the rest of those pumps that went in the water at exactly the same time and saw exactly the same conditions? Shouldn't they have gotten yanked out too, and right quick?" After all, much of the damage is simply not visible by removing an inspection hatch and looking down into the pump for oil leaks. For example, the Rineer motor corrosion on pump 5E was internal to the motor. The only way to see it was to pull the entire water pump, cart the pump to a shop on a flatbed truck, and cut the piping leading to the Rineer because the bolts holding the piping to the motor had rusted to nothing. Then the motor itself had to be cracked open and inspected.
Well, the Corps did none of that. They did the repair work on water pumps 5E, 7E and the other two pumps - 8W and 9W - laid out in Healtheon task order #3, and then stopped. The hopelessly corroded carbon steel pipe sections were removed and replaced with stainless steel (what should have gone in originally). 5E and 7E went back in the water in early September and underwent some cursory on-site performance testing. The other two pumps went through the same repair process during September (oddly there's no QRI task order for spill cleanup during the removal of those two pumps), finally getting back in the canal in early October.
According to publicly available contracting data for 2009, the work on those four 17th Street pumps was the last repair work done on any of the hydraulic pump systems for the 2009 hurricane season. Despite having a report from their own contractor warning them that all the other water pumps were imminently close to failure, the Corps New Orleans District (yes, the local Corps office) apparently did nothing additional for the entire 2009 hurricane season to address the situation. And of course, they never informed the public about any of this, despite a series of public meetings and presentations about the outfall canals.
Take a look at those pictures above, which show the damage after just over two years immersed in the canals. Imagine what another year in the water has done to the pumps. How does that make you feel? How does that inaction reflect on the local leadership and decisionmakers of the New Orleans District?
So what do we have here? We've got the Corps knowing that hydraulic water pumps are rusting to bits and doing practically nothing about it for the whole 2009 storm season. We've got oil pouring into the canals through those rusty bits, and no evidence of anyone reporting it to the federal government. We've got the Corps refusing to turn the rusty pumps on during a big storm in December, despite water depths in not one but two of the canals exceeding the Corps' own criteria for activation (more on that one later).
And finally, we've got the Corps embarking on what can only be called a crash repair program to belatedly fix the problem, but not telling anyone about it despite numerous contemporaneous public meetings and interviews.
As I said at the beginning of this post, for the last couple of months the Corps has been yanking nearly every hydraulic pump out of the 17th Street and London Avenue sites and sending them out for corrosion-related overhauls. The first task order of the year - number 4 on the Healtheon contract - was issued February 1, 2010 and is valued at $238,846.95. It is for the repair of 17th Street Phase 1 pumps W5 and W6. The scope of repairs for those two pumps is more extensive than for anything done during 2009. Not only was all the interior and exterior carbon steel piping to be replaced with stainless steel, but also both external oil coolers on each pump were to be completely swapped out, switching out carbon steel for stainless steel. When one considers that other replacements include seals, bearings, Rineer motors, gland packing plates, hoses and fittings, as well as an estimated 24 man-hours of weld repairs on each pump, it's easy to conclude that there is almost nothing usable left on these pumps after a little over 2 years in the water.
The Corps is in such a hurry that they are pulling out more than two at a time at 17th Street. I've had a FOIA request in for that information for nearly two months, but it has not been fulfilled in full yet. I got task order number 4, but they still have not transmitted task orders covering the numerous other pumps that have gotten yanked this spring. Despite that, I'll have more on that work in an upcoming post.
[Update, 6/12/10: I wrote about the rest of this spring's work in "This year's scramble" on June 3. The Corps transmitted contract information for the other pumps pulled out this spring on June 11. I posted about it in my June 11 post, "Worse than previously known". Despite the work this spring, the vast majority of the large hydraulic pumps remain corroding and unrepaired today.]
But that's enough for now. I'm going to have much more on this story over the coming days, because it has a lot of tentacles. Next up is how the Corps got into this mess. Stay tuned...
[Postscript: The Corps closed out task order #3 on the first Healtheon contract in September, 2010 with modification 2. They reconciled the Healtheon/Conhagen invoices and subtracted $20,005.63, meaning the total cost of rebuild for the four 17th Street 60" pumps was $385,591.32]
Only in recent weeks has the Corps' New Orleans District embarked on a crash repair initiative hidden from the public in plain sight, in an effort to make up for years of deferred - but not unfunded - preventive maintenance on pumps which they knew from the very beginning would fall apart within a matter of a couple of years, if not quicker. That lack of maintenance likely left New Orleans' front line flood defenses in a precarious state of decay during the entire length of last year's hurricane season, and may have actually played a part in a year-end flooding event. [UPDATE: It is more likely that the December 8, 2009 systemwide failure of the Corps SCADA control system for the pumps and gates played a larger part in the decision not to lower the gates and run the pumps during widespread flooding and elevated lake levels on December 12-13, 2009. Details can be found in my subsequent post, "SCADA."] It also may have been a pawn in the Corps recent negotiations with the state to impose their inadequate Option 1 scheme. I'll get to the bits about the current work, the year-end flooding event, and Option 1 in future posts. For now, I'm just going to concentrate on the 2009 repairs.
In spring, 2009, the Corps issued their first work order on contract W912P8-08-D-0090, a potentially five-year-long pump maintenance they inked with a company named Healtheon in 2008.
That $28,138 work order, whose value was later increased by $1487.43 made reference to pulling three pumps at 17th Street because of "hydraulic oil leaking problems:"
"Repair hydraulic oil leaking problems of one 60" hydraulic pump #10W and two 40"[sic - should be 42"] bridge hydraulic pumps #1 and #6 at 17th Street Canal. One MWI technician is needed for two days on May 28 and May 29 and three days during the week of June 1-5. The exact date of these three days will be determined later due to the availability of the crane. Healtheon will coordinate closely with Corps employees, Ms Lynn Tinto and Jack Rouquette, during pump pullup, repair, and reinstallation of the pumps."
As a quick reminder, most of the pumps at all three interim closure sites on the lakefront are hydraulically powered. Hydraulic oil is raised to approximately 3000 psi by hydraulic pumps manufactured by Denison Hydraulics at engine skids placed in sheds remote from the pumps (though at 17th Street, some of the pumps have their engine skids adjacent to them). That pressurized oil flows along hundreds of feet of pipe (or a few dozen feet of rubber hose, in the case of the 17th Street skid-adjacent pumps) to the water pumps in the canals. Inside the water pumps, the energy of the pressurized hydraulic oil is translated to rotational energy by a hydraulic motor manufactured by a Texas outfit known as Rineer.
The Rineer motor (the original was four port model 125, though it has since been modified - see below) is attached to a shaft which spins an impeller, pushing stormwater out of the canal, through the pump, and down 9-foot-diameter pipes which travel past the gates to the lake. The depressurized hydraulic oil travels through coolers mounted on the water pump, and then back through more piping to a reservoir at the drive unit. From there, the cycle repeats.
There are 18 60" hydraulic pumps at 17th Street, and 14 42" models. The size (60" or 42") refers to diameter of the pump housing. The original order to MWI for the hydraulic pumps in January, 2006 was for 34 60" pumps, with 12 each going in at 17th Street and London Avenue, and the remaining 10 going in at Orleans Avenue. Those were the "Phase 1" pumps. Another six 60" pumps were installed at 17th Street later in 2006 ("Phase 2"), and the 14 42" pumps went in during the spring of 2007 ("Phase 3").
There are also 11 direct drive units at 17th Street and 8 direct drive units at London Avenue, installed during the spring and summer of 2007. Those pumps have their drives directly coupled to their shafts, eliminating all the problems inherent in a hydraulically powered system. Here's the layout of the pumps at 17th Street:
Here's the numbering of each pump at 17th Street:
Here's the London Ave layout, with the original 12 hydraulic pumps noted as "Phase 1:"
And the numbering for the pumps at London Avenue:
The numbering at Orleans Avenue is similar to at London Avenue, with hydraulic pumps E1 through E5 on the east side and W1 through W5 on the west side.
The piping which conveys the hydraulic oil is coupled to the water pumps via hoses. To repair the water pumps, the hoses must be decoupled. There is no way to blow the hydraulic oil back to the drive units, so the oil spills in the canal when the hoses are detached. As it turns out, these "planned" oil spills are excellent indicators of work (or trouble) happening on the hydraulic pumps.
That's because the Corps also has a contract for oil spill cleanups. It is with Quaternary Resource Investigations (QRI) of Baton Rouge. Every time a pump is pulled, oil is spilled, so QRI should be on site to clean it up. QRI also provides and maintains the permanent oil protection booms that are at each site.
Thus, if a pump (or pumps) is pulled, QRI gets a task order on their contract. According to federal law and the Corps own oil spill response plans (linked: 17th Street SPCC; Orleans Avenue SPCC; London Avenue SPCC), the oil spill should also be reported to the National Response Center (NRC), and it should show up on the NRC report query website. Depending on the size of the spill (the state only requires spill reports for spills over 42 gallons, or 1 barrel), the state should also have a record of the spill on the LDEQ Electronic Document Management Service website. Reporting to the state is also a requirement under the law and the Corps' written plans.
So back to the May, 2009 pump repairs...
The work was - according to the task order - to take place in late May and early June. Every year in late May, the Corps holds a hurricane preparedness exercise, part of which they put on in front of the cameras of the local news crews. It is mostly a dog-and-pony show. In 2009, the Corps had the media out to the London Avenue canal gates on May 27, 2009 (Fox 8 story here), which was a departure from their usual M.O. Whenever the New Orleans District wants to emphasize how wonderful the gates and pumps are, they have the press out to the much larger 17th Street site. No one in the local press seemed to question why they were at London instead.
However, the Corps themselves obliquely acknowledged the 17th Street work in their post-exercise press release on May 27, 2009, which was the same day the repair work was due to commence:
"Because of previously scheduled maintenance on some of the pumps at the 17th St. Canal (due to be complete next week), the pumps could not be turned on immediately following the gate closure for the safety of the contractors working in the area. However, each of the pumps, excluding those affected by the ongoing maintenance, were run through the start up cycle at some point during the exercise and effectively pumped. Those affected by the ongoing maintenance will be run when the work is complete."
The "previously scheduled" thing is a canard. The work appears to have begun in a hurry, since the Healtheon task order #1 has an effective date of June 17, 2009, or three weeks after the work began (likely with some kind of unofficial Notice to Proceed like an email or a phone call). Other contracts issued for the work all had dates after the work began, except for the crane rental contract, which went out less than a week before (see below). Also note how vague the press release is, giving the impression of "nothing to see here, move along." That was made easier by keeping the prying eyes of the local TV crews miles away from the 17th Street site.
A more detailed account of what was going on May 27, 2009 at 17th Street did get created: the QRI-written spill report for the pulling of 60" hydraulic pump W10. It was generated as part of QRI's work under task order #5. It gives an extensive narrative of that day's events:
"At 0630 hours on May 27, 2009, QRI's Field Supervisor and Field Technician arrived at the Orleans Canal storage (connex) box to retrieve staged spill response materials. At 0645 hours QRI's Field Manager and Site Safety Officer made contact with Corps employee Ms. Lynn Tinto. Following an 0715 Safety Meeting, QRI personnel launched from the 17th Street Canal boat launch and proceeded to the vicinity of the W 10 pump.
Pump W10 was isolated by installing both hard and soft boom[s] - to form a double boom around the perimeter of pump W10. Following placement of the boom, the QRI crew and boat staged on the lake side of the canal to allow USACE personnel to access pump W10.
At 1240 hours the pump was lifted out and approximately 40 gallons of hydraulic fluid was spilled. QRI's response crew placed napkins within the double boom area to absorb the spilled fluid. An additional 150 foot section of soft boom was placed outside the isolated area to absorb any oil escaping from the double-boomed section. QRI personnel used 12 1/2 additional bags of napkins during the cleanup process.
QRI's safety officer and USACE personnel discussed regulations outlined in the 17th Street Canal SPCC Plan and determined that notification to LDEQ was necessary. Incident number T115306 was assigned by LDEQ."
Note that notification of the National Response Center is oddly omitted. That's strange, because the state reportable quantity (RQ) for oil is 1 barrel (42 gallons), while the federal government RQ is any amount, no matter how tiny. Indeed, there is no report of this spill on the NRC website.
Anyway, the report describes a lot of activity. QRI took date-stamped pictures too:
It would have aroused questions. Keeping the cameras away sidestepped that possibility.
So they lifted out W10 and then what? It's not easy to tell. There's a number of other contract actions related to the work which went out before the time of the Healtheon task order, with its effective date of June 17, 2009 but its text requiring work before that date.
There's the $16,370 (note: FPDS-NG reports the value as $31,492) contract for rental of the crane seen in the photos above that pulled pump W10, W912P8-09-P-0210, issued to RPL Oil Distributor and Supply, LLC of Gretna, LA, a frequent go-to company for the New Orleans District for all manner of things. That contract was issued with an effective date of May 21, 2009:
"Furnish one (1) 240-ton capacity hydraulic truck crane, with operator and maintenance to work as directed by the Government to remove Pump No. 10 for repairs and replace same. Based on 5 days at 10 hours. Also includes Rental of Crane Mats for 5 days."
The Corps used their own, rented crane to pull the bridge pumps.
Speaking of those bridge pumps, there's also the $25,850 contract for scaffolding around those smaller 42" bridge pumps nos. 1 and 6 that were also pulled. That was contract W912P8-09-P-0219, also issued to RPL Oil Distributor and Supply with an effective date of June 1, 2009. The contract text reads:
"Rent and install scaffolding for bridge pumps #1 and # 6 at 17th Street Canal interim pumping station. Work will involve providing and installing 1 level of hanging scaffolding under the bridge at the interim structure of approximately 10 feet by 10 feet by 13 feet. The scaffolding and installation will meet OSHA codes. The government will provide crane service to load and unload material. Work will start on 1 Jun and will be completed by 31 Jul or sooner."
Also on the topic of the bridge pumps, there's the $9000 task order 56 on the Corps contract with their diving company, Independent Divers, dated June 3, 2009:
"Provide Dive Team to inspect bridge pumps 1 and 6, and its foundations, and remove any existing obstructions. All work shall be performed in accordance with the contract terms and conditions, this task order, and the attached Scope of Work. Diving shall be performed on 4 June 2009."The scope of work reiterates the same basic text.
Finally, there was the $15,551.19 contract for all the stainless steel hardware, as well as hose assemblies, needed to replace all the carbon steel fittings and hose on all three pumps. That contract, W921P8-09-P-0222 was issued to QPL, Inc of Thibodaux, LA with an effective date of June 4, 2009. That contract text is a long listing of model numbers for hydraulic piping fittings and other bits of hardware, so I won't reproduce it here. Suffice to say that I figured out it is the exact requirement for replacement of three pumps' worth of piping, which matches with the fact that three pumps were pulled.
That's what this is about - all the carbon steel piping on the water pumps which conveys the hydraulic fluid had to be replaced because of severe corrosion. That's where the oil in the water was coming from - holes in the pipes. There's more on this below when I talk about the second round of 2009 repairs.
Based on all those individual pieces-and-parts contracts, I believe the Corps did the repairs on these pumps themselves and did not send them out to a shop (as they would later in the summer, see below). Indeed, with all that other contract activity, it's not clear exactly what Healtheon brought to the table. A report about W10 and the other two pumps pulled under this work order (QRI was also tasked with the cleanup on those two bridge pumps' removal under task order #6 of their contract) was required under the Healtheon contract, but it is unknown if was really written. If it was, the Corps hasn't released it yet - despite a pending FOIA request for the information.
There's scant information on the repairs to these three pumps beyond what I've gotten, but there is one more data point. A Flickr user named "billread" visited the 17th Street site at the time, apparently on July 12, 2009. He snapped a couple of shots (on Flickr here and here):
The tipoff that something is up in these shots is the presence of the Corps' rental crane up on the main structure. Unless it is needed, it is parked off the structure on the Jefferson Parish side of the site.
Unfortunately one cannot see pump W10 in these shots, but we have a good sight line at many of the bridge pumps, including what I believe to be then-recently repaired bridge pump number 6.
Zooming in, take a closer look:
Here's the same area as viewed from the other photo:
Given that bridge pumps #1 and #6 were pulled, it appears the pump I've marked as "Clean" is pump #6 in the water immediately after its refit. The cleanliness is an indication of a fresh paint job, while the rust or marine growth on the adjacent units indicates they've been in the water a while.
Combined with 1) the position of the crane toward the west end of the structure, and 2) the lack of scaffolding which was called for in the RPL contract (though that's not conclusive because we don't have definitive evidence that scaffolding was built; but it most likely was), and I believe these pictures were taken very soon after replacement of one or both of the bridge pumps. That would make sense, because it shouldn't take more than about a month to refit those pumps, and they were apparently pulled around the first week of June. The inspection dive by independent Divers was on June 4, and the QRI oil spill cleanup for the pump pull was likely on June 8 (according to QRI task order number 6)
But more important is that W10 and bridge pumps #1 and #6 were not the last pumps with corrosion-related hydraulic oil leaks - not by far.
In early July, 2009, there was another flurry of activity at 17th Street. Another task order (number 3, linked here) was issued to Healtheon for hydraulic pump repairs, this time for four more 60" pumps at 17th Street. The value was $349,357.35, and was later increased to $405,596.95 (in modification 1, to include "additional expense that could not have been anticipated until commencement of the work.") On this new work order, a lot more detail was provided.
The four pumps were 5E, 7E, 8W, and 9W. The problems with the original order of 34 pumps - of which pump 5E is a part - are well documented and legion.
As mentioned above, the main Healtheon contract called for creation of reports documenting the work they and their subs did on each task order. For task order #3, a report was generated in (I'm guessing) early September, 2009 after the work was complete. However, since it summarizes all the work happening on pumps 5E and 7E from July to September, including the initial evaluations, it's fair to assume that key Corps personnel were informally made aware of Conhagen's findings all throughout July. Though, considering they were already in the middle of replacing the piping on three other pumps because of corrosion, it's doubtful those findings came as a surprise.
You WANT to read this report and look at the pictures in it. Trust me.
The report includes passages such as the following:
"The first thing we did was to remove the inspection covers from the pumps and check the inside of the pumps for oil. Of the 4 units overhauled, only pump 5E had a significant amount of oil inside the pump (several hundred gallons). The oil was pump[ed] out and disposed of."
"Several hundred gallons:" verbiage similar to the Healtheon task order #3. However, in the task order, it was pump 7E that had the spill of approximately 700 gallons. Note that quantity is unlikely because the reservoirs on the engine skids hold less than half that volume of hydraulic oil (unless they were continually filling the oil reservoirs after test runs - yikes!). It probably represents an estimate of internal volume of the pump. In any case, while it is unclear if it was 7E, or 5E, or both that had major oil spills, it is crystal clear there was a bunch of oil in the water.
A spill of this size would certainly need to be reported to the state, and a spill of any size must reported to the National Response Center. Searches of both databases (including through public records requests) do not turn up reports for the spill, though QRI was dispatched - and presumably paid - under task order #7 of their spill cleanup contract. Curiously, the QRI task order claims the spill was small - between 10 and 50 gallons. The Healtheon task order for the repairs and the Conhagen repair report do not seem to describe a small spill. Given what Conhagen found, I'm strongly inclined to believe their narrative over the QRI task order.
And just to clarify (or confuse): it's not clear whether the pictures in the report below are of pump 5E or pump 7E, but the point is really moot. All the pumps sit in the same water, see the same weather conditions, and are exposed to the same salinity. Any problem that strikes one of them in such a catastrophic fashion would strike all of them similarly. And when I say, "all," I mean ALL the pumps across all three sites.
Other excerpts from Conhagen report:
"The motors and motor stands were removed from the diffuser housing. These were initially supposed to remain mounted, but the corrosion on the piping connection on the motor stand required that they be removed to make the repairs. One of the motors (5E) also had to be replaced due to being contaminated with water and having internal corrosion."
The Rineer motors were blamed for the severe vibration problems that struck these pumps upon their startup during the summer and fall of 2006. All of the motors were supposedly fixed by installing sturdier springs. It is extremely unclear if the Conhagen replacement of the Rineer motor on pump 5E included the sturdier springs, or if they went with the standard Rineer "Code 61" motors with flimsier springs that caused the original problems. Note that a beefier, high-pressure "Code 62" Rineer would work as well, and original installation of Code 62's in 2006 would have likely avoided the vibration problems altogether. That's because the standard Code 61 model has a maximum pressure of 3000 psi, which is also the operating pressure of the system. The maximum pressure of the Code 62 is 4500-5000 psi, well above the operating pressure and a much wiser choice (that's why they had the Code 62 spring package installed eventually). The Corps has not responded to a FOIA request for the purchasing details on the replacement Rineer motor. The motor (and other later replacements this year) were likely pulled from the Corps' stock of spare motors, but the lack of response to the FOIA request leaves that as speculation.
[Update, 7/23/12: The Corps eventually did respond to this FOIA request in July, 2011, and confirmed that the Rineer motor placed in pump 5E was from their stock of spares, and that all their spares had previously been upgraded with the Code 62 spring package.]
Then there's the final two paragraphs of the report:
"All of the piping was removed from the pump and motor. All of the piping showed corrosion. The piping and flanges internal of the pump had the most severe corrosion. Wall thickness in some of the areas was down to zero. The flanges were severely corroded with the heads of some of the flange bolts completely corroded away.
The severe corrosion seen on the piping internal of the pump should be cause for concern in the remaining pumps at this site. Leaks that were imminent on these pumps are sure to be imminent on the remaining."
One does not have to be an engineer to grasp the meaning of what Conhagen wrote there. In fact, they are even understating the matter, since they note that hundreds of gallons of oil had already leaked, so the corrosion failures were no longer imminent - they were existent. Hydraulic oil that was supposed to be going to the motor to turn the pump impeller was going straight in the water.
Pump 5E was part of the original order of 34 pumps. Pump 7E went in shortly after the original order as part of the Phase 2 installation. After periodic dunkings in 2006, both had been in the water continuously since about March or April of 2007. They, along with the rest of the original 34 and the later six at 17th St, spent most of the fall and winter of 2006-07 in the shop getting new Rineer springs and having piping extensions installed.
So under the most generous interpretation, the corrosion clock started ticking in about April of 2007, with the pumps having the same paint as originally specified. By June of 2009, one or both of the pumps' internals had completely corroded, causing catastrophic failure and an inability to drain water from the city during a storm event when the gates were closed.
Reading about it is one thing. Seeing the pictures is another. For comparison purposes, let's look at the pumps before they were installed. These pictures were taken in the spring of 2006:
and then just to the left of those:
Let's zoom into the pump on the left:
And then zoom in further for a closer look at the Rineer motor:
"HP" means high pressure, "LP" means low pressure. All the piping is carbon steel. The case drain line transports small amounts of oil away from the motor. Keep in mind these photos were in 2006. All the hydraulic pumps went into the shop in late 2006 and early 2007 for refits on the Rineer motors and addition of piping extensions to the points where the oil lines go through the pump housing. The paint system used for those refits was identical to the original system applied during manufacture in early 2006.
Here's pictures of either 5E or 7E at 17th Street when it was pulled in July 2009:
To make comparisons with the pictures above easier, I've labeled the pipes:
Here's a section of pipe removed from the pump sitting on the table at Conhagen (the fitting at the bottom indicates it may be from the inlet or outlet of one of the hydraulic coolers)
It appears that a large chunk of the pipe is completely gone in this photo. Also note the rusty hose fitting on the table:
Damage like that is the result of two things: 1) poor initial design related to corrosion protection and painting and 2) an utter lack of preventative maintenance. It is inconceivable that a properly specified and maintained piece of equipment - especially one required for life and safety like these pumps - could have a hand-sized hole corrode away before someone responsible did something. Inconceivable for anyone but the Corps New Orleans District.
The report mentions rust inside the Rineer motor, requiring its replacement. Besides the rust on the pipes causing holes, it's also likely that a failed, rusty seal on the bottom of the motor might have also contributed:
There was marine growth all over the things. Here's one of the oil coolers mounted to the sides of the pumps.
The still-hot hydraulic oil passes through these after leaving the Rineer motor but before returning to the reservoir on the engine skid. The residual heat is supposed to be transferred to the canal water. How much heat could really be transferred with all that fouling?
Here's some of the piping installed over the winter of 2006-07, specifically the return piping (two larger lines) and the smaller case drain line:
Is it any wonder the lines started corroding?
The same lines, with a bit more of the pump housing:
It's a wonder there's not giant holes in the sides of the pump, rather than just the pipes!
One doesn't need to be a genius to look at these pictures - and others I haven't highlighted - in the Conhagen report and wonder, "If this is what just one or two of the pumps looked like after just two years, what about the rest of those pumps that went in the water at exactly the same time and saw exactly the same conditions? Shouldn't they have gotten yanked out too, and right quick?" After all, much of the damage is simply not visible by removing an inspection hatch and looking down into the pump for oil leaks. For example, the Rineer motor corrosion on pump 5E was internal to the motor. The only way to see it was to pull the entire water pump, cart the pump to a shop on a flatbed truck, and cut the piping leading to the Rineer because the bolts holding the piping to the motor had rusted to nothing. Then the motor itself had to be cracked open and inspected.
Well, the Corps did none of that. They did the repair work on water pumps 5E, 7E and the other two pumps - 8W and 9W - laid out in Healtheon task order #3, and then stopped. The hopelessly corroded carbon steel pipe sections were removed and replaced with stainless steel (what should have gone in originally). 5E and 7E went back in the water in early September and underwent some cursory on-site performance testing. The other two pumps went through the same repair process during September (oddly there's no QRI task order for spill cleanup during the removal of those two pumps), finally getting back in the canal in early October.
According to publicly available contracting data for 2009, the work on those four 17th Street pumps was the last repair work done on any of the hydraulic pump systems for the 2009 hurricane season. Despite having a report from their own contractor warning them that all the other water pumps were imminently close to failure, the Corps New Orleans District (yes, the local Corps office) apparently did nothing additional for the entire 2009 hurricane season to address the situation. And of course, they never informed the public about any of this, despite a series of public meetings and presentations about the outfall canals.
Take a look at those pictures above, which show the damage after just over two years immersed in the canals. Imagine what another year in the water has done to the pumps. How does that make you feel? How does that inaction reflect on the local leadership and decisionmakers of the New Orleans District?
So what do we have here? We've got the Corps knowing that hydraulic water pumps are rusting to bits and doing practically nothing about it for the whole 2009 storm season. We've got oil pouring into the canals through those rusty bits, and no evidence of anyone reporting it to the federal government. We've got the Corps refusing to turn the rusty pumps on during a big storm in December, despite water depths in not one but two of the canals exceeding the Corps' own criteria for activation (more on that one later).
And finally, we've got the Corps embarking on what can only be called a crash repair program to belatedly fix the problem, but not telling anyone about it despite numerous contemporaneous public meetings and interviews.
As I said at the beginning of this post, for the last couple of months the Corps has been yanking nearly every hydraulic pump out of the 17th Street and London Avenue sites and sending them out for corrosion-related overhauls. The first task order of the year - number 4 on the Healtheon contract - was issued February 1, 2010 and is valued at $238,846.95. It is for the repair of 17th Street Phase 1 pumps W5 and W6. The scope of repairs for those two pumps is more extensive than for anything done during 2009. Not only was all the interior and exterior carbon steel piping to be replaced with stainless steel, but also both external oil coolers on each pump were to be completely swapped out, switching out carbon steel for stainless steel. When one considers that other replacements include seals, bearings, Rineer motors, gland packing plates, hoses and fittings, as well as an estimated 24 man-hours of weld repairs on each pump, it's easy to conclude that there is almost nothing usable left on these pumps after a little over 2 years in the water.
[Update, 6/12/10: I wrote about the rest of this spring's work in "This year's scramble" on June 3. The Corps transmitted contract information for the other pumps pulled out this spring on June 11. I posted about it in my June 11 post, "Worse than previously known". Despite the work this spring, the vast majority of the large hydraulic pumps remain corroding and unrepaired today.]
But that's enough for now. I'm going to have much more on this story over the coming days, because it has a lot of tentacles. Next up is how the Corps got into this mess. Stay tuned...
[Postscript: The Corps closed out task order #3 on the first Healtheon contract in September, 2010 with modification 2. They reconciled the Healtheon/Conhagen invoices and subtracted $20,005.63, meaning the total cost of rebuild for the four 17th Street 60" pumps was $385,591.32]
Labels: Rusty pumps
posted by mcbrid35 at
5:24 PM
1 Comments:
I am the photographer, billread, who took some of the photos you mention. I can verify the date of the photos was indeed as recorded, July 12th, 2009.
By
BillRead, at May 18, 2010 8:39 AM
Post a Comment
<< Home