Rusty pumps - the summing up
[Update, September 3, 2011: A pump at the Orleans Avenue structure - E3 - sprung a corrosion-related oil leak on August 10, 2011 and was pulled out shortly afterward. Details can be found at this subsequent entry.]
There's lots of matters the Corps of Engineers' New Orleans District would rather the public didn't know about, or at least pay attention to. Many times, it's the details of things. They say on June 1, 2011 they've met their goal of hundred year protection, but you find out there's dozens of unfinished projects. They point to how great the West Closure Complex is, but then you learn there were only 8 of 11 pumps ready at the beginning of hurricane season and an early season storm could force local pump stations to shut down. They say the levee system is better than ever, and then you find out it's filled with concrete, wood, steel, and other flotsam.
So it goes with the 54 hydraulic pumps installed at the three outfall canal sites in New Orleans - 32 at 17th Street, 10 at Orleans Avenue and 12 at London Avenue. 40 of them are the 60" type, while the other 14 - all installed at 17th Street - are the smaller 42" version (this post focuses on the 60" pumps, due to a severe lack of contracting information for the repairs to the 42" pumps). The Corps would rather the public didn't bother with such details as whether the pumps work under storm conditions, or if they have crippling design flaws, or if the salt water in Lake Pontchartrain is turning them into rusty heaps. That's because they know if the public were privy to all those pesky details, they might freak out over how scary the story is.
Thank goodness for the Freedom of Information Act, though, because it allows us to push past the well financed Public Affairs curtain and see the entire rusty pump story for what it is: a fiasco five years in the making.
The first signs - summer, 2006
It started nearly instantly after the pumps first went in the canals in June, 2006, with corrosion appearing on the hose connections from the hard piping to the pumps. Hydraulic oil is driven from hydraulic power units onshore through the piping and hoses to the pumps which sit in the canals. Here's a picture of those rusty hose connections: (via SCPR Flickr):
And here's a closeup of one of those rusty hose connections (via SCPR Flickr):
This was a hint that the environment around the waters of Lake Pontchartrain was very salty, and materials susceptible to corrosion - such as the carbon steel making up the pumps - would perform quite poorly. This shouldn't have surprised anyone, since Lake Pontchartrain is the second largest salt water body in the country.
Missing the big picture - 2006 through 2008
But over the next few months, the Corps and its paid consultants (many of which were former Corps themselves) would not look at the systemwide problem augered by that early rust, but would instead focus narrowly on a tiny sliver of the corrosion problem. They concentrated on the hose connections to the pumps which sat beneath or near the waterline as the pumps rested in the canals (these hose connections were at the opposite end of the hoses from those shown in the June, 2006 photos above):
Remarkably, their "solution" to preventing the rusting of those connections in early 2007 was the addition of more carbon steel to the pumps, in the form of piping extensions:
This raised the lower hose connections above the waterline, but also had the effect of giving much more surface area for rust and marine organisms to attack. It was like discovering your car's engine was on fire, and deciding to check the spark plugs - while wearing a suit covered in gasoline!
With all that yummy carbon steel waiting like a buffet, the marine organisms in the Lake Pontchartrain water started chomping away at the coar tar epoxy paint, exposing the bare metal below in no time. Corrosion of the steel quickly followed. An oil spill from pump E5 at the 17th Street site on June 26, 2008 was a hint of things to come. The Corps paid over $10,000 to clean it up, but never bothered to report it to the federal or state authorities.
A month later a solicitation for comprehensive pump maintenance for pumps across all three sites was issued. It was a small business Section 8(A) set aside, which is a rather odd choice for such a critical function as maintenance of pumps upon which the residents of the City of New Orleans relied for protection of life and property. Nonetheless, the contract was awarded August 28, 2008 to small business Healtheon.
Early self-repair efforts prove inadequate - Spring, 2009
By early 2009, the hydraulic oil which powered the pumps and which was contained within the pipes was seeping regularly from multiple pumps, and sometimes gushing into the canals. A halting repair effort began in March, at first concentrating only on pumps with known leaks.
The Corps attempted to repair them on their own, sometimes going to interesting lengths to conceal their efforts from the public. But after six pumps had to be pulled out in three months, accompanied by five oil spills (four of which went unreported) costing over $30,000 to clean up, it was clear professional help was needed.
Professionals are brought in - summer, 2009
Task orders started going out on the Healtheon contract in July, 2009. Healtheon hired Conhagen, Inc of Kenner, LA to perform the repairs. Conhagen had been performing similar services for the New Orleans Sewerage & Water Board for many years, and indeed it is unclear why the Corps simply didn't hire them directly.
The first task order, on July 21, 2009, called for the partial repair of four 60" pumps at the 17th Street site. New and more plentiful zinc anodes would be attached to the pumps to fight future corrosion, and bits of the pump housings eaten away by rust would be weld repaired.
Also, all the carbon steel piping on the pumps would be removed and replaced. However, only some of it - the piping inside the pump bore - would be upgraded to more corrosion resistant 316L stainless steel. The piping and oil coolers on the outside of the pumps, which sit in the same salty water as that inside the pumps, would remain carbon steel. Critical moving parts inside the pumps, like bearings and the Rineer hydraulic motors, would also not be replaced on a comprehensive basis. The pumps the Corps had pulled themselves earlier in 2009 also received this rather limited set of repairs, including two other 60" pumps. In sum, this work only addressed about a third of corrosion damage with upgrades, replacing the rest of the rusted or damaged pieces in kind with exactly the same materials.
Fortunately, one of the contractual requirements was generation of reports by the contractor on what they found on these pumps when they came out of the water. The pictures in those reports were disturbing:
Here's a section of pipe removed from the pump sitting on the table at Conhagen (the fitting at the bottom indicates it may be from the inlet or outlet of one of the hydraulic coolers)
It appears that a large chunk of the pipe is completely gone in this photo. Also note the rusty hose fitting on the table:
Here's one of the oil coolers mounted to the sides of the pumps, with a generous coating of marine growth:
These coolers are meant to lower the temperature of still-hot hydraulic oil passing through after leaving the Rineer hydraulic motor but before returning to the reservoir on the engine skid back on the landside of the canal. The residual heat is supposed to be transferred to the canal water, but such severe fouling would undoubtedly hamper that from happening.
Here's some of the piping extensions installed over the winter of 2006-07, specifically the return piping (two larger lines) and the smaller case drain line:
It was clear there was an urgent problem facing the Corps - systemwide corrosion of all 54 pumps within two years of their acceptance for service. Indeed, Conhagen said so in their summary of the repairs, terming the corrosion-related failure of the Corps' other hydraulic pumps "imminent." However, the Corps continued on a rather slow pace of repairs, only contracting with a single repair shop and never sending more than four pumps out at any given time, even outside of hurricane season.
Repair scope widens - spring, 2010
Indeed, because of the Corps' slowness to respond - despite the dire warning from Conhagen - nearly seven months went by before any more pump repairs took place. In February, 2010, a task order for further repairs was finally issued, calling for two more pumps to be pulled out at the 17th Street site. They would receive a greatly enhanced set of repairs compared to the pumps fixed up in 2009. All the carbon steel piping - inside and out - would be replaced with stainless steel. Rotating parts like the Rineer motors and the bearings would also be replaced. Seals would be replaced, and new gland plates holding those seals would be manufactured of sturdier materials. All fasteners would also be upgraded to stainless steel. Essentially, the pumps would be completely rebuilt, inside and out.
It's a good thing the Corps finally expanded their focus to include the external piping, because here's one of the oil coolers on those first two pumps pulled in 2010:
It's also a good thing they started systematically replacing bearings, seals, and Rineer motors, because they were taking serious damage. Here's one of the Rineers from that same pair of pumps:
And here is the area around the shaft and bearings, showing obvious evidence of water infiltration:
From the amount of rust inside the bearing housing, as well as the significant discoloration along the shaft, it is clear the seals completely failed. This would have exposed the bearings and shaft to salt water, causing failure of the pump. And there is no doubt similar damage had been done to the seals on the Rineer motor, damaging it as well.
The pace of repairs would increase throughout the spring of 2010, with eight more 60" pumps going to Conhagen's shop for the upgraded rebuilds spread over two additional task orders. The pictures from the repair reports continued to show extensive corrosion of all pumps. Rust covered the outside of the pumps, such as 17th Street pump E2...
...as well as the interior, as seen inside London Avenue pump E4:
As with 17th Street pumps W5 and W6 earlier in 2010, there was also evidence of water infiltration into the parts of the mechanism that were supposed to be sealed. Here's part of the drive of 17th Street pump E6 showing corrosion on surfaces supposedly protected by seals:
This should not be showing any signs of rust, but there they are.
There was substantial corrosion damage to the main pump housings as well. At the very bottom of the pump is the suction bell, the tapered portion of the pump which funnels water up to the impeller. The funnelling effect is assisted by tapered vanes. The suction bell is always submerged, and the brackish water took its toll. Here's 17th Street pump E2's suction bell:
and 17th Street pump E6's suction bell:
Conhagen addressed all these areas of concern in their repairs that spring.
Work slows but corrosion continues - summer & fall 2010
The work on these 8 pumps was performed under a new contract given to Healtheon, this one with a scope specifically tailored to the corrosion repairs. But its funding - a potential value of just under $3 million - was horribly inadequate to deal with the entire problem. As the 2010 hurricane season dawned on June 1, there were still 30 of the 60" pumps in the water with either partial (5 of them) or no repairs to arrest the ongoing corrosion damage, and the Corps had already spent half of the $3 million repairing just 8 pumps. Fixing the other 30 would take at least $4.8 million, meaning a shortfall of at least $3 million. The effects of that lassitude would be seen immediately.
The next day, June 2, 2010, pump W1 at the 17th Street site would spring a leak, dumping 15 gallons of hydraulic fluid in the canal. It would be pulled out two weeks later and sent to Conhagen's shop for a rebuild. The leak should not have been a surprise in light of the extreme corrosion found on the pump's external piping:
The next month, pump W1 at London Avenue came out, and pump W4 at Orleans Avenue was sent to the shop in October. Both showed typical corrosion damage. For example, the Rineer motor in Orleans pump W4 looked particularly rusty:
While there were no spill reports for these two pumps, I bet the Corps pulled them out because they found oil in the water and didn't report it.
These three were the only 60" pump repairs during the 2010 hurricane season, since the Corps shut down their scheduled repair effort when the season started in June, another sign of a lack of urgency. In this case, it was brought about by another poor Corps decision, one made in 2006 and 2007 during the construction of the interim closure sturctures. The Corps did not install any spare pumping capacity at the structures, meaning they are unable to repair pumps while providing the required ability to drain the canals when the gates are closed against a storm surge. They have duplicated this error at the massive West Closure Complex.
Repairs restart in earnest - fall, 2010 through spring, 2011
After the end of the 2010 hurricane season, more scheduled repairs with the enhanced scope were tasked to Healtheon. Two pumps were pulled out at the London Avenue site in December, an event accompanied by not one, but two oil spills, totalling 125 gallons. Again, there was major corrosion found on the external piping:
With the Corps approaching the cost ceiling on their second pump repair contract, they solicited for a third one in October, 2010 and awarded it in February, 2011. Once again, Healtheon was the winner, with Conhagen doing the actual repairs. This time, the cost ceiling was $6.75 million, over double the potential value of the prior contract. This should be more than enough to finish off the repairs (they average about $160,000 per pump) and have about $3 million left over until the permanent pumps come online in 2015. Why did it take them over two years to figure out they needed more cash for these repairs? I don't know.
This new contract was put to use immediately, with eight more pumps getting pulled out under two task orders on it in the spring of 2011. Again, the pictures in the Conhagen reports demonstrated these pumps should have been repaired years earlier:
And while these sorts of pictures and repairs were becoming depressingly typical, there was a twist in one of those spring 2011 task orders.
One of the pumps pulled out in March, 2011 was E1 at the London Avenue site. When the pictures from the repair report for that pump were examined, it was revealed that London Avenue E1 had already been out for repairs two years earlier, in the spring of 2009:
This showed the Corps did not have confidence in the 2009 partial repairs, and considered the pumps fixed that year as just as vulnerable to corrosion-related failure as those with no repairs. That would be proven just months later.
Earlier repairs prove inadequate - summer, 2011
As in 2010, the Corps shut down their scheduled pump repairs just before the beginning of the 2011 hurricane season. At that point 17 of the forty 60" pumps either had partial or no repairs. And just as in 2010, some of those pumps started spewing oil nearly immediately.
It is tough to nail down the exact date due to lack of federally required reporting, but sometime in May or June, 2011, pump W8 at the 17th Street site was leaking oil. It got yanked and sent to Conhagen's shop for a rebuild. W8 was a pump that had been part of the incomplete 2009 repairs, proving those repairs were effectively money flushed down the drain. While the parts of the pumps which received upgrades in 2009, like the interior piping, appeared to do well...
...the other components which were replaced in kind in 2009, like the oil coolers made of carbon steel, didn't do so well:
The total expense for both times this pump was pulled out would come to approximately $227,000. The average expense for these pump rebuildings has been around $160,000, meaning about $67,000 was wasted on this pump.
This story would be repeated just a month later, as pump W9 at 17th Street would also start leaking (the Corps reported that one) and get sent to Conhagen for a rebuilt. It too was a 2009 partial repair. Three more 60" pumps "fixed" in 2009 remain in service at 17th Street, along with an unknown number of 42" pumps.
Where we stand - hurricane season 2011
Once 17th Street pump W9 goes back in the water near the end of this month, the Corps will have rebuilt 25 of the 60" hydraulic pumps over a span of about two years (three of them twice), leaving 15 of them effectively in jeopardy of failure during a storm due to leakage of hydraulic oil, which is used to power the pumps. Here's the individual statuses at each site as of the most recent confirmed repairs (17th Street's W9 will not be included until its repair is confirmed through documentation):
Remember they were aware of this problem within a month of installing the pumps in 2006 but only started addressing it in 2009. And a very long two years after that, 38% of the large pumps (15 out of 40) remain in imminent danger of failure when called upon in a storm because of corrosion. Along the way there have been at least nine oil spills totalling hundreds of gallons which have gone unreported to federal authorities, as required by law.
Then there's the dollars. The Corps have spent over $4.3 million on these repairs, and that's just on the 60" pumps. There's likely another $2.2 million to be spent rebuilding the remaining 15 60" pumps. That's on top of the $37.8 million they spent originally on those pumps buying them and trying to make them work in 2006 and 2007, which included a $4.7 million delivery incentive.
That makes the total spent on each rebuilt 60" pump an average of $1.1 million. The original contract price for each pump - before all the modifications on the original contract and all the repairs over the last two years - was $750,000. Thus - on average - the Corps will have spent, once the repairs are done, nearly an additional 50% of the original purchase price since 2006 on each of the 60" pumps.
This repair "effort" has seen zero urgency, inadequate funding, a remarkable ignorance of the Clean Water Act, and months of wasted work in 2009 which must now be redone at extra expense. From beginning to end, the Corps has shown a deep reluctance to do the right thing: yank all the pumps out and send them to wherever they need to go to be repaired. Once they started fixing them in 2009, everything should have been done before June 1, 2010. Instead, at the lazy pace they've set, all the pumps will not be rebuilt until just before the 2013 hurricane season, two years from now and seven years after the initial identification of corrosion as a serious problem. This is unacceptable.
Many but not all of the prior entries in this series are linked throughout the report above. For convenience, here's the complete set:
Imminent, originally posted May 13, 2010, covers the 2009 repairs
How did the pumps get from..., originally posted May 27, 2010, covers the original 2006-07 decisions on corrosion
This year's scramble, originally posted June 3, 2010, covers the spring 2010 repairs
Worse than previously known, originally posted June 11, 2010, also covers the spring 2010 repairs
Corps of oil, Part 1, originally posted June 20, 2010, covers oil spills from 2006 and early 2007.
Corps of oil, Part 2, originally posted June 21, 2010, covers oil spills in later 2007
Corps of oil, Part 3, originally posted June 22, 2010, covers oil spills from 2008 through 2010
No urgency, originally posted August 18, 2010, reports on repairs during summer, 2010
Quick update (with pretty new trees!), originally posted August 30, 2010, gives a summary of corrosion repair events to date at the fifth anniversary of Katrina, as well as revealing the Corps' wasteful spending on tree planting around the closure sites
The latest on lakefront pump repairs, originally posted March 23, 2011, looks at the repairs from summer 2010 through early 2011
The 2011 pump rebuild scramble, originally posted May 9, 2011, covers the spring, 2011 repairs
Brief update on London Avenue pumps, originally posted June 24, 2011, also covers the spring, 2011 repairs
Finally, originally posted July 13, 2011, provides an update on receipt of FOIA-requested documents
No surprise, originally posted July 15, 2011, looks at the summer, 2011 re-repairs of pumps also pulled out in 2009
No surprise, Part 2, originally posted July 20, 2011, covers possible contamination of the pumps' hydraulic fluid with water
There's lots of matters the Corps of Engineers' New Orleans District would rather the public didn't know about, or at least pay attention to. Many times, it's the details of things. They say on June 1, 2011 they've met their goal of hundred year protection, but you find out there's dozens of unfinished projects. They point to how great the West Closure Complex is, but then you learn there were only 8 of 11 pumps ready at the beginning of hurricane season and an early season storm could force local pump stations to shut down. They say the levee system is better than ever, and then you find out it's filled with concrete, wood, steel, and other flotsam.
So it goes with the 54 hydraulic pumps installed at the three outfall canal sites in New Orleans - 32 at 17th Street, 10 at Orleans Avenue and 12 at London Avenue. 40 of them are the 60" type, while the other 14 - all installed at 17th Street - are the smaller 42" version (this post focuses on the 60" pumps, due to a severe lack of contracting information for the repairs to the 42" pumps). The Corps would rather the public didn't bother with such details as whether the pumps work under storm conditions, or if they have crippling design flaws, or if the salt water in Lake Pontchartrain is turning them into rusty heaps. That's because they know if the public were privy to all those pesky details, they might freak out over how scary the story is.
Thank goodness for the Freedom of Information Act, though, because it allows us to push past the well financed Public Affairs curtain and see the entire rusty pump story for what it is: a fiasco five years in the making.
The first signs - summer, 2006
It started nearly instantly after the pumps first went in the canals in June, 2006, with corrosion appearing on the hose connections from the hard piping to the pumps. Hydraulic oil is driven from hydraulic power units onshore through the piping and hoses to the pumps which sit in the canals. Here's a picture of those rusty hose connections: (via SCPR Flickr):
And here's a closeup of one of those rusty hose connections (via SCPR Flickr):
This was a hint that the environment around the waters of Lake Pontchartrain was very salty, and materials susceptible to corrosion - such as the carbon steel making up the pumps - would perform quite poorly. This shouldn't have surprised anyone, since Lake Pontchartrain is the second largest salt water body in the country.
Missing the big picture - 2006 through 2008
But over the next few months, the Corps and its paid consultants (many of which were former Corps themselves) would not look at the systemwide problem augered by that early rust, but would instead focus narrowly on a tiny sliver of the corrosion problem. They concentrated on the hose connections to the pumps which sat beneath or near the waterline as the pumps rested in the canals (these hose connections were at the opposite end of the hoses from those shown in the June, 2006 photos above):
Remarkably, their "solution" to preventing the rusting of those connections in early 2007 was the addition of more carbon steel to the pumps, in the form of piping extensions:
This raised the lower hose connections above the waterline, but also had the effect of giving much more surface area for rust and marine organisms to attack. It was like discovering your car's engine was on fire, and deciding to check the spark plugs - while wearing a suit covered in gasoline!
With all that yummy carbon steel waiting like a buffet, the marine organisms in the Lake Pontchartrain water started chomping away at the coar tar epoxy paint, exposing the bare metal below in no time. Corrosion of the steel quickly followed. An oil spill from pump E5 at the 17th Street site on June 26, 2008 was a hint of things to come. The Corps paid over $10,000 to clean it up, but never bothered to report it to the federal or state authorities.
A month later a solicitation for comprehensive pump maintenance for pumps across all three sites was issued. It was a small business Section 8(A) set aside, which is a rather odd choice for such a critical function as maintenance of pumps upon which the residents of the City of New Orleans relied for protection of life and property. Nonetheless, the contract was awarded August 28, 2008 to small business Healtheon.
Early self-repair efforts prove inadequate - Spring, 2009
By early 2009, the hydraulic oil which powered the pumps and which was contained within the pipes was seeping regularly from multiple pumps, and sometimes gushing into the canals. A halting repair effort began in March, at first concentrating only on pumps with known leaks.
The Corps attempted to repair them on their own, sometimes going to interesting lengths to conceal their efforts from the public. But after six pumps had to be pulled out in three months, accompanied by five oil spills (four of which went unreported) costing over $30,000 to clean up, it was clear professional help was needed.
Professionals are brought in - summer, 2009
Task orders started going out on the Healtheon contract in July, 2009. Healtheon hired Conhagen, Inc of Kenner, LA to perform the repairs. Conhagen had been performing similar services for the New Orleans Sewerage & Water Board for many years, and indeed it is unclear why the Corps simply didn't hire them directly.
The first task order, on July 21, 2009, called for the partial repair of four 60" pumps at the 17th Street site. New and more plentiful zinc anodes would be attached to the pumps to fight future corrosion, and bits of the pump housings eaten away by rust would be weld repaired.
Also, all the carbon steel piping on the pumps would be removed and replaced. However, only some of it - the piping inside the pump bore - would be upgraded to more corrosion resistant 316L stainless steel. The piping and oil coolers on the outside of the pumps, which sit in the same salty water as that inside the pumps, would remain carbon steel. Critical moving parts inside the pumps, like bearings and the Rineer hydraulic motors, would also not be replaced on a comprehensive basis. The pumps the Corps had pulled themselves earlier in 2009 also received this rather limited set of repairs, including two other 60" pumps. In sum, this work only addressed about a third of corrosion damage with upgrades, replacing the rest of the rusted or damaged pieces in kind with exactly the same materials.
Fortunately, one of the contractual requirements was generation of reports by the contractor on what they found on these pumps when they came out of the water. The pictures in those reports were disturbing:
Here's a section of pipe removed from the pump sitting on the table at Conhagen (the fitting at the bottom indicates it may be from the inlet or outlet of one of the hydraulic coolers)
It appears that a large chunk of the pipe is completely gone in this photo. Also note the rusty hose fitting on the table:
Here's one of the oil coolers mounted to the sides of the pumps, with a generous coating of marine growth:
These coolers are meant to lower the temperature of still-hot hydraulic oil passing through after leaving the Rineer hydraulic motor but before returning to the reservoir on the engine skid back on the landside of the canal. The residual heat is supposed to be transferred to the canal water, but such severe fouling would undoubtedly hamper that from happening.
Here's some of the piping extensions installed over the winter of 2006-07, specifically the return piping (two larger lines) and the smaller case drain line:
It was clear there was an urgent problem facing the Corps - systemwide corrosion of all 54 pumps within two years of their acceptance for service. Indeed, Conhagen said so in their summary of the repairs, terming the corrosion-related failure of the Corps' other hydraulic pumps "imminent." However, the Corps continued on a rather slow pace of repairs, only contracting with a single repair shop and never sending more than four pumps out at any given time, even outside of hurricane season.
Repair scope widens - spring, 2010
Indeed, because of the Corps' slowness to respond - despite the dire warning from Conhagen - nearly seven months went by before any more pump repairs took place. In February, 2010, a task order for further repairs was finally issued, calling for two more pumps to be pulled out at the 17th Street site. They would receive a greatly enhanced set of repairs compared to the pumps fixed up in 2009. All the carbon steel piping - inside and out - would be replaced with stainless steel. Rotating parts like the Rineer motors and the bearings would also be replaced. Seals would be replaced, and new gland plates holding those seals would be manufactured of sturdier materials. All fasteners would also be upgraded to stainless steel. Essentially, the pumps would be completely rebuilt, inside and out.
It's a good thing the Corps finally expanded their focus to include the external piping, because here's one of the oil coolers on those first two pumps pulled in 2010:
It's also a good thing they started systematically replacing bearings, seals, and Rineer motors, because they were taking serious damage. Here's one of the Rineers from that same pair of pumps:
And here is the area around the shaft and bearings, showing obvious evidence of water infiltration:
From the amount of rust inside the bearing housing, as well as the significant discoloration along the shaft, it is clear the seals completely failed. This would have exposed the bearings and shaft to salt water, causing failure of the pump. And there is no doubt similar damage had been done to the seals on the Rineer motor, damaging it as well.
The pace of repairs would increase throughout the spring of 2010, with eight more 60" pumps going to Conhagen's shop for the upgraded rebuilds spread over two additional task orders. The pictures from the repair reports continued to show extensive corrosion of all pumps. Rust covered the outside of the pumps, such as 17th Street pump E2...
...as well as the interior, as seen inside London Avenue pump E4:
As with 17th Street pumps W5 and W6 earlier in 2010, there was also evidence of water infiltration into the parts of the mechanism that were supposed to be sealed. Here's part of the drive of 17th Street pump E6 showing corrosion on surfaces supposedly protected by seals:
This should not be showing any signs of rust, but there they are.
There was substantial corrosion damage to the main pump housings as well. At the very bottom of the pump is the suction bell, the tapered portion of the pump which funnels water up to the impeller. The funnelling effect is assisted by tapered vanes. The suction bell is always submerged, and the brackish water took its toll. Here's 17th Street pump E2's suction bell:
and 17th Street pump E6's suction bell:
Conhagen addressed all these areas of concern in their repairs that spring.
Work slows but corrosion continues - summer & fall 2010
The work on these 8 pumps was performed under a new contract given to Healtheon, this one with a scope specifically tailored to the corrosion repairs. But its funding - a potential value of just under $3 million - was horribly inadequate to deal with the entire problem. As the 2010 hurricane season dawned on June 1, there were still 30 of the 60" pumps in the water with either partial (5 of them) or no repairs to arrest the ongoing corrosion damage, and the Corps had already spent half of the $3 million repairing just 8 pumps. Fixing the other 30 would take at least $4.8 million, meaning a shortfall of at least $3 million. The effects of that lassitude would be seen immediately.
The next day, June 2, 2010, pump W1 at the 17th Street site would spring a leak, dumping 15 gallons of hydraulic fluid in the canal. It would be pulled out two weeks later and sent to Conhagen's shop for a rebuild. The leak should not have been a surprise in light of the extreme corrosion found on the pump's external piping:
The next month, pump W1 at London Avenue came out, and pump W4 at Orleans Avenue was sent to the shop in October. Both showed typical corrosion damage. For example, the Rineer motor in Orleans pump W4 looked particularly rusty:
While there were no spill reports for these two pumps, I bet the Corps pulled them out because they found oil in the water and didn't report it.
These three were the only 60" pump repairs during the 2010 hurricane season, since the Corps shut down their scheduled repair effort when the season started in June, another sign of a lack of urgency. In this case, it was brought about by another poor Corps decision, one made in 2006 and 2007 during the construction of the interim closure sturctures. The Corps did not install any spare pumping capacity at the structures, meaning they are unable to repair pumps while providing the required ability to drain the canals when the gates are closed against a storm surge. They have duplicated this error at the massive West Closure Complex.
Repairs restart in earnest - fall, 2010 through spring, 2011
After the end of the 2010 hurricane season, more scheduled repairs with the enhanced scope were tasked to Healtheon. Two pumps were pulled out at the London Avenue site in December, an event accompanied by not one, but two oil spills, totalling 125 gallons. Again, there was major corrosion found on the external piping:
With the Corps approaching the cost ceiling on their second pump repair contract, they solicited for a third one in October, 2010 and awarded it in February, 2011. Once again, Healtheon was the winner, with Conhagen doing the actual repairs. This time, the cost ceiling was $6.75 million, over double the potential value of the prior contract. This should be more than enough to finish off the repairs (they average about $160,000 per pump) and have about $3 million left over until the permanent pumps come online in 2015. Why did it take them over two years to figure out they needed more cash for these repairs? I don't know.
This new contract was put to use immediately, with eight more pumps getting pulled out under two task orders on it in the spring of 2011. Again, the pictures in the Conhagen reports demonstrated these pumps should have been repaired years earlier:
And while these sorts of pictures and repairs were becoming depressingly typical, there was a twist in one of those spring 2011 task orders.
One of the pumps pulled out in March, 2011 was E1 at the London Avenue site. When the pictures from the repair report for that pump were examined, it was revealed that London Avenue E1 had already been out for repairs two years earlier, in the spring of 2009:
This showed the Corps did not have confidence in the 2009 partial repairs, and considered the pumps fixed that year as just as vulnerable to corrosion-related failure as those with no repairs. That would be proven just months later.
Earlier repairs prove inadequate - summer, 2011
As in 2010, the Corps shut down their scheduled pump repairs just before the beginning of the 2011 hurricane season. At that point 17 of the forty 60" pumps either had partial or no repairs. And just as in 2010, some of those pumps started spewing oil nearly immediately.
It is tough to nail down the exact date due to lack of federally required reporting, but sometime in May or June, 2011, pump W8 at the 17th Street site was leaking oil. It got yanked and sent to Conhagen's shop for a rebuild. W8 was a pump that had been part of the incomplete 2009 repairs, proving those repairs were effectively money flushed down the drain. While the parts of the pumps which received upgrades in 2009, like the interior piping, appeared to do well...
...the other components which were replaced in kind in 2009, like the oil coolers made of carbon steel, didn't do so well:
The total expense for both times this pump was pulled out would come to approximately $227,000. The average expense for these pump rebuildings has been around $160,000, meaning about $67,000 was wasted on this pump.
This story would be repeated just a month later, as pump W9 at 17th Street would also start leaking (the Corps reported that one) and get sent to Conhagen for a rebuilt. It too was a 2009 partial repair. Three more 60" pumps "fixed" in 2009 remain in service at 17th Street, along with an unknown number of 42" pumps.
Where we stand - hurricane season 2011
Once 17th Street pump W9 goes back in the water near the end of this month, the Corps will have rebuilt 25 of the 60" hydraulic pumps over a span of about two years (three of them twice), leaving 15 of them effectively in jeopardy of failure during a storm due to leakage of hydraulic oil, which is used to power the pumps. Here's the individual statuses at each site as of the most recent confirmed repairs (17th Street's W9 will not be included until its repair is confirmed through documentation):
Remember they were aware of this problem within a month of installing the pumps in 2006 but only started addressing it in 2009. And a very long two years after that, 38% of the large pumps (15 out of 40) remain in imminent danger of failure when called upon in a storm because of corrosion. Along the way there have been at least nine oil spills totalling hundreds of gallons which have gone unreported to federal authorities, as required by law.
Then there's the dollars. The Corps have spent over $4.3 million on these repairs, and that's just on the 60" pumps. There's likely another $2.2 million to be spent rebuilding the remaining 15 60" pumps. That's on top of the $37.8 million they spent originally on those pumps buying them and trying to make them work in 2006 and 2007, which included a $4.7 million delivery incentive.
That makes the total spent on each rebuilt 60" pump an average of $1.1 million. The original contract price for each pump - before all the modifications on the original contract and all the repairs over the last two years - was $750,000. Thus - on average - the Corps will have spent, once the repairs are done, nearly an additional 50% of the original purchase price since 2006 on each of the 60" pumps.
This repair "effort" has seen zero urgency, inadequate funding, a remarkable ignorance of the Clean Water Act, and months of wasted work in 2009 which must now be redone at extra expense. From beginning to end, the Corps has shown a deep reluctance to do the right thing: yank all the pumps out and send them to wherever they need to go to be repaired. Once they started fixing them in 2009, everything should have been done before June 1, 2010. Instead, at the lazy pace they've set, all the pumps will not be rebuilt until just before the 2013 hurricane season, two years from now and seven years after the initial identification of corrosion as a serious problem. This is unacceptable.
Many but not all of the prior entries in this series are linked throughout the report above. For convenience, here's the complete set:
Imminent, originally posted May 13, 2010, covers the 2009 repairs
How did the pumps get from..., originally posted May 27, 2010, covers the original 2006-07 decisions on corrosion
This year's scramble, originally posted June 3, 2010, covers the spring 2010 repairs
Worse than previously known, originally posted June 11, 2010, also covers the spring 2010 repairs
Corps of oil, Part 1, originally posted June 20, 2010, covers oil spills from 2006 and early 2007.
Corps of oil, Part 2, originally posted June 21, 2010, covers oil spills in later 2007
Corps of oil, Part 3, originally posted June 22, 2010, covers oil spills from 2008 through 2010
No urgency, originally posted August 18, 2010, reports on repairs during summer, 2010
Quick update (with pretty new trees!), originally posted August 30, 2010, gives a summary of corrosion repair events to date at the fifth anniversary of Katrina, as well as revealing the Corps' wasteful spending on tree planting around the closure sites
The latest on lakefront pump repairs, originally posted March 23, 2011, looks at the repairs from summer 2010 through early 2011
The 2011 pump rebuild scramble, originally posted May 9, 2011, covers the spring, 2011 repairs
Brief update on London Avenue pumps, originally posted June 24, 2011, also covers the spring, 2011 repairs
Finally, originally posted July 13, 2011, provides an update on receipt of FOIA-requested documents
No surprise, originally posted July 15, 2011, looks at the summer, 2011 re-repairs of pumps also pulled out in 2009
No surprise, Part 2, originally posted July 20, 2011, covers possible contamination of the pumps' hydraulic fluid with water
posted by mcbrid35 at
9:50 AM
0 Comments:
Post a Comment
<< Home