No surprise
This post is one in a series I've been running since May, 2010 about the corrosion-related repairs to the 54 hydraulic pumps at the Corps' three New Orleans outfall canal sites. I've only recently started to receive contract documents and reports on these repairs via the Frreedom of Information Act.
A post from earlier this week provides links to the earlier articles in the series. Since the documents released date back almost two years, many of those earlier articles have been heavily revised to reflect the newly gained knowledge from the documents.
But the released documents also included information on very recent events surrounding these pumps...
In the last post, I noted that two pumps at 17th Street (W8 and W9) had been pulled out in the last couple of months. Those two pumps were already pulled out in 2009 for repairs, yet here they are coming out again. Why is that happening?
Because the parts that weren't overhauled in the 2009 repairs - which make up most of the pumps - gave out.
Just as a reminder, the repairs to pumps in 2009 (covered in earlier posts here and here) only concerned themselves with replacing carbon steel piping on the inside of the pumps with stainless steel:
The piping on the outside of the pumps was replaced, but with the same carbon steel material that had rusted in the first place. Then it was repainted with the same paint as before as well:
Had there been new stainless steel piping on the outside of the pumps, it would appear the same as the silvery stainless steel piping shown in the picture of the inside of the pump above.
In addition, all the moving parts like the bearings and the Rineer motors were also put back.
So essentially, about a third of what needed to be fixed was fixed, and the pumps were put back in the brackish Lake Pontchartrain waters to start corroding again. Six 60" pumps - five at 17th Street and one at London Avenue - as well as an unknown number of 42" pumps at 17th Street received this suite of repairs. It is no surprise that these pumps would be subject to the same corrosion damage as any other, mostly-carbon steel piece of equipment sitting in salty water.
We saw the first evidence of this earlier this spring, when pump E1 at the London Avenue site was removed as part of a planned pump pull. I only recently figured out that this pump was repaired in 2009 in addition to the five 60" pumps at 17th Street.
The London E1 stainless steel interior pipes, installed in 2009 and seen in the photo above, did not appear severely rusted in spring, 2011...
...while the carbon steel external piping and coolers looked like they took some serious corrosion damage since June, 2009:
[More details on the 2011 repairs to this pump are in my earlier post, "The 2011 pump rebuild scramble."]
That this pump was taken out as part of a planned complete overhaul in the spring of 2011 is evidence the Corps recognized the 2009 repairs were inadequate. But with many other pumps completely untouched since 2007, they triaged and focused on the more damaged ones rather than the other five pumps repaired in 2009.
However, it was only a matter of time before the pumps partially fixed in 2009 also started breaking. That's what happened with 17th Street pumps W8 and W9 in the last two months.
The first failure appears to have occurred in May or June of this year. We learned this from $89,489.55 task order #8 on the second Healtheon contract, issued on June 7, 2011. Prior to this, the second Healtheon contract had lain fallow since the issuance of task order #7 and its immediate follow-up modification on December 9th and 10th, 2010. After that, the 60" hydraulic pump repair work had been moved over to the third Healtheon contract, which was inked in January, 2011. Issuance of a task order on a contract previously thought ended was a little odd, but not unprecedented. The first Healtheon contract has continued to get task orders issued on it as well.
Anyhow, the body of task order #8 reads,
So the Corps had discovered an oil leak on pump W8. Did they report it to the National Response Center, as required under federal law? Apparently not, because searches on the NRC website show no reports for such an incident prior to June 7, 2011. That's really no surprise - the Corps has little regard for the law on reporting oil spills.
Pump W8 was pulled out of the 17th Street site and sent to Conhagen's shop. A $30,178.32 modification to task order #8 was issued June 20, 2011:
I would have expected the scope to simply be a restatement of the normal scope of work, with perhaps the exception of replacement of the interior piping. Instead, we find this oddly worded scope, which is rather lacking in detail. Nevertheless, it appears to cover much of the work which would be expected.
Fortunately, Conhagen had a better grasp on what needed to be done. The Conhangen repair report for pump W8's second time in their shop gives a detailed scope which covers everything needed to bring this pump up to same status as other completely rebuilt units.
That report shows the areas which one would expect to be the most damaged - those not upgraded in the 2009 repairs - were indeed the most damaged. The carbon steel oil coolers installed in 2009 had unsurprisingly accumulated a great deal of marine growth and had rusted:
Admittedly, the damage wasn't as great as we have seen on other pumps, but it is evident. Here's a similar area on one of the pumps after it was repaired in 2009:
The Rineer motor and its mount were also rusty:
Here's the "before" picture of that area in 2009:
Conhagen fixed up the pump, leak tested it, and put it back in the water on June 6, 2011, according to the acceptance testing record included in Conhagen's report.
Wait, what?
The original task order for the removal of this pump was dated June 7, 2011, and the follow-up modification with the scope of repairs was dated June 20, 2011. So how did the pump go back in the water before there was any contract for repairs?
It appears this is once again the Corps' Contracting Division issuing after-the-fact task orders and modifications for work already completed. In fact, the cover page of the Conhagen repair report indicates their work on the pump actually started three weeks before they received their first contractual documents:
One has to wonder how exactly contracts are run at the New Orleans District, when we find stuff like this.
So apart from the contracting slight-of-hand, I think we can take the Conhagen photos of W8's repairs in May and June of 2011 as proof that it has been fully rebuilt, meaning the 17th Street pump status can be updated again. However, the fact that W8 after previously being partially repaired still sprung a leak means that the remaining 60" pumps fixed in 2009 are now not just questionable, but effectively unrepaired. So the revised 17th Street status is:
This now means there are sixteen 60" hydraulic pumps across all three sites which cannot be trusted to be free from corrosion-related failures in the future:
17th Street: E5, E7, W2, W3, W9, W10 (W2 and W3 never previously repaired)
Orleans Avenue: E1, E2, E3, E4, E5, W1, W2, W3, W5 (all never previously repaired)
London Avenue: W2 (not previously repaired)
Indeed, this assertion was confirmed just last week, as 17th Street pump W9, another of the pumps only partially repaired in 2009, also sprung a leak.
This time, there was an oil spill report to the National Response Center, on July 5, 2011:
Two days later, on July 7, 2011, the Corps issued $89,489.55 task order #9 on the second Healtheon contract. That dollar amount is exactly the same as the initial task order (#8) on 17th Street pump W8 in June. This time, the task order reads:
Like the dollar amount, this is a copy and paste from the initial task order for pump W8, down to the typo on the work "leaked."
Since my receipt of documents via FOIA only covered stuff through last week, I don't have the task order modification which covers the actual repairs of 17th Street pump W9, but I am certain it looks nearly identical to the corresponding modification for pump W8. Right now, pump W9 is sitting on Conhagen's shop floor, and it will probably go back in the water before July 31.
Update, 7/6/12
The modification that covered the additional scope of repairs on 17th Street W9 was issued July 19, 2011 for $29,803.42. Again, this appears to be an after-the-fact contract action, since documents attached to the Conhagen repair report indicate the pump went back in the water on July 15, 2011.
That report includes the usual sorts of pictures of excessive marine growth and corrosion on the pump:
Original post resumes
The pattern of pump repairs over the past two years has been for scheduled rebuild work to only be undertaken outside hurricane season, meaning from December 1 to May 31st. If individual pumps fail during or close to hurricane season, as 17th Street pumps W8 and W9 have in the past couple of months - or as three other pumps did during the 2010 hurricane season - they are yanked out and rebuilt. Everything else - including pumps that have been rusting for nearly five years now - is left in the canals. This is a remnant of a foolish Corps decision made after Katrina - and carried out for two years of engineering and construction in 2006 and 2007 - not to supply any spare pumping capacity at their closure structures.
At the pace demonstrated by the repair contractor over the past two years, it seems impossible all 16 remaining 60" pumps will be completely rebuilt by June 1, 2012. Only ten pumps each were rebuilt in the 2009-10 and 2010-11 hurricane off seasons. So it seems likely - barring further (admittedly very likely) oil spills from unrebuilt pumps - there will still be rusty pumps in the canals all the way through the 2012 hurricane season.
A post from earlier this week provides links to the earlier articles in the series. Since the documents released date back almost two years, many of those earlier articles have been heavily revised to reflect the newly gained knowledge from the documents.
But the released documents also included information on very recent events surrounding these pumps...
In the last post, I noted that two pumps at 17th Street (W8 and W9) had been pulled out in the last couple of months. Those two pumps were already pulled out in 2009 for repairs, yet here they are coming out again. Why is that happening?
Because the parts that weren't overhauled in the 2009 repairs - which make up most of the pumps - gave out.
Just as a reminder, the repairs to pumps in 2009 (covered in earlier posts here and here) only concerned themselves with replacing carbon steel piping on the inside of the pumps with stainless steel:
The piping on the outside of the pumps was replaced, but with the same carbon steel material that had rusted in the first place. Then it was repainted with the same paint as before as well:
Had there been new stainless steel piping on the outside of the pumps, it would appear the same as the silvery stainless steel piping shown in the picture of the inside of the pump above.
In addition, all the moving parts like the bearings and the Rineer motors were also put back.
So essentially, about a third of what needed to be fixed was fixed, and the pumps were put back in the brackish Lake Pontchartrain waters to start corroding again. Six 60" pumps - five at 17th Street and one at London Avenue - as well as an unknown number of 42" pumps at 17th Street received this suite of repairs. It is no surprise that these pumps would be subject to the same corrosion damage as any other, mostly-carbon steel piece of equipment sitting in salty water.
We saw the first evidence of this earlier this spring, when pump E1 at the London Avenue site was removed as part of a planned pump pull. I only recently figured out that this pump was repaired in 2009 in addition to the five 60" pumps at 17th Street.
The London E1 stainless steel interior pipes, installed in 2009 and seen in the photo above, did not appear severely rusted in spring, 2011...
...while the carbon steel external piping and coolers looked like they took some serious corrosion damage since June, 2009:
[More details on the 2011 repairs to this pump are in my earlier post, "The 2011 pump rebuild scramble."]
That this pump was taken out as part of a planned complete overhaul in the spring of 2011 is evidence the Corps recognized the 2009 repairs were inadequate. But with many other pumps completely untouched since 2007, they triaged and focused on the more damaged ones rather than the other five pumps repaired in 2009.
However, it was only a matter of time before the pumps partially fixed in 2009 also started breaking. That's what happened with 17th Street pumps W8 and W9 in the last two months.
The first failure appears to have occurred in May or June of this year. We learned this from $89,489.55 task order #8 on the second Healtheon contract, issued on June 7, 2011. Prior to this, the second Healtheon contract had lain fallow since the issuance of task order #7 and its immediate follow-up modification on December 9th and 10th, 2010. After that, the 60" hydraulic pump repair work had been moved over to the third Healtheon contract, which was inked in January, 2011. Issuance of a task order on a contract previously thought ended was a little odd, but not unprecedented. The first Healtheon contract has continued to get task orders issued on it as well.
Anyhow, the body of task order #8 reads,
"Remove pump # 8W at 17th St. Canal to inspect, recommend what need to be repaired and reinstall it.
Completion of this work will be six weeks after notice of award.
Note:
1. Pump # 8W was overhauled in Aug 2009 and is currently leaked [sic] outside of the pump.
2. After the contractor inspects and finds out what the pump problems are, we will provide Contracting Division a separate scope of work for repair."
So the Corps had discovered an oil leak on pump W8. Did they report it to the National Response Center, as required under federal law? Apparently not, because searches on the NRC website show no reports for such an incident prior to June 7, 2011. That's really no surprise - the Corps has little regard for the law on reporting oil spills.
Pump W8 was pulled out of the 17th Street site and sent to Conhagen's shop. A $30,178.32 modification to task order #8 was issued June 20, 2011:
"The following scope of work was found necessary upon removal and inspection of Pump #8W at the 17th St. ICS:
- Remove hydraulic motor and motor adapter. Replace motor with gov-furnished new one.
- Build up weld eroded and corroded areas on motor adapter and other pump surfaces.
- Replace oil coolers and connected piping. Spot prep minor corrosion on other exterior piping.
- Replace small bypass hose for motor case drain. Replace exterior pipe u-bolts.
- Provide a full coat of epoxy paint on pump assembly in addition to touch up of original scope.
- Have mechanical seal reconditioned at subcontractor’s shop (to be returned to gov). Install new gov-furnished seal."
I would have expected the scope to simply be a restatement of the normal scope of work, with perhaps the exception of replacement of the interior piping. Instead, we find this oddly worded scope, which is rather lacking in detail. Nevertheless, it appears to cover much of the work which would be expected.
Fortunately, Conhagen had a better grasp on what needed to be done. The Conhangen repair report for pump W8's second time in their shop gives a detailed scope which covers everything needed to bring this pump up to same status as other completely rebuilt units.
That report shows the areas which one would expect to be the most damaged - those not upgraded in the 2009 repairs - were indeed the most damaged. The carbon steel oil coolers installed in 2009 had unsurprisingly accumulated a great deal of marine growth and had rusted:
Admittedly, the damage wasn't as great as we have seen on other pumps, but it is evident. Here's a similar area on one of the pumps after it was repaired in 2009:
The Rineer motor and its mount were also rusty:
Here's the "before" picture of that area in 2009:
Conhagen fixed up the pump, leak tested it, and put it back in the water on June 6, 2011, according to the acceptance testing record included in Conhagen's report.
Wait, what?
The original task order for the removal of this pump was dated June 7, 2011, and the follow-up modification with the scope of repairs was dated June 20, 2011. So how did the pump go back in the water before there was any contract for repairs?
It appears this is once again the Corps' Contracting Division issuing after-the-fact task orders and modifications for work already completed. In fact, the cover page of the Conhagen repair report indicates their work on the pump actually started three weeks before they received their first contractual documents:
One has to wonder how exactly contracts are run at the New Orleans District, when we find stuff like this.
So apart from the contracting slight-of-hand, I think we can take the Conhagen photos of W8's repairs in May and June of 2011 as proof that it has been fully rebuilt, meaning the 17th Street pump status can be updated again. However, the fact that W8 after previously being partially repaired still sprung a leak means that the remaining 60" pumps fixed in 2009 are now not just questionable, but effectively unrepaired. So the revised 17th Street status is:
This now means there are sixteen 60" hydraulic pumps across all three sites which cannot be trusted to be free from corrosion-related failures in the future:
17th Street: E5, E7, W2, W3, W9, W10 (W2 and W3 never previously repaired)
Orleans Avenue: E1, E2, E3, E4, E5, W1, W2, W3, W5 (all never previously repaired)
London Avenue: W2 (not previously repaired)
Indeed, this assertion was confirmed just last week, as 17th Street pump W9, another of the pumps only partially repaired in 2009, also sprung a leak.
This time, there was an oil spill report to the National Response Center, on July 5, 2011:
"Official Material Name: HYDRAULIC OIL
Qty Released: 8 GALLON(S)
Qty in Water: 8 GALLON(S)
CALLER IS REPORTING A DISCHARGE OF HYDRAULIC OIL FROM A HYDRAULIC PUMP DUE TO UNKNOWN CAUSES.
REMEDIAL ACTIONS: CLEANING UP SPILL WITH ABSORBENT PADS WITHIN THE BOOM SYSTEM."
Two days later, on July 7, 2011, the Corps issued $89,489.55 task order #9 on the second Healtheon contract. That dollar amount is exactly the same as the initial task order (#8) on 17th Street pump W8 in June. This time, the task order reads:
"Remove pump # 9W at 17th St. Canal to inspect, recommend what need to be repaired and reinstall it.
Note:
1. Pump # 9W was overhauled in Aug 2009 and is currently leaked [sic] outside of the pump.
2. After the contractor inspects and finds out what the pump problems are, we will provide Contracting Division a separate scope of work for repair.
3. Estimated date to start is 7/8/ 2011 and overtime for the contractors to work on Saturday 7/9/11 and Sunday 7/10/11 should be authorized."
Like the dollar amount, this is a copy and paste from the initial task order for pump W8, down to the typo on the work "leaked."
Since my receipt of documents via FOIA only covered stuff through last week, I don't have the task order modification which covers the actual repairs of 17th Street pump W9, but I am certain it looks nearly identical to the corresponding modification for pump W8. Right now, pump W9 is sitting on Conhagen's shop floor, and it will probably go back in the water before July 31.
Update, 7/6/12
The modification that covered the additional scope of repairs on 17th Street W9 was issued July 19, 2011 for $29,803.42. Again, this appears to be an after-the-fact contract action, since documents attached to the Conhagen repair report indicate the pump went back in the water on July 15, 2011.
That report includes the usual sorts of pictures of excessive marine growth and corrosion on the pump:
Original post resumes
The pattern of pump repairs over the past two years has been for scheduled rebuild work to only be undertaken outside hurricane season, meaning from December 1 to May 31st. If individual pumps fail during or close to hurricane season, as 17th Street pumps W8 and W9 have in the past couple of months - or as three other pumps did during the 2010 hurricane season - they are yanked out and rebuilt. Everything else - including pumps that have been rusting for nearly five years now - is left in the canals. This is a remnant of a foolish Corps decision made after Katrina - and carried out for two years of engineering and construction in 2006 and 2007 - not to supply any spare pumping capacity at their closure structures.
At the pace demonstrated by the repair contractor over the past two years, it seems impossible all 16 remaining 60" pumps will be completely rebuilt by June 1, 2012. Only ten pumps each were rebuilt in the 2009-10 and 2010-11 hurricane off seasons. So it seems likely - barring further (admittedly very likely) oil spills from unrebuilt pumps - there will still be rusty pumps in the canals all the way through the 2012 hurricane season.
Labels: Rusty pumps
posted by mcbrid35 at
6:56 AM
0 Comments:
Post a Comment
<< Home