This year's scramble
After I received a large number of documents related to the New Orleans Corps' corrosion rebuilds of the hydraulic pumps in July, 2011, I went back and rewrote all the prior blog entries I posted on the effort since May, 2010, including the one below. I incorporated photos and text from these new documents in order to give a fuller picture of what was happening.
At the time these entries were written, I was mainly relying on photography of the sites by friends to track the movements of pumps in and out of each canal. The Corps had turned over a few documents to me, but the bulk of the knowledge was coming from studying the site pictures.
With the receipt of all my requested documents, we now have the complete picture of the rebuild effort. During the time covered by the post (February through early June, 2010), 10 of the 40 large 60" hydraulic pumps were pulled out in three batches for complete rebuilds. At the time I wrote this entry (May, 2010), I wasn't aware of the complete extent of the work to fix corrosion damage. I knew more carbon steel piping was replaced with stainless steel in the 2010 work than in 2009, but I didn't know the critical internal components including bearings, seals, and Rineer hydraulic motors were also getting changed out. When I discovered in June, 2010 that was the case for the later spring, 2010 repairs, I wrote the entry after this one, "Worse than previously known." But even that knowledge was incomplete.
The complete set of contract documents and reports now shows that bearings, seals, and Rineers were being changed out in the first set of 2010 repairs as well. I include that information below, as well as information on the bearing, seal, and Rineer replacements during the other spring, 2010 repairs. This somewhat blunts the original tone of "discovery" in "Worse than previously known."
More important than changing tones, though, is the finding of such repairs on the first set of pumps pulled out in 2010. That moves that part of the rebuilding chronology back further than previously known, and shows how quickly these parts were rendered nonfunctional by the brackish water in Lake Pontchartrain, an important consideration with the installation of the permanent pumps intended to replace these hydraulic units now delayed until 2015.
Remarkably, many of the pumps at the three sites as of this writing - late July, 2011 - have still not been pulled out, over two years after these first extensive replacements. They have been sitting in the salty water of the canals for five years now, corroding into uselessness.
So we've detailed the repair work done on the rusty pumps last year, and the Corps decisions in 2006-07 which led to that work. The repair work is being funded with dollars that were supposed to go toward placement of a permanent pumping system at the lakefront. Instead, they're fixing their three-year-old screwups.
March 16, 2010
The first (and only public) inkling there was repair work happening in 2010 was from a TV news story about those permanent pumps: Fox 8's March 16, 2010 piece about the signing ceremony for the permanent pump station Project Partnership Agreement between the state and the Corps, held at the 17th Street site. Fox 8 was one of three media outlets there, along with WWL-TV (story here) and the Times-Picayune (story here). Unmentioned by any of the stories was the fact the document had actually been signed four days previous to the ceremony.
But the Project Partnership Agreement is beside the issue. What came out of the Fox story was the first real-time footage showing the repair work to the pumps. Fox 8 got two shots of 60" 17th Street pump W5 being placed back in the platform after its earlier removal for repairs. Here's a screencap from the first shot:
You can see the "W5" label on the elbow, clearly identifying this pump.
Here's another shot of W5 going back into the structure. The new zinc anodes on the pump's suction bell installed as part of the repairs are clear in this shot.
This March 16, 2010 replacement was the end of the work on $238,846.95 task order #4 of the first Healtheon contract, which involved 17th Street pumps W5 and W6. This task order, issued February 1, 2010, was a watershed in the pump rebuild effort. Whereas the pumps pulled out in 2009 only had piping on their insides replaced with stainless steel...
...while the piping outside the pumps and the oil coolers were left as carbon steel and painted black, as shown in these photos from the Conhagen report on the repairs to 17th Street pumps E5 and E7 under task order #3 to the first Healtheon contract...
...this time around, not only was the internal piping changed out for stainless, so were the oil coolers and the external piping. That new external stainless steel piping can be seen in the Fox 8 footage:
That is, for the 2010 repairs to 17th Street pumps W5 and W6 under task order #4, every bit of piping touching the water or in the splash zone was upgraded to 316L stainless, even the oil coolers.
Here's a summary of the differences between the 2009 and 2010 repairs:
Piping internal to pump:
2009: 316L stainless steel
2010: 316L stainless steel
Piping external to pump:
2009: Carbon steel
2010: 316L stainless steel
All pipe fitting fasteners:
2009: Grade 8 (a medium carbon alloy steel)
2010: 316L stainless steel
And, the oil coolers:
2009: sandblasted and weld repaired, but original carbon steel remained
2010: completely replaced with new 316L stainless steel
Here's one of the pumps with the new stainless oil coolers and external piping on the Conhagen shop floor, immediately before its March 16, 2010 reinstallation:
Per standard practice, the stainless steel is left unpainted, appearing a silvery gray. That is why the new piping and oil coolers are lighter in color than the painted portions of the pump, as also shown in the Fox 8 pictures of W5 above.
And while the Fox 8 video tipped me off to the expanded work on the piping, it wasn't until the July, 2011 receipt of other contracting documents that we could know just how extensive the work was on W5 and W6.
One of those documents allows us to see the condition of the pumps when they went to Conhagen's shop in early February, 2010. The Conhagen repair report for 17th Street pumps W5 and W6 shows pumps corroded to the point of failure just two and a half years after being accepted by the Corps.
This picture explicitly tells the story of why the external piping had to be changed out:
That hole is on one of the oil coolers, which is on the exterior of the pump:
We already knew about the external piping changes. The repair report shows that stuff inside the pump, subject to the same corrosion from brackish water, had also rotted. Here's one of the Rineer hydraulic motors:
And even further into the innards, sections of the pump mechanism which had multiple seals protecting them from water showed obvious evidence of water infiltration. Here is the area around the shaft and bearings:
From the amount of rust inside the bearing housing, as well as the significant discoloration along the shaft, it is clear the seals completely failed. This would have exposed the bearings and shaft to salt water, causing failure of the pump. And there is no doubt similar damage had been done to the seals on the Rineer motor, damaging it as well.
So it makes sense that $23,511.72 modification to task order #4 on the first Healtheon contract was issued to address this issue. In addition to the original expansion of work to include the replacement of the external piping and oil coolers, this modification further expanded the rework to include bearing, seal, and Rineer motor repairs and replacements:
This modification was issued April 27, 2010, over a month after the pumps went back in the water. Considering the relatively small dollar value, this modification appears to be backdated. That is, if it represented new work - which would have involved crane work and taking the pumps back to Conhagen's shop for a second visit - it would have been valued much higher. A final September, 2010 cost reconciling brought the final cost of the work on this task order down by $10,263.75 to $252,094.92, indicating the only crane work was that paid for in the intial task order, and that the first modification was indeed only accounting for extra work done in the shop.
So when W5 and W6 went back in the water in March, 2010, they had essentially been completely remanufactured, with only the cast stainless steel impellers and the main carbon steel housings (which themselves had to have significant weld repairs) remaining from when the pumps were originally accepted by the Corps in September, 2007. That is an extraordinarily short service life, and it pointed toward the extensive threat posed by corrosion to all the other hydraulic pumps still sitting in the salty water, corroding away. That's why pumps started getting yanked in bulk for the rest of the 2010 spring.
Last week of March, 2010
Through 2009 and most of 2010, it was easy to tell when any of the 60" pumps got pulled. The Corps' had a crane contractor lift the pump and the elbow atop it in one piece. They unbolted the elbow and left it at the site, while the pump was trucked offsite to a shop. The hole left in the piping on the structure was obvious.
Here's an example from October, 2006, when I took pictures at the Orleans Avenue gates. All five of the pumps from the east side of the site had been removed the previous month due to internal and external oil leaks. They had been placed up on the deck, and the elbows were on the levee next to them:
It was pretty easy to see that the pumps were out:
So to see if pumps are out at any of the sites, all we had to look for are elbows on the ground and missing elbows on the structure. Soon after the March 16, 2010 replacement of 17th Street pumps W5 and W6, those telltale signs appeared elsewhere.
At 17th Street, pumps E1 and E2 were pulled in mid-March:
At London Avenue, pumps E3 and E4 were pulled at the same time:
This work was carried out under $632,308.77 task order #2 of the second Healtheon repair contract, issued March 19, 2010. These repairs followed the same pattern as that of the prior work on 17th Street pumps W5 and W6. The complete replacement of all carbon steel piping and hardware, along with pump housing weld repairs, was covered under the original task order, and a modification to task order #2 (worth $65,910.82 and again postdated, this time to April 26, 2010) called for replacement of all the bearings, seals, Rineer motors on all four pumps.
We can see why such extensive work was necessary in the pages of the repair reports from Conhagen for these four pumps. Conhagen wrote a report for 17th Street pumps E1 and E2 and a separate report for London Avenue pumps E3 and E4. According to the photos in those reports, the pumps were in bad shape when they came out of the water.
The oil coolers and external piping were covered with rust and marine growth. Here's 17th Street pump E1:
And here's 17th Street pump E2:
The London Avenue pumps were just as bad. Here's London Avenue pump E3:
And London Avenue pump E4:
This particular view shows the severe corrosion buildup on the piping which was normally submerged when compared to the piping which stayed above the waterline.
The interior guts didn't fare any better. Here's the inside of 17th Street pump E2:
These are vertical pumps. The picture above is taken looking down toward the pump's inlet. So oil would have leaked out of the Rineer toward the impeller, away from the viewer.
The damage inside London Avenue pump E4 is nearly identical, with rust everywhere:
As with 17th Street pumps W5 and W6 earlier in 2010, there was also evidence of water infiltration into the parts of the mechanism that were supposed to be sealed. Here's part of the drive of London Avenue pump E3 with its cover removed:
This should not be showing any signs of rust, but there they are.
There was substantial corrosion damage to the main pump housings as well. At the very bottom of the pump is the suction bell, the tapered portion of the pump which funnels water up to the impeller. The funnelling effect is assisted by tapered vanes. The suction bell is always submerged, and the brackish water took its toll. Here's 17th Street pump E2's suction bell:
The single pair of anodes on that same pump had long stopped working by the time it came out:
So had the 17th Street pump E1 anodes:
All these problems were addressed in the repairs. The carbon steel coolers and piping were replaced with stainless steel. Here's London pump E4, showing a typical cooler and piping replacement:
They also got new Rineer hydraulic motors, as shown on London Avenue pump E4:
Remember those rusty bits on the internal drive assembly? They were completely replaced (photo shows 17th Street pump E1):
And here's that same shaft assembly - upside down on a table, with a cleaned up shaft - just before it was placed in the bearing housing shown above:
Note the shiny new bearings.
So all four of these pumps got the full rebuild treatment during their stay in Conhagen's shop in March and April. According to acceptance testing records attached to each report, 17th Street pumps E1 and E2 went back in the water April 13, 2010. and London Avenue pumps E3 and E4 went back in April 16, 2010. Here's pictures of both sets back in their places:
The main 60" pumps were not the only ones getting attention in early 2010. Some of the fourteen 42" bridge pumps were coming out at 17th Street too. The March 16, 2010 Fox 8 report had captured evidence of that. The Corps had taped off the area around the bridge pump drives and there were hydraulic oil hoses up on the deck; they had been disconnected from one of the bridge pumps, possibly #2 or #3:
Also, the Corps' rental crane - normally kept off the deck unless needed - was parked looming over the gathered guests. From its position with its boom above the midpoint of the structure, it appeared there was work happening on bridge pump #6, #7, or #8:
Additional evidence of work on the bridge pumps comes from the Corps putting up scaffolding around to disconnect the hoses and unbolt the pumps from their elbows. As of mid-March, it looked like at least bridge pumps #7 and #8 had been pulled:
Here's a clearer view of the scaffolding:
Bridge pump #3 would come out later in March, but I don't have a good contemporaneous picture of that.
April 15, 2010
By mid-April, more pumps were getting rebuilt. At 17th Street, pumps E6 and E8 came out:
And at London Avenue, pumps W3 and W4 were pulled. This work was carried out under $619,642.17 task order #3 to the second Healtheon repair contract, issued April 12, 2010. The work under task order #3 was similar to that performed on the other six 60" hydraulic pumps earlier in 2010, with the major effort in replacing all carbon steel piping, oil coolers, and piping hardware with 316L stainless steel, as well as replacement of bearings, seals, and Rineer hydraulic motors. The details on the repairs to 17th Street pump E6 and E8 and London Avenue pumps W3 and W4 can be found in the subsequent entry, "Worse than previously known."
April 28, 2010
By late April, E6 and E8 were still out at 17th Street, and were due back in by mid-to-late-May (though that would be delayed until early June).
Back at 17th Street, it looked like they were still working on the same bridge pumps as back in mid-March, nos. 7 and 8:
While we can only see that #7 is definitely out, the scaffolding is still there for #8.
There's evidence that #3 was also still out (scaffolding), but it's not conclusive due to the angle of the photo.
The repairs so far
So the final tally that we know of:
2009 repairs (with only some stainless steel replacement):
17th Street: 60 inch pumps E5, E7, W8, W9, and W10, Bridge pumps #1 and #6
London Avenue: none
Orleans Avenue: none
2010 repairs (possibly definitely with total stainless steel replacement):
17th Street: 60 inch pumps W5, W6, E1, E2, E6, E8, Bridge pumps #3, #7, #8 (unknown scope of repairs), possibly other bridge pumps
London Avenue: 60 inch pumps E3, E4, W3, W4possibly two other 60 inch pumps
Orleans Avenue: none
Or, in graphic form...
17th Street:
London Avenue:
Orleans Avenue:
Let me explain these graphics. The green pumps are the ones I'm pretty positive have gotten as much corrosion repair as possible, i.e. those fixed this year. I'm coming to that conclusion through FOIA results and deduction from which pumps were pulled since February.
The yellow pumps are either those that were repaired in 2009 (and thus still have carbon steel piping) or those pumps for which I don't have adequate documentation (see my comments about FOIA frustrations below). The red pumps are ones that haven't yet been pulled and had their corrosion damage repaired or replaced. All this is based on the best information I could get, which hasn't been a lot, thanks to the Corps New Orleans District.
These number and pictures mean that - by my best estimates - 75%80% (32 30 of 40) of the 60" hydraulic pumps are unrepaired or partially repaired and continue to corrode far faster than they should:
17th Street: 12 (7 pumps unrepaired and 5 pumps repaired in 2009 with carbon steel piping)
Orleans Avenue: all 10
London Avenue:10 (possibly definitely 8
(note: the 11 direct drive pumps at 17th Street and 8 direct drive pumps at London Avenue are likely reliable.)
And just a reminder here, these are what one of those unrepaired pumps likely looked like one year ago, after just two years in the water:
Imagine how they look now, after yet another year in the same brackish water without repairs.
As mentioned above, the totals of unrepaired and questionable pumps do not include an unknown number of 17th Street bridge pumps, which represent a total of 1600 cfs at that site. They are harder to get a handle on. The Corps is doing the repairs themselves, and they are not as visible to the public.
FOIA difficulties
And speaking of visibility, that leads me to how difficult getting this information has been. I have been forced to rely on reporter friends and acquaintances as primary sources when tracking the repairs, because the Corps failed at the time to reveal almost anything about the work. Fortunately, those friends and acquaintances are wonderful folks. Without the yeoman work of my good friend and award-winning reporter Karen Gadbois and her editor Steve Beatty of The Lens, as well as my good friend (and also award-winning reporter) Molly Peterson of KPCC-FM, this post would have been impossible, and this information would have been lost completely. It is grass-roots, on-the-ground reporting like this that truly makes good journalism, and I'm in their debt for their assistance.
However, it should not take sustained investigative reporting efforts and FOIA requests to find out the extent of damage and the status of repairs to critical, taxpayer-funded flood protection works - the Corps should be publicizing it. How can people judge their level of risk (hint - it's quite high) if they don't know the Corps' pumps are rusting to bits?
And while the fact-gathering on the ground was great, the FOIA process was terrible and grueling. My initial request for the 2009 repair information was filed in September, 2009. I had to wait until September because the government's central public contracting database, FPDS-NG, has a 90 day delay on displaying all Defense Department contract actions. So stuff that happens in June - like the 2009 repairs - isn't made public until September. I had to wait until I had the exact contract numbers and dates to make sure the Corps wouldn't worm out of my requests; if you don't craft them extremely specificly, you don't get anything. The Corps New Orleans District General Counsel's office does not volunteer anything for a FOIA request - you have to already know what you're asking for before you ask for it.
In response to that September, 2009 request, I did not receive contract documents until January, 2010. I then filed a second round of requests to get Conhagen's repair reports associated with that contract work, and did not receive a response to that (the Conhagen report on the repairs to 17th Street E5 and E7) until March. I didn't receive any other repair reports on the rest of the 2009 work until July, 2011, and never got emails associated with the first Conhagen report I got.
It didn't get any better when it came to 2010's round of repairs. After the Fox 8 report in March, 2010, when I repeatedly asked for everything related to the 2010 pump work, I had to wait another two months just to get task order #4 to the first Healtheon contract, covering the repairs to just two pumps - 17th Street pumps W5 and W6. Thanks only to all the photography and fact-gathering displayed above, we knew there was MUCH more contracting and repair activity in 2010 than just the work on those two pumps. I received a few additional documents on June 11, 2010. The remainder of my requested documents arrived over a year later, in July, 2011.
With all these delays, I concluded the Corps withheld as much information as possible on these repairs, because they knew how embarrassing they are. They didn't issued a single press release after May, 2009 (in which they only obliquely referred to the repair work, never using the word "corrosion"), they didn't mentioned the repairs once at any public meeting related to the gates and pumps, and they refused my own repeated requests for comment in connection with this latest series of posts.
Coming up next
At some point, someone in the mainstream media will pick up on this story, and the Corps will be forced to say something (because, you know, no one reads blogs). It'll probably be total spin. I suspect that may come about after my next post, in which I plan to detail the approximately 2000 gallons of oil (at least) spilled at the gates sites over the last four years, and how much of it in the last two years has simply gone unreported by the Corps to the state and federal authorities. That's another set of FOIA requests unfulfilled, at least by the Corps, which I will also detail. Fortunately, other agencies have been far more forthcoming in responding to my document requests. For now, I'll give you the list of 20 spills I've compiled [Update, 6/9/10: It turns out this list is very incomplete. I've come across 17 more incidents]. Stay tuned.
Karen Gadbois, Steve Beatty, and Molly Peterson contributed to this report
At the time these entries were written, I was mainly relying on photography of the sites by friends to track the movements of pumps in and out of each canal. The Corps had turned over a few documents to me, but the bulk of the knowledge was coming from studying the site pictures.
With the receipt of all my requested documents, we now have the complete picture of the rebuild effort. During the time covered by the post (February through early June, 2010), 10 of the 40 large 60" hydraulic pumps were pulled out in three batches for complete rebuilds. At the time I wrote this entry (May, 2010), I wasn't aware of the complete extent of the work to fix corrosion damage. I knew more carbon steel piping was replaced with stainless steel in the 2010 work than in 2009, but I didn't know the critical internal components including bearings, seals, and Rineer hydraulic motors were also getting changed out. When I discovered in June, 2010 that was the case for the later spring, 2010 repairs, I wrote the entry after this one, "Worse than previously known." But even that knowledge was incomplete.
The complete set of contract documents and reports now shows that bearings, seals, and Rineers were being changed out in the first set of 2010 repairs as well. I include that information below, as well as information on the bearing, seal, and Rineer replacements during the other spring, 2010 repairs. This somewhat blunts the original tone of "discovery" in "Worse than previously known."
More important than changing tones, though, is the finding of such repairs on the first set of pumps pulled out in 2010. That moves that part of the rebuilding chronology back further than previously known, and shows how quickly these parts were rendered nonfunctional by the brackish water in Lake Pontchartrain, an important consideration with the installation of the permanent pumps intended to replace these hydraulic units now delayed until 2015.
Remarkably, many of the pumps at the three sites as of this writing - late July, 2011 - have still not been pulled out, over two years after these first extensive replacements. They have been sitting in the salty water of the canals for five years now, corroding into uselessness.
So we've detailed the repair work done on the rusty pumps last year, and the Corps decisions in 2006-07 which led to that work. The repair work is being funded with dollars that were supposed to go toward placement of a permanent pumping system at the lakefront. Instead, they're fixing their three-year-old screwups.
March 16, 2010
The first (and only public) inkling there was repair work happening in 2010 was from a TV news story about those permanent pumps: Fox 8's March 16, 2010 piece about the signing ceremony for the permanent pump station Project Partnership Agreement between the state and the Corps, held at the 17th Street site. Fox 8 was one of three media outlets there, along with WWL-TV (story here) and the Times-Picayune (story here). Unmentioned by any of the stories was the fact the document had actually been signed four days previous to the ceremony.
But the Project Partnership Agreement is beside the issue. What came out of the Fox story was the first real-time footage showing the repair work to the pumps. Fox 8 got two shots of 60" 17th Street pump W5 being placed back in the platform after its earlier removal for repairs. Here's a screencap from the first shot:
You can see the "W5" label on the elbow, clearly identifying this pump.
Here's another shot of W5 going back into the structure. The new zinc anodes on the pump's suction bell installed as part of the repairs are clear in this shot.
This March 16, 2010 replacement was the end of the work on $238,846.95 task order #4 of the first Healtheon contract, which involved 17th Street pumps W5 and W6. This task order, issued February 1, 2010, was a watershed in the pump rebuild effort. Whereas the pumps pulled out in 2009 only had piping on their insides replaced with stainless steel...
...while the piping outside the pumps and the oil coolers were left as carbon steel and painted black, as shown in these photos from the Conhagen report on the repairs to 17th Street pumps E5 and E7 under task order #3 to the first Healtheon contract...
...this time around, not only was the internal piping changed out for stainless, so were the oil coolers and the external piping. That new external stainless steel piping can be seen in the Fox 8 footage:
That is, for the 2010 repairs to 17th Street pumps W5 and W6 under task order #4, every bit of piping touching the water or in the splash zone was upgraded to 316L stainless, even the oil coolers.
Here's a summary of the differences between the 2009 and 2010 repairs:
Piping internal to pump:
2009: 316L stainless steel
2010: 316L stainless steel
Piping external to pump:
2009: Carbon steel
2010: 316L stainless steel
All pipe fitting fasteners:
2009: Grade 8 (a medium carbon alloy steel)
2010: 316L stainless steel
And, the oil coolers:
2009: sandblasted and weld repaired, but original carbon steel remained
2010: completely replaced with new 316L stainless steel
Here's one of the pumps with the new stainless oil coolers and external piping on the Conhagen shop floor, immediately before its March 16, 2010 reinstallation:
Per standard practice, the stainless steel is left unpainted, appearing a silvery gray. That is why the new piping and oil coolers are lighter in color than the painted portions of the pump, as also shown in the Fox 8 pictures of W5 above.
And while the Fox 8 video tipped me off to the expanded work on the piping, it wasn't until the July, 2011 receipt of other contracting documents that we could know just how extensive the work was on W5 and W6.
One of those documents allows us to see the condition of the pumps when they went to Conhagen's shop in early February, 2010. The Conhagen repair report for 17th Street pumps W5 and W6 shows pumps corroded to the point of failure just two and a half years after being accepted by the Corps.
This picture explicitly tells the story of why the external piping had to be changed out:
That hole is on one of the oil coolers, which is on the exterior of the pump:
We already knew about the external piping changes. The repair report shows that stuff inside the pump, subject to the same corrosion from brackish water, had also rotted. Here's one of the Rineer hydraulic motors:
And even further into the innards, sections of the pump mechanism which had multiple seals protecting them from water showed obvious evidence of water infiltration. Here is the area around the shaft and bearings:
From the amount of rust inside the bearing housing, as well as the significant discoloration along the shaft, it is clear the seals completely failed. This would have exposed the bearings and shaft to salt water, causing failure of the pump. And there is no doubt similar damage had been done to the seals on the Rineer motor, damaging it as well.
So it makes sense that $23,511.72 modification to task order #4 on the first Healtheon contract was issued to address this issue. In addition to the original expansion of work to include the replacement of the external piping and oil coolers, this modification further expanded the rework to include bearing, seal, and Rineer motor repairs and replacements:
"Scope of Work in addition to original Task Order.
For Pumps – 17th St. 5W and 6W
TAKE DELIVERY OF EQUIPMENT AS FOLLOWS:
- Pick up Equipment at Job Site With our Truck
- Develop Complete Job Work Scope
MANUFACTURE/FURNISH THE FOLLOWING:
- Replace Radial Bearing
- Replace Hydraulic Motor (Customer Furnished)
- Replace Thrust Bearings (Customer Furnished)
- Provide replacement parts as required to recondition Mechanical Seals (2 Each)
REPAIR THE FOLLOWING:
- Remove All Cooling Lines And Hoses
- Remove Hydraulic Motor
- Disassemble And Inspect Motor
- Remove Pump Shaft And Bearings And Inspect
- Change Out Pump Shaft Bearings
- Clean All Parts For Assembly
- Reinstall Pump Shaft And Bearings
- Reinstall Hydraulic Motor
- Reinstall Lube Lines And Hoses
- Reinstall Seals And Impeller
- Retest System For Leaks
- Retouch Up Tar set Disturb During Disassembly And Reassembly
- Prep For Shipping
PREPARE FOR SHIPPING AS FOLLOWS:
- Bring Pump Back To 17th Street
- Retest Pump"
This modification was issued April 27, 2010, over a month after the pumps went back in the water. Considering the relatively small dollar value, this modification appears to be backdated. That is, if it represented new work - which would have involved crane work and taking the pumps back to Conhagen's shop for a second visit - it would have been valued much higher. A final September, 2010 cost reconciling brought the final cost of the work on this task order down by $10,263.75 to $252,094.92, indicating the only crane work was that paid for in the intial task order, and that the first modification was indeed only accounting for extra work done in the shop.
So when W5 and W6 went back in the water in March, 2010, they had essentially been completely remanufactured, with only the cast stainless steel impellers and the main carbon steel housings (which themselves had to have significant weld repairs) remaining from when the pumps were originally accepted by the Corps in September, 2007. That is an extraordinarily short service life, and it pointed toward the extensive threat posed by corrosion to all the other hydraulic pumps still sitting in the salty water, corroding away. That's why pumps started getting yanked in bulk for the rest of the 2010 spring.
Last week of March, 2010
Through 2009 and most of 2010, it was easy to tell when any of the 60" pumps got pulled. The Corps' had a crane contractor lift the pump and the elbow atop it in one piece. They unbolted the elbow and left it at the site, while the pump was trucked offsite to a shop. The hole left in the piping on the structure was obvious.
Here's an example from October, 2006, when I took pictures at the Orleans Avenue gates. All five of the pumps from the east side of the site had been removed the previous month due to internal and external oil leaks. They had been placed up on the deck, and the elbows were on the levee next to them:
It was pretty easy to see that the pumps were out:
So to see if pumps are out at any of the sites, all we had to look for are elbows on the ground and missing elbows on the structure. Soon after the March 16, 2010 replacement of 17th Street pumps W5 and W6, those telltale signs appeared elsewhere.
At 17th Street, pumps E1 and E2 were pulled in mid-March:
At London Avenue, pumps E3 and E4 were pulled at the same time:
This work was carried out under $632,308.77 task order #2 of the second Healtheon repair contract, issued March 19, 2010. These repairs followed the same pattern as that of the prior work on 17th Street pumps W5 and W6. The complete replacement of all carbon steel piping and hardware, along with pump housing weld repairs, was covered under the original task order, and a modification to task order #2 (worth $65,910.82 and again postdated, this time to April 26, 2010) called for replacement of all the bearings, seals, Rineer motors on all four pumps.
We can see why such extensive work was necessary in the pages of the repair reports from Conhagen for these four pumps. Conhagen wrote a report for 17th Street pumps E1 and E2 and a separate report for London Avenue pumps E3 and E4. According to the photos in those reports, the pumps were in bad shape when they came out of the water.
The oil coolers and external piping were covered with rust and marine growth. Here's 17th Street pump E1:
And here's 17th Street pump E2:
The London Avenue pumps were just as bad. Here's London Avenue pump E3:
And London Avenue pump E4:
This particular view shows the severe corrosion buildup on the piping which was normally submerged when compared to the piping which stayed above the waterline.
The interior guts didn't fare any better. Here's the inside of 17th Street pump E2:
These are vertical pumps. The picture above is taken looking down toward the pump's inlet. So oil would have leaked out of the Rineer toward the impeller, away from the viewer.
The damage inside London Avenue pump E4 is nearly identical, with rust everywhere:
As with 17th Street pumps W5 and W6 earlier in 2010, there was also evidence of water infiltration into the parts of the mechanism that were supposed to be sealed. Here's part of the drive of London Avenue pump E3 with its cover removed:
This should not be showing any signs of rust, but there they are.
There was substantial corrosion damage to the main pump housings as well. At the very bottom of the pump is the suction bell, the tapered portion of the pump which funnels water up to the impeller. The funnelling effect is assisted by tapered vanes. The suction bell is always submerged, and the brackish water took its toll. Here's 17th Street pump E2's suction bell:
The single pair of anodes on that same pump had long stopped working by the time it came out:
So had the 17th Street pump E1 anodes:
All these problems were addressed in the repairs. The carbon steel coolers and piping were replaced with stainless steel. Here's London pump E4, showing a typical cooler and piping replacement:
They also got new Rineer hydraulic motors, as shown on London Avenue pump E4:
Remember those rusty bits on the internal drive assembly? They were completely replaced (photo shows 17th Street pump E1):
And here's that same shaft assembly - upside down on a table, with a cleaned up shaft - just before it was placed in the bearing housing shown above:
Note the shiny new bearings.
So all four of these pumps got the full rebuild treatment during their stay in Conhagen's shop in March and April. According to acceptance testing records attached to each report, 17th Street pumps E1 and E2 went back in the water April 13, 2010. and London Avenue pumps E3 and E4 went back in April 16, 2010. Here's pictures of both sets back in their places:
The main 60" pumps were not the only ones getting attention in early 2010. Some of the fourteen 42" bridge pumps were coming out at 17th Street too. The March 16, 2010 Fox 8 report had captured evidence of that. The Corps had taped off the area around the bridge pump drives and there were hydraulic oil hoses up on the deck; they had been disconnected from one of the bridge pumps, possibly #2 or #3:
Also, the Corps' rental crane - normally kept off the deck unless needed - was parked looming over the gathered guests. From its position with its boom above the midpoint of the structure, it appeared there was work happening on bridge pump #6, #7, or #8:
Additional evidence of work on the bridge pumps comes from the Corps putting up scaffolding around to disconnect the hoses and unbolt the pumps from their elbows. As of mid-March, it looked like at least bridge pumps #7 and #8 had been pulled:
Here's a clearer view of the scaffolding:
Bridge pump #3 would come out later in March, but I don't have a good contemporaneous picture of that.
April 15, 2010
By mid-April, more pumps were getting rebuilt. At 17th Street, pumps E6 and E8 came out:
And at London Avenue, pumps W3 and W4 were pulled. This work was carried out under $619,642.17 task order #3 to the second Healtheon repair contract, issued April 12, 2010. The work under task order #3 was similar to that performed on the other six 60" hydraulic pumps earlier in 2010, with the major effort in replacing all carbon steel piping, oil coolers, and piping hardware with 316L stainless steel, as well as replacement of bearings, seals, and Rineer hydraulic motors. The details on the repairs to 17th Street pump E6 and E8 and London Avenue pumps W3 and W4 can be found in the subsequent entry, "Worse than previously known."
April 28, 2010
By late April, E6 and E8 were still out at 17th Street, and were due back in by mid-to-late-May (though that would be delayed until early June).
Back at 17th Street, it looked like they were still working on the same bridge pumps as back in mid-March, nos. 7 and 8:
While we can only see that #7 is definitely out, the scaffolding is still there for #8.
There's evidence that #3 was also still out (scaffolding), but it's not conclusive due to the angle of the photo.
The repairs so far
So the final tally that we know of:
2009 repairs (with only some stainless steel replacement):
17th Street: 60 inch pumps E5, E7, W8, W9, and W10, Bridge pumps #1 and #6
London Avenue: none
Orleans Avenue: none
2010 repairs (
17th Street: 60 inch pumps W5, W6, E1, E2, E6, E8, Bridge pumps #3, #7, #8 (unknown scope of repairs), possibly other bridge pumps
London Avenue: 60 inch pumps E3, E4, W3, W4
Orleans Avenue: none
Or, in graphic form...
17th Street:
London Avenue:
Orleans Avenue:
Let me explain these graphics. The green pumps are the ones I'm pretty positive have gotten as much corrosion repair as possible, i.e. those fixed this year. I'm coming to that conclusion through FOIA results and deduction from which pumps were pulled since February.
The yellow pumps are either those that were repaired in 2009 (and thus still have carbon steel piping) or those pumps for which I don't have adequate documentation (see my comments about FOIA frustrations below). The red pumps are ones that haven't yet been pulled and had their corrosion damage repaired or replaced. All this is based on the best information I could get, which hasn't been a lot, thanks to the Corps New Orleans District.
These number and pictures mean that - by my best estimates - 75%
17th Street: 12 (7 pumps unrepaired and 5 pumps repaired in 2009 with carbon steel piping)
Orleans Avenue: all 10
London Avenue:
(note: the 11 direct drive pumps at 17th Street and 8 direct drive pumps at London Avenue are likely reliable.)
And just a reminder here, these are what one of those unrepaired pumps likely looked like one year ago, after just two years in the water:
Imagine how they look now, after yet another year in the same brackish water without repairs.
As mentioned above, the totals of unrepaired and questionable pumps do not include an unknown number of 17th Street bridge pumps, which represent a total of 1600 cfs at that site. They are harder to get a handle on. The Corps is doing the repairs themselves, and they are not as visible to the public.
FOIA difficulties
And speaking of visibility, that leads me to how difficult getting this information has been. I have been forced to rely on reporter friends and acquaintances as primary sources when tracking the repairs, because the Corps failed at the time to reveal almost anything about the work. Fortunately, those friends and acquaintances are wonderful folks. Without the yeoman work of my good friend and award-winning reporter Karen Gadbois and her editor Steve Beatty of The Lens, as well as my good friend (and also award-winning reporter) Molly Peterson of KPCC-FM, this post would have been impossible, and this information would have been lost completely. It is grass-roots, on-the-ground reporting like this that truly makes good journalism, and I'm in their debt for their assistance.
However, it should not take sustained investigative reporting efforts and FOIA requests to find out the extent of damage and the status of repairs to critical, taxpayer-funded flood protection works - the Corps should be publicizing it. How can people judge their level of risk (hint - it's quite high) if they don't know the Corps' pumps are rusting to bits?
And while the fact-gathering on the ground was great, the FOIA process was terrible and grueling. My initial request for the 2009 repair information was filed in September, 2009. I had to wait until September because the government's central public contracting database, FPDS-NG, has a 90 day delay on displaying all Defense Department contract actions. So stuff that happens in June - like the 2009 repairs - isn't made public until September. I had to wait until I had the exact contract numbers and dates to make sure the Corps wouldn't worm out of my requests; if you don't craft them extremely specificly, you don't get anything. The Corps New Orleans District General Counsel's office does not volunteer anything for a FOIA request - you have to already know what you're asking for before you ask for it.
In response to that September, 2009 request, I did not receive contract documents until January, 2010. I then filed a second round of requests to get Conhagen's repair reports associated with that contract work, and did not receive a response to that (the Conhagen report on the repairs to 17th Street E5 and E7) until March. I didn't receive any other repair reports on the rest of the 2009 work until July, 2011, and never got emails associated with the first Conhagen report I got.
It didn't get any better when it came to 2010's round of repairs. After the Fox 8 report in March, 2010, when I repeatedly asked for everything related to the 2010 pump work, I had to wait another two months just to get task order #4 to the first Healtheon contract, covering the repairs to just two pumps - 17th Street pumps W5 and W6. Thanks only to all the photography and fact-gathering displayed above, we knew there was MUCH more contracting and repair activity in 2010 than just the work on those two pumps. I received a few additional documents on June 11, 2010. The remainder of my requested documents arrived over a year later, in July, 2011.
With all these delays, I concluded the Corps withheld as much information as possible on these repairs, because they knew how embarrassing they are. They didn't issued a single press release after May, 2009 (in which they only obliquely referred to the repair work, never using the word "corrosion"), they didn't mentioned the repairs once at any public meeting related to the gates and pumps, and they refused my own repeated requests for comment in connection with this latest series of posts.
Coming up next
At some point, someone in the mainstream media will pick up on this story, and the Corps will be forced to say something (because, you know, no one reads blogs). It'll probably be total spin. I suspect that may come about after my next post, in which I plan to detail the approximately 2000 gallons of oil (at least) spilled at the gates sites over the last four years, and how much of it in the last two years has simply gone unreported by the Corps to the state and federal authorities. That's another set of FOIA requests unfulfilled, at least by the Corps, which I will also detail. Fortunately, other agencies have been far more forthcoming in responding to my document requests. For now, I'll give you the list of 20 spills I've compiled [Update, 6/9/10: It turns out this list is very incomplete. I've come across 17 more incidents]. Stay tuned.
Karen Gadbois, Steve Beatty, and Molly Peterson contributed to this report
Labels: Rusty pumps
posted by mcbrid35 at
10:54 AM
0 Comments:
Post a Comment
<< Home