Credibility
The Corps internal investigation is the gift that keeps on giving. And, as with any government report, the tasty tidbits are in the appendices. Here's three that I've found so far.
Note that I refer to page numbers by "Adobe page so-and-so" That's the page number in the box at the bottom of Adobe Reader.
1) The hydraulic oil is not environmentally friendly.
Back on September 16, 2006, when there was hydraulic oil spilling everywhere, Colonel Bedey was quoted in the Times-Picayune as saying, "We're using environmentally friendly hydraulic oil."
That may have been true back then. The National Response Center spill report for the September 14, 2006 spill shows the material spilled as "MOBIL 67 BIODEGRADABLE HYDRAULIC OIL." (the 67 is probably a typo; the next nearest ISO grade is 68). Possibly, it was one of the lubricants on this Mobil page, all of which are advertised as environmentally friendly. But with such a paucity of information, I could never narrow it down, or even verify the claim.
Since that time, I've been curious to find out exactly what kind of hydraulic oil is being used, in order to find out if it really is "environmentally friendly." Thanks to the internal report we now know. On Adobe page 100 is a May 7, 2007 memo from Denison to the Corps' Dan Bradley. It calls out the specific brand and type of hydraulic oil. According to the memo, the hydraulic oil is Exxon-Mobil Nuto H 68.
Taking a look at the product information page for Exxon Mobil Nuto H, I can't see anything in there that mentions "environmental friendliness." In fact, it appears on a webpage with a family of products that is notable for its lack of "environmental friendliness." Such a quality is a big marketing tool, and if it were part of the makeup of the oil, Exxon Mobil would definitely mention it. It appears the Corps and MWI are no longer using environmentally friendly oil. Someone should tell Colonel Bedey.
2) Water in the hydraulic oil
Buried inside a field report by a contractor about a visit this past February to the hydraulic motor manufacturer, Rineer, is a yummy morsel about the hydraulic oil having water in it.
On Adobe page 88, Corps contractor John Ehlers describes the teardown of three Rineer hydraulic motors. These are three of the motors which were vibrating severely, and which have had stronger springs installed in them since then. This was, for months, the only problem to which the Corps would admit when it came to the floodgate pumps. What we have been told publicly was that springs in the motors were replaced with stronger ones at MWI's rep, Associated Pump & Supply in Houma, LA.
What has not been publicized until now is that the motors were not in as-new condition when they were reinstalled at Associated. Specifically, Mr. Ehlers writes this:
He also wrote in his conclusions section:
So the hydraulic motors were put back together with rusty parts inside them.
But what's more interesting is the fact that at some point between installation of the pumps in the May-June-July timeframe and their initial startup testing during the same period, the hydraulic pipes were contaminated with water.
This has large consequences. The following is just theory, but it's what I think would normally happen in the case of contamination of the hydraulic fluid. There's only one source for water to get into the hydraulic lines. It was there to start with.
You might say, "What if the system had leaks?" A system running at any appreciable internal pressure is not going to allow water in while it's running. And while it's conceivable that water leaked into the system, it seems unlikely to have done so in three separate pump systems in exactly the same concentration (Mr. Ehlers writes of similar "slight" amounts of rust in each motor). So I'm going with the theory that the water was in the pipes to begin with.
As everyone knows, water and oil don't mix. When the pipes arrived on site they were supposed to have been pickled and oiled, according to the specification. The pickling process involves dipping the pipe in vats of acid and other chemicals to passivate it (I think - I'm a little hazy on pickling). Then I believe the pipes are supposed to be coated in oil and then capped and shipped. Theoretically (I think), the lines should be flushed completely before startup to ensure there was no debris or other crap in there before the final charging of hydraulic oil was placed. Feel free to correct me on this.
I'm betting that the lines were either not pickled and oiled properly or not flushed properly. What would this lead to? Once it was determined there was water in the oil (it would have showed itself in other ways during startup operations), all of the pipes would have had to have been completely flushed and refilled with new hydraulic oil. The change in oil described above - from the "environmentally friendly" grade to the Nuto H grade - could be explained by this. This work would have been at great expense and would have been backcharged to MWI if they didn't do it themselves.
All of the above is theory, but I like to think of it as deductive reasoning based on the available facts. Anyone who has evidence to back up or dispute my theory about flushing of the lines is encouraged to contact me.
3) When was it decided to raise the hydraulic reservoirs, the so-called "critical fix?"
The critical fix that remains undone is the raising of the hydraulic reservoirs. With that modification, hydraulic oil would flow downward from reservoirs into the Denison hydraulic pumps, instead of having to be sucked out the top of the reservoirs. Pulling the oil out the top of the tanks guaranteed entrainment of air in the lines. Denison pumps do not tolerate air very well. They were failing like crazy in the factory with the lines coming out of the tops of the reservoirs. And apparently, they continued to fail after they arrived on the sites (from Adobe page 24):
This single sentence is huge in its implications. It totally rebuts the Corps' entire defense that they were installing "some capacity instead of none." Colonel Bedey and others have repeatedly used this line with the press, with the public, with policymakers in Congress, and even got the GAO to buy it.
That defense is BS. How could you have any capacity if the most critical part of your system was continuing to fail even after you installed it? It is very important that we find out the extent of the damage to these Denison pumps after their arrival on site. 74 (two on each of the 34 skids) were installed across the three canal sites. How many were determined to have failed after they were installed in New Orleans? 3? 30? 60? Knowing that would blow a giant hole in the Corps' credibility, because it would mean that there really wasn't any capacity to begin with - and there probably still isn't any, because the reservoirs haven't been raised in order to put the kibosh on the air-induced failures.
Yes they can turn them on for a little while, but how about for six, or eight, or 12, or 24 hours? What happens when air gets in the lines and the Denison pumps fail and send their bits and pieces down the line to the Rineer motors to gobble up? What then?
Anyway, as far as the time that the real solution (raising the reservoirs) was reached, it appears to be in an email found on Adobe page 114. It was from Jim St. Germain and addressed to nearly everyone on the project. It is dated June 18, 2006 and says,
Well, that work never took place, at least on the original 34 pumps. The roofs were put on the buildings and the fences were erected. Somehow, the reservoirs were raised on the extra six pumps puchased long after the original 34.
This June 18, 2006 email shows that Jim St. Germain - the guy responsible for driving the project - and the manufacturer - MWI, who apparently proposed the idea - had a clear appreciation of what needed to be done. Heck, St. Germain's the author of the email. Yet it took over a year to get this work going (we're still not sure what its status is), never mind how the public was in the dark about it. Why didn't it get done? Unfortunately, the report is silent on the why's of many things, including this mystery.
And it's not like they were pulling this solution (raising the reservoirs) out of the air. Two weeks earlier, the New Orleans Corps had discussed how they didn't like the quick fix of evacuating the hydraulic lines during startup. This comes from a May 31 email from Jim Bartek to Jim St. Germain, Dan Bradley, and Steve Farkas (but oddly, not Maria Garzino, the Pump Team leader on site):
The priming procedure referred to by Mr. Bartek over a year ago is the same procedure under which the Corps is now operating the pumps.
Mr Bartek goes on to presciently rebut Colonel Bedey's current assertions that the field testing the Corps is doing today (turning the pumps on and then turning them off in less than hour with low water in the canals) is adequate:
Like I said, this report is the gift that keeps giving. I'll be back with more analysis in future posts.
Note that I refer to page numbers by "Adobe page so-and-so" That's the page number in the box at the bottom of Adobe Reader.
1) The hydraulic oil is not environmentally friendly.
Back on September 16, 2006, when there was hydraulic oil spilling everywhere, Colonel Bedey was quoted in the Times-Picayune as saying, "We're using environmentally friendly hydraulic oil."
That may have been true back then. The National Response Center spill report for the September 14, 2006 spill shows the material spilled as "MOBIL 67 BIODEGRADABLE HYDRAULIC OIL." (the 67 is probably a typo; the next nearest ISO grade is 68). Possibly, it was one of the lubricants on this Mobil page, all of which are advertised as environmentally friendly. But with such a paucity of information, I could never narrow it down, or even verify the claim.
Since that time, I've been curious to find out exactly what kind of hydraulic oil is being used, in order to find out if it really is "environmentally friendly." Thanks to the internal report we now know. On Adobe page 100 is a May 7, 2007 memo from Denison to the Corps' Dan Bradley. It calls out the specific brand and type of hydraulic oil. According to the memo, the hydraulic oil is Exxon-Mobil Nuto H 68.
Taking a look at the product information page for Exxon Mobil Nuto H, I can't see anything in there that mentions "environmental friendliness." In fact, it appears on a webpage with a family of products that is notable for its lack of "environmental friendliness." Such a quality is a big marketing tool, and if it were part of the makeup of the oil, Exxon Mobil would definitely mention it. It appears the Corps and MWI are no longer using environmentally friendly oil. Someone should tell Colonel Bedey.
2) Water in the hydraulic oil
Buried inside a field report by a contractor about a visit this past February to the hydraulic motor manufacturer, Rineer, is a yummy morsel about the hydraulic oil having water in it.
On Adobe page 88, Corps contractor John Ehlers describes the teardown of three Rineer hydraulic motors. These are three of the motors which were vibrating severely, and which have had stronger springs installed in them since then. This was, for months, the only problem to which the Corps would admit when it came to the floodgate pumps. What we have been told publicly was that springs in the motors were replaced with stronger ones at MWI's rep, Associated Pump & Supply in Houma, LA.
What has not been publicized until now is that the motors were not in as-new condition when they were reinstalled at Associated. Specifically, Mr. Ehlers writes this:
"The writer notes that the coil spring material is 'piano wire', which is a high carbon alloy. Any high carbon alloy is a candidate for rust when exposed to water and warm temperature and other contamination. The rust evidence on the timing plates suggests that there was some water contamination within the motor. Since the motor seals were all good, it is probable that the water was introduced by hydraulic fluid which had been exposed to water or moisture."
He also wrote in his conclusions section:
"All 3 motors had slight signs of rust on the timing plates. There was evidence that the hydraulic oil was less than perfect, in that a contaminant, most likely water, was present in the hydraulic oil."
So the hydraulic motors were put back together with rusty parts inside them.
But what's more interesting is the fact that at some point between installation of the pumps in the May-June-July timeframe and their initial startup testing during the same period, the hydraulic pipes were contaminated with water.
This has large consequences. The following is just theory, but it's what I think would normally happen in the case of contamination of the hydraulic fluid. There's only one source for water to get into the hydraulic lines. It was there to start with.
You might say, "What if the system had leaks?" A system running at any appreciable internal pressure is not going to allow water in while it's running. And while it's conceivable that water leaked into the system, it seems unlikely to have done so in three separate pump systems in exactly the same concentration (Mr. Ehlers writes of similar "slight" amounts of rust in each motor). So I'm going with the theory that the water was in the pipes to begin with.
As everyone knows, water and oil don't mix. When the pipes arrived on site they were supposed to have been pickled and oiled, according to the specification. The pickling process involves dipping the pipe in vats of acid and other chemicals to passivate it (I think - I'm a little hazy on pickling). Then I believe the pipes are supposed to be coated in oil and then capped and shipped. Theoretically (I think), the lines should be flushed completely before startup to ensure there was no debris or other crap in there before the final charging of hydraulic oil was placed. Feel free to correct me on this.
I'm betting that the lines were either not pickled and oiled properly or not flushed properly. What would this lead to? Once it was determined there was water in the oil (it would have showed itself in other ways during startup operations), all of the pipes would have had to have been completely flushed and refilled with new hydraulic oil. The change in oil described above - from the "environmentally friendly" grade to the Nuto H grade - could be explained by this. This work would have been at great expense and would have been backcharged to MWI if they didn't do it themselves.
All of the above is theory, but I like to think of it as deductive reasoning based on the available facts. Anyone who has evidence to back up or dispute my theory about flushing of the lines is encouraged to contact me.
3) When was it decided to raise the hydraulic reservoirs, the so-called "critical fix?"
The critical fix that remains undone is the raising of the hydraulic reservoirs. With that modification, hydraulic oil would flow downward from reservoirs into the Denison hydraulic pumps, instead of having to be sucked out the top of the reservoirs. Pulling the oil out the top of the tanks guaranteed entrainment of air in the lines. Denison pumps do not tolerate air very well. They were failing like crazy in the factory with the lines coming out of the tops of the reservoirs. And apparently, they continued to fail after they arrived on the sites (from Adobe page 24):
"The hydraulic oil pumps, since the delivery to the site [emphasis mine] have been removed and inspected by the manufacturer to determine premature failures, similar to the ones that were happening at factory testing of the pumping units."
This single sentence is huge in its implications. It totally rebuts the Corps' entire defense that they were installing "some capacity instead of none." Colonel Bedey and others have repeatedly used this line with the press, with the public, with policymakers in Congress, and even got the GAO to buy it.
That defense is BS. How could you have any capacity if the most critical part of your system was continuing to fail even after you installed it? It is very important that we find out the extent of the damage to these Denison pumps after their arrival on site. 74 (two on each of the 34 skids) were installed across the three canal sites. How many were determined to have failed after they were installed in New Orleans? 3? 30? 60? Knowing that would blow a giant hole in the Corps' credibility, because it would mean that there really wasn't any capacity to begin with - and there probably still isn't any, because the reservoirs haven't been raised in order to put the kibosh on the air-induced failures.
Yes they can turn them on for a little while, but how about for six, or eight, or 12, or 24 hours? What happens when air gets in the lines and the Denison pumps fail and send their bits and pieces down the line to the Rineer motors to gobble up? What then?
Anyway, as far as the time that the real solution (raising the reservoirs) was reached, it appears to be in an email found on Adobe page 114. It was from Jim St. Germain and addressed to nearly everyone on the project. It is dated June 18, 2006 and says,
"A meeting was held with MWI to discuss weld repairs and the Denison hydraulic motor. MWI has proposed to redesign the hydraulic tank and hoses to form a flooded suction for the two Denison hydraulic pumps. MWI will raise the hydraulic fluid tank and reposition the hoses to the Denison pump. This fix should prevent air from entering the motor and eleminate [sic] the need for the priming valve. MWI will also remove the suction strainer.
The proposal says that MWI will perform the work at no cost to the Government provided Government onsite contracts load and inload the skids. They propose to take the engines skids at 17th that are not installed and the spares first and then swap engins [sic] at other sites to try to avoid pump outages. They estimate a week to do a set (6 to 10) units.
This presents some problems. First, the preliminary time frame is not going to allow completion without impacts on scheduled pump completion dates. Second, once the fencing on the engine deck and building is installed it may be difficult to remove the engines. We should immediate[ly] have MWI retrofit the engines at 17th that are not installed. Coordinate with the three contractors on swap outs to minimize impacts to buildings. Avoid any impacts to pumping capacity. Have MWI develop a plan to retrofit in place."
Well, that work never took place, at least on the original 34 pumps. The roofs were put on the buildings and the fences were erected. Somehow, the reservoirs were raised on the extra six pumps puchased long after the original 34.
This June 18, 2006 email shows that Jim St. Germain - the guy responsible for driving the project - and the manufacturer - MWI, who apparently proposed the idea - had a clear appreciation of what needed to be done. Heck, St. Germain's the author of the email. Yet it took over a year to get this work going (we're still not sure what its status is), never mind how the public was in the dark about it. Why didn't it get done? Unfortunately, the report is silent on the why's of many things, including this mystery.
And it's not like they were pulling this solution (raising the reservoirs) out of the air. Two weeks earlier, the New Orleans Corps had discussed how they didn't like the quick fix of evacuating the hydraulic lines during startup. This comes from a May 31 email from Jim Bartek to Jim St. Germain, Dan Bradley, and Steve Farkas (but oddly, not Maria Garzino, the Pump Team leader on site):
"Just a few thoughts...It appears to me that MWI is just trying to 'make it work' to get by. Everyone agrees that the Denison pumps are operating in a dry run condition. While the priming procedure described below may work as a one time or short term fix. I feel, and I think Steve agrees, it will not hold up over the long term. There will be entrained air that will percolate out and also air leaking in through the tank etc that will get trapped. They need to look more at root causes than quick fixes of the problems. This could be pump sizing or relocating the suction intake to a more suitable location. Why has this design successfully worked on other pumps but not these units?"
The priming procedure referred to by Mr. Bartek over a year ago is the same procedure under which the Corps is now operating the pumps.
Mr Bartek goes on to presciently rebut Colonel Bedey's current assertions that the field testing the Corps is doing today (turning the pumps on and then turning them off in less than hour with low water in the canals) is adequate:
"These pumps must be field tested with design water levels to ensure acceptable operation."
Like I said, this report is the gift that keeps giving. I'll be back with more analysis in future posts.
posted by mcbrid35 at
9:56 PM
0 Comments:
Post a Comment
<< Home