What MVN thinks of the internal
There were ten technical recommendations in the Corps' internal investigation into the floodgate pumps. Since General Crear ignored almost all of them in his press release cover letter, it fell to the local media to find out their disposition.
Courtesy of WWL-TV, below you will find the New Orleans District's responses to the internal investigation's technical recommendations. I can't verify the truth of anything but part of the first one. The hydraulic reservoirs are being raised (see photos below).
The idea behind raising the hydraulic reservoirs is to pull hydraulic fluid from lower in the reservoirs, preventing entrained air from entering the delicate Denison hydraulic pumps and wrecking them. MWI's original scheme (the one which has been in existence up until last month) was to pull hydraulic fluid off the top of the reservoirs, working against gravity and guaranteeing air would get trapped in the inlet lines to the Denison pumps. This wrecked Denison pump after Denison pump. Considering that the scheme has been in operation up until last month, it is a strong possibility there are now wrecked Denison pumps on the skids, a conclusion reached in the internal investigation (see my commentary on internal recommendation #1 below).
The new scheme raises the reservoirs and reworks the piping and hoses between the reservoir and the Denison pumps, with the hydraulic fluid coming out of a point lower on the reservoir (hopefully the very bottom, but that's not clear).
They've started raising the reservoirs with Orleans Avenue for some unknown reason (I would have thought 17th St and London would be far more urgent, since those floodgates gates are intended to be dropped long before Orleans). All of the Orleans Avenue reservoirs appear to be raised.
17th Street does not have raised reservoirs on any of the original 12 pumps, as of June 20 (the six newer pumps came with raised reservoirs last year). Considering how much importance the Corps puts on 17th Street - every major press availability is held there, and it's the only site the Corps took the GAO when they visited, not that it mattered to the GAO - you'd think they would have started there.
So how does one tell if the reservoirs have been raised? It's fairly simple. First, let's get oriented to where the hydraulic reservoirs can be found on the drive skids. We'll use a picture from Dennis Strecker's April 18, 2006 report, which is included in the internal investigation (Adobe page 136):
The "day tank" is another name for the fuel tank. One can see the hydraulic reservoir is located inboard - or behind - the fuel tank. One can also see that the hydraulic reservoir is shorter than the fuel tank. This makes for a handy reference to determine whether the reservoirs have been raised. If the top of the reservoir is above the fuel tank, they've gone up. If the fuel tank is higher than the reservoir, they haven't been raised.
Here's a "before" picture of a drive skid at the east drive shed at Orleans Avenue:
As you can see, the top of the fuel tank was above the top of the reservoir. Thus, the reservoir was at its original, as-delivered height on March 25, 2007.
Now look at an "after" shot of the same skid:
The top of the reservoir is above the top of the fuel tank. The reservoir is now sitting up on a new set of approximately foot-high legs. This work was done in late May or early June.
All five drive skids on the east side of Orleans Avenue had gotten this treatment as of June 20th:
I don't have a picture of it, but I did see that the five skids on the west side of Orleans also had their reservoirs raised as of June 20, 2007.
Here's a picture of the actual legs being used to raise the reservoirs, along with hoses that have been removed from skids:
Moving over to London Avenue...
Here's the three southernmost skids in the west drive shed at London Avenue, none of which had their reservoirs raised at the time of this photo:
Here's the other three skids in that shed (sorry about cutting off the skid on the right):
How do we know the reservoirs are unraised in this shed? By examining the relative heights of the fuel tanks and the reservoirs. Here's a detail from the first London Avenue shot above, which shows the reservoir unraised:
All of the rest of the London Avenue - west drive skids were exactly the same as of June 20, 2007.
I don't have pictures of the London Avenue - east drive skids, but I did see they were also unraised as of June 20, 2007.
Finally, let's do the same treatment on 17th Street. First, the east drive platform:
Below is a detail shot of the drive skid on the left. While it's a little hard to make out (it's actually easier to see in the long shot above), you can see that the top of the fuel tank remains above the top of the reservoir:
Here's the best picture I could get of the 17th Street west drive shed:
And here's a detail, showing that the reservoirs hadn't been raised in that shed as of June 20, 2007 (I'm using the "daylight" argument):
My camera doesn't pick up as much detail as the naked eye, so you'll have to trust that I could clearly see the reservoirs in that shed were still at their original heights that day.
The internal recommendations follow, along with the New Orleans' District's responses. I've included a little commentary inside brackets. "MVN" is the Corps code for the New Orleans District.
"1) Flooded suction for all hydraulic oil pumps to prevent equipment failures and unsafe priming by operating personnel. Simultaneously with this action is the removal of the Denison Hydraulic pumps to inspect for any failed or in process of failing components. Any unsuitable components shall be replaced."
MVN responds: In process.
[It's important to note the second sentence of this recommendation. The internal team believes there could be "failed or in process of failing" components inside the Denison hydraulic pumps - even today. This is pretty huge, since the Denison pumps are the heart of the drive skids, and their failure leads to the failure of a water pump. There are 80 Denison pumps installed at the three canals, along with another six installed on spare drive skids. That's 86 units that are still in jeopardy of going kablooey.
Such Denison pump failures happened all the time during the MWI factory testing, and they continued during the initial field testing in the spring of 2006 (note: not complete field performance testing - which the Corps has no intention of conducting - just turning the pumps on to see if they worked).
We know the Denison hydraulic pump failures continued during the field testing because the internal investigation folks included data on field tests in Attachment #10 to the internal report. It's only two pages, but's it's sure disturbing. Here's the results for the tests of just two drive unit (DU)/pump assembly (PA) combos at Orleans Avenue in May and June of 2006:
DU 8845/PA 4590 (test date: 5/31/06)
Total run time: 25 minutes
Component failure: Filter side Denison hydraulic pump
DU 8842/PA 4581 (test date: 5/24/06)
Total run time: none - static pressure test only
Component failure: Filter side Denison hydraulic pump
DU 8842/PA 4581 (test date: 6/1/06)
Total run time: none - 20.2 minutes
Component failure: Control panel side Denison hydraulic pump
What's interesting about DU 8842 is that it was the skid that was used for the only 24 hour "endurance" test performed during the MWI factory testing in the third week of April, 2006. Except that test had the skid running at 1000 psi, less than a third of the pressure the drive was supposed to generate in the field. But after 24 hours of running on cruise control (and a brief repair to the gear oil circulation pump), 8842 was good enough to send out to Orleans Avenue, where we can see it promptly broke - twice - after less than an hour of runtime.
None of the information about the poor performance of the pumps in the field after they were installed was published in the media at the time it happened. Articles from May, June, July, and August of 2006 show no acknowledgement by the Corps of any problems on the MWI pumps, until the vibration problems emerged in mid-August. Those vibration problems are also documented in those Orleans Avenue tests three months previously (more withholding of information).
The same withholding by the Corps goes for public meetings attended by in-the-know Corps New Orleans officials such as Jim St. Germain, Colonel Bedey, Colonel Wagenaar, and Walter Baumy, some of which I have on tape. No one in the public - and I mean NO ONE - was clued in to the fact that drive skids were going kablooey and pump assemblies were vibrating practically on a daily basis throughout the summer of 2006. That's a horrible way to treat the people of New Orleans, but it's a great way to treat a manufacturer that doesn't want bad press.
Now, a year later the Corps is doing the exact same thing, because it's the exact same people - St. Germain, Bedey, Bradley. They run the pumps and drive units for 45 minutes, assumedly at 3000 psi (though we have no idea, and the freaking media won't ask), get worthless flow numbers from flowmeters that exemplify GIGO, and then they say, "Everything's fine!" Everything is not fine. Qualified, non- New Orleans individuals in the Corps believe there are still vital pieces of equipment primed for failure on these skids today. That's serious.
We're talking about protecting peoples' lives here. It's not a game. This is why comprehensive, extensive testing is required of this equipment, not the crappy crud with which MWI and the New Orleans District bamboozles the local media. The contract says so, reality says so, the public says so, and common decency says so. The only people that say it isn't are the New Orleans District and MWI.]
"2) Recommend that a certified hydraulic systems inspector, per ASME B31.1 Power Piping, inspect the piping system and certify that the hydraulic piping system is safe to operate for the intended use. The inspector may add operating requirements due to the reduced factor of safety. Any additional operating requirements must be included in the training of, and provided to any pump system operators."
MVN responds: They analyzed...said not true.
[This one cracks me up, and kind of makes me cry a bit. The Corps is, I believe, relying on a static test of the piping at 4500 psi to make this statement, as well as some really crummy calcs included in the internal report (found on the last two pages).
Such a static test is inadequate to determine if the pipes are suitable for dynamic conditions lasting hours on end. Considering how much the piping vibrates when the pumps are run, and how rusty the pipes are now, and how undersized they are according to code, it's clear the New Orleans District and its Metairie-based consultants at NY Associates, Linfield Hunter Junius, and URS likely have no idea what they're talking about when it comes to the pipes failing. They're playing with peoples' lives for God's sake. Are they really willing to leave clearly deteriorating equipment in the field for use during a hurricane?]
"3) Evaluate to determine if the automatic clutch system should be required to be installed to reduce shockwaves within the hydraulic piping system as required per contract documents to provide a smooth, soft start."
MVN responds: At first said don't think need to be done....then said in process.
[If they think the piping is okay, then why are they looking at the clutch to prevent damage to the pipes?]
"4) Information on welds that have been previously repaired by the Government and other welds which were not repaired should be reviewed. Further review is needed to determine whether additional welds should be repaired."
MVN responds: Done.
[We will likely never see information about this.]
"5) Provide mechanical computations for flange bolts to insure they are sized properly to handle the weight of the components."
MVN responds: Done.
[We will likely never see these calculations.]
"6) Remove inlet strainers per Denison recommendations."
MVN responds: Evaluating.
[How can they still be "evaluating" this one? All the information was laid out over a year ago on this issue, as shown in the internal report and the shop inspection records. Denison themselves says not to use strainers on the inlets to their pumps. What is there to evaluate?]
"7) Installation of the hydraulic cones in London and Orleans, along with vortex suppressors as determined by the Model Studies conducted at ERDC. The cones are currently being installed at London Avenue Canal."
MVN responds: Done.
"8) Each pump should be provided with a hydraulic system monitoring device (as required by the contract) to allow diagnosing hydraulic system behavior even while the pump is submerged. If a monitoring device is not provided, then a credit to the Government is due."
MVN responds: Evaluating.
[After all the hydraulic system problems, you'd think this one would be a no-brainer. You'd think...]
"9) Calibrate and locate the ultra-sonic equipment in the proper locations as required by the manufacturer with direction as required by Dr. Maynord at ERDC. This is to determine the correct amount of flow capacity for the pump at all three Canal sites."
MVN responds: In process.
[As I mentioned a few posts back, there's very little reason to believe the flowmeters will ever work. In ERDC's own April 2007 tests at London Avenue, they weren't used to find the capacity of the pumps. Instead, ERDC "backed into" the capacity by getting the static head in the pumps and used a derived pump curve (with its own significant scatter errors) to interpolate to the capacity. From an engineering standpoint, this is far less reliable than a direct measurement of the flow. It also subverts the entire floodgate pump control system, which is based on the flowmeters, not some jury-rigged piezometer-and-pump-curve smashup.
This would appear to be another source of a tremendous amount of wasted taxpayer funds - probably millions of dollars. Here we are over a year after the installation of these control systems, and at their most fundamental level, they don't work. What a mess.
This one lands smack at Dan Bradley's feet. He's the electrical engineer on this job, and was responsible for the controls system.]
"10) Have MWI provide manufacturer’s information and shop drawings (including Durst information) as discussed earlier in this report."
MVN responds: Done.
[I have a hard time believing this is "done," since the ERDC folks didn't have MWI's certified information on which to rely just over two months ago.]
Courtesy of WWL-TV, below you will find the New Orleans District's responses to the internal investigation's technical recommendations. I can't verify the truth of anything but part of the first one. The hydraulic reservoirs are being raised (see photos below).
The idea behind raising the hydraulic reservoirs is to pull hydraulic fluid from lower in the reservoirs, preventing entrained air from entering the delicate Denison hydraulic pumps and wrecking them. MWI's original scheme (the one which has been in existence up until last month) was to pull hydraulic fluid off the top of the reservoirs, working against gravity and guaranteeing air would get trapped in the inlet lines to the Denison pumps. This wrecked Denison pump after Denison pump. Considering that the scheme has been in operation up until last month, it is a strong possibility there are now wrecked Denison pumps on the skids, a conclusion reached in the internal investigation (see my commentary on internal recommendation #1 below).
The new scheme raises the reservoirs and reworks the piping and hoses between the reservoir and the Denison pumps, with the hydraulic fluid coming out of a point lower on the reservoir (hopefully the very bottom, but that's not clear).
They've started raising the reservoirs with Orleans Avenue for some unknown reason (I would have thought 17th St and London would be far more urgent, since those floodgates gates are intended to be dropped long before Orleans). All of the Orleans Avenue reservoirs appear to be raised.
17th Street does not have raised reservoirs on any of the original 12 pumps, as of June 20 (the six newer pumps came with raised reservoirs last year). Considering how much importance the Corps puts on 17th Street - every major press availability is held there, and it's the only site the Corps took the GAO when they visited, not that it mattered to the GAO - you'd think they would have started there.
So how does one tell if the reservoirs have been raised? It's fairly simple. First, let's get oriented to where the hydraulic reservoirs can be found on the drive skids. We'll use a picture from Dennis Strecker's April 18, 2006 report, which is included in the internal investigation (Adobe page 136):
The "day tank" is another name for the fuel tank. One can see the hydraulic reservoir is located inboard - or behind - the fuel tank. One can also see that the hydraulic reservoir is shorter than the fuel tank. This makes for a handy reference to determine whether the reservoirs have been raised. If the top of the reservoir is above the fuel tank, they've gone up. If the fuel tank is higher than the reservoir, they haven't been raised.
Here's a "before" picture of a drive skid at the east drive shed at Orleans Avenue:
As you can see, the top of the fuel tank was above the top of the reservoir. Thus, the reservoir was at its original, as-delivered height on March 25, 2007.
Now look at an "after" shot of the same skid:
The top of the reservoir is above the top of the fuel tank. The reservoir is now sitting up on a new set of approximately foot-high legs. This work was done in late May or early June.
All five drive skids on the east side of Orleans Avenue had gotten this treatment as of June 20th:
I don't have a picture of it, but I did see that the five skids on the west side of Orleans also had their reservoirs raised as of June 20, 2007.
Here's a picture of the actual legs being used to raise the reservoirs, along with hoses that have been removed from skids:
Moving over to London Avenue...
Here's the three southernmost skids in the west drive shed at London Avenue, none of which had their reservoirs raised at the time of this photo:
Here's the other three skids in that shed (sorry about cutting off the skid on the right):
How do we know the reservoirs are unraised in this shed? By examining the relative heights of the fuel tanks and the reservoirs. Here's a detail from the first London Avenue shot above, which shows the reservoir unraised:
All of the rest of the London Avenue - west drive skids were exactly the same as of June 20, 2007.
I don't have pictures of the London Avenue - east drive skids, but I did see they were also unraised as of June 20, 2007.
Finally, let's do the same treatment on 17th Street. First, the east drive platform:
Below is a detail shot of the drive skid on the left. While it's a little hard to make out (it's actually easier to see in the long shot above), you can see that the top of the fuel tank remains above the top of the reservoir:
Here's the best picture I could get of the 17th Street west drive shed:
And here's a detail, showing that the reservoirs hadn't been raised in that shed as of June 20, 2007 (I'm using the "daylight" argument):
My camera doesn't pick up as much detail as the naked eye, so you'll have to trust that I could clearly see the reservoirs in that shed were still at their original heights that day.
The internal recommendations follow, along with the New Orleans' District's responses. I've included a little commentary inside brackets. "MVN" is the Corps code for the New Orleans District.
"1) Flooded suction for all hydraulic oil pumps to prevent equipment failures and unsafe priming by operating personnel. Simultaneously with this action is the removal of the Denison Hydraulic pumps to inspect for any failed or in process of failing components. Any unsuitable components shall be replaced."
MVN responds: In process.
[It's important to note the second sentence of this recommendation. The internal team believes there could be "failed or in process of failing" components inside the Denison hydraulic pumps - even today. This is pretty huge, since the Denison pumps are the heart of the drive skids, and their failure leads to the failure of a water pump. There are 80 Denison pumps installed at the three canals, along with another six installed on spare drive skids. That's 86 units that are still in jeopardy of going kablooey.
Such Denison pump failures happened all the time during the MWI factory testing, and they continued during the initial field testing in the spring of 2006 (note: not complete field performance testing - which the Corps has no intention of conducting - just turning the pumps on to see if they worked).
We know the Denison hydraulic pump failures continued during the field testing because the internal investigation folks included data on field tests in Attachment #10 to the internal report. It's only two pages, but's it's sure disturbing. Here's the results for the tests of just two drive unit (DU)/pump assembly (PA) combos at Orleans Avenue in May and June of 2006:
DU 8845/PA 4590 (test date: 5/31/06)
Total run time: 25 minutes
Component failure: Filter side Denison hydraulic pump
DU 8842/PA 4581 (test date: 5/24/06)
Total run time: none - static pressure test only
Component failure: Filter side Denison hydraulic pump
DU 8842/PA 4581 (test date: 6/1/06)
Total run time: none - 20.2 minutes
Component failure: Control panel side Denison hydraulic pump
What's interesting about DU 8842 is that it was the skid that was used for the only 24 hour "endurance" test performed during the MWI factory testing in the third week of April, 2006. Except that test had the skid running at 1000 psi, less than a third of the pressure the drive was supposed to generate in the field. But after 24 hours of running on cruise control (and a brief repair to the gear oil circulation pump), 8842 was good enough to send out to Orleans Avenue, where we can see it promptly broke - twice - after less than an hour of runtime.
None of the information about the poor performance of the pumps in the field after they were installed was published in the media at the time it happened. Articles from May, June, July, and August of 2006 show no acknowledgement by the Corps of any problems on the MWI pumps, until the vibration problems emerged in mid-August. Those vibration problems are also documented in those Orleans Avenue tests three months previously (more withholding of information).
The same withholding by the Corps goes for public meetings attended by in-the-know Corps New Orleans officials such as Jim St. Germain, Colonel Bedey, Colonel Wagenaar, and Walter Baumy, some of which I have on tape. No one in the public - and I mean NO ONE - was clued in to the fact that drive skids were going kablooey and pump assemblies were vibrating practically on a daily basis throughout the summer of 2006. That's a horrible way to treat the people of New Orleans, but it's a great way to treat a manufacturer that doesn't want bad press.
Now, a year later the Corps is doing the exact same thing, because it's the exact same people - St. Germain, Bedey, Bradley. They run the pumps and drive units for 45 minutes, assumedly at 3000 psi (though we have no idea, and the freaking media won't ask), get worthless flow numbers from flowmeters that exemplify GIGO, and then they say, "Everything's fine!" Everything is not fine. Qualified, non- New Orleans individuals in the Corps believe there are still vital pieces of equipment primed for failure on these skids today. That's serious.
We're talking about protecting peoples' lives here. It's not a game. This is why comprehensive, extensive testing is required of this equipment, not the crappy crud with which MWI and the New Orleans District bamboozles the local media. The contract says so, reality says so, the public says so, and common decency says so. The only people that say it isn't are the New Orleans District and MWI.]
"2) Recommend that a certified hydraulic systems inspector, per ASME B31.1 Power Piping, inspect the piping system and certify that the hydraulic piping system is safe to operate for the intended use. The inspector may add operating requirements due to the reduced factor of safety. Any additional operating requirements must be included in the training of, and provided to any pump system operators."
MVN responds: They analyzed...said not true.
[This one cracks me up, and kind of makes me cry a bit. The Corps is, I believe, relying on a static test of the piping at 4500 psi to make this statement, as well as some really crummy calcs included in the internal report (found on the last two pages).
Such a static test is inadequate to determine if the pipes are suitable for dynamic conditions lasting hours on end. Considering how much the piping vibrates when the pumps are run, and how rusty the pipes are now, and how undersized they are according to code, it's clear the New Orleans District and its Metairie-based consultants at NY Associates, Linfield Hunter Junius, and URS likely have no idea what they're talking about when it comes to the pipes failing. They're playing with peoples' lives for God's sake. Are they really willing to leave clearly deteriorating equipment in the field for use during a hurricane?]
"3) Evaluate to determine if the automatic clutch system should be required to be installed to reduce shockwaves within the hydraulic piping system as required per contract documents to provide a smooth, soft start."
MVN responds: At first said don't think need to be done....then said in process.
[If they think the piping is okay, then why are they looking at the clutch to prevent damage to the pipes?]
"4) Information on welds that have been previously repaired by the Government and other welds which were not repaired should be reviewed. Further review is needed to determine whether additional welds should be repaired."
MVN responds: Done.
[We will likely never see information about this.]
"5) Provide mechanical computations for flange bolts to insure they are sized properly to handle the weight of the components."
MVN responds: Done.
[We will likely never see these calculations.]
"6) Remove inlet strainers per Denison recommendations."
MVN responds: Evaluating.
[How can they still be "evaluating" this one? All the information was laid out over a year ago on this issue, as shown in the internal report and the shop inspection records. Denison themselves says not to use strainers on the inlets to their pumps. What is there to evaluate?]
"7) Installation of the hydraulic cones in London and Orleans, along with vortex suppressors as determined by the Model Studies conducted at ERDC. The cones are currently being installed at London Avenue Canal."
MVN responds: Done.
"8) Each pump should be provided with a hydraulic system monitoring device (as required by the contract) to allow diagnosing hydraulic system behavior even while the pump is submerged. If a monitoring device is not provided, then a credit to the Government is due."
MVN responds: Evaluating.
[After all the hydraulic system problems, you'd think this one would be a no-brainer. You'd think...]
"9) Calibrate and locate the ultra-sonic equipment in the proper locations as required by the manufacturer with direction as required by Dr. Maynord at ERDC. This is to determine the correct amount of flow capacity for the pump at all three Canal sites."
MVN responds: In process.
[As I mentioned a few posts back, there's very little reason to believe the flowmeters will ever work. In ERDC's own April 2007 tests at London Avenue, they weren't used to find the capacity of the pumps. Instead, ERDC "backed into" the capacity by getting the static head in the pumps and used a derived pump curve (with its own significant scatter errors) to interpolate to the capacity. From an engineering standpoint, this is far less reliable than a direct measurement of the flow. It also subverts the entire floodgate pump control system, which is based on the flowmeters, not some jury-rigged piezometer-and-pump-curve smashup.
This would appear to be another source of a tremendous amount of wasted taxpayer funds - probably millions of dollars. Here we are over a year after the installation of these control systems, and at their most fundamental level, they don't work. What a mess.
This one lands smack at Dan Bradley's feet. He's the electrical engineer on this job, and was responsible for the controls system.]
"10) Have MWI provide manufacturer’s information and shop drawings (including Durst information) as discussed earlier in this report."
MVN responds: Done.
[I have a hard time believing this is "done," since the ERDC folks didn't have MWI's certified information on which to rely just over two months ago.]
posted by mcbrid35 at
3:11 PM
0 Comments:
Post a Comment
<< Home