That's all folks

As some of you may know, I'm trying to step away from all of this. It's taken a huge toll on me, as if Katrina weren't damaging enough.

So this is my last post. For a while, I'd considered this a closed loop process, where a defined end would be reached at some point.

However, I've concluded "Katrina" will never end, and at this point the only way to find my end is to declare it.

I had thought about continuing for a while, despite some major transitions in my life, but I just don't think it's worth it. I've got to move on. I've put out as much information as I can get, and if people don't understand the issues at hand by now, they'll never understand.

Thanks to all those who have supported me. I wish the best for New Orleans.

What MVN thinks of the internal

There were ten technical recommendations in the Corps' internal investigation into the floodgate pumps. Since General Crear ignored almost all of them in his press release cover letter, it fell to the local media to find out their disposition.

Courtesy of WWL-TV, below you will find the New Orleans District's responses to the internal investigation's technical recommendations. I can't verify the truth of anything but part of the first one. The hydraulic reservoirs are being raised (see photos below).

The idea behind raising the hydraulic reservoirs is to pull hydraulic fluid from lower in the reservoirs, preventing entrained air from entering the delicate Denison hydraulic pumps and wrecking them. MWI's original scheme (the one which has been in existence up until last month) was to pull hydraulic fluid off the top of the reservoirs, working against gravity and guaranteeing air would get trapped in the inlet lines to the Denison pumps. This wrecked Denison pump after Denison pump. Considering that the scheme has been in operation up until last month, it is a strong possibility there are now wrecked Denison pumps on the skids, a conclusion reached in the internal investigation (see my commentary on internal recommendation #1 below).

The new scheme raises the reservoirs and reworks the piping and hoses between the reservoir and the Denison pumps, with the hydraulic fluid coming out of a point lower on the reservoir (hopefully the very bottom, but that's not clear).

They've started raising the reservoirs with Orleans Avenue for some unknown reason (I would have thought 17th St and London would be far more urgent, since those floodgates gates are intended to be dropped long before Orleans). All of the Orleans Avenue reservoirs appear to be raised.

17th Street does not have raised reservoirs on any of the original 12 pumps, as of June 20 (the six newer pumps came with raised reservoirs last year). Considering how much importance the Corps puts on 17th Street - every major press availability is held there, and it's the only site the Corps took the GAO when they visited, not that it mattered to the GAO - you'd think they would have started there.

So how does one tell if the reservoirs have been raised? It's fairly simple. First, let's get oriented to where the hydraulic reservoirs can be found on the drive skids. We'll use a picture from Dennis Strecker's April 18, 2006 report, which is included in the internal investigation (Adobe page 136):


The "day tank" is another name for the fuel tank. One can see the hydraulic reservoir is located inboard - or behind - the fuel tank. One can also see that the hydraulic reservoir is shorter than the fuel tank. This makes for a handy reference to determine whether the reservoirs have been raised. If the top of the reservoir is above the fuel tank, they've gone up. If the fuel tank is higher than the reservoir, they haven't been raised.

Here's a "before" picture of a drive skid at the east drive shed at Orleans Avenue:


As you can see, the top of the fuel tank was above the top of the reservoir. Thus, the reservoir was at its original, as-delivered height on March 25, 2007.

Now look at an "after" shot of the same skid:


The top of the reservoir is above the top of the fuel tank. The reservoir is now sitting up on a new set of approximately foot-high legs. This work was done in late May or early June.

All five drive skids on the east side of Orleans Avenue had gotten this treatment as of June 20th:


I don't have a picture of it, but I did see that the five skids on the west side of Orleans also had their reservoirs raised as of June 20, 2007.

Here's a picture of the actual legs being used to raise the reservoirs, along with hoses that have been removed from skids:


Moving over to London Avenue...

Here's the three southernmost skids in the west drive shed at London Avenue, none of which had their reservoirs raised at the time of this photo:


Here's the other three skids in that shed (sorry about cutting off the skid on the right):


How do we know the reservoirs are unraised in this shed? By examining the relative heights of the fuel tanks and the reservoirs. Here's a detail from the first London Avenue shot above, which shows the reservoir unraised:


All of the rest of the London Avenue - west drive skids were exactly the same as of June 20, 2007.

I don't have pictures of the London Avenue - east drive skids, but I did see they were also unraised as of June 20, 2007.

Finally, let's do the same treatment on 17th Street. First, the east drive platform:


Below is a detail shot of the drive skid on the left. While it's a little hard to make out (it's actually easier to see in the long shot above), you can see that the top of the fuel tank remains above the top of the reservoir:


Here's the best picture I could get of the 17th Street west drive shed:


And here's a detail, showing that the reservoirs hadn't been raised in that shed as of June 20, 2007 (I'm using the "daylight" argument):

My camera doesn't pick up as much detail as the naked eye, so you'll have to trust that I could clearly see the reservoirs in that shed were still at their original heights that day.

The internal recommendations follow, along with the New Orleans' District's responses. I've included a little commentary inside brackets. "MVN" is the Corps code for the New Orleans District.

"1) Flooded suction for all hydraulic oil pumps to prevent equipment failures and unsafe priming by operating personnel. Simultaneously with this action is the removal of the Denison Hydraulic pumps to inspect for any failed or in process of failing components. Any unsuitable components shall be replaced."

MVN responds: In process.

[It's important to note the second sentence of this recommendation. The internal team believes there could be "failed or in process of failing" components inside the Denison hydraulic pumps - even today. This is pretty huge, since the Denison pumps are the heart of the drive skids, and their failure leads to the failure of a water pump. There are 80 Denison pumps installed at the three canals, along with another six installed on spare drive skids. That's 86 units that are still in jeopardy of going kablooey.

Such Denison pump failures happened all the time during the MWI factory testing, and they continued during the initial field testing in the spring of 2006 (note: not complete field performance testing - which the Corps has no intention of conducting - just turning the pumps on to see if they worked).

We know the Denison hydraulic pump failures continued during the field testing because the internal investigation folks included data on field tests in Attachment #10 to the internal report. It's only two pages, but's it's sure disturbing. Here's the results for the tests of just two drive unit (DU)/pump assembly (PA) combos at Orleans Avenue in May and June of 2006:

DU 8845/PA 4590 (test date: 5/31/06)
Total run time: 25 minutes
Component failure: Filter side Denison hydraulic pump

DU 8842/PA 4581 (test date: 5/24/06)
Total run time: none - static pressure test only
Component failure: Filter side Denison hydraulic pump

DU 8842/PA 4581 (test date: 6/1/06)
Total run time: none - 20.2 minutes
Component failure: Control panel side Denison hydraulic pump

What's interesting about DU 8842 is that it was the skid that was used for the only 24 hour "endurance" test performed during the MWI factory testing in the third week of April, 2006. Except that test had the skid running at 1000 psi, less than a third of the pressure the drive was supposed to generate in the field. But after 24 hours of running on cruise control (and a brief repair to the gear oil circulation pump), 8842 was good enough to send out to Orleans Avenue, where we can see it promptly broke - twice - after less than an hour of runtime.

None of the information about the poor performance of the pumps in the field after they were installed was published in the media at the time it happened. Articles from May, June, July, and August of 2006 show no acknowledgement by the Corps of any problems on the MWI pumps, until the vibration problems emerged in mid-August. Those vibration problems are also documented in those Orleans Avenue tests three months previously (more withholding of information).

The same withholding by the Corps goes for public meetings attended by in-the-know Corps New Orleans officials such as Jim St. Germain, Colonel Bedey, Colonel Wagenaar, and Walter Baumy, some of which I have on tape. No one in the public - and I mean NO ONE - was clued in to the fact that drive skids were going kablooey and pump assemblies were vibrating practically on a daily basis throughout the summer of 2006. That's a horrible way to treat the people of New Orleans, but it's a great way to treat a manufacturer that doesn't want bad press.

Now, a year later the Corps is doing the exact same thing, because it's the exact same people - St. Germain, Bedey, Bradley. They run the pumps and drive units for 45 minutes, assumedly at 3000 psi (though we have no idea, and the freaking media won't ask), get worthless flow numbers from flowmeters that exemplify GIGO, and then they say, "Everything's fine!" Everything is not fine. Qualified, non- New Orleans individuals in the Corps believe there are still vital pieces of equipment primed for failure on these skids today. That's serious.

We're talking about protecting peoples' lives here. It's not a game. This is why comprehensive, extensive testing is required of this equipment, not the crappy crud with which MWI and the New Orleans District bamboozles the local media. The contract says so, reality says so, the public says so, and common decency says so. The only people that say it isn't are the New Orleans District and MWI.]

"2) Recommend that a certified hydraulic systems inspector, per ASME B31.1 Power Piping, inspect the piping system and certify that the hydraulic piping system is safe to operate for the intended use. The inspector may add operating requirements due to the reduced factor of safety. Any additional operating requirements must be included in the training of, and provided to any pump system operators."

MVN responds: They analyzed...said not true.

[This one cracks me up, and kind of makes me cry a bit. The Corps is, I believe, relying on a static test of the piping at 4500 psi to make this statement, as well as some really crummy calcs included in the internal report (found on the last two pages).

Such a static test is inadequate to determine if the pipes are suitable for dynamic conditions lasting hours on end. Considering how much the piping vibrates when the pumps are run, and how rusty the pipes are now, and how undersized they are according to code, it's clear the New Orleans District and its Metairie-based consultants at NY Associates, Linfield Hunter Junius, and URS likely have no idea what they're talking about when it comes to the pipes failing. They're playing with peoples' lives for God's sake. Are they really willing to leave clearly deteriorating equipment in the field for use during a hurricane?]

"3) Evaluate to determine if the automatic clutch system should be required to be installed to reduce shockwaves within the hydraulic piping system as required per contract documents to provide a smooth, soft start."

MVN responds: At first said don't think need to be done....then said in process.

[If they think the piping is okay, then why are they looking at the clutch to prevent damage to the pipes?]

"4) Information on welds that have been previously repaired by the Government and other welds which were not repaired should be reviewed. Further review is needed to determine whether additional welds should be repaired."

MVN responds: Done.

[We will likely never see information about this.]

"5) Provide mechanical computations for flange bolts to insure they are sized properly to handle the weight of the components."

MVN responds: Done.

[We will likely never see these calculations.]

"6) Remove inlet strainers per Denison recommendations."

MVN responds: Evaluating.

[How can they still be "evaluating" this one? All the information was laid out over a year ago on this issue, as shown in the internal report and the shop inspection records. Denison themselves says not to use strainers on the inlets to their pumps. What is there to evaluate?]

"7) Installation of the hydraulic cones in London and Orleans, along with vortex suppressors as determined by the Model Studies conducted at ERDC. The cones are currently being installed at London Avenue Canal."

MVN responds: Done.

"8) Each pump should be provided with a hydraulic system monitoring device (as required by the contract) to allow diagnosing hydraulic system behavior even while the pump is submerged. If a monitoring device is not provided, then a credit to the Government is due."

MVN responds: Evaluating.

[After all the hydraulic system problems, you'd think this one would be a no-brainer. You'd think...]

"9) Calibrate and locate the ultra-sonic equipment in the proper locations as required by the manufacturer with direction as required by Dr. Maynord at ERDC. This is to determine the correct amount of flow capacity for the pump at all three Canal sites."

MVN responds: In process.

[As I mentioned a few posts back, there's very little reason to believe the flowmeters will ever work. In ERDC's own April 2007 tests at London Avenue, they weren't used to find the capacity of the pumps. Instead, ERDC "backed into" the capacity by getting the static head in the pumps and used a derived pump curve (with its own significant scatter errors) to interpolate to the capacity. From an engineering standpoint, this is far less reliable than a direct measurement of the flow. It also subverts the entire floodgate pump control system, which is based on the flowmeters, not some jury-rigged piezometer-and-pump-curve smashup.

This would appear to be another source of a tremendous amount of wasted taxpayer funds - probably millions of dollars. Here we are over a year after the installation of these control systems, and at their most fundamental level, they don't work. What a mess.

This one lands smack at Dan Bradley's feet. He's the electrical engineer on this job, and was responsible for the controls system.]

"10) Have MWI provide manufacturer’s information and shop drawings (including Durst information) as discussed earlier in this report."

MVN responds: Done.

[I have a hard time believing this is "done," since the ERDC folks didn't have MWI's certified information on which to rely just over two months ago.]

Cover up, Part 3: GAO SUX

Updated 10/8/07. See below

In Parts 1 and 2 of this report, we looked at the lies, extremes in poor judgement, and skirting of the law in which the Corps New Orleans District regularly trades. Today, we'll expand our focus to what passes for oversight over the Corps, and why that can't even be trusted.

When I heard this past March that, at Louisiana Senator Mary Landrieu's request, the Government Accountability Office (GAO) was investigating the floodgate pumps mess, I was hopeful but skeptical. GAO's previous work on Katrina issues has been spotty at best. It seems, based on comparisons to what the Corps internal investigation turned up, that my skepticism was justified.

What GAO gave to Senator Landrieu's office was a steaming turd. It doesn't even track what the Senator asked for in her letter to the GAO. And now that we have the Corps internal investigation in hand, we can see the blatant mistakes by GAO clearly.

The GAO report doesn't mention the numerous contract administration improprieties (including - but not confined to - troubling use of implied authorities by individuals like Jim St. Germain and Dan Bradley who did not possess such authorities), it only glancingly mentions the critical technical difficulties with the hydraulic systems (accepting the inadequate interim fix on the reservoirs as adequate, instead of bringing up the fact that the permanent fix was critical and had not been implemented for nearly a year, and still hasn't been implemented on most of the drive units), and I have to wonder if they even read Ms. Garzino's correspondence and the shop inspection reports. In fact, Ms. Garzino's name inexplicably does not appear anywhere in a report that would not exist without her memo.

In addition, there's zero mention of the continuing non-execution of the performance testing section of the contract, which is at the heart of the cover-up (see Part 1 and Part 2 of this report). In fact, there's only parrotting of the Corps' line: "Any performance problems with the pump systems could be resolved after they were installed on the canals."

Except (and this is only 9 pages later in the GAO report), "According to the Corps, it is not possible to test each pump system for an extended period of time or to test the entire system due to limitations in simulating the amount of water that would be present in the canals and the lake during an actual storm event."

So the performance problems could be resolved later, presumably through contractually-mandated full performance testing. Except that "later," there's no way to actually perform those tests? Huh?

How in the world did the GAO not catch and highlight this clear contradiction in the Corps' statements, especially since it appears the GAO's own report? Why didn't they catch it and perhaps think about whether the Corps ever intended to do that performance testing, or if the Corps has never intended to do it (see the earlier parts of this report)? And then why wouldn't they think about possible motivations for such inaction, and really dig in? Why couldn't they do that bit of thinking?

This is what I mean when I say the GAO investigation is junk. They appear to have taken whatever the Corps New Orleans District handed them and did no critical reasoning on it. Hey GAO, here's critical reasoning on the Corps statement about the impossibility of extended testing of the pumps on site: What about taking the pumps out of their platforms and running them elsewhere, where they could be lowered and raised - thus eliminating the "limitations in simulating the amount of water?" I'm not the first person to think of it. That's exactly what the Corps themselves did last August when the vibration problems emerged! They hooked pumps to a crane and dunked them in the canal at different depths. Surely everyone remembers that. Here's an August 6, 2006 picture of that widely publicized testing, taken from the Corps' own press release (dated December 1, 2006) (it also flashes by in the multi-image GIF on the Corps New Orleans Hurricane Protection Office front page, which is where I got the date):


The caption cracks me up. It makes it seem like this was routine testing, not everyone at the Corps and MWI grasping at straws.

You can see the drive unit for the pump on the deck at the lower left of the photo behind the mist cloud. Now, if this was possible last August, why couldn't a barge be rigged up with a crane (to raise and lower the pump), some kind of framework (to hold the pump), and a set of appropriately long discharge piping in which to measure flow? And why couldn't such a system be run for a legitimately long amount of time to measure the entire range of flows and heads, as well as a long enough time to simulate storm conditions?

The answer is: of course it could, but I'm sure MWI would refuse to do it, even though they are contractually obligated to do so. Because it costs too much. And the Corps New Orleans District refuses to do it because it would show how poor their contract administration has been, i.e. they've paid millions of taxpayer dollars for equipment which does not meet spec. (Side question: I wonder if MWI paid for all the crane rental time, labor, and materials for the testing shown in the photo above?)

And that's how you do critical reasoning.

There's even flat-out lies in the GAO report that should have been obvious. On Adobe page 35, there's a chart of Orleans Avenue canal capacity dating back to June of 2006:


It shows a constant capacity for Orleans at just under 2000 cfs from June, 2006 until mid-May of this year, when there's a small, unexplained bump up to just over 2000 cfs. I'd like to know how the Corps could have gotten 2000 cfs out of 10 pumps at Orleans when five of them were sitting out of the water up on the deck from October, 2006 through March, 2007, 2007 (read the linked blog entries for more details and pictures):




The year on the picture above should have read "2007." Oops.

Yes, I understand that the period the pumps were out at Orleans was outside the height of hurricane season. But the pumps weren't even in the damn water for over six months! When basic, easily checkable facts like that are screwed up, why should the rest of the GAO investigation carry any credibility?

The whole GAO thing seems less like an investigation and more like a mistake-ridden, propaganda-laced summary of current conditions. GAO themselves needs to be investigated by their inspector general to determine how Senator Landrieu could write the following as their marching orders...

"I request that the Government Accountability Office immediately undertake an investigation of the Army Corps of Engineers rationale for the the installation of faulty flood control pumps despite the internal memoranda indicating their insufficiency for the task. Furthermore, the investigation should examine the contract arrangements with the company in question and should determine whether or not any improprieties exist in the award or fulfillment of these contracts."

...and then the GAO could interpret/twist that into the following "questions to be answered:"

"- What were the specifications and requirements of the contract and what was the basis for the selection of the supplier for the pump systems?
- In light of the factory test failures known to the Corps, why did it install the pump systems?
- What pumping capacity existed on June 1, 2006, and what actions has the Corps taken to address the known problems with the pump systems?
- What is the current pumping capacity at the drainage canals and what are the Corps’ plans to increase capacity during the 2007 hurricane season?"

They're kinda close, but GAO's missing a large chunk of what the Senator asked for. Where are the GAO's questions about the contract arrangements and improprieties in the fulfillment of the contract? "Fulfillment" entails looking at the entire span of the contract (which means up to the day GAO issued their report), not the "basis for selection of the supplier." That is a tremendous narrowing of the scope of the Senator's request, and GAO needs to be taken to task for it.

Why is there zero mention of the substantial improprieties on the financial and contract administration side of the project? The Corps internal investigation uncovered suspiciously inadequate record-keeping, an uncooperative Contracting Officer, and millions of dollars in payments to MWI that couldn't (or wouldn't) be accounted for. Isn't that precisely what the General Accountability Office should have also found? After all, they've got the word "account" right there in their name.

Even taking the GAO's questions on their own terms, how can they ask their last question about current pumping capacity without mentioning the persistent, documented capacity shortfalls revealed by Corps-overseen testing in April 2006, November 2006, and once again in April 2007? The November 2006 and April 2007 testing was performed at the behest of the Corps internal investigative team. GAO themselves say they met with "members of the Mississippi Valley Division’s independent team reviewing issues related to the contract and pump performance." If they listened at all to that team, how could GAO not reach the same conclusions (on all matters, not just capacities) the internal team was reaching?

You can probably see that I think very little of the GAO investigation. I can say now that while they were investigating in March, April and May, they were reading this blog every day, sometimes for hours at a stretch. That gave me limited hopes, because one of the reasons I started blogging was to make documents easily available to the eventual investigations I knew would come. Even today, someone at the GAO checks in regularly. How they got from what I've reported to their stuff is a mystery. And when you consider the everlasting, gob-stopping treasure trove of the Corps internal investigation (which is light years beyond any of my little rhetorical bottle rockets), the GAO report is measurably inferior.

There's only three possibilities for the radical differences between the GAO's report and the Corps internal investigation:

1) GAO didn't get access to the same data the Corps internal team did. This is very, very unlikely, despite what they told Senator Landrieu's office. There's no way GAO could have written a report without looking at the contract, the modifications, and the correspondence. Also, did they not request the same documents the Corps internal team was using?

2) The Corps and GAO had access to the same data, but GAO actively ignored it for some unknown reason. This is the scariest possibility, because is speaks to a compromised investigatory process.

3) The Corps and GAO looked at the same data, but GAO didn't realize its significance. In other words, the GAO team was incompetent.

In any case, whatever the reason for the inadequacy of the GAO report, the consequence is clear. It has muddied the waters into what happened, extending the cover-up. Indeed, after its release, both MWI and Colonel Bedey crowed about how they were in the clear.

MWI (in the May 21st Palm Beach Post letters to the editor):

"...The Associated Press further reported on Thursday that Sen. Mary Landrieu, D-La., said an investigation and forthtcoming [sic] report by the Government Accountability Office found no evidence of fraud or improper influence in the pump project."

Colonel Bedey (in the May 17th AP article about the briefing GAO gave Senator Landrieu's office):

"'We always said there was no smoking gun,' Col. Jeffrey Bedey, who is overseeing reconstruction of the city's levee system for the corps, said in reaction to the report."

As we now know from the internal investigation, there's an armory of smoking guns. GAO either ignored them or was told to ignore them.

In some ways, I wish the GAO had just kept their own counsel. Fortunately, Senator Landrieu is the chair of the Senate Appropriations Subcommittee that controls GAO's budget, so GAO had better do a much more thorough job when they dig back into this fiasco. And, as I mentioned above, I hope the GAO Inspector General takes a close look at this one too. It's certainly destroyed my faith in yet another agency of the government, which I didn't think was possible after all that Katrina has wrought.

Update, 10/8/07:
After the internal came out, and it became readily apparent that the GAO investigation was inadequate, Senator Landrieu said she strongly encouraged GAO to revisit their findings:

"...the GAO must keep digging to ensure that Congress and the people of New Orleans have the truth"

That press release is here. That was on June 8th.

It took a while, but the GAO did restart their inquiry. Oddly, it was MWI themselves that first revealed this publicly. On September 12, 2007, they responded to an article posted on a website called CorpWatch around Katrina's 2nd anniversary. The original article summarized the pump story to date. MWI's response attempted to rebut it, and accidentally revealed:

"The GAO is now in the process of pursuing an additional inquiry and we welcome
the opportunity to meet with them again."

I seriously doubt they "welcomed" the opportunity to be investigated, but that's standard corporate-speak.

This was confirmed on October 8th, along with some other huge news:

Times-Picayune, Monday, October 8:
Third probe of pumps on canals requested

Associated Press, Monday, October 8:
Report: New Orleans pump contract likely handled improperly

Ms. Garzino went to the Office of Special Counsel, an independent federal investigatory agency. Her referral raised enough eyebrows at OSC that they ordered Secretary of Defense Robert Gates to further investigate the entire imbroglio. Going through OSC provides an extra degree of protection to ensure the integrity of the investigation.

The existence of the Defense Department investigation was revealed by an illegal leak of its existence from within the Pentagon to a whistleblower group called PEER. Their source has likely lost his or her job, because PEER did not bother to submit a FOIA request for the documents they leaked to the Times-Picayune. Had they done so, their source would have been protected, and the DoD investigation would have likely gone a lot smoother.

Be that as it may, the documents they gave to the Times-Picayune are attached to their October 8, 2007 press release here. The direct link to the documents is here.