Fix the pumps

Tuesday, July 03, 2007

That's all folks

As some of you may know, I'm trying to step away from all of this. It's taken a huge toll on me, as if Katrina weren't damaging enough.

So this is my last post. For a while, I'd considered this a closed loop process, where a defined end would be reached at some point.

However, I've concluded "Katrina" will never end, and at this point the only way to find my end is to declare it.

I had thought about continuing for a while, despite some major transitions in my life, but I just don't think it's worth it. I've got to move on. I've put out as much information as I can get, and if people don't understand the issues at hand by now, they'll never understand.

Thanks to all those who have supported me. I wish the best for New Orleans.

What MVN thinks of the internal

There were ten technical recommendations in the Corps' internal investigation into the floodgate pumps. Since General Crear ignored almost all of them in his press release cover letter, it fell to the local media to find out their disposition.

Courtesy of WWL-TV, below you will find the New Orleans District's responses to the internal investigation's technical recommendations. I can't verify the truth of anything but part of the first one. The hydraulic reservoirs are being raised (see photos below).

The idea behind raising the hydraulic reservoirs is to pull hydraulic fluid from lower in the reservoirs, preventing entrained air from entering the delicate Denison hydraulic pumps and wrecking them. MWI's original scheme (the one which has been in existence up until last month) was to pull hydraulic fluid off the top of the reservoirs, working against gravity and guaranteeing air would get trapped in the inlet lines to the Denison pumps. This wrecked Denison pump after Denison pump. Considering that the scheme has been in operation up until last month, it is a strong possibility there are now wrecked Denison pumps on the skids, a conclusion reached in the internal investigation (see my commentary on internal recommendation #1 below).

The new scheme raises the reservoirs and reworks the piping and hoses between the reservoir and the Denison pumps, with the hydraulic fluid coming out of a point lower on the reservoir (hopefully the very bottom, but that's not clear).

They've started raising the reservoirs with Orleans Avenue for some unknown reason (I would have thought 17th St and London would be far more urgent, since those floodgates gates are intended to be dropped long before Orleans). All of the Orleans Avenue reservoirs appear to be raised.

17th Street does not have raised reservoirs on any of the original 12 pumps, as of June 20 (the six newer pumps came with raised reservoirs last year). Considering how much importance the Corps puts on 17th Street - every major press availability is held there, and it's the only site the Corps took the GAO when they visited, not that it mattered to the GAO - you'd think they would have started there.

So how does one tell if the reservoirs have been raised? It's fairly simple. First, let's get oriented to where the hydraulic reservoirs can be found on the drive skids. We'll use a picture from Dennis Strecker's April 18, 2006 report, which is included in the internal investigation (Adobe page 136):


The "day tank" is another name for the fuel tank. One can see the hydraulic reservoir is located inboard - or behind - the fuel tank. One can also see that the hydraulic reservoir is shorter than the fuel tank. This makes for a handy reference to determine whether the reservoirs have been raised. If the top of the reservoir is above the fuel tank, they've gone up. If the fuel tank is higher than the reservoir, they haven't been raised.

Here's a "before" picture of a drive skid at the east drive shed at Orleans Avenue:


As you can see, the top of the fuel tank was above the top of the reservoir. Thus, the reservoir was at its original, as-delivered height on March 25, 2007.

Now look at an "after" shot of the same skid:


The top of the reservoir is above the top of the fuel tank. The reservoir is now sitting up on a new set of approximately foot-high legs. This work was done in late May or early June.

All five drive skids on the east side of Orleans Avenue had gotten this treatment as of June 20th:


I don't have a picture of it, but I did see that the five skids on the west side of Orleans also had their reservoirs raised as of June 20, 2007.

Here's a picture of the actual legs being used to raise the reservoirs, along with hoses that have been removed from skids:


Moving over to London Avenue...

Here's the three southernmost skids in the west drive shed at London Avenue, none of which had their reservoirs raised at the time of this photo:


Here's the other three skids in that shed (sorry about cutting off the skid on the right):


How do we know the reservoirs are unraised in this shed? By examining the relative heights of the fuel tanks and the reservoirs. Here's a detail from the first London Avenue shot above, which shows the reservoir unraised:


All of the rest of the London Avenue - west drive skids were exactly the same as of June 20, 2007.

I don't have pictures of the London Avenue - east drive skids, but I did see they were also unraised as of June 20, 2007.

Finally, let's do the same treatment on 17th Street. First, the east drive platform:


Below is a detail shot of the drive skid on the left. While it's a little hard to make out (it's actually easier to see in the long shot above), you can see that the top of the fuel tank remains above the top of the reservoir:


Here's the best picture I could get of the 17th Street west drive shed:


And here's a detail, showing that the reservoirs hadn't been raised in that shed as of June 20, 2007 (I'm using the "daylight" argument):

My camera doesn't pick up as much detail as the naked eye, so you'll have to trust that I could clearly see the reservoirs in that shed were still at their original heights that day.

The internal recommendations follow, along with the New Orleans' District's responses. I've included a little commentary inside brackets. "MVN" is the Corps code for the New Orleans District.

"1) Flooded suction for all hydraulic oil pumps to prevent equipment failures and unsafe priming by operating personnel. Simultaneously with this action is the removal of the Denison Hydraulic pumps to inspect for any failed or in process of failing components. Any unsuitable components shall be replaced."

MVN responds: In process.

[It's important to note the second sentence of this recommendation. The internal team believes there could be "failed or in process of failing" components inside the Denison hydraulic pumps - even today. This is pretty huge, since the Denison pumps are the heart of the drive skids, and their failure leads to the failure of a water pump. There are 80 Denison pumps installed at the three canals, along with another six installed on spare drive skids. That's 86 units that are still in jeopardy of going kablooey.

Such Denison pump failures happened all the time during the MWI factory testing, and they continued during the initial field testing in the spring of 2006 (note: not complete field performance testing - which the Corps has no intention of conducting - just turning the pumps on to see if they worked).

We know the Denison hydraulic pump failures continued during the field testing because the internal investigation folks included data on field tests in Attachment #10 to the internal report. It's only two pages, but's it's sure disturbing. Here's the results for the tests of just two drive unit (DU)/pump assembly (PA) combos at Orleans Avenue in May and June of 2006:

DU 8845/PA 4590 (test date: 5/31/06)
Total run time: 25 minutes
Component failure: Filter side Denison hydraulic pump

DU 8842/PA 4581 (test date: 5/24/06)
Total run time: none - static pressure test only
Component failure: Filter side Denison hydraulic pump

DU 8842/PA 4581 (test date: 6/1/06)
Total run time: none - 20.2 minutes
Component failure: Control panel side Denison hydraulic pump

What's interesting about DU 8842 is that it was the skid that was used for the only 24 hour "endurance" test performed during the MWI factory testing in the third week of April, 2006. Except that test had the skid running at 1000 psi, less than a third of the pressure the drive was supposed to generate in the field. But after 24 hours of running on cruise control (and a brief repair to the gear oil circulation pump), 8842 was good enough to send out to Orleans Avenue, where we can see it promptly broke - twice - after less than an hour of runtime.

None of the information about the poor performance of the pumps in the field after they were installed was published in the media at the time it happened. Articles from May, June, July, and August of 2006 show no acknowledgement by the Corps of any problems on the MWI pumps, until the vibration problems emerged in mid-August. Those vibration problems are also documented in those Orleans Avenue tests three months previously (more withholding of information).

The same withholding by the Corps goes for public meetings attended by in-the-know Corps New Orleans officials such as Jim St. Germain, Colonel Bedey, Colonel Wagenaar, and Walter Baumy, some of which I have on tape. No one in the public - and I mean NO ONE - was clued in to the fact that drive skids were going kablooey and pump assemblies were vibrating practically on a daily basis throughout the summer of 2006. That's a horrible way to treat the people of New Orleans, but it's a great way to treat a manufacturer that doesn't want bad press.

Now, a year later the Corps is doing the exact same thing, because it's the exact same people - St. Germain, Bedey, Bradley. They run the pumps and drive units for 45 minutes, assumedly at 3000 psi (though we have no idea, and the freaking media won't ask), get worthless flow numbers from flowmeters that exemplify GIGO, and then they say, "Everything's fine!" Everything is not fine. Qualified, non- New Orleans individuals in the Corps believe there are still vital pieces of equipment primed for failure on these skids today. That's serious.

We're talking about protecting peoples' lives here. It's not a game. This is why comprehensive, extensive testing is required of this equipment, not the crappy crud with which MWI and the New Orleans District bamboozles the local media. The contract says so, reality says so, the public says so, and common decency says so. The only people that say it isn't are the New Orleans District and MWI.]

"2) Recommend that a certified hydraulic systems inspector, per ASME B31.1 Power Piping, inspect the piping system and certify that the hydraulic piping system is safe to operate for the intended use. The inspector may add operating requirements due to the reduced factor of safety. Any additional operating requirements must be included in the training of, and provided to any pump system operators."

MVN responds: They analyzed...said not true.

[This one cracks me up, and kind of makes me cry a bit. The Corps is, I believe, relying on a static test of the piping at 4500 psi to make this statement, as well as some really crummy calcs included in the internal report (found on the last two pages).

Such a static test is inadequate to determine if the pipes are suitable for dynamic conditions lasting hours on end. Considering how much the piping vibrates when the pumps are run, and how rusty the pipes are now, and how undersized they are according to code, it's clear the New Orleans District and its Metairie-based consultants at NY Associates, Linfield Hunter Junius, and URS likely have no idea what they're talking about when it comes to the pipes failing. They're playing with peoples' lives for God's sake. Are they really willing to leave clearly deteriorating equipment in the field for use during a hurricane?]

"3) Evaluate to determine if the automatic clutch system should be required to be installed to reduce shockwaves within the hydraulic piping system as required per contract documents to provide a smooth, soft start."

MVN responds: At first said don't think need to be done....then said in process.

[If they think the piping is okay, then why are they looking at the clutch to prevent damage to the pipes?]

"4) Information on welds that have been previously repaired by the Government and other welds which were not repaired should be reviewed. Further review is needed to determine whether additional welds should be repaired."

MVN responds: Done.

[We will likely never see information about this.]

"5) Provide mechanical computations for flange bolts to insure they are sized properly to handle the weight of the components."

MVN responds: Done.

[We will likely never see these calculations.]

"6) Remove inlet strainers per Denison recommendations."

MVN responds: Evaluating.

[How can they still be "evaluating" this one? All the information was laid out over a year ago on this issue, as shown in the internal report and the shop inspection records. Denison themselves says not to use strainers on the inlets to their pumps. What is there to evaluate?]

"7) Installation of the hydraulic cones in London and Orleans, along with vortex suppressors as determined by the Model Studies conducted at ERDC. The cones are currently being installed at London Avenue Canal."

MVN responds: Done.

"8) Each pump should be provided with a hydraulic system monitoring device (as required by the contract) to allow diagnosing hydraulic system behavior even while the pump is submerged. If a monitoring device is not provided, then a credit to the Government is due."

MVN responds: Evaluating.

[After all the hydraulic system problems, you'd think this one would be a no-brainer. You'd think...]

"9) Calibrate and locate the ultra-sonic equipment in the proper locations as required by the manufacturer with direction as required by Dr. Maynord at ERDC. This is to determine the correct amount of flow capacity for the pump at all three Canal sites."

MVN responds: In process.

[As I mentioned a few posts back, there's very little reason to believe the flowmeters will ever work. In ERDC's own April 2007 tests at London Avenue, they weren't used to find the capacity of the pumps. Instead, ERDC "backed into" the capacity by getting the static head in the pumps and used a derived pump curve (with its own significant scatter errors) to interpolate to the capacity. From an engineering standpoint, this is far less reliable than a direct measurement of the flow. It also subverts the entire floodgate pump control system, which is based on the flowmeters, not some jury-rigged piezometer-and-pump-curve smashup.

This would appear to be another source of a tremendous amount of wasted taxpayer funds - probably millions of dollars. Here we are over a year after the installation of these control systems, and at their most fundamental level, they don't work. What a mess.

This one lands smack at Dan Bradley's feet. He's the electrical engineer on this job, and was responsible for the controls system.]

"10) Have MWI provide manufacturer’s information and shop drawings (including Durst information) as discussed earlier in this report."

MVN responds: Done.

[I have a hard time believing this is "done," since the ERDC folks didn't have MWI's certified information on which to rely just over two months ago.]

Monday, July 02, 2007

Cover up, Part 3: GAO SUX

Updated 10/8/07. See below

In Parts 1 and 2 of this report, we looked at the lies, extremes in poor judgement, and skirting of the law in which the Corps New Orleans District regularly trades. Today, we'll expand our focus to what passes for oversight over the Corps, and why that can't even be trusted.

When I heard this past March that, at Louisiana Senator Mary Landrieu's request, the Government Accountability Office (GAO) was investigating the floodgate pumps mess, I was hopeful but skeptical. GAO's previous work on Katrina issues has been spotty at best. It seems, based on comparisons to what the Corps internal investigation turned up, that my skepticism was justified.

What GAO gave to Senator Landrieu's office was a steaming turd. It doesn't even track what the Senator asked for in her letter to the GAO. And now that we have the Corps internal investigation in hand, we can see the blatant mistakes by GAO clearly.

The GAO report doesn't mention the numerous contract administration improprieties (including - but not confined to - troubling use of implied authorities by individuals like Jim St. Germain and Dan Bradley who did not possess such authorities), it only glancingly mentions the critical technical difficulties with the hydraulic systems (accepting the inadequate interim fix on the reservoirs as adequate, instead of bringing up the fact that the permanent fix was critical and had not been implemented for nearly a year, and still hasn't been implemented on most of the drive units), and I have to wonder if they even read Ms. Garzino's correspondence and the shop inspection reports. In fact, Ms. Garzino's name inexplicably does not appear anywhere in a report that would not exist without her memo.

In addition, there's zero mention of the continuing non-execution of the performance testing section of the contract, which is at the heart of the cover-up (see Part 1 and Part 2 of this report). In fact, there's only parrotting of the Corps' line: "Any performance problems with the pump systems could be resolved after they were installed on the canals."

Except (and this is only 9 pages later in the GAO report), "According to the Corps, it is not possible to test each pump system for an extended period of time or to test the entire system due to limitations in simulating the amount of water that would be present in the canals and the lake during an actual storm event."

So the performance problems could be resolved later, presumably through contractually-mandated full performance testing. Except that "later," there's no way to actually perform those tests? Huh?

How in the world did the GAO not catch and highlight this clear contradiction in the Corps' statements, especially since it appears the GAO's own report? Why didn't they catch it and perhaps think about whether the Corps ever intended to do that performance testing, or if the Corps has never intended to do it (see the earlier parts of this report)? And then why wouldn't they think about possible motivations for such inaction, and really dig in? Why couldn't they do that bit of thinking?

This is what I mean when I say the GAO investigation is junk. They appear to have taken whatever the Corps New Orleans District handed them and did no critical reasoning on it. Hey GAO, here's critical reasoning on the Corps statement about the impossibility of extended testing of the pumps on site: What about taking the pumps out of their platforms and running them elsewhere, where they could be lowered and raised - thus eliminating the "limitations in simulating the amount of water?" I'm not the first person to think of it. That's exactly what the Corps themselves did last August when the vibration problems emerged! They hooked pumps to a crane and dunked them in the canal at different depths. Surely everyone remembers that. Here's an August 6, 2006 picture of that widely publicized testing, taken from the Corps' own press release (dated December 1, 2006) (it also flashes by in the multi-image GIF on the Corps New Orleans Hurricane Protection Office front page, which is where I got the date):


The caption cracks me up. It makes it seem like this was routine testing, not everyone at the Corps and MWI grasping at straws.

You can see the drive unit for the pump on the deck at the lower left of the photo behind the mist cloud. Now, if this was possible last August, why couldn't a barge be rigged up with a crane (to raise and lower the pump), some kind of framework (to hold the pump), and a set of appropriately long discharge piping in which to measure flow? And why couldn't such a system be run for a legitimately long amount of time to measure the entire range of flows and heads, as well as a long enough time to simulate storm conditions?

The answer is: of course it could, but I'm sure MWI would refuse to do it, even though they are contractually obligated to do so. Because it costs too much. And the Corps New Orleans District refuses to do it because it would show how poor their contract administration has been, i.e. they've paid millions of taxpayer dollars for equipment which does not meet spec. (Side question: I wonder if MWI paid for all the crane rental time, labor, and materials for the testing shown in the photo above?)

And that's how you do critical reasoning.

There's even flat-out lies in the GAO report that should have been obvious. On Adobe page 35, there's a chart of Orleans Avenue canal capacity dating back to June of 2006:


It shows a constant capacity for Orleans at just under 2000 cfs from June, 2006 until mid-May of this year, when there's a small, unexplained bump up to just over 2000 cfs. I'd like to know how the Corps could have gotten 2000 cfs out of 10 pumps at Orleans when five of them were sitting out of the water up on the deck from October, 2006 through March, 2007, 2007 (read the linked blog entries for more details and pictures):




The year on the picture above should have read "2007." Oops.

Yes, I understand that the period the pumps were out at Orleans was outside the height of hurricane season. But the pumps weren't even in the damn water for over six months! When basic, easily checkable facts like that are screwed up, why should the rest of the GAO investigation carry any credibility?

The whole GAO thing seems less like an investigation and more like a mistake-ridden, propaganda-laced summary of current conditions. GAO themselves needs to be investigated by their inspector general to determine how Senator Landrieu could write the following as their marching orders...
"I request that the Government Accountability Office immediately undertake an investigation of the Army Corps of Engineers rationale for the the installation of faulty flood control pumps despite the internal memoranda indicating their insufficiency for the task. Furthermore, the investigation should examine the contract arrangements with the company in question and should determine whether or not any improprieties exist in the award or fulfillment of these contracts."

...and then the GAO could interpret/twist that into the following "questions to be answered:"
"- What were the specifications and requirements of the contract and what was the basis for the selection of the supplier for the pump systems?
- In light of the factory test failures known to the Corps, why did it install the pump systems?
- What pumping capacity existed on June 1, 2006, and what actions has the Corps taken to address the known problems with the pump systems?
- What is the current pumping capacity at the drainage canals and what are the Corps’ plans to increase capacity during the 2007 hurricane season?"

They're kinda close, but GAO's missing a large chunk of what the Senator asked for. Where are the GAO's questions about the contract arrangements and improprieties in the fulfillment of the contract? "Fulfillment" entails looking at the entire span of the contract (which means up to the day GAO issued their report), not the "basis for selection of the supplier." That is a tremendous narrowing of the scope of the Senator's request, and GAO needs to be taken to task for it.

Why is there zero mention of the substantial improprieties on the financial and contract administration side of the project? The Corps internal investigation uncovered suspiciously inadequate record-keeping, an uncooperative Contracting Officer, and millions of dollars in payments to MWI that couldn't (or wouldn't) be accounted for. Isn't that precisely what the General Accountability Office should have also found? After all, they've got the word "account" right there in their name.

Even taking the GAO's questions on their own terms, how can they ask their last question about current pumping capacity without mentioning the persistent, documented capacity shortfalls revealed by Corps-overseen testing in April 2006, November 2006, and once again in April 2007? The November 2006 and April 2007 testing was performed at the behest of the Corps internal investigative team. GAO themselves say they met with "members of the Mississippi Valley Division’s independent team reviewing issues related to the contract and pump performance." If they listened at all to that team, how could GAO not reach the same conclusions (on all matters, not just capacities) the internal team was reaching?

You can probably see that I think very little of the GAO investigation. I can say now that while they were investigating in March, April and May, they were reading this blog every day, sometimes for hours at a stretch. That gave me limited hopes, because one of the reasons I started blogging was to make documents easily available to the eventual investigations I knew would come. Even today, someone at the GAO checks in regularly. How they got from what I've reported to their stuff is a mystery. And when you consider the everlasting, gob-stopping treasure trove of the Corps internal investigation (which is light years beyond any of my little rhetorical bottle rockets), the GAO report is measurably inferior.

There's only three possibilities for the radical differences between the GAO's report and the Corps internal investigation:

1) GAO didn't get access to the same data the Corps internal team did. This is very, very unlikely, despite what they told Senator Landrieu's office. There's no way GAO could have written a report without looking at the contract, the modifications, and the correspondence. Also, did they not request the same documents the Corps internal team was using?

2) The Corps and GAO had access to the same data, but GAO actively ignored it for some unknown reason. This is the scariest possibility, because is speaks to a compromised investigatory process.

3) The Corps and GAO looked at the same data, but GAO didn't realize its significance. In other words, the GAO team was incompetent.

In any case, whatever the reason for the inadequacy of the GAO report, the consequence is clear. It has muddied the waters into what happened, extending the cover-up. Indeed, after its release, both MWI and Colonel Bedey crowed about how they were in the clear.

MWI (in the May 21st Palm Beach Post letters to the editor):
"...The Associated Press further reported on Thursday that Sen. Mary Landrieu, D-La., said an investigation and forthtcoming [sic] report by the Government Accountability Office found no evidence of fraud or improper influence in the pump project."

Colonel Bedey (in the May 17th AP article about the briefing GAO gave Senator Landrieu's office):
"'We always said there was no smoking gun,' Col. Jeffrey Bedey, who is overseeing reconstruction of the city's levee system for the corps, said in reaction to the report."

As we now know from the internal investigation, there's an armory of smoking guns. GAO either ignored them or was told to ignore them.

In some ways, I wish the GAO had just kept their own counsel. Fortunately, Senator Landrieu is the chair of the Senate Appropriations Subcommittee that controls GAO's budget, so GAO had better do a much more thorough job when they dig back into this fiasco. And, as I mentioned above, I hope the GAO Inspector General takes a close look at this one too. It's certainly destroyed my faith in yet another agency of the government, which I didn't think was possible after all that Katrina has wrought.

Update, 10/8/07:
After the internal came out, and it became readily apparent that the GAO investigation was inadequate, Senator Landrieu said she strongly encouraged GAO to revisit their findings:
"...the GAO must keep digging to ensure that Congress and the people of New Orleans have the truth"
That press release is here. That was on June 8th.

It took a while, but the GAO did restart their inquiry. Oddly, it was MWI themselves that first revealed this publicly. On September 12, 2007, they responded to an article posted on a website called CorpWatch around Katrina's 2nd anniversary. The original article summarized the pump story to date. MWI's response attempted to rebut it, and accidentally revealed:
"The GAO is now in the process of pursuing an additional inquiry and we welcome
the opportunity to meet with them again."

I seriously doubt they "welcomed" the opportunity to be investigated, but that's standard corporate-speak.

This was confirmed on October 8th, along with some other huge news:

Times-Picayune, Monday, October 8:
Third probe of pumps on canals requested

Associated Press, Monday, October 8:
Report: New Orleans pump contract likely handled improperly

Ms. Garzino went to the Office of Special Counsel, an independent federal investigatory agency. Her referral raised enough eyebrows at OSC that they ordered Secretary of Defense Robert Gates to further investigate the entire imbroglio. Going through OSC provides an extra degree of protection to ensure the integrity of the investigation.

The existence of the Defense Department investigation was revealed by an illegal leak of its existence from within the Pentagon to a whistleblower group called PEER. Their source has likely lost his or her job, because PEER did not bother to submit a FOIA request for the documents they leaked to the Times-Picayune. Had they done so, their source would have been protected, and the DoD investigation would have likely gone a lot smoother.

Be that as it may, the documents they gave to the Times-Picayune are attached to their October 8, 2007 press release here. The direct link to the documents is here.

Thursday, June 28, 2007

Cover up, Part 2

In Part 1 of this report, we detailed Colonel Bedey's lie on national television about full performance testing of the pumps and drives. Bedey claimed the Corps had "decided" in the spring of 2006 to move that testing from the Florida factory of MWI to the worksites in New Orleans.

We demonstrated the transparent falsity of that lie by looking at actual contract documents, emails, and analysis within the internal Corps floodgate pumps report. All of those sources show that Corps had no intention of conducting full performance testing at the worksites. They also show that there has not been any full performance testing at the sites for over a year, a clear violation of the contract. Colonel Bedey made up out of whole cloth the fiction that full performance testing would be conducted in New Orleans. The lack of such testing is a serious shortfall in the performance of the contract.

In this part, we'll look at the implications of the lack of full performance testing, as well as the Corps' actions in trying to avoid doing such testing.

What does the lack of full performance testing mean?

For one thing, it completely undercuts all of the Corps New Orleans District's reassurances about how great the pumps are working. The fact is they have no way to determine whether all of the pumps are working up to spec, because through their inaction they have illegally de facto eliminated that part of the contract.

Remember what the internal investigation says about full performance testing of all the pumps:
"This testing was not performed in accordance with the contract requirements and a significant credit is due to the Government for the nonperformance of it"

This is in the present tense, i.e. the full performance testing was not performed as of June, 2007. Perhaps that's why Bedey said earlier in the interview (see my June 13th post) that the internal report was a snapshot of September, 2006 conditions: he was clumsily trying to undercut the fact that for over a year, he and the people under his command and their supplier have likely colluded to avoid enforcement of a vital aspect of the pump supply contract. Enforcement of that provision - and its probable results that the pumps don't meet the spec (see below for backup on that) - would likely land the Corps and MWI in court, and due to the New Orleans District's cruddy recordkeeping, MWI might win.

But what about all the testing the Corps says they've run? They say the pumps are ready

All those dog and pony shows the Corps has run for the local media, which the media then turns into b-roll footage for the nightly news... none of them were actual performance tests over the entire range of the pumps' required flows and heads. Bedey admitted in the press briefing when the internal report was released that none of the latest round of "tests" went longer than 45 minutes. There's no way one could run a satisfactory flow test in such a brief time.

The Corps' own press release from May 31, 2007 (just a few weeks ago) refers to them as "demonstrations." That's also a term that appears in the very vague section on field testing in the original contract (a section that was taken straight from MWI's own specifications). It's probably a key semantic difference in Corps-speak which is conveniently left unexplained to the layman. Dollars to doughnuts, a "demonstration" does not qualify as a full performance test. But the Corps is perfectly happy muddying the waters for the unitiated.

In running all these "demonstrations," and in not performing the contract as it is written, the New Orleans District has decided not to determine if what the taxpayers have spent millions of dollars on actually meets the specification! It also appears that in portraying the demonstrations as legitimate performance tests (schizophrenic, or maybe duplicitous, quote from Corps May 31 press release: "they've been successfully tested"), there has been a coordinated effort to cover up the lack of full performance testing.

Why not run the full performace tests?

Why wouldn't the New Orleans District and MWI want to know for sure if the pumps meet the spec? Why not yank every one of the pumps out one at a time, along with their twinned drive unit, mount them on a barge in the lake, and run 'em like they're supposed to be run under Hydraulic Institute testing standards? That is, why not test the pumps through their entire range of performance as they were specified?

The internal investigation also provides that answer: the pumps don't meet the specification. According to previously unreported limited, but pretty rigorous, pump testing performed late last year (internal report Attachment #3) and earlier this year (Attachment #13) by Corps hydraulic scientists, all the pumps fall below the requirements for flow. In factory testing last fall, when a new pump curve was generated (where was the certified version from MWI?), the shortfall was in some cases over 5%.

In field tests just two months ago at London Avenue, the numbers were much worse, with capacity shortfalls of over 50%. The Corps researcher guessed that the April, 2007 London Avenue results were due to poorly calibrated flowmeters or perhaps bad placement of the flowmeters, and used the pump curve generated in the fall testing to determine flow (using the pump curve, the Corps researcher again called out about a 5% flow shortfall for the April test). But that pump curve itself has errors and scatter, meaning the error in the April test was compounded.

[Special aside on flowmeters:

The London Avenue tests used acoustic flowmeters placed on the 9'-0" manifolds, just as I had suggested was the best place for them in my May 1 post. Even so, the Corps researcher got pretty cruddy results.

It is instructive that the Corps researcher in April, 2007 (just two months ago) did not depend upon the readouts from the individual flowmeters already installed on 5'-0" discharges of the pumps, but instead installed new flowmeters just for the test. That says to me that the meters on the 5'-0" discharges are - as I suspected - worthless. Keep in mind those are the meters whose results give Bedey the Wheaties to make his pronouncements that everything's shipshape. Thus, Bedey's full of it.

The poor showing of the meters on the 9'-0" tubes also reinforces the notion that the ones on the 5'-0" discharges aren't giving good data. If the Corps researcher couldn't get good results with 4 diameters of runup to the meters (the approximate amount at London's 9'-0" manifold pipes), how in the world could meters with just 1 diameter of runup (those that are installed on the individual 5'-0" pump discharges) be expected to give anything but garbage?

Besides all that, though, there remains the question of whether the meters in the April test were way off - and their results can be dismissed - or whether the Corps researcher was just engaged in guesswork as to the results of a 50% capacity decrease. He refused to express confidence in any flowmeter readings at the floodgates when I spoke with him recently, and said it was still a subject of much debate. He also said he didn't want to seem like he was "covering anything up," but that he needed to go through channels before he discussed anything further. I told him that we already deep into hurricane season, and the time for "channels" was long past.]

Frankly, the Corps has no real idea how much water is flowing through the pumps (because they don't trust the flowmeters that are in place now - where have I heard that before?), but they are sure it's less than what it's supposed to be. In any case, there's no real basis for Colonel Bedey to pronounce that the pumps are working as designed.

Spread over the 18 units at 17th Street, a 5% capacity decrease means there's effectively only 17 pumps there. A 50% capacity decrease ... well I think you can figure out how bad that is. That's hardly good value, and is indicative of very poor engineering and even worse stewardship over the funds of the taxpayers.

Any other reasons the Corps wouldn't want full testing?

There's the obvious one: when subjected to similar testing for a week in Florida, the pumps failed at a spectacularly high rate.

They also failed to meet the specification for flow and head in that testing.

All of that is laid out in Ms. Garzino's memo, the shop inspection records, and in the internal report. All of them describe numerous failures throughout use of the pumps over a long timespan. Again, that's hardly good value.

Pulling the threads together

Colonel Bedey lied repeatedly in his interview with CNN's Anderson Cooper on June 11th. His biggest whopper - that the pumps would be performance tested in New Orleans instead of Florida - is undercut by a mountain of direct and indirect evidence. In fact, not only was the performance testing not moved, it has never been performed at all.

Bedey's lie also unspools - and perhaps help explain - a disturbing series of events.

1) After just one week of full performance testing, that testing was stopped on the authority of individuals who were not allowed to make such a change.

2) Then, for the last year, the Corps New Orleans District (NOD) has decided not to enforce the provision of the contract requiring full performance testing of all pumps, and they have not documented such a substantial change in the normal Corps way - i.e. they are operating outside of normal Corps channels without authorization to do so.

3) Instead, the Corps NOD is holding public "demonstrations" of the pump systems which are portrayed to the public (through the media and the New Orleans District Public Affairs office) as adequate performace tests in conformance with the contract.

4) The few semi-rigorous, non-"demonstration" tests done on the pumps since their installation show them falling below the required head and flow as called out in the contract. The first testing that showed the shortfall happened April, 2006. And then there was testing in November, 2006, and then more testing in April, 2007. Shouldn't they be following up on this?

6) The New Orleans District is making millions of dollars in poorly documented payments to a supplier that - according to the Corps' own hydraulic scientists and engineers - did not deliver what was specified.

7) The New Orleans District contracting office is acting as extremely poor stewards of public funds, leaving out - or perhaps not even creating - the required justifications for many of the over 30 modifications to the pump supply contract. They then refused to turn over a complete accounting of the project to the internal investigators:
"The Contracting Officer was requested on 11 April 2007 to provide a total dispursement screen from CEFMS to verify the actual expenditures to date on this contract. This was not provided to the Artman Team by the Contracting Officer." [Adobe pages 21-22, internal report]

Does any of this sound like the normal way to run a project? No, of course not.

But it does sound like the way potential indictees act when they are trying to hide their actions.

All of this - the cruddy accounting, the poor recordkeeping, the willingness to turn a blind eye to supplier inadequecies, the lies to the public - it all points to a cover-up of likely motivations - people in the Corps (Bedey, St. Germain, Bradley) were so interested in getting MWI's equipment on site and portraying it as non-defective that they would do anything for over a year to dupe the public into believing they were protected. Surely they should lose their jobs for this.

How in the world can the people of New Orleans, the federal government, those within the Corps, and anyone else trust these folks to build a functioning flood protection system when they engage in unethical behavior and skirt the law and their own regulations?

In light of the lives and property at stake in New Orleans, what could possibly motivate a person so strongly to do such things? I'm pretty sure only subpoenas are going to let us know the complete truth. I hope the Justice Department is paying attention.

If the Corps wishes to dispute this, then they need to make the involved persons (specifically Maria Garzino, Jim St. Germain, and Dan Bradley) available to do so. Anything else is just bloviation and spin.

Saturday, June 23, 2007

Cover up, Part 1

Introduction

Today I'm starting a multipart series of reporting and analysis of the facts revealed in the Corps' internal report on the floodgate pumps and drives. Today's part focuses on what I believe is the latch to Pandora's box: a lie by Colonel Bedey on national television.

That lie on CNN leads to a horrible conclusion: the Corps has deliberately avoided enforcing its own contract to keep egg off its face and to protect its supplier. Doing so is, well, crazy, and possibly illegal. One would think this is enough for people to lose their jobs.

The Quote

On the June 11th edition of CNN's Anderson Cooper 360 (transcript here) there was the following exchange between Cooper and Colonel Bedey:



"COOPER: There was a report last year by an Army Corps of Engineers engineer who said it's not working. The pumps won't work. And just this past Friday three engineers backed up that report. So do the pumps work?

[...lots of other stuff about how much Bedey loved and appreciated being investigated...]

BEDEY: "...And in doing so [putting in the floodgates and pumps], we had to make some hard decisions. Some of those decisions were to forego all of the factory testing and get the pumps in the water, knowing that we would have to do all the testing and commissioning here on site.

"We've done that. We've been successful."

I've added the emphasis, because it opens up a can of worms. It is a slight variation on the standard line the Corps New Orleans District has used to explain away the installation of defective pumps last year, events revealed by the release of Ms. Maria Garzino's memo this past March 13th. Usually they say they wanted to get pumps in the water, and then "troubleshoot" them later. However, this time Bedey switched it up a bit and mentioned the testing that was so thoroughly eviscerated during the pumps' and drive units' time in Florida.

Bedey's used this line before, though not in such a prominent forum. In a piece of Corps propaganda from April of this year, he said, "We decided we would work out the final testing on the pumps in place."

The language for that testing in the contract calls for full performance testing of every pump system in accordance with Hydraulic Institute standards. This would mean running the pumps through their entire range of flows and pressures under as close to "real world" conditions as possible. This was the segment of the testing which was unofficially "eliminated" by unauthorized Corps personnel and MWI in Florida after just a week of running the pumps last April. The exact day of the decision was April 18, 2006. The modification to the contract dealing with changes in the testing did not come until two weeks later, on May 2, 2006. As we will see below, the official modification did not eliminate the contractual requirement for full performance testing.

Bedey is saying the Corps had/has every intention of still doing that contractually obligated full performance testing; they would just do it on site in New Orleans.

What's interesting is that, unlike many of the other standard excuses the Corps uses, this one can be truth-squaded with documentation now available courtesy of the internal investigation. So in this part of the report, we'll examine the truthfulness of Colonel Bedey's statement. In the next part of the report, we'll take a look at its implications and consequences...

We'll look at three sources, all from the Corps themselves: 1) the text of the internal report, 2) the relevant contract modification, and 3) contemporaneous internal emails.

Exhibit 1: The internal report

The report has extensive passages on the elimination of contract-required full performance testing. It has this to say about the current state of affairs:

"Modification P00004 revised the static test procedures by clarifying the actual steps to follow but no where did it delete the requirements of the full size water testing for each pump. This testing was not performed in accordance with the contract requirements and a significant credit is due to the Government for the nonperformance of it."

So the requirement for full size testing has yet to be fulfilled. That's odd, because Bedey said on CNN they've done that testing and it's been "successful." Hmmm...

Exhibit 2: P00004

As the quote above from the internal report shows, Bedey was talking about a May, 2006 modification to the floodgate pump contract called P00004, which dealt with testing of the pumps and drives. The entire text of P00004 is included as Attachment #11 to the internal report. It is the only modification given such treatment, which attests to its importance.

For your convenience, I've broken it out as its own file. We know that P00004 is the only modification relating to changes in testing because the internal report acerbically notes, "Modification P00004 (See Attachment #11) was the alleged modification that revised the testing requirements of the contract."

Let me cut to the chase on this one - there's no mention in P00004 of eliminating the complete factory testing and replacing it with complete field testing in accordance with HI standards.

Assumedly, if Bedey were telling the truth, some mention of changing the testing from the factory to the site would have been included P00004.

So the only relevant modification to the contract does not mention what Bedey said it mentions. Hmm...

Exhibit 3: The emails

The internal report also contains contemporaneous internal emails among Corps personnel assigned to the Pump Team. Were Colonel Bedey telling the truth, the movement of the full performance testing of all pumps from Florida to New Orleans would surely show up in these emails.

Instead, three critical emails from late May, 2006 (when these pumps were supposedly - according to Bedey - to be fully tested) show no evidence of a testing plan for the site. They also show no evidence of a paperwork trail to back up Bedey's assertion.

The emails are on Adobe pages 122, 123, and 124. I have broken them out as their own link here.

First is a May 17, 2006 email from Joe Thomas to Maria Garzino and Dan Bradley. It states:

"Would like to establish a QC/QA program for field testing of the engines and pumps. Need some help to determine what testing is needed; parameters of the field test and what should be recorded for these tests and what is deemed an acceptable field trail [sic] run. What field tests will be run? How long? Acceptance criteria? Flow measurements? Can anyone develop a document so that we may record pertinent data and information. Such items as pressure; temperatures of system ??? Any help would be great."

Mr. Thomas appears unaware of Colonel Bedey's supposed (read: fictional) plans for full performance testing of all pumps.

Next is a May 17, 2006 response email from Maria Garzino, the Pump Team Leader who was then on site in New Orleans, and would obviously be the person best qualified to know if there had been any plans to move the factory testing to the field. It is sent to a bunch of people, including Jim St. Germain's cohort, Dan Bradley. The tone of the email is almost pleading.

"I do not have any information on how we are to proceed with field testing – I am unaware of any direction given by MWI as to what they envision for field testing (I have asked on many occasions for this info and have not to date received it – if it has been sent, possibly Dan [Bradley] has it then?) – I am also unaware of any direction that Task Force Guardian has given on how we are to proceed on our own as far as field testing of the Pumps and Drive Units – I have heard there is discussion in this area, but I am unaware of any direction given."

The person who would be responsible for overseeing field testing that should mimic the factory testing (which she herself witnessed) had no information about field testing.

There still remains the remote possibility that Ms. Garzino just didn't have the right memo in her inbox. That possibility is foreclosed by the email to which her email is attached. That email, dated May 31, 2007, is from Jim Bartek to Steve Farkas and Bob Hoffman (why in the world is Ms. Garzino not included?). It states,

"I have not received a memo from Jim [St. Germain] regarding testing. All I have received is the email below that I think you sent out [Ms. Garzino's May 17th email]: I would agree with the recommendations except to add monitor vibration. Keep in mind this was a supply contract and the spec is very vague regarding field testing. It just says 'The pump manufacturer shall provide for final inspection and testing of the system and shall make necessary adjustments to the control system prior to actual start-up tests. Start-up tests and demonstration shall be performed by the pump manufacturer’s representative and the Contractor, and witnessed by the Government…'"

Notice what Mr. Bartek is citing when referring to field testing: the original contract! He's not referring to this phantom modification that Bedey is making up that somehow justified speeding up and eviscerating the factory testing. Bartek's citing the original contract.

So the contemporaneous emails show no evidence of what Bedey asserts.

The exhibits lead to one conclusion

Thus, we have the internal investigation, the relevant contract modification, and internal emails all saying the same thing: the Corps New Orleans District willfully ignored a vital part of its own contract; after P00004, there was no plan for full performance testing of the pumps, and there still isn't a plan for full pump-and-drive performance testing to this day.

Put another way, Colonel Bedey has made up a fairy tale to cover up something far more disturbing. The Corps was flying (is flying?) by the seat of their pants when it came (comes?) to the on-site testing, working without the paperwork to back themselves up and safeguard the government's (and the taxpayers') interests. Bedey, St. Germain, and Bradley know this. This is a major, major violation of construction contract administration. But it gets worse.

In the next part we'll look at why they're doing this and the consequences of not conducting full performance tests.

Wednesday, June 13, 2007

AC360

Colonel Bedey popped up on CNN's Anderson Cooper 360 Monday night. You can find the transcript here. It was his first interview after the release of the Corps internal investigation into the floodgate pumps. Cooper interviewed him while standing on the deck of the 17th Street gates (talk about home field advantage).

He put out more than a few falsehoods in the interview.

Lie #1: [speaking about the internal investigation] "BEDEY: That report -- that report that was just published last week was actually a snapshot in time of September of last year."

What? Um, the date on the front of the report is May 11, 2007. There are pieces of correspondence attached to the report which are as recent as May 7, 2007. There are ten technical recommendations for work which (according to the report) have not been completed, including raising the hydraulic reservoirs - which was supposed to be done over a year ago.

My guess at what Bedey meant? I'm not really sure. Frankly, it sounds like something a crazy person would say, like pointing at the sky and saying, "What a lovely shade of green!" Maybe he slipped and let out an old talking point that referred to a previous draft of the report which only examined September conditions (the report shows a flurry of work last September, including site visits and document collections). But the report now is most definitely written in the present tense, not the past.

Lie #2: "Well, we're absolutely stronger today than we were pre-Katrina. This pump, this interim closure structure is an example of that."

Again I say, huh? While the gates themselves might be stronger than the outfall canal walls (emphasis on might, since the system has yet to be battle tested), everyone locally knows there's less pumping capacity at the gates than at the Sewerage & Water Board pumping stations. That's why there's more pumps being put in at 17th Street and London Avenue, and even then there won't be enough pumps to match the interior stations' outputs. So in a fundamental way, we're weaker today than we were pre-Katrina. We can't evacuate rainwater during a hurricane as fast as we could pre-Katrina.

It's easy to slip stuff like this past reporters that are only in New Orleans for the day.

By the way, what's this "we?" Bedey lives on the North Shore. He should really refer to "New Orleans" rather than grouping himself in with the city.

Lie #3: "The commitment of the administration of the United States and the Congress of the United States was to provide the 100-year level of protection for the people of this great community and of the nation."

I really don't like straying into political territory, but I have to just put a toe in here.

This is what President Bush said in Jackson Square on September 15, 2005:"City and parish officials in New Orleans, and state officials in Louisiana will have a large part in the engineering decisions to come. And the Army Corps of Engineers will work at their side to make the flood protection system stronger than it has ever been."

Now, if the flood protection system was supposed to be able to handle a 100 year storm (the basis for the FEMA flood maps), then wouldn't "stronger than it has ever been" refer to greater protection than a 100-year storm?

I know I'm being a little obtuse here. I recognize the realities of Congress, the Administration, budgeting, and politics have whittled the stated goals of improving New Orleans' flood protection down to, "We're going to give you what we said you had for 40 years." But it's still wrong.

There's more to come from this interview...