London Avenue canal tests

Corrections included 6/1/07
Updated 7/28/07, see below

[For the latest on London Avenue canal testing, see my June 11, 2007 post, "Testing, testing." It has been updated quite often.]

This past Saturday, the Times-Picayune printed a detailed article about upcoming tests along the east bank of the London Avenue canal. It was buried on page 2 of the Metro section, and didn't appear online until yesterday. The article's not too long, and I strongly suggest you read it.

The gist of it is that the Corps will be isolating a 150 foot long section of the eastern wall of the canal in exactly the location I have written about previously: about 4000 feet north of pump station 3, around the Dillard University tennis courts, or perhaps just a bit north of there. [correction, 6/1/07: I've been using the wrong location. The article says, "just south of Burbank Street on Warrington Drive," which would put the location of the testing here. I regret the error, which sprung from a January conversation I had with a well placed member of the Corps' New Orleans District. A fuller correction can be found here.]

They will drive a sheet pile coffer dam inside the canal to surround the area of concern. This will look exactly the same as the coffer dams driven around the breach sites, including the one still at the 17th Street breach. They will then pump water into the dammed area in a controlled manner, raising the level of water in six inch increments and letting the level stay for four hours each time. There will be lots of sensors, penetrometers, piezometers, and doo-dads to measure what the wall and levee are doing the entire time these increasing levels of water are placed against them. They'll take all that data back to the batcave in Vicksburg for analysis.

According to this article, none of this will happen until May, which is the end of the rainy season around here. Since the top reason all of this work is happening is to determine the safe water level which is acceptable for drainage from rainstorms (tropical or not), this is troubling. What's the delay? Paperwork:

"Bedey's office is pushing to get necessary agreements executed with the New Orleans Sewerage & Water Board and the Southeast Louisiana Flood Protection Authority-East to get the work started as quickly as possible."

There's no indication the T-P called the S&WB or the SLFPA-E to confirm this; they just took Col. Bedey's statement at face value.

There's also the usual Corps happy talk, again unrebutted by the T-P:

"The results of this so-called "site-specific testing" could result in a higher water level during the coming hurricane season that opens June 1, Bedey suggested.

"Even an additional six to 12 inches of water would allow the gates to remain up longer in the face of a tropical storm or hurricane. It also would allow the New Orleans Sewerage and Water Board to pump, without interference, more storm water out of the canal during a heavy, non-tropical rain event."

Of course, it's also possible that the testing could result in a lower Safe Water Level, or no significant change. That's why the testing is occurring. But of course, the Corps presents only the possibility that makes them look best. This is not how reputable engineers do things. Engineers don't assume anything before a test, for fear of skewing the results or the analysis. We definitely don't publicly hype one particular result weeks before the test even starts. But we all know what the Corps thinks of testing (see: pumps, defective).

What is also interesting about the second paragraph above is how the Corps has changed their tune now that the importance of the Safe Water Level has been revealed, via the shutoff of a 1000 cfs pump at S&WB pump station 3 during the December 30, 2006 storm, when the London Avenue SWL was exceeded. Now they say that 6 to 12 inches is very significant.

However, last September, when the four foot Safe Water Level at London Avenue was announced (it was a drop from the previously announced five foot level), they were downplaying it big time. In a September 12, 2006 article, here's what Corps pump person Jim St. Germain said:

"St. Germain would not predict just where the London canal's safe water level ultimately may be established, but he cautioned the public not to read too much into a foot or half-foot of change either way.

"It's good that the 17th went up a foot, but if London does go down, I wouldn't get too excited. We'll use the best technology we have in determining when to close those gates, but it isn't exact," St. Germain said. Only the specifics of each storm, including the rate of rise in the water level of the lake and each canal, would suggest when gates should be closed."

First off, you'll notice St. Germain is not quoted in this latest article. In fact, he seems to have gone underground since the pump testing imbroglio broke on March 13. He even gave a "no comment" to WWL-TV while escorting a Dutch delegation around the 17th Street floodgates (I watched the report, and the "member of the Corps of Engineers giving the tour" was St. Germain).

More importantly, as we now know the Corps has been very "excited" about the London Avenue Safe Water Level for a long time, including probably at the time of that September article. That is, they've known the exact location of the weakest point in the London Avenue canal walls since at least September of last year. They have allowed residents, businesses, and institutions (including Dillard University) behind those walls to rebuild for the last seven months without doing a thing to reinforce - or at least isolate - that section of wall. Instead, they've puttered around, gone out on boats to look at it, shuffled paper, and planned tests [correction, 6/1/07: see correction above for real location of load test].

Here's an idea: drive the sheet piling now, then worry about the tests later. We've already had an incident, almost three months ago (not to mention the December 21, 2006 storm, when the canal depth got within five inches of the Safe Water Level). What does it take to make these people move, an actual levee breach?

By the way, the most menacing part of the Saturday article is the last paragraph:

"Depending on the results of the field work, the corps team could decide to test the entire canal. But Bedey said it was premature to speculate on whether a test that broad will be needed."

Prepare for a test of the entire canal.

Update, 7/28/07

For the latest on London Avenue canal testing, see my June 11, 2007 post, "Testing, testing."

The fire

This week's Gambit Weekly contains a lengthy article about the floodgate pumps. Mentioned for the first time is the fatal fire that occurred at MWI on March 23, 2006. This fire occurred about 2.5 weeks before pump testing commenced.on April 11, 2006.

On March 23, 2006, a fire erupted in a warehouse at 33 NW 2nd Street, Deerfield Beach, FL. It was reported on briefly by the local TV news (courtesy Lexis-Nexis AlaCarte):

"Global Broadcast Database - English - 3/24/2006 - 46 words Danger on the job for a welder in deerfield beach. The man working at pump [sic] when splashes cause insulation to catch fire. The welder suffered serious burns and taken to broward general medical center. He was transferred to jackson memorial hospital. Crews are now investigating...."

That man died on March 28th of second and third degree burns at Jackson Memorial Hospital in Miami. I have received his mother's permission to report upon these events.

33 NW 2nd St. is MWI's production and testing facility. It is the site of the "wet" testing mentioned in Maria Garzino's memo. As with any fatal accident, The Department of Labor's Occupational, Safety, and Health Administration (OSHA) investigated. Their inspection number was 309430627. As you can see, they eventually levied a fine of $6876.00 upon Industrial Diversified Services. Industrial Diversified Services was a subcontractor to MWI hired to install an overhead bridge crane in the testing facility. The crane was meant to lift pump units in and out of MWI's testing tank. The bridge crane (or its absence) is mentioned by the Corps QA inspector from Jacksonville in the April 18, 2006 report (page 46 of Ms. Garzino's memo):

"5. To date, the Contractor's subcontractor is still trying to set up the bridge crane for pump testing in their wet tank. They indicate that this will allow them to cut time out of the setup process. With the crane that is currently being used, the setup has takenapproximately 8-10 hours. If they can get the bridge crane installed, the contractor believes that it will cut setup time down to 4-6 hours."

The Florida State Fire Marshall also looked into the fire and, after a brief investigation, ruled the fire accidental. Material samples of the insulation were not taken by the fire marshall's office.

I received both the OSHA report and the fire marshall report through records requests. The OSHA report had a lot of duplicative information, so I culled it down to a manageable length. Should anyone want to see the pages I didn't scan in, they're welcome to ask me for them or get them from OSHA in Ft. Lauderdale. The OSHA report came exactly as you see it, with the redactions. The OSHA report is here.

The fire marshall report came unredacted, with witnesses names and addresses. I have removed those myself, out of respect for people who happened to see a horrible tragedy unfold before their eyes. Should you be interested in receiving the complete report, you can contact the fire marshall's office in Tallahassee and pay your $10, and they'll send it along. The fire marshall report is here.

I also received the pictures below with the OSHA investigation report. I have not altered the pictures except for adding dates.

The pictures show extensive fire damage to the entire building, including the steel to which the bridge crane would be attached.

As described in Ms. Garzino's memo, and the Quality Assurance inspectors' reports attached to the memo, testing began on April 11, about 2.5 weeks after the fire. For her account of April 19, 2006, almost a month after the fire, Ms. Garzino wrote the following:

"During the evening Drive Unit 8840 was set up to undergo an E/R test - the test facility is pitch dark - I have asked MWI if we can have 30 minutes to go to the drug store and buy flash lights as I was concerned that we could not see any of the plethora of machinery that was running around us, the obstacles on the ground, and the drive units themselves (see my pics of this condition) - I was especially concerned for the safety of the QA's as the Drive Units have been experiencing excessive hydraulic failures and injury from not being able to see the failure occur and get out of the way in time was a large concern for me - MWI informed us they were 30 + minutes from starting so all Corps personnel left to get flash lights."

Now it is clear why "all Corps personnel" had to get flash lights. The building was without lights and electrical power for many - if not all - "wet" tests conducted on the pump units and the drive units after dark for many weeks. One only has to read the memo and the attachments to see that there were such tests conducted nearly every night. All of this was confirmed by MWI's spokesperson, Mike Powers, in the Gambit article. Powers claims generators for lights were brought in later (assumedly after the timespan of Ms. Garzino's memo, which covers April, 2006. Of course, the testing was about 2/3rd's done by the time she handed in her memo, so we'd only be talking about around two more weeks before testing was finished.). However, even a single night without lighting should be unacceptable to any company that truly values its' employees' safety and the safety of federal employees working in its facility. That such a situation continued for weeks - apparently without any additional attention from the state fire marshall or OSHA - is scary. Finally, if this haphazard approach to testing is typical, what questions does it raise about the construction and assembly of the pumps?

My address

I just realized that this entire time, while I've thought my email address was available on my Blogger profile, it actually wasn't. It's there now.

The Corps defense

The Corps has been playing defense on the defective pump story since it broke last Tuesday. I thought it was time to deconstruct some of their defenses.

Defense number 1 Corps says they went to the media within a week or two of Ms. Garzino handing in her memo on May 4th to let the public know about these problems. Col. Bedey used this one on the Garland Robinette "Think Tank" show on WWL radio on March 15. You can download the entire interview from WWL's podcast page here.

What really happened:
I've been following these stories very closely, and I've been archiving nearly every pertinent article. I can only assume they are referring to this article in the May 12, 2006 Times-Picayune:
Gates won't be ready by June 1

Go ahead and read that article. See if you can find anything in there about problems with the pumps' functionality. In fact, all that's mentioned are the following:

a) There were problems installing the gates at the London Avenue site. Nothing is mentioned about pump functionality there.

b) The Corps says the 17th Street site is small, and there wasn't enough time to get the MWI pumps in.

c) Regarding Orleans Avenue, which has the exact same pumps as London and 17th St., the article says the following, "The floodgate-and-pump project in the Orleans Avenue Canal was a less complex job and will be ready on June 1, Setliff said."

I don't see it. Do you?

Defense number 2 The Corps says they've open and honest about all this. Colonel Bedey also said this on WWL, and it's been put out there by other Corps officials as well. They point to the reportage of vibration problems from last August as a prime example of that openness and honesty. Incidentally, the Times-Picayune also said the same thing (with regards to their own coverage) last week during their near-complete burial of this story. Let's take a closer look at that coverage.

What really happened:

First off, there were no Corps press releases at the time of those problems. All information was reported second hand by the media or other sources.

Secondly, the vibration problems last August are not what Ms. Garzino's memo was about. Her memo and the independently authored attachments were mostly about numerous design problems with the hydraulic power units and the testing methodology itself, as well as deficiencies in contract oversight by New Orleans. The Corps is attempting to confuse the justifiably enraged public by conflating two completely different stories. They will not succeed.

Thirdly, I have found only one stray mention of any problems besides the pump unit vibrations. It is startling, but it is just a single mention in one article. Otherwise, all other reporting was strictly about vibrations. The hydraulic drive units were never mentioned before or again.

But to give proper due, let's look at that single mention:

Problems emerge on new pumps - Times-Picayune, 8-24-06:

This article is incredibly interesting, not for what it says (mostly stuff like vibration problems, we're working on it, deadlines slip, etc), but for what it tantalizingly whispers about the now infamous testing at MWI in Florida. Buried deep in the article (and oddly presented as a Corps defense for the vibrations) is this passage:

"The problem pumps were specially manufactured for the canal projects, and corps officials were unable to explain why the vibrations were not detected during tests performed in the manufacturer's lab under corps scrutiny.

"We have to find out why the pumps did what they did, but until we do further investigation, we won't know," St. Germain said.

During testing in the manufacturer's laboratory, St. Germain said, corps engineers saw problems with the drive units, but not the pumps themselves. "Because of what we saw, we had them do more drive testing and less pump testing in order to meet our schedule."

Had problems with the pumps themselves been detected, St. Germain said, the corps would have ordered further testing on that as well. St. Germain said all the pumps were tested to determine if they properly held hydraulic pressure, but only a few were performance-tested in a tank of water in the manufacturing lab that is large enough to hold a 20,000-pound pump while it runs."

Remarkably, the T-P just lets this information lie there with no follow-up, either in the article itself or in any subsequent articles since then. Read carefully what is said in those four paragraphs. It is the bare-bones outline of last weeks' story, along with a stunning bit of candidness that Jim St. Germain snuck past the T-P. But I would hardly call it the same as explaining that information to the reading public, or the extent of what was described. In fact, when read within the totality of the article, it's almost an aside.

Still, when you look at just those paragraphs, you find:
a) He actually said they saw problems with the drive units, not the pump units. It's clear from context he's referring to problems which were not vibration-related. Thanks to Ms. Garzino, we now know the scope of some of those problems.
b) He actually said that they did "less pump testing" to meet the schedule. This is a shocking admission in the press of the Corps deliberately bypassing their own specs.
c) He also said that it wasn't the common-sense - and contractual - need to load test ALL the pumps with water that would have made the Corps actually load test the pumps. He actually said they only would have load tested all the pumps if problems had somehow announced themselves on the few pumps MWI and the Corps deigned to run water through. Riddle me this: how does one know if there are problems when water is put through the pumps, unless water is put through the pumps? That's through-the-looking-glass thinking if I've ever heard it.

Of course everyone now recognizes the significance of these statements, but back then only the people making them and those inside the Corps realized what they meant. Frankly, I didn't notice it myself until I got Ms. Garzino's memo and started going back through the news coverage, anticipating that the Corps would claim, "that's old news."

As I said above, those paragraphs in that single Times-Picayune story are the only mention I can find of any drive unit problems and other difficulties besides the widely reported August vibrations (which are really not what Ms. Garzino's memo and the attachments are about, as anyone who's actually read the whole thing can plainly see). Even then, they're referred to elliptically, as if it was okay that the pumps weren't being factory tested. If that's what the Corps and the Times-Picayune want to camp out on as "we already truthfully and openly reported that," I think they need to reevaluate their standards for communicating with the public.

[Side note: yes I realize this is a long post, but hang in there...]

For comparison, check out WWL-TV's coverage of the August problems:
17th St., London Avenue pumps have to be shut down during test run, 8-21-06

The key paragraph is here:

"[Colonel] Wagenaar said the problems could be mechanical or hydraulic or it could be related to the low water level in the canal. The pumps are designed to work in water that is three to four feet deep and the water in the canal was only about a foot deep. Tests were scheduled with pumps pushed further down into the water to simulate what actual conditions would be like during a storm."

Do you see any ackowledgement by the Corps of what they already knew: there were serious problems with the drive units and pumps, all of which had been explained in detail to them by their own employees (remember, the QA reports attached to Ms. Garzino's memo were written independently by other Corps employees)? I don't. Imagine if Wagenaar had said, "We're not surprised by this, since we had seen many other problems with these systems and have attempting to address them fully. In fact, our own people have reported those problems and we've taken their concerns very seriously." Of course, for him to be able to say that without it shocking everyone, there would have had to have been a series of statements all summer along the same lines, and the Corps would never, ever be that candid, even though they're spending the people's money on the people's protection.

It's also instructive to read the other stuff that was coming out throughout the summer of '06, before the vibrations came up but after they received Ms. Garzino's memo. For example, there's a detail-laden Times-Picayune article from July 19, 2006:
Drainage forecast cut at London Ave. Canal

Read that and tell me if there's anything about pump testing in there. There's not.

Then, there's the release of the inundation maps on July 26, 2006:
The actual press release and maps are here.
Corps issues maps of rainfall's impact, T-P, 7/27/06
Corps releases new maps showing flood risks this hurricane season, WWL-TV, 7/26/06

Read those articles and the press release. Anything about the functioning of the pumps? Nope.

Those maps show what happens when the gates drop. If there was ever a time to put out the caveat, "this data is based upon full pump functionality for the length of the storm depicted on the map," that was it. Instead, as with every other time the Corps spoke with the public, the media, and Congress, the pump flow capacities are very declaritively stated as fact. The Corps had opportunity after opportunity to tell the public they had serious problems with the functionality of the pumps, and they skipped every opportunity. They were not open and honest.

You might say, "Well, that's how the media covered it, but they leave a lot of stuff out." That's true. So where else was the Corps doing their song and dance last year? There were the monthly meetings of the now-defunct Orleans Levee District (OLD) for one.

OLD archived all their meeting minutes online. Here's what was reported by the Corps to OLD on July 19, 2006:

"Mr. Fred Young, U.S. Army Corps of Engineers (Corps) Project Manager for the Orleans East Basin under Task Force Guardian, reported on the status of the temporary pumps on the interim structures. All the gates can be closed, and work is continuing on the floodwall repair at the three breaches. Currently, pumping capacity is 2200 cfs at the Orleans Avenue Canal and 2800 cfs at the London Avenue Canal. The Corps is still working on bringing pumps on line at the 17th Street Canal structure.... The Corps feels it has what is needed with the 2200 and 2800 cfs capacity at the Orleans and London Avenue Canals to get through this hurricane season."

Here's the August 16, 2006 report:

"Mr. Young reported the pumping stations at the Orleans and London Avenue Canals are approximately 98 percent complete; the pumps can be run, however, some electrical and mechanical work remains."

September 20, 2006:

"Mr. Frederick Young, Project Manager for the U.S. Army Corps of Engineers(Corps), reported work is on-going on the pumps at the control structures at the three outfall canals. There appears to be better control over the pump vibrations, and the Corps anticipates having some type of resolution of this problem relatively soon. Additionally, by June 1st, the Corps anticipates having an additional set of hydraulic pumps at the 17th Street Canal to provide 7300 cfs capacity, and increasing the capacity on the London Avenue Canal from 2800 to 4300-4800 cfs. The capacity at the Orleans Avenue Canal is already at or above what can come out of that pumping station."

You get the idea. Every time they had a chance to mention the mechanical problems, to temper expectations, to let people know that the capacities they were publishing were theoretical, they passed on it. I've highlighted the passages dealing with the Orleans Avenue canal because - even though the same problems plagued all 34 pumps and drive units across all three floodgate sites - the Corps acted like each site was different. When it comes to the pumps, they're all the same. If there's a design problem with one drive unit, there's a design problem with them all. So the fact that construction was completed at Orleans Avenue meant absolutely nothing, because the pumps were still malfunctioning elsewhere, and the Corps knew it. That means that Orleans wasn't any better than any other site, no matter how much lipstick the Corps tried to smear on the site through their public pronouncements. In fact, the east pumps at Orleans Avenue remain out of the water to this day, as they have for the last six months. That's not open and honest.

The people of this city and this country deserved to know the risks they were taking coming back here. And every time the Corps talked pumps last year, even when the capacities were woefully low, they never included the fact that even those capacities were questionable at best according to their own experts. They withheld crucial information, and as a result, thousands of people spent money, time, and moved heaven and earth to come back. Yet all the time, the Corps knew that there was a strong likelihood of non-functionality - and any engineer outside the Corps' New Orleans District will tell you that equipment as critical as these pumps has to be absolutely, unquestionably functional under all conditions - a conclusion that can only be reached by full, complete, and professionally-conducted factory testing in accordance with industry-accepted standards. Field testing under conditions very far from reality doesn't cut it. They could have gotten a lot closer to storm conditions in the factory, but they chose not to. The fact that they didn't even test most of the pump units, the fact that the drive units appeared to have deeply serious design flaws, and the fact that they continually downgraded the testing requirements until they were practically meaningless (and behind their own Contract Officer's back, no less!) was information essential to citizens' and policy-makers' decision-making processes. All that was held back simply for them to retain what tiny shred of credibility they believed they possessed.

When it comes down to it, they were playing CYA by lying to all of us.

Calling BS

Now that the news about Maria Garzino's memo has broken all over the place, I thought a few points needed to be made.

Yesterday, the Corps flew Colonel Bedey, the head of the Hurricane Protection Office, back from Washington so he could be there when two local TV affiliates (Fox 8 and ABC 26) filmed four of the 17th Street pumps being run. The Corps pointed to this demonstration as proof that everything's just A-OK.

1) The Corps already ran these pumps last Saturday. The Times-Picayune buried a tiny article about it on page B-3 in Sunday's local section. Here's the article:

Corps pump tests get thumbs up

This article (and possibly the Corps' alerting the media about the test) was also a bid by the Corps to get out ahead of the news that broke yesterday. They obviously failed in that bid.

2) Those four pumps were ordered long after Ms. Garzino's memo and after the other 34 pumps started failing last summer. They were part of an order of six extra pumps to MWI (total charge, about $4 million for all six). These four were actually the last pumps installed - they went just a couple of months ago (they weren't fully hooked up as of January 26, 2007) and are not the subject of Ms. Garzino's memo. That is, they were not in the original order of 34. Using these pumps as a demonstration to show that the other 34 are working is pure lying. The Corps is probably going to turn on those pumps again tomorrow for the Mayor. They represent about 800 cubic feet per second of capacity (theoretically). Pre-Katrina, over 10,000 cfs flowed down the 17th Street canal.

3) The test last Saturday, like the one last night, only went for about an hour. That was not mentioned in the Times-Picayune article. But ALL the floodgate pumps (there are no spares in case one or more fails during a storm) will have to run for 12 or maybe 24 hours during a tropical storm or hurricane. Why isn't the Corps running tests for the media that last that long? Because they know the pumps and their drive units probably can't hold out for that long.

Don't buy what the Corps is selling.

Finally, I have one other hunk of information to add.

As a result of another FOIA request I made in May of last year, the Corps uploaded a number of documents to the IPET website this past February (the site that was set up to allow the public to understand the Corps' investigation into Katrina). I just found out about this two days ago when they responded to my FOIA lawsuit. You can find them by going to the IPET website, clicking on "Post-Katrina" on the left, and then "Project Informations Reports." You'll see the files listed there.

The documents are all of the revisions of the Project Information Report ("PIR" in Corps-speak) that authorized the floodgates and the pumps. PIR's are detailed scopes of work and budgets written by the Corps to detail ther projects. They are not generally released. I had previously obtained Revisions 0 and 1 of this document, despite the Corps ignoring my FOIA request for them. I did not know that Revisions 2, 3, and 4 existed until a few days ago.

Revision 0 (Oct., 2005)
Revision 1 (Jan., 2006)
Revision 2 (May, 2006)
Revision 3 (Jul., 2006)
Revision 4 (Oct., 2006)

In Revision 1, which came out in January of 2006, the Corps set up their entire strategy for protecting the three outfall canals. It was to be a two pronged strategy. There would be floodgates to protect against storm surge, and there would also be repairs to the walls along the canals. They set aside $29 million to do this at all three canals. I've blogged repeatedly about these bank stabilization contracts before, wondering what happened to the money and the plans. Well, now that I've seen Revisions 2, 3, and 4, I know part of the answer: the plans simply disappeared.

Revision 2, issued May 17, 2006, makes absolutely no mention of the status of the bank stabilization contracts (other than mentioning them in the budgetary chart - so apparently they still existed as ideas in May, 2006), or any plans for implementing them. Revisions 3 and 4 are similarly silent on those repair efforts - again, other than calling them out as line items in the budgetary chart (Note: at some point bank stabilization for Orleans Avenue canal was eliminated, since it disappears entirely between January and May, 2006). But since the walls weren't fixed (and still aren't), New Orleans now has problems with drainage even when there isn't a tropical storm. That is because the walls are so weak along the outfal canals (most severely along the London Avenue canal), the Corps has forbidden the local drainage authority from filling the canals above a level at which they deem a breach is imminent, called the Safe Water Level. At London Avenue, this level is four feet, which is the height of the base of the wall atop the levee. If that level is exceeded by the interior drainage pumps, some of those pumps must be turned off. This has already happened once, on December 30th. I blogged about this a few weeks back, and the Times-Picayune finally caught up to the story a few days later. The Corps, specifically Acting Branch Chief (Existing Pump Stations Branch) Jim St. Germain, was present in Pump Station 3 for that shutdown. In fact, it was Mr. St. Germain who recommended that shutdown.

That is the same Jim St. Germain around whom many of the events of the past few days are swirling. He's one of the Corps' New Orleans District's pumping guys, along with Dan Bradley and Dennis Strecker. All three play prominent roles in Ms. Garzino's memo, specifically in the downgrading and elimination of testing requirements without the prior knowledge of the Corps Contracting Officer, Cynthia Nicholas. Ms. Nicholas was the only person legally authorized to make such changes. You can see a picture of Messrs. St. Germain and Bradley in the Corps' February 6, 2006 newsletter. A picture of Mr. Strecker at his retirement in December, 2005 can be seen here. Mr. Strecker was evidently brought back from retirement, possibly as a contractor, just to deal with the pumps. He was at an April 18, 2006 meeting at MWI's Florida facility, representing the Corps.

Frankly, if the Corps had simply spent the money fixing the walls, which are part of the federally authorized hurricane protection system, there would have been no need for the floodgates at all.

The reason for Revision 2 was to propose a plan for expanding pumping capacity at the floodgates, specifically at 17th Street and London Ave. The initial order of MWI pumps did not give New Orleans nearly enough pumping capacity to pump out rainwater during a storm (I know, I know...). What is also really interesting about Revision 2 of the PIR is its ... surprising ... recommendation to perform that expansion by buying 32 more hydraulic pumps just like the 34 they'd already bought from MWI. 16 would be installed at 17th Street and 16 would be installed at London Avenue. This particular expansion never did take place, thank goodness.

But notice the dates of what was happening. The Corps upper-ups at New Orleans got Ms. Garzino's memo on May 4, 2006. Just 13 days later, they were recommending buying 32 more of the exact same pumps that they had been told wouldn't work! That's crazy talk! They pooh-poohed another idea to install much more robust, direct-drive, non-hydraulically-powered pumps that would have cost more (maybe) and been delivered perhaps a month later. Considering the original MWI pumps weren't working anyway, would a month have really mattered? Besides, they ended up buying the direct drive pumps anyway this past January, as I've also written about January 12, January 18, and February 11. The Corps finally reported the news in their February 21st newsletter, along with some admittedly snazzy graphics.

So to make this as clear as possible... Not only did the Corps install pumps they knew wouldn't work, not only did they simply give up throughout the entire year of 2006 on repairing the canal walls, they also recommended doubling the order of the known-to-them-to-be-defective pumps and drive units, assumedly with the same manufacturer - MWI.

The complete memo

Here's the May 4, 2006 memorandum for record, with all the attachments, from Ms. Garzino to Colonel Setliff. It's about 12.5 megs.

I received it as a result of a Freedom of Information Act request placed with the Corps. Unfortunately, the Corps blacked out all the names. I will be appealing this. Even so, the events described leave little to the imagination.

The memo

Defective

Lots of people have been finding their way to my blog today. It's because of this:
New Orleans pumps were faulty

"The Army Corps of Engineers, rushing to meet President Bush's promise to protect New Orleans by the start of the 2006 hurricane season, installed defective flood-control pumps last year despite warnings from its own expert that the equipment would fail during a storm, according to documents obtained by The Associated Press."

"The 2006 hurricane season turned out to be mild, and the new pumps were never pressed into action. But the Corps and the politically connected manufacturer of the equipment are still struggling to get the 34 heavy-duty pumps working properly."

And lots more where that came from.

So I assume people want to read this memo.

[Updated March 19, 2007]
First, an important bit of background...

This memo was not leaked to the press. I received it through a Freedom of Information Act request placed with the Corps of Engineers' New Orleans District (actually, it was originally placed in Los Angeles, and then L.A. transferred it to N.O.). The Corps released this memo of their own accord.
[end update]

Now, here's why I haven't posted it...

I actually tried to go to Kinko's last week, but they wanted $5.99 for the first page and $1.99 each additional page, and they said it would take three days to get it scanned. I wasn't about to pay over $150 to scan a document. I cast around the neighborhood looking for a scanner that could handle a big document, but I couldn't find one.

So in the interim, here's the text of the first three pages of Ms. Garzino's email. These are the main body of her memorandum for record. Behind this, there is a list of all pumps which failed before the pump testing requirements were eliminated (one week into the testing), then a listing of the testing results of all the power units, then a day-by-day chronology of her time at MWI's Deerfield Beach facility during the month of April, 2006.

Following all that are the official quality assurance (aka "QA") reports filed by other Corps personnel during their time at MWI. The Corps loaned personnel form their Jacksonville office to this effort. Most of the time, there were two QA's working at MWI, one working a day shift and one working a night shift. They would file reports of what they observed each day.

So here's the beginning of the memo:

Sent: Thursday, May 04, 2006 9:25 AM
To: Setliff, Lewis F COL MVS
Subject: FW: Defective Pumping Equipment
Importance: High
Sensitivity: Confidential

Col Setliff,

I have submitted the message below, with complete attachments, to Ms. Nicholas - Contracting Officer for the pumping equipment in question. I have called and visited her contract specialist, Ms. Rouse, to give her a "heads up" regarding general content of this message, and, have offered my assistance when/where ever necessary to answer questions/provide additional information should she feel it necessary. I have also dropped off a full copy of the MFR with complete attachments at your office this morning (I attempted to send this message last night with complete attachments and the server would not let me - I am attaching here only the docs that were generated by myself...).

I understand your schedule is extraordinarily busy and your time valuable, with this in mind, I wish to ask of you a few moments of your time so I may discuss information that is not contained in this memo and is more private in nature.

Thank you for your time and consideration in this matter.

Respectfully,

Maria Garzino
USACE Mechanical Engineer
Task Force Guardian
________________________________________

Sent: Wednesday, May 03, 2006 5:26 PM
To: Nicholas, Cindy A MVN
Cc: Rouse, Gayle E MVN
Subject: Defective Pumping Equipment
Importance: High
Sensitivity: Confidential

MEMORANDUM FOR RECORD

DEFECTIVE PUMPING EQUIPMENT

W912P8-06-R-0089 - Temporary Pumps for Three Outfall Canals, New Orleans LA
Date: May 3, 2006

To: Ms. Cynthia A. Nicholas, USACE Contracting Officer, Task Force Guardian

Subj: Defective Pumping Equipment Supplied by MWI - Pumping Equipment Not In Accordance With Contract Requirements

Ms. Nicholas:

It is the opinion of the undersigned that pump equipment supplied by the above cited contract, specifically the pump assemblies and drive units, that are arriving in the field and being installed daily, are not capable of fulfilling their function as intended by the original contract requirements - are defective - and will experience failure should they be tasked to run, under normal use, as would be required in the event of a hurricane.

Put simply, if the intent of the contract requirements is to have pump equipment capable of being turned on and used for prolonged periods of time in the event of a hurricane, then I believe the pump equipment will not function as intended. In fact, without extensive follow on supplemental operation of the pumping equipment after installation, and subsequent likely follow on repair of failed pumping equipment components (hydraulic pumps, hydraulic pump motors, failed high pressure hydraulic lines, etc.), I believe significant failure of the pump equipment can be expected. In addition, as the original contract deliverables (as provided by the written specifications, oral negotiations, and bid proposal) have changed significantly due to revised testing procedures, and without formal modifications to the contract, it is recommended that requests for payment from MWI for this pumping equipment should be extensively reviewed prior to such payment to ensure the government receives fair value for services/equipment received - especially if the governments intent was to receive fully functional pumping equipment that could be pressed into full service after installation without the threat of imminent failure - something the current contract documents/specifications impart.

This opinion is derived from events transpiring during continuous and diligent observance by the undersigned of full sized testing of these pump assemblies and drive units at the manufacturer's Deerfield Beach, Florida facility from the period April 7, 2006 thru May 01, 2006. These events have been documented, by myself and independently by Jacksonville District QA personnel, in reports as follows - and to be provided as attachments:

Listing of Pump Assemblies That Have 'Seen' the Testing Tank (actually pumped water) Failures/Successes - 4/6 thru 4/26
Listing of Drive Units Failures/Successes - 4/6 thru 4/26
Florida Trip 4/07 to 4/16 - MWI - Testing - Report No. 1
Florida Trip 4/17 to 4/23 - MWI - Testing - Report No. 2
Florida Trip 4/24 to 4/30 - MWI - Testing - Report No. 3
QA Shop Inspection Report #6 (4/11), #7 (4/12), #8 (4/12), #9 (4/13), #10 (4/14), #11 (4/15-17), #12 (4/18), #13 (4/19-20), #14 (4/21), and #15 (4/21)
QA-TFG- Reports for 4/25, 4/26, and 4/27

Finally, events outlined in the above reports highlight two disturbing, and still unresolved, pumping equipment failure issues - that being:

1) Cause of the voluminous failures of the hydraulic pumps on the drive units is still unknown at this time - the manufacturer of the hydraulic pumps (Denison) has not yet provided any official input as to the failures being caused by a plethora of "bad" pumps, or, point to an as yet unknown design deficiency with the hydraulic system. This situation would provide for the possibility of future failures of the drive units at 100% until a design deficiency can be ruled out - in addition, there is the very likely possibility, more probable actually, that damaged hydraulic pumps starting the failure process have "passed" testing and are currently slated to be, or have been, installed.

2) The original contract specifications required 100% load testing of all pump assemblies - this requirement has subsequently been eliminated, and to date, less than 25% of all pump assemblies have been load tested (leaving potentially 75% not load tested), and, of the eight (8) pump assemblies that have been load tested, one has only been run for a few minutes at best and one other was run at 1/3 operating pressure (the hydraulic oil barely got warm enough to register). Of the remaining six (6) pump assemblies actually undergoing load testing (actually pumping water), three (3) - 50% - have experienced catastrophic failure. Of note, these three failed pump assemblies have also been the pump assemblies that have the most run time on them - leading me to the logical conclusion that, barring some extraordinary anomaly, the more you run them, the more likely catastrophic failures will occur.

For these reasons, and because I am fully aware the U.S. Army Corps of Engineers intends to proceed with the utmost care and diligence in all tasks associated with Task Force Guardian, I am writing this memorandum for record to ensure this situation is communicated as best as I can to the ultimate responsible authority.

Respectfully submitted,

Maria Garzino
USACE, Mechanical Engineer
Task Force Guardian


Matt here again. You can read a little more about Maria Garzino in this Task Force Hope newsletter from last July. Here's another link to it. Back then, the Corps had this to say about her: "Eventually, Garzino was sent back to New Orleans and made Pumping Systems Installation Team Leader. This is currently one of the most important jobs in the hurricane protection system with the New Orleans District Corps of Engineers." [emphasis mine]

It seems that the Corps - at least publicly - had a pretty high opinion of Ms. Garzino.

Coincidentally, that's the same newsletter that shows the consequences of the pumps at the floodgates having less capacity that the Orleans Parish pumps in the interior of the city. Lots of areas would get flooding if the floodgate pumps don't work.

Floodgate pump specifications

In light of some upcoming news, I thought it would be instructive to highlight two particular sections of the original Corps specifications issued for the purchase of the floodgate pumps from MWI. The first section deals with the Corps requirements for testing:

2.5 Pump Testing:
2.5.1 Each pump and hydraulic power transmission system shall be factory pressure tested statically to maximum design psi for a minimum of 90 minutes at design operating temperatures with every plumbing connection checked for possible leaks. In the event a leak is observed or detected, it shall be repaired and the test be repeated until all leaks are eliminated.
2.5.2 Each pump and hydraulic power transmission system shall be factory pressure tested dynamically to maximum operating speeds, pressures and temperatures for a minimum period of 15 minutes. The dynamic test shall be conducted in a horizontal variable speed dynanometer that is capable of varying torque loads from 0 to maximum required horsepower as specified.
2.5.3 The field test shall be witnessed by the Government. All plumbing fittings and hydraulic equipment shall be inspected again for leakage. Should leakage be detected or observed, repairs shall be made and tests performed again, until all leaks or losses are detected and repaired.
2.5.4 Full size factory testing shall be witnessed by the Government prior to shipment of the pumps. All tests shall be in accordance with Hydraulic Institute Standards.
2.5.5 Pump discharge and head testing shall be conducted in an open sump at the manufacturer’s testing facility in accordance with the Hydraulic Institute Standards and in the presence of a registered professional engineer. Data from previous pump tests, if available, can be furnished in lieu of new tests.

The second section is titled "Incentive:"

1.7 Incentive
The complete deliverable is required 104 calendar days from the notice to proceed. For each complete pump system (hydraulic motor, pump, drive unit, discharge elbow, and discharge piping) delivered and accepted by the Government 104 days from notice to proceed, the contractor will earn an incentive of $9,800 per pump per calendar day. No incentives are provided for delivery prior to 88 calendar days from notice to proceed. The maximum amount of the incentive for the 17th Street Canal pump system is $1,764,000. The maximum amount of the incentive for the London Avenue Canal is $1,764,000. The maximum incentive for the Orleans Avenue Canal is $1,470,000. For each pump system delivered after 104 calendar days from the notice to proceed, the supplier shall pay the Government as damages $1,700 per day per pump not delivered to and accepted by the Government.

MWI collected their $5,000,000 bonus, as shown on row 13077 (on the sheet marked "KATRINA CONTRACTS") of this government spreadsheet, which tracks all Katrina contracts issued by the government.

Background on MWI pumps

I know I've been away for a while. Real life has reared its head, as it often does in post-Katrina New Orleans.

However, I've been keeping a very close eye on the Corps' work around town recently. In anticipation of some news about the floodgate pumps, I wanted to give readers (new and old) background information on the inner workings of those pumps.

The pumps are intended to remove rainwater from the city when the floodgates are dropped. The floodgates would be dropped to prevent storn surge from an approaching hurricane or tropical storm from entering New Orleans' three outfall drainage canals: 17th Street, Orleans Avenue, and London Avenue.

Currently, there are ten pumps at the Orleans Avenue canal floodgates, twelve at London Avenue, and eighteen pumps at 17th Street. All are the same size: 60" diameter. All are factory rated for about 200 cubic feet per second (cfs), though the actual capacity may be somewhat lower.

The pumps (60" Hydraflo HAC360 models) and their diesel drive units were manufactured by MWI Corporation of Deerfield Beach, Florida, under a Corps of Engineers contract issued January 27, 2006. The contract announcement is here, while the original specifications sent out for bid are here. MWI's sales literature for the Hydraflo can be found at this link.

The pumps are made up of two main components. The first is the pump assembly itself. Here's one of the Orleans Avenue pumps last October, after it and four others at Orleans Ave. had been pulled from the canal due to still-unexplained problems:


So you can see what's inside, here's some of the pumps on the ground at Orleans Avenue last spring before they were installed:


The above picture is looking into the discharge of the pumps, which is the top when they are installed. Here's a blowup with some of the major parts labeled:


The pumps are powered with hydraulic oil. The oil is supposed to be raised to 3000 pounds per square inch (psi) and then sent via pipe and hose to the hydraulic motor, which translates the energy of the hydraulic fluid to rotational energy on the shaft of the pump. The propeller blades attached to the shaft move the water upward through the pump and into large pipes, which discharge on the lake side of the floodgates.

The pump power units are far more complicated pieces of equipment than the pumps. They have large 735 horsepower, 1800 rpm Caterpillar diesel engines on them. The engines power the hydraulic power system through a gearbox, also known as a pump drive.

The heart of the hydraulic system is a pair of hydraulic pumps made by Denison Hydraulics. The pumps draw hydraulic fluid from a large reservoir on the power unit, raise the pressure of the fluid to 3000 psi, and send it to the pump. At the pump, the hydraulic motor spins the propeller shaft. After the hydraulic fluid passes through the motor, it goes through the coolers on the pump, and then returns to the reservoir on the power unit.

According to the Corps' Floodgates Operating Manual, here are the manufacturers and model numbers of these major components (where I've been able to, I've linked to the literature for the particular model):

Drive unit engine - Mfr: Caterpillar model: model 3412
Pump drive - Mfr: Durst model: 2PD10
Hydraulic pump - Mfr: Denison model: T7ED
Hydraulic motor - Mfr: Rineer model: 125 four-port (not enough info in manual to figure out exact model)

What can we do with this information?

I ran some numbers. If the Caterpillar engine is rated for 735 hp at 1800 rpm, that converts to a rated output torque of about 2144 foot-pounds. Then I looked at the rating sheet for the Durst pump drives.

According to that sheet, one is supposed to take the rated torque of the input to the pump drive, multiply it by a service factor, and then compare that number to the maximum rated torque of the chosen Durst Model (a 2PD10, in this case).

In our case, the service factor would probably be 1.5, which corresponds to a multiple cylinder internal combustion engine with uniform (i.e. no shock) loading. So we multiply 1.5 times 2144, giving 3216 ft-lbs as the number which the Durst drive's rating must beat.

According the rating sheet, the Durst 2PD10 is only rated for 1996 ft-lbs, which is less than the nominal torque from the engine, let alone the torque modified by the Durst service factor. The 2PD10 is the biggest two-pad pump drive in the Durst product line. It would appear that the Durst pump drives on the MWI drive units were severely undersized.