Oh. My. God.

Corrections have been made to the post below. They can be found here. Note that conclusions remain true, it is merely the methodology which was partially in error.

This one is aimed at the engineers in the audience, but everyone has to read it.

We know the following information about the welded pipe that runs from the drive units to the pump units, conveying the high pressure hydraulic oil to spin the pump impellers:

Max working pressure = 3000 psi (actually can go higher, see Ms. Garzinos's memo)
Diameter of high pressure line = 3" (see the Floodgates Operating Manual)
Thickness of high pressure line = Schedule 80 (nominally 0.3") (See the original specs, or the job sites. "Schedule 80" is written all over the pipes, which are plainly visible from the levees at Orleans Avenue. I have pictures, which I will post later)
Material of high pressure line = A106, probably grade B (See the original specs), with Maximum Allowable Stress in Tension of 15,000 psi [corrected to 20,000 psi here]
Pipe supplier = MWI (See the original specs)

Engineers in the audience will know to run the numbers through equation (4) in section 104.1 of ASME B31.1 [corrected here]. This is an equation that has been around for decades and is well known to every engineer who has ever specified pipe.

When you run that calculation with just the standard numbers (assuming a standard 12.5% wall deviation, joint efficiency = 1, y = 0.4), you find something absolutely horrifying.

The pipes are too thin to take the normal operating pressures! They're too thin by a lot. When you run the numbers, you find that the maximum burst pressure for 3" Schedule 80 A106B pipe is only 2393 psi [corrected here]. Even if the material is actually A106, Grade C, that still only gets you to 2791 psi. Even when you knock the wall allowance out completely, you still don't get there. When I did this calculation, I actually felt physically sick when I got the result. I've redone it a few times, hoping the numbers were different. They're not. And keep in mind that the pressure can exceed 3000 psi during operations, as also shown in Ms. Garzino's memo.

There are two of these lines on every pump/drive unit pair, each of them about 250 feet long. There is over 17,000 feet of this pipe installed at all three floodgate sites. All of it is unfit for the service it will see. All of it is prone to bursting under normal service [corrected here]. How this could have slipped through so many layers of review is unknown, but I can tell you that if there was a "Fitness for Service" declaration signed, it must be either fraudulent or signed under duress. I bet that the welders working these jobs probably asked for some kind of release from liability, so they wouldn't be blamed when the pipes burst. This is the worst kind of engineering mistake. This is "crash-into-Mars-because-we-didn't-convert-the-units-correctly" stuff.

This is equivalent to driving the sheet piles too shallow on the levees. It is a fundamental flaw in the system that has not been addressed by anyone, and has most likely been covered up to have gone this long. And it could have been remedied so simply: the Corps should have specified Schedule 160 pipe instead of Schedule 80 (by the way, there are questions to be answered as to why a thickness for the pipe was specified, but no diameter. That's very unusual. It's almost like the Corps knew in advance what they were going to get before they issued the specification...).

For reference, I suggest you take a look at this company's website:
Redox, Inc.

This page on their site is particularly instructive.

I have no association with them, other than they happened to have all the information that makes my point - conveniently organized and well written.

This is so incredibly shocking it bends the brain. It appears that the Corps and/or MWI didn't do the simplest of calculations involved in sizing a pipe. Any newly minted engineer who does anything with piping becomes familiar with this equation within five minutes of being handed a job. By the way, that calculation is also called for in the Corps' own engineering manual dealing with piping, EM-1110-1-4008, "Engineering and Design - Liquid Process Piping." It shows up on page 5 [correction: page 16] of chapter 3 (note that the Corps wrote the manual on WordPerfect, so the characters for plus, minus, and equals won't show up properly in Adobe Reader) [correction:

Put simply, all of the pumping systems, as designed, procured, and built, have a strong likelihood of failure when placed under normal operating conditions.

[Corrections to this post are available here]

Would you buy a used car from these folks?

[UPDATE, June 4, 2010: Many of the conclusions in this post have been superceded by my May 27, 2010 post, "How did the pumps get from...". I have seeded updates through the post to provide perspective.]

There's been a lot of talk since the story about the defective MWI pumps broke on March 12. And while the media and the Corps have covered various aspects of the story around the pumps, no one has publicly looked at the system itself to figure out exactly what problems may still need to be addressed. Fortunately, nearly everything about the system is fully on display for anyone to walk up to and examine.

Over the next few posts, I'm going to be concentrating on specific deficiencies in the design of the pumping system. Some relate to fabrication, some relate to engineering design, and some relate to installation. This is nitty-gritty stuff that engineers love (or should love) to dig into. Keep in mind that engineering is not a static field - it is subject to opinion on how to do things the best way. These are my opinions, backed up with my experience as a mechanical engineer and consultations with others familiar with similar installations. Some of this may seem a bit didactic, but I'll always show that there's a point.

I'm going to start with the pump units themselves. As a refresher, you may wish to read my earlier post about the fundamentals of how the pumps and their drive units are set up.

Out at the Orleans canal floodgates, all ten pumps have been pulled in anticipation of refits by MWI. The five west pumps were pulled Wednesday, March 22. The five east pumps have been on the deck since last October. They're all up on the deck on the east (Marconi Drive) side, and anyone can go up and check them out. So I did so last Sunday. So did at least five other people while I was there, including two joggers with their dogs.

Frankly, the workmanship is ... not great. Let me lead off by showing you something that I had to laugh at to keep from crying:


Yes, that's duct tape wrapped around a hose meant to hold at least 1000 psi of hydraulic oil (I think - I'm not sure of the pressure on the outlet of the coolers). To top it all off, that particular fitting sits below the waterline when the pump is in the canal. Are they kidding us with this?

[UPDATE: This hose fitting appears to be one of the ones on the east pumps that were sandblasted and painted during October, 2006 as a first cut at repairing corrosion damage on the hose fittings. The duct tape may have been acting as masking for the painting and sandblasting, or it may have been part of the protection. One can see what also appears to be duct tape on other painted fittings in this shot from October, 2006:


There's more details on this hose fitting painting scheme and how it fits into the overall corrosion chronology in my May 27, 2010 post, "How did the pumps get from...". I don't think this particular method of corrosion repair ever ended up back in the water, though see the UPDATE further down that speaks to confusion over this.]

Allow me to further explain what you're looking at in that picture. The black piece is the fitting between the hydraulic hose coming out of the discharge of the Rineer hydraulic motor and the inlet of one of the two hydraulic coolers. The hose comes down from the top of the frame, and the duct tape is around the very section where the hose joins the fitting. I couldn't believe what I was seeing when I snapped this picture.

Let's step back a bit. Here's a picture of three of the five west pumps (the ones that were just pulled out of the canal last week):


I've noted the location of the typical waterline, which is based on the growth of those little barnicle things all over the pumps. As you can see, the fittings joining the hoses to the hydraulic cooler inlets on these pumps have all rusted. Here's a close-up of another cooler fitting from a west pump:


What is more surprising [UPDATE: This should read "perplexing," and it is explained in the next UPDATE paragraph as being in the "splash zone"] is that the fittings on the discharges of the coolers - which for most of the last year have been above the waterline, and thus only exposed to air and humidity - have also rusted. How could this happen? These pumps have only been in the canal since last June.

It's pretty simple. It appears that the manufacturer, MWI, skimped on the purchase of those fittings, buying carbon steel instead of stainless steel despite knowing they would be exposed to water all the time. Curiously, the Corps' specifications are oddly silent on the material of the hose fittings:

"2.4 Hydraulic Piping and Hose: ... All reinforced supply hose shall be double wire braid reinforcement and shall have minimum safe working pressure of 3,000 psi. All pipe fittings shall be socket weld type (with socket weld to thread fittings at conversion point of pipe to reinforced hose)..."

Considering the criticality of the installation (protection of life and property), use of stainless steel fittings in this application to prevent corrosion should have been a no-brainer.

[UPDATE: This very issue was the subject of considerable debate between the Corps and MWI at the time. The fittings are zinc-plated steel, and obviously could not hold up to brackish water, whether immersed or in the splash zone (the area above the water line, but subject to spray and waves, which are just as corrosive). However, MWI stated that since the plated fittings were part of their standard pump package, they were under no obligation to supply anything better. Of course, since the Corps copied and pasted MWI's specs for the standard package, it shouldn't have been a surprise that the pumps wouldn't work very well in conditions for which the standard package is not designed. Also, MWI claimed they didn't know the water was salty, so why would they provide anything better than what they thought was needed for fresh water?

Again, it's best to read over my May 27, 2010 post, "How did the pumps get from...". Also, the letter MWI sent to the Corps at the same time the Orleans Avenue east pumps were initially on the deck (during the period when the Corps and MWI were trying to figure out what to do about corrosion) is quite illuminating. It can be found starting on page 201 of the complete MWI contract file]

You may be asking, why is the fitting in the duct tape photo not rusted? I believe that fitting is on one of the pumps which has been out of the canal since October (the ones from the east side of the project), so it was only in the water for a little over four months. The pumps shown in the other photos have been in the water for approximately ten months. This is troubling, since the expected service life of these pumps is about four years - the Corps has promised permanent pump stations at the lakefront by the end of 2010.

[UPDATE: We now know the fitting in that picture was painted after being removed from the water in September. It was rusty before that. Also, thanks to tons of delays by the Corps, the permanent pumps are now not scheduled to be on line until the 2015 hurricane season.]

There is another possibility why the fittings on the east pumps don't show rust: they leak so much hydraulic oil that rust cannot form. There is some evidence for this. Below is a picture of one of the east pumps (which has been on the deck since October). I've highlighted the fitting on the discharge of the cooler.


Here's the highlighted area blown up:


I've noted the places where oil has apparently leaked.

[UPDATE: This is curious, because this photo is of one of the pumps from the west side of the site. It was pulled up in March, 2007 ahead of the piping extension repairs mentioned below and detailed in my May 27, 2010 post "How did the pumps get from...". Yet the hose fitting I've highlighted appears to have been sandblasted and painted like was done to the fittings on the east side pumps back in October, 2006. It's difficult to figure out, since the contract documentation gives the impression the sandblasting and painting technique was only done on the five east pumps. I'm not sure what to make of this.

However, the idea that the photo also shows evidence of leaks on the oil cooler is eminently possible, especially after reading about the repairs to the pumps in 2009 and 2010. The 2009 Conhagen report of repairs to 17th Street pumps E5 and E7 mentions weld repairs to the coolers. In the 2010 repairs, the coolers were apparently so corroded that they were completely replaced with stainless steel units.]

This is not good manufacturing. Material selection is a pretty easy part of the engineering on this job, and it appears the ball was dropped. It certainly brings into question the quality of the work elsewhere on the job.

This part of the story may have a happy ending. As part of the refits the Corps and MWI are undertaking - apparently at the shop of Associated Pump & Supply in Houma, LA - all of the below-the-waterline connections which previously were between hose and pipe have been replaced with hard pipe, whioch hopefully won't leak. Now all the pumps have pieces of pipe sticking up out their sides, terminating with connection points which will be above the deck on the support platforms. Frankly, this should have been done in the first place by MWI, and if they knew there were going to be connections which remained underwater constantly (which they did, since the pumps are designed to be immersed in the canal), those fittings should have been corrosion-proof. That means using stainless steel, which is the industry standard. Keep in mind that none of this has ever been acknowledged as a problem by the Corps.

[UPDATE: It was never acknowledged, but it was paid for, to the tune of over $500,000 for the piping modifications. And those modifications turned out to be nearly an entire waste of money, because while the elevated hose fittings may have been a little further from the brackish water, the entire rest of the pumps were completely corroding to bits just two years later.

See my May 13, 2010 post, "Imminent," my May 27, 2010 post, "How did the pumps get from..." and my June 3, 2010 post "This year's scramble" for the results of the total lack of consideration on dealing with corrosion in the late 2006/early 2007 timeframe. They screwed this one up big time, and then used permanent pump station money for the "fix," which in some cases was no fix at all.]

In future entries, I'll be looking at other problems with the pumping system. There are many, and none of them have been acknowledged by the Corps.

London Avenue canal tests

Corrections included 6/1/07
Updated 7/28/07, see below

[For the latest on London Avenue canal testing, see my June 11, 2007 post, "Testing, testing." It has been updated quite often.]

This past Saturday, the Times-Picayune printed a detailed article about upcoming tests along the east bank of the London Avenue canal. It was buried on page 2 of the Metro section, and didn't appear online until yesterday. The article's not too long, and I strongly suggest you read it.

The gist of it is that the Corps will be isolating a 150 foot long section of the eastern wall of the canal in exactly the location I have written about previously: about 4000 feet north of pump station 3, around the Dillard University tennis courts, or perhaps just a bit north of there. [correction, 6/1/07: I've been using the wrong location. The article says, "just south of Burbank Street on Warrington Drive," which would put the location of the testing here. I regret the error, which sprung from a January conversation I had with a well placed member of the Corps' New Orleans District. A fuller correction can be found here.]

They will drive a sheet pile coffer dam inside the canal to surround the area of concern. This will look exactly the same as the coffer dams driven around the breach sites, including the one still at the 17th Street breach. They will then pump water into the dammed area in a controlled manner, raising the level of water in six inch increments and letting the level stay for four hours each time. There will be lots of sensors, penetrometers, piezometers, and doo-dads to measure what the wall and levee are doing the entire time these increasing levels of water are placed against them. They'll take all that data back to the batcave in Vicksburg for analysis.

According to this article, none of this will happen until May, which is the end of the rainy season around here. Since the top reason all of this work is happening is to determine the safe water level which is acceptable for drainage from rainstorms (tropical or not), this is troubling. What's the delay? Paperwork:

"Bedey's office is pushing to get necessary agreements executed with the New Orleans Sewerage & Water Board and the Southeast Louisiana Flood Protection Authority-East to get the work started as quickly as possible."

There's no indication the T-P called the S&WB or the SLFPA-E to confirm this; they just took Col. Bedey's statement at face value.

There's also the usual Corps happy talk, again unrebutted by the T-P:

"The results of this so-called "site-specific testing" could result in a higher water level during the coming hurricane season that opens June 1, Bedey suggested.

"Even an additional six to 12 inches of water would allow the gates to remain up longer in the face of a tropical storm or hurricane. It also would allow the New Orleans Sewerage and Water Board to pump, without interference, more storm water out of the canal during a heavy, non-tropical rain event."

Of course, it's also possible that the testing could result in a lower Safe Water Level, or no significant change. That's why the testing is occurring. But of course, the Corps presents only the possibility that makes them look best. This is not how reputable engineers do things. Engineers don't assume anything before a test, for fear of skewing the results or the analysis. We definitely don't publicly hype one particular result weeks before the test even starts. But we all know what the Corps thinks of testing (see: pumps, defective).

What is also interesting about the second paragraph above is how the Corps has changed their tune now that the importance of the Safe Water Level has been revealed, via the shutoff of a 1000 cfs pump at S&WB pump station 3 during the December 30, 2006 storm, when the London Avenue SWL was exceeded. Now they say that 6 to 12 inches is very significant.

However, last September, when the four foot Safe Water Level at London Avenue was announced (it was a drop from the previously announced five foot level), they were downplaying it big time. In a September 12, 2006 article, here's what Corps pump person Jim St. Germain said:

"St. Germain would not predict just where the London canal's safe water level ultimately may be established, but he cautioned the public not to read too much into a foot or half-foot of change either way.

"It's good that the 17th went up a foot, but if London does go down, I wouldn't get too excited. We'll use the best technology we have in determining when to close those gates, but it isn't exact," St. Germain said. Only the specifics of each storm, including the rate of rise in the water level of the lake and each canal, would suggest when gates should be closed."

First off, you'll notice St. Germain is not quoted in this latest article. In fact, he seems to have gone underground since the pump testing imbroglio broke on March 13. He even gave a "no comment" to WWL-TV while escorting a Dutch delegation around the 17th Street floodgates (I watched the report, and the "member of the Corps of Engineers giving the tour" was St. Germain).

More importantly, as we now know the Corps has been very "excited" about the London Avenue Safe Water Level for a long time, including probably at the time of that September article. That is, they've known the exact location of the weakest point in the London Avenue canal walls since at least September of last year. They have allowed residents, businesses, and institutions (including Dillard University) behind those walls to rebuild for the last seven months without doing a thing to reinforce - or at least isolate - that section of wall. Instead, they've puttered around, gone out on boats to look at it, shuffled paper, and planned tests [correction, 6/1/07: see correction above for real location of load test].

Here's an idea: drive the sheet piling now, then worry about the tests later. We've already had an incident, almost three months ago (not to mention the December 21, 2006 storm, when the canal depth got within five inches of the Safe Water Level). What does it take to make these people move, an actual levee breach?

By the way, the most menacing part of the Saturday article is the last paragraph:

"Depending on the results of the field work, the corps team could decide to test the entire canal. But Bedey said it was premature to speculate on whether a test that broad will be needed."

Prepare for a test of the entire canal.

Update, 7/28/07

For the latest on London Avenue canal testing, see my June 11, 2007 post, "Testing, testing."

The fire

This week's Gambit Weekly contains a lengthy article about the floodgate pumps. Mentioned for the first time is the fatal fire that occurred at MWI on March 23, 2006. This fire occurred about 2.5 weeks before pump testing commenced.on April 11, 2006.

On March 23, 2006, a fire erupted in a warehouse at 33 NW 2nd Street, Deerfield Beach, FL. It was reported on briefly by the local TV news (courtesy Lexis-Nexis AlaCarte):

"Global Broadcast Database - English - 3/24/2006 - 46 words Danger on the job for a welder in deerfield beach. The man working at pump [sic] when splashes cause insulation to catch fire. The welder suffered serious burns and taken to broward general medical center. He was transferred to jackson memorial hospital. Crews are now investigating...."

That man died on March 28th of second and third degree burns at Jackson Memorial Hospital in Miami. I have received his mother's permission to report upon these events.

33 NW 2nd St. is MWI's production and testing facility. It is the site of the "wet" testing mentioned in Maria Garzino's memo. As with any fatal accident, The Department of Labor's Occupational, Safety, and Health Administration (OSHA) investigated. Their inspection number was 309430627. As you can see, they eventually levied a fine of $6876.00 upon Industrial Diversified Services. Industrial Diversified Services was a subcontractor to MWI hired to install an overhead bridge crane in the testing facility. The crane was meant to lift pump units in and out of MWI's testing tank. The bridge crane (or its absence) is mentioned by the Corps QA inspector from Jacksonville in the April 18, 2006 report (page 46 of Ms. Garzino's memo):

"5. To date, the Contractor's subcontractor is still trying to set up the bridge crane for pump testing in their wet tank. They indicate that this will allow them to cut time out of the setup process. With the crane that is currently being used, the setup has takenapproximately 8-10 hours. If they can get the bridge crane installed, the contractor believes that it will cut setup time down to 4-6 hours."

The Florida State Fire Marshall also looked into the fire and, after a brief investigation, ruled the fire accidental. Material samples of the insulation were not taken by the fire marshall's office.

I received both the OSHA report and the fire marshall report through records requests. The OSHA report had a lot of duplicative information, so I culled it down to a manageable length. Should anyone want to see the pages I didn't scan in, they're welcome to ask me for them or get them from OSHA in Ft. Lauderdale. The OSHA report came exactly as you see it, with the redactions. The OSHA report is here.

The fire marshall report came unredacted, with witnesses names and addresses. I have removed those myself, out of respect for people who happened to see a horrible tragedy unfold before their eyes. Should you be interested in receiving the complete report, you can contact the fire marshall's office in Tallahassee and pay your $10, and they'll send it along. The fire marshall report is here.

I also received the pictures below with the OSHA investigation report. I have not altered the pictures except for adding dates.

The pictures show extensive fire damage to the entire building, including the steel to which the bridge crane would be attached.

As described in Ms. Garzino's memo, and the Quality Assurance inspectors' reports attached to the memo, testing began on April 11, about 2.5 weeks after the fire. For her account of April 19, 2006, almost a month after the fire, Ms. Garzino wrote the following:

"During the evening Drive Unit 8840 was set up to undergo an E/R test - the test facility is pitch dark - I have asked MWI if we can have 30 minutes to go to the drug store and buy flash lights as I was concerned that we could not see any of the plethora of machinery that was running around us, the obstacles on the ground, and the drive units themselves (see my pics of this condition) - I was especially concerned for the safety of the QA's as the Drive Units have been experiencing excessive hydraulic failures and injury from not being able to see the failure occur and get out of the way in time was a large concern for me - MWI informed us they were 30 + minutes from starting so all Corps personnel left to get flash lights."

Now it is clear why "all Corps personnel" had to get flash lights. The building was without lights and electrical power for many - if not all - "wet" tests conducted on the pump units and the drive units after dark for many weeks. One only has to read the memo and the attachments to see that there were such tests conducted nearly every night. All of this was confirmed by MWI's spokesperson, Mike Powers, in the Gambit article. Powers claims generators for lights were brought in later (assumedly after the timespan of Ms. Garzino's memo, which covers April, 2006. Of course, the testing was about 2/3rd's done by the time she handed in her memo, so we'd only be talking about around two more weeks before testing was finished.). However, even a single night without lighting should be unacceptable to any company that truly values its' employees' safety and the safety of federal employees working in its facility. That such a situation continued for weeks - apparently without any additional attention from the state fire marshall or OSHA - is scary. Finally, if this haphazard approach to testing is typical, what questions does it raise about the construction and assembly of the pumps?

My address

I just realized that this entire time, while I've thought my email address was available on my Blogger profile, it actually wasn't. It's there now.

The Corps defense

The Corps has been playing defense on the defective pump story since it broke last Tuesday. I thought it was time to deconstruct some of their defenses.

Defense number 1 Corps says they went to the media within a week or two of Ms. Garzino handing in her memo on May 4th to let the public know about these problems. Col. Bedey used this one on the Garland Robinette "Think Tank" show on WWL radio on March 15. You can download the entire interview from WWL's podcast page here.

What really happened:
I've been following these stories very closely, and I've been archiving nearly every pertinent article. I can only assume they are referring to this article in the May 12, 2006 Times-Picayune:
Gates won't be ready by June 1

Go ahead and read that article. See if you can find anything in there about problems with the pumps' functionality. In fact, all that's mentioned are the following:

a) There were problems installing the gates at the London Avenue site. Nothing is mentioned about pump functionality there.

b) The Corps says the 17th Street site is small, and there wasn't enough time to get the MWI pumps in.

c) Regarding Orleans Avenue, which has the exact same pumps as London and 17th St., the article says the following, "The floodgate-and-pump project in the Orleans Avenue Canal was a less complex job and will be ready on June 1, Setliff said."

I don't see it. Do you?

Defense number 2 The Corps says they've open and honest about all this. Colonel Bedey also said this on WWL, and it's been put out there by other Corps officials as well. They point to the reportage of vibration problems from last August as a prime example of that openness and honesty. Incidentally, the Times-Picayune also said the same thing (with regards to their own coverage) last week during their near-complete burial of this story. Let's take a closer look at that coverage.

What really happened:

First off, there were no Corps press releases at the time of those problems. All information was reported second hand by the media or other sources.

Secondly, the vibration problems last August are not what Ms. Garzino's memo was about. Her memo and the independently authored attachments were mostly about numerous design problems with the hydraulic power units and the testing methodology itself, as well as deficiencies in contract oversight by New Orleans. The Corps is attempting to confuse the justifiably enraged public by conflating two completely different stories. They will not succeed.

Thirdly, I have found only one stray mention of any problems besides the pump unit vibrations. It is startling, but it is just a single mention in one article. Otherwise, all other reporting was strictly about vibrations. The hydraulic drive units were never mentioned before or again.

But to give proper due, let's look at that single mention:

Problems emerge on new pumps - Times-Picayune, 8-24-06:

This article is incredibly interesting, not for what it says (mostly stuff like vibration problems, we're working on it, deadlines slip, etc), but for what it tantalizingly whispers about the now infamous testing at MWI in Florida. Buried deep in the article (and oddly presented as a Corps defense for the vibrations) is this passage:

"The problem pumps were specially manufactured for the canal projects, and corps officials were unable to explain why the vibrations were not detected during tests performed in the manufacturer's lab under corps scrutiny.

"We have to find out why the pumps did what they did, but until we do further investigation, we won't know," St. Germain said.

During testing in the manufacturer's laboratory, St. Germain said, corps engineers saw problems with the drive units, but not the pumps themselves. "Because of what we saw, we had them do more drive testing and less pump testing in order to meet our schedule."

Had problems with the pumps themselves been detected, St. Germain said, the corps would have ordered further testing on that as well. St. Germain said all the pumps were tested to determine if they properly held hydraulic pressure, but only a few were performance-tested in a tank of water in the manufacturing lab that is large enough to hold a 20,000-pound pump while it runs."

Remarkably, the T-P just lets this information lie there with no follow-up, either in the article itself or in any subsequent articles since then. Read carefully what is said in those four paragraphs. It is the bare-bones outline of last weeks' story, along with a stunning bit of candidness that Jim St. Germain snuck past the T-P. But I would hardly call it the same as explaining that information to the reading public, or the extent of what was described. In fact, when read within the totality of the article, it's almost an aside.

Still, when you look at just those paragraphs, you find:
a) He actually said they saw problems with the drive units, not the pump units. It's clear from context he's referring to problems which were not vibration-related. Thanks to Ms. Garzino, we now know the scope of some of those problems.
b) He actually said that they did "less pump testing" to meet the schedule. This is a shocking admission in the press of the Corps deliberately bypassing their own specs.
c) He also said that it wasn't the common-sense - and contractual - need to load test ALL the pumps with water that would have made the Corps actually load test the pumps. He actually said they only would have load tested all the pumps if problems had somehow announced themselves on the few pumps MWI and the Corps deigned to run water through. Riddle me this: how does one know if there are problems when water is put through the pumps, unless water is put through the pumps? That's through-the-looking-glass thinking if I've ever heard it.

Of course everyone now recognizes the significance of these statements, but back then only the people making them and those inside the Corps realized what they meant. Frankly, I didn't notice it myself until I got Ms. Garzino's memo and started going back through the news coverage, anticipating that the Corps would claim, "that's old news."

As I said above, those paragraphs in that single Times-Picayune story are the only mention I can find of any drive unit problems and other difficulties besides the widely reported August vibrations (which are really not what Ms. Garzino's memo and the attachments are about, as anyone who's actually read the whole thing can plainly see). Even then, they're referred to elliptically, as if it was okay that the pumps weren't being factory tested. If that's what the Corps and the Times-Picayune want to camp out on as "we already truthfully and openly reported that," I think they need to reevaluate their standards for communicating with the public.

[Side note: yes I realize this is a long post, but hang in there...]

For comparison, check out WWL-TV's coverage of the August problems:
17th St., London Avenue pumps have to be shut down during test run, 8-21-06

The key paragraph is here:

"[Colonel] Wagenaar said the problems could be mechanical or hydraulic or it could be related to the low water level in the canal. The pumps are designed to work in water that is three to four feet deep and the water in the canal was only about a foot deep. Tests were scheduled with pumps pushed further down into the water to simulate what actual conditions would be like during a storm."

Do you see any ackowledgement by the Corps of what they already knew: there were serious problems with the drive units and pumps, all of which had been explained in detail to them by their own employees (remember, the QA reports attached to Ms. Garzino's memo were written independently by other Corps employees)? I don't. Imagine if Wagenaar had said, "We're not surprised by this, since we had seen many other problems with these systems and have attempting to address them fully. In fact, our own people have reported those problems and we've taken their concerns very seriously." Of course, for him to be able to say that without it shocking everyone, there would have had to have been a series of statements all summer along the same lines, and the Corps would never, ever be that candid, even though they're spending the people's money on the people's protection.

It's also instructive to read the other stuff that was coming out throughout the summer of '06, before the vibrations came up but after they received Ms. Garzino's memo. For example, there's a detail-laden Times-Picayune article from July 19, 2006:
Drainage forecast cut at London Ave. Canal

Read that and tell me if there's anything about pump testing in there. There's not.

Then, there's the release of the inundation maps on July 26, 2006:
The actual press release and maps are here.
Corps issues maps of rainfall's impact, T-P, 7/27/06
Corps releases new maps showing flood risks this hurricane season, WWL-TV, 7/26/06

Read those articles and the press release. Anything about the functioning of the pumps? Nope.

Those maps show what happens when the gates drop. If there was ever a time to put out the caveat, "this data is based upon full pump functionality for the length of the storm depicted on the map," that was it. Instead, as with every other time the Corps spoke with the public, the media, and Congress, the pump flow capacities are very declaritively stated as fact. The Corps had opportunity after opportunity to tell the public they had serious problems with the functionality of the pumps, and they skipped every opportunity. They were not open and honest.

You might say, "Well, that's how the media covered it, but they leave a lot of stuff out." That's true. So where else was the Corps doing their song and dance last year? There were the monthly meetings of the now-defunct Orleans Levee District (OLD) for one.

OLD archived all their meeting minutes online. Here's what was reported by the Corps to OLD on July 19, 2006:

"Mr. Fred Young, U.S. Army Corps of Engineers (Corps) Project Manager for the Orleans East Basin under Task Force Guardian, reported on the status of the temporary pumps on the interim structures. All the gates can be closed, and work is continuing on the floodwall repair at the three breaches. Currently, pumping capacity is 2200 cfs at the Orleans Avenue Canal and 2800 cfs at the London Avenue Canal. The Corps is still working on bringing pumps on line at the 17th Street Canal structure.... The Corps feels it has what is needed with the 2200 and 2800 cfs capacity at the Orleans and London Avenue Canals to get through this hurricane season."

Here's the August 16, 2006 report:

"Mr. Young reported the pumping stations at the Orleans and London Avenue Canals are approximately 98 percent complete; the pumps can be run, however, some electrical and mechanical work remains."

September 20, 2006:

"Mr. Frederick Young, Project Manager for the U.S. Army Corps of Engineers(Corps), reported work is on-going on the pumps at the control structures at the three outfall canals. There appears to be better control over the pump vibrations, and the Corps anticipates having some type of resolution of this problem relatively soon. Additionally, by June 1st, the Corps anticipates having an additional set of hydraulic pumps at the 17th Street Canal to provide 7300 cfs capacity, and increasing the capacity on the London Avenue Canal from 2800 to 4300-4800 cfs. The capacity at the Orleans Avenue Canal is already at or above what can come out of that pumping station."

You get the idea. Every time they had a chance to mention the mechanical problems, to temper expectations, to let people know that the capacities they were publishing were theoretical, they passed on it. I've highlighted the passages dealing with the Orleans Avenue canal because - even though the same problems plagued all 34 pumps and drive units across all three floodgate sites - the Corps acted like each site was different. When it comes to the pumps, they're all the same. If there's a design problem with one drive unit, there's a design problem with them all. So the fact that construction was completed at Orleans Avenue meant absolutely nothing, because the pumps were still malfunctioning elsewhere, and the Corps knew it. That means that Orleans wasn't any better than any other site, no matter how much lipstick the Corps tried to smear on the site through their public pronouncements. In fact, the east pumps at Orleans Avenue remain out of the water to this day, as they have for the last six months. That's not open and honest.

The people of this city and this country deserved to know the risks they were taking coming back here. And every time the Corps talked pumps last year, even when the capacities were woefully low, they never included the fact that even those capacities were questionable at best according to their own experts. They withheld crucial information, and as a result, thousands of people spent money, time, and moved heaven and earth to come back. Yet all the time, the Corps knew that there was a strong likelihood of non-functionality - and any engineer outside the Corps' New Orleans District will tell you that equipment as critical as these pumps has to be absolutely, unquestionably functional under all conditions - a conclusion that can only be reached by full, complete, and professionally-conducted factory testing in accordance with industry-accepted standards. Field testing under conditions very far from reality doesn't cut it. They could have gotten a lot closer to storm conditions in the factory, but they chose not to. The fact that they didn't even test most of the pump units, the fact that the drive units appeared to have deeply serious design flaws, and the fact that they continually downgraded the testing requirements until they were practically meaningless (and behind their own Contract Officer's back, no less!) was information essential to citizens' and policy-makers' decision-making processes. All that was held back simply for them to retain what tiny shred of credibility they believed they possessed.

When it comes down to it, they were playing CYA by lying to all of us.

Calling BS

Now that the news about Maria Garzino's memo has broken all over the place, I thought a few points needed to be made.

Yesterday, the Corps flew Colonel Bedey, the head of the Hurricane Protection Office, back from Washington so he could be there when two local TV affiliates (Fox 8 and ABC 26) filmed four of the 17th Street pumps being run. The Corps pointed to this demonstration as proof that everything's just A-OK.

1) The Corps already ran these pumps last Saturday. The Times-Picayune buried a tiny article about it on page B-3 in Sunday's local section. Here's the article:

Corps pump tests get thumbs up

This article (and possibly the Corps' alerting the media about the test) was also a bid by the Corps to get out ahead of the news that broke yesterday. They obviously failed in that bid.

2) Those four pumps were ordered long after Ms. Garzino's memo and after the other 34 pumps started failing last summer. They were part of an order of six extra pumps to MWI (total charge, about $4 million for all six). These four were actually the last pumps installed - they went just a couple of months ago (they weren't fully hooked up as of January 26, 2007) and are not the subject of Ms. Garzino's memo. That is, they were not in the original order of 34. Using these pumps as a demonstration to show that the other 34 are working is pure lying. The Corps is probably going to turn on those pumps again tomorrow for the Mayor. They represent about 800 cubic feet per second of capacity (theoretically). Pre-Katrina, over 10,000 cfs flowed down the 17th Street canal.

3) The test last Saturday, like the one last night, only went for about an hour. That was not mentioned in the Times-Picayune article. But ALL the floodgate pumps (there are no spares in case one or more fails during a storm) will have to run for 12 or maybe 24 hours during a tropical storm or hurricane. Why isn't the Corps running tests for the media that last that long? Because they know the pumps and their drive units probably can't hold out for that long.

Don't buy what the Corps is selling.

Finally, I have one other hunk of information to add.

As a result of another FOIA request I made in May of last year, the Corps uploaded a number of documents to the IPET website this past February (the site that was set up to allow the public to understand the Corps' investigation into Katrina). I just found out about this two days ago when they responded to my FOIA lawsuit. You can find them by going to the IPET website, clicking on "Post-Katrina" on the left, and then "Project Informations Reports." You'll see the files listed there.

The documents are all of the revisions of the Project Information Report ("PIR" in Corps-speak) that authorized the floodgates and the pumps. PIR's are detailed scopes of work and budgets written by the Corps to detail ther projects. They are not generally released. I had previously obtained Revisions 0 and 1 of this document, despite the Corps ignoring my FOIA request for them. I did not know that Revisions 2, 3, and 4 existed until a few days ago.

Revision 0 (Oct., 2005)
Revision 1 (Jan., 2006)
Revision 2 (May, 2006)
Revision 3 (Jul., 2006)
Revision 4 (Oct., 2006)

In Revision 1, which came out in January of 2006, the Corps set up their entire strategy for protecting the three outfall canals. It was to be a two pronged strategy. There would be floodgates to protect against storm surge, and there would also be repairs to the walls along the canals. They set aside $29 million to do this at all three canals. I've blogged repeatedly about these bank stabilization contracts before, wondering what happened to the money and the plans. Well, now that I've seen Revisions 2, 3, and 4, I know part of the answer: the plans simply disappeared.

Revision 2, issued May 17, 2006, makes absolutely no mention of the status of the bank stabilization contracts (other than mentioning them in the budgetary chart - so apparently they still existed as ideas in May, 2006), or any plans for implementing them. Revisions 3 and 4 are similarly silent on those repair efforts - again, other than calling them out as line items in the budgetary chart (Note: at some point bank stabilization for Orleans Avenue canal was eliminated, since it disappears entirely between January and May, 2006). But since the walls weren't fixed (and still aren't), New Orleans now has problems with drainage even when there isn't a tropical storm. That is because the walls are so weak along the outfal canals (most severely along the London Avenue canal), the Corps has forbidden the local drainage authority from filling the canals above a level at which they deem a breach is imminent, called the Safe Water Level. At London Avenue, this level is four feet, which is the height of the base of the wall atop the levee. If that level is exceeded by the interior drainage pumps, some of those pumps must be turned off. This has already happened once, on December 30th. I blogged about this a few weeks back, and the Times-Picayune finally caught up to the story a few days later. The Corps, specifically Acting Branch Chief (Existing Pump Stations Branch) Jim St. Germain, was present in Pump Station 3 for that shutdown. In fact, it was Mr. St. Germain who recommended that shutdown.

That is the same Jim St. Germain around whom many of the events of the past few days are swirling. He's one of the Corps' New Orleans District's pumping guys, along with Dan Bradley and Dennis Strecker. All three play prominent roles in Ms. Garzino's memo, specifically in the downgrading and elimination of testing requirements without the prior knowledge of the Corps Contracting Officer, Cynthia Nicholas. Ms. Nicholas was the only person legally authorized to make such changes. You can see a picture of Messrs. St. Germain and Bradley in the Corps' February 6, 2006 newsletter. A picture of Mr. Strecker at his retirement in December, 2005 can be seen here. Mr. Strecker was evidently brought back from retirement, possibly as a contractor, just to deal with the pumps. He was at an April 18, 2006 meeting at MWI's Florida facility, representing the Corps.

Frankly, if the Corps had simply spent the money fixing the walls, which are part of the federally authorized hurricane protection system, there would have been no need for the floodgates at all.

The reason for Revision 2 was to propose a plan for expanding pumping capacity at the floodgates, specifically at 17th Street and London Ave. The initial order of MWI pumps did not give New Orleans nearly enough pumping capacity to pump out rainwater during a storm (I know, I know...). What is also really interesting about Revision 2 of the PIR is its ... surprising ... recommendation to perform that expansion by buying 32 more hydraulic pumps just like the 34 they'd already bought from MWI. 16 would be installed at 17th Street and 16 would be installed at London Avenue. This particular expansion never did take place, thank goodness.

But notice the dates of what was happening. The Corps upper-ups at New Orleans got Ms. Garzino's memo on May 4, 2006. Just 13 days later, they were recommending buying 32 more of the exact same pumps that they had been told wouldn't work! That's crazy talk! They pooh-poohed another idea to install much more robust, direct-drive, non-hydraulically-powered pumps that would have cost more (maybe) and been delivered perhaps a month later. Considering the original MWI pumps weren't working anyway, would a month have really mattered? Besides, they ended up buying the direct drive pumps anyway this past January, as I've also written about January 12, January 18, and February 11. The Corps finally reported the news in their February 21st newsletter, along with some admittedly snazzy graphics.

So to make this as clear as possible... Not only did the Corps install pumps they knew wouldn't work, not only did they simply give up throughout the entire year of 2006 on repairing the canal walls, they also recommended doubling the order of the known-to-them-to-be-defective pumps and drive units, assumedly with the same manufacturer - MWI.

The complete memo

Here's the May 4, 2006 memorandum for record, with all the attachments, from Ms. Garzino to Colonel Setliff. It's about 12.5 megs.

I received it as a result of a Freedom of Information Act request placed with the Corps. Unfortunately, the Corps blacked out all the names. I will be appealing this. Even so, the events described leave little to the imagination.

The memo

Defective

Lots of people have been finding their way to my blog today. It's because of this:
New Orleans pumps were faulty

"The Army Corps of Engineers, rushing to meet President Bush's promise to protect New Orleans by the start of the 2006 hurricane season, installed defective flood-control pumps last year despite warnings from its own expert that the equipment would fail during a storm, according to documents obtained by The Associated Press."

"The 2006 hurricane season turned out to be mild, and the new pumps were never pressed into action. But the Corps and the politically connected manufacturer of the equipment are still struggling to get the 34 heavy-duty pumps working properly."

And lots more where that came from.

So I assume people want to read this memo.

[Updated March 19, 2007]
First, an important bit of background...

This memo was not leaked to the press. I received it through a Freedom of Information Act request placed with the Corps of Engineers' New Orleans District (actually, it was originally placed in Los Angeles, and then L.A. transferred it to N.O.). The Corps released this memo of their own accord.
[end update]

Now, here's why I haven't posted it...

I actually tried to go to Kinko's last week, but they wanted $5.99 for the first page and $1.99 each additional page, and they said it would take three days to get it scanned. I wasn't about to pay over $150 to scan a document. I cast around the neighborhood looking for a scanner that could handle a big document, but I couldn't find one.

So in the interim, here's the text of the first three pages of Ms. Garzino's email. These are the main body of her memorandum for record. Behind this, there is a list of all pumps which failed before the pump testing requirements were eliminated (one week into the testing), then a listing of the testing results of all the power units, then a day-by-day chronology of her time at MWI's Deerfield Beach facility during the month of April, 2006.

Following all that are the official quality assurance (aka "QA") reports filed by other Corps personnel during their time at MWI. The Corps loaned personnel form their Jacksonville office to this effort. Most of the time, there were two QA's working at MWI, one working a day shift and one working a night shift. They would file reports of what they observed each day.

So here's the beginning of the memo:

Sent: Thursday, May 04, 2006 9:25 AM
To: Setliff, Lewis F COL MVS
Subject: FW: Defective Pumping Equipment
Importance: High
Sensitivity: Confidential

Col Setliff,

I have submitted the message below, with complete attachments, to Ms. Nicholas - Contracting Officer for the pumping equipment in question. I have called and visited her contract specialist, Ms. Rouse, to give her a "heads up" regarding general content of this message, and, have offered my assistance when/where ever necessary to answer questions/provide additional information should she feel it necessary. I have also dropped off a full copy of the MFR with complete attachments at your office this morning (I attempted to send this message last night with complete attachments and the server would not let me - I am attaching here only the docs that were generated by myself...).

I understand your schedule is extraordinarily busy and your time valuable, with this in mind, I wish to ask of you a few moments of your time so I may discuss information that is not contained in this memo and is more private in nature.

Thank you for your time and consideration in this matter.

Respectfully,

Maria Garzino
USACE Mechanical Engineer
Task Force Guardian
________________________________________

Sent: Wednesday, May 03, 2006 5:26 PM
To: Nicholas, Cindy A MVN
Cc: Rouse, Gayle E MVN
Subject: Defective Pumping Equipment
Importance: High
Sensitivity: Confidential

MEMORANDUM FOR RECORD

DEFECTIVE PUMPING EQUIPMENT

W912P8-06-R-0089 - Temporary Pumps for Three Outfall Canals, New Orleans LA
Date: May 3, 2006

To: Ms. Cynthia A. Nicholas, USACE Contracting Officer, Task Force Guardian

Subj: Defective Pumping Equipment Supplied by MWI - Pumping Equipment Not In Accordance With Contract Requirements

Ms. Nicholas:

It is the opinion of the undersigned that pump equipment supplied by the above cited contract, specifically the pump assemblies and drive units, that are arriving in the field and being installed daily, are not capable of fulfilling their function as intended by the original contract requirements - are defective - and will experience failure should they be tasked to run, under normal use, as would be required in the event of a hurricane.

Put simply, if the intent of the contract requirements is to have pump equipment capable of being turned on and used for prolonged periods of time in the event of a hurricane, then I believe the pump equipment will not function as intended. In fact, without extensive follow on supplemental operation of the pumping equipment after installation, and subsequent likely follow on repair of failed pumping equipment components (hydraulic pumps, hydraulic pump motors, failed high pressure hydraulic lines, etc.), I believe significant failure of the pump equipment can be expected. In addition, as the original contract deliverables (as provided by the written specifications, oral negotiations, and bid proposal) have changed significantly due to revised testing procedures, and without formal modifications to the contract, it is recommended that requests for payment from MWI for this pumping equipment should be extensively reviewed prior to such payment to ensure the government receives fair value for services/equipment received - especially if the governments intent was to receive fully functional pumping equipment that could be pressed into full service after installation without the threat of imminent failure - something the current contract documents/specifications impart.

This opinion is derived from events transpiring during continuous and diligent observance by the undersigned of full sized testing of these pump assemblies and drive units at the manufacturer's Deerfield Beach, Florida facility from the period April 7, 2006 thru May 01, 2006. These events have been documented, by myself and independently by Jacksonville District QA personnel, in reports as follows - and to be provided as attachments:

Listing of Pump Assemblies That Have 'Seen' the Testing Tank (actually pumped water) Failures/Successes - 4/6 thru 4/26
Listing of Drive Units Failures/Successes - 4/6 thru 4/26
Florida Trip 4/07 to 4/16 - MWI - Testing - Report No. 1
Florida Trip 4/17 to 4/23 - MWI - Testing - Report No. 2
Florida Trip 4/24 to 4/30 - MWI - Testing - Report No. 3
QA Shop Inspection Report #6 (4/11), #7 (4/12), #8 (4/12), #9 (4/13), #10 (4/14), #11 (4/15-17), #12 (4/18), #13 (4/19-20), #14 (4/21), and #15 (4/21)
QA-TFG- Reports for 4/25, 4/26, and 4/27

Finally, events outlined in the above reports highlight two disturbing, and still unresolved, pumping equipment failure issues - that being:

1) Cause of the voluminous failures of the hydraulic pumps on the drive units is still unknown at this time - the manufacturer of the hydraulic pumps (Denison) has not yet provided any official input as to the failures being caused by a plethora of "bad" pumps, or, point to an as yet unknown design deficiency with the hydraulic system. This situation would provide for the possibility of future failures of the drive units at 100% until a design deficiency can be ruled out - in addition, there is the very likely possibility, more probable actually, that damaged hydraulic pumps starting the failure process have "passed" testing and are currently slated to be, or have been, installed.

2) The original contract specifications required 100% load testing of all pump assemblies - this requirement has subsequently been eliminated, and to date, less than 25% of all pump assemblies have been load tested (leaving potentially 75% not load tested), and, of the eight (8) pump assemblies that have been load tested, one has only been run for a few minutes at best and one other was run at 1/3 operating pressure (the hydraulic oil barely got warm enough to register). Of the remaining six (6) pump assemblies actually undergoing load testing (actually pumping water), three (3) - 50% - have experienced catastrophic failure. Of note, these three failed pump assemblies have also been the pump assemblies that have the most run time on them - leading me to the logical conclusion that, barring some extraordinary anomaly, the more you run them, the more likely catastrophic failures will occur.

For these reasons, and because I am fully aware the U.S. Army Corps of Engineers intends to proceed with the utmost care and diligence in all tasks associated with Task Force Guardian, I am writing this memorandum for record to ensure this situation is communicated as best as I can to the ultimate responsible authority.

Respectfully submitted,

Maria Garzino
USACE, Mechanical Engineer
Task Force Guardian


Matt here again. You can read a little more about Maria Garzino in this Task Force Hope newsletter from last July. Here's another link to it. Back then, the Corps had this to say about her: "Eventually, Garzino was sent back to New Orleans and made Pumping Systems Installation Team Leader. This is currently one of the most important jobs in the hurricane protection system with the New Orleans District Corps of Engineers." [emphasis mine]

It seems that the Corps - at least publicly - had a pretty high opinion of Ms. Garzino.

Coincidentally, that's the same newsletter that shows the consequences of the pumps at the floodgates having less capacity that the Orleans Parish pumps in the interior of the city. Lots of areas would get flooding if the floodgate pumps don't work.

Floodgate pump specifications

In light of some upcoming news, I thought it would be instructive to highlight two particular sections of the original Corps specifications issued for the purchase of the floodgate pumps from MWI. The first section deals with the Corps requirements for testing:

2.5 Pump Testing:
2.5.1 Each pump and hydraulic power transmission system shall be factory pressure tested statically to maximum design psi for a minimum of 90 minutes at design operating temperatures with every plumbing connection checked for possible leaks. In the event a leak is observed or detected, it shall be repaired and the test be repeated until all leaks are eliminated.
2.5.2 Each pump and hydraulic power transmission system shall be factory pressure tested dynamically to maximum operating speeds, pressures and temperatures for a minimum period of 15 minutes. The dynamic test shall be conducted in a horizontal variable speed dynanometer that is capable of varying torque loads from 0 to maximum required horsepower as specified.
2.5.3 The field test shall be witnessed by the Government. All plumbing fittings and hydraulic equipment shall be inspected again for leakage. Should leakage be detected or observed, repairs shall be made and tests performed again, until all leaks or losses are detected and repaired.
2.5.4 Full size factory testing shall be witnessed by the Government prior to shipment of the pumps. All tests shall be in accordance with Hydraulic Institute Standards.
2.5.5 Pump discharge and head testing shall be conducted in an open sump at the manufacturer’s testing facility in accordance with the Hydraulic Institute Standards and in the presence of a registered professional engineer. Data from previous pump tests, if available, can be furnished in lieu of new tests.

The second section is titled "Incentive:"

1.7 Incentive
The complete deliverable is required 104 calendar days from the notice to proceed. For each complete pump system (hydraulic motor, pump, drive unit, discharge elbow, and discharge piping) delivered and accepted by the Government 104 days from notice to proceed, the contractor will earn an incentive of $9,800 per pump per calendar day. No incentives are provided for delivery prior to 88 calendar days from notice to proceed. The maximum amount of the incentive for the 17th Street Canal pump system is $1,764,000. The maximum amount of the incentive for the London Avenue Canal is $1,764,000. The maximum incentive for the Orleans Avenue Canal is $1,470,000. For each pump system delivered after 104 calendar days from the notice to proceed, the supplier shall pay the Government as damages $1,700 per day per pump not delivered to and accepted by the Government.

MWI collected their $5,000,000 bonus, as shown on row 13077 (on the sheet marked "KATRINA CONTRACTS") of this government spreadsheet, which tracks all Katrina contracts issued by the government.

Background on MWI pumps

I know I've been away for a while. Real life has reared its head, as it often does in post-Katrina New Orleans.

However, I've been keeping a very close eye on the Corps' work around town recently. In anticipation of some news about the floodgate pumps, I wanted to give readers (new and old) background information on the inner workings of those pumps.

The pumps are intended to remove rainwater from the city when the floodgates are dropped. The floodgates would be dropped to prevent storn surge from an approaching hurricane or tropical storm from entering New Orleans' three outfall drainage canals: 17th Street, Orleans Avenue, and London Avenue.

Currently, there are ten pumps at the Orleans Avenue canal floodgates, twelve at London Avenue, and eighteen pumps at 17th Street. All are the same size: 60" diameter. All are factory rated for about 200 cubic feet per second (cfs), though the actual capacity may be somewhat lower.

The pumps (60" Hydraflo HAC360 models) and their diesel drive units were manufactured by MWI Corporation of Deerfield Beach, Florida, under a Corps of Engineers contract issued January 27, 2006. The contract announcement is here, while the original specifications sent out for bid are here. MWI's sales literature for the Hydraflo can be found at this link.

The pumps are made up of two main components. The first is the pump assembly itself. Here's one of the Orleans Avenue pumps last October, after it and four others at Orleans Ave. had been pulled from the canal due to still-unexplained problems:


So you can see what's inside, here's some of the pumps on the ground at Orleans Avenue last spring before they were installed:


The above picture is looking into the discharge of the pumps, which is the top when they are installed. Here's a blowup with some of the major parts labeled:


The pumps are powered with hydraulic oil. The oil is supposed to be raised to 3000 pounds per square inch (psi) and then sent via pipe and hose to the hydraulic motor, which translates the energy of the hydraulic fluid to rotational energy on the shaft of the pump. The propeller blades attached to the shaft move the water upward through the pump and into large pipes, which discharge on the lake side of the floodgates.

The pump power units are far more complicated pieces of equipment than the pumps. They have large 735 horsepower, 1800 rpm Caterpillar diesel engines on them. The engines power the hydraulic power system through a gearbox, also known as a pump drive.

The heart of the hydraulic system is a pair of hydraulic pumps made by Denison Hydraulics. The pumps draw hydraulic fluid from a large reservoir on the power unit, raise the pressure of the fluid to 3000 psi, and send it to the pump. At the pump, the hydraulic motor spins the propeller shaft. After the hydraulic fluid passes through the motor, it goes through the coolers on the pump, and then returns to the reservoir on the power unit.

According to the Corps' Floodgates Operating Manual, here are the manufacturers and model numbers of these major components (where I've been able to, I've linked to the literature for the particular model):

Drive unit engine - Mfr: Caterpillar model: model 3412
Pump drive - Mfr: Durst model: 2PD10
Hydraulic pump - Mfr: Denison model: T7ED
Hydraulic motor - Mfr: Rineer model: 125 four-port (not enough info in manual to figure out exact model)

What can we do with this information?

I ran some numbers. If the Caterpillar engine is rated for 735 hp at 1800 rpm, that converts to a rated output torque of about 2144 foot-pounds. Then I looked at the rating sheet for the Durst pump drives.

According to that sheet, one is supposed to take the rated torque of the input to the pump drive, multiply it by a service factor, and then compare that number to the maximum rated torque of the chosen Durst Model (a 2PD10, in this case).

In our case, the service factor would probably be 1.5, which corresponds to a multiple cylinder internal combustion engine with uniform (i.e. no shock) loading. So we multiply 1.5 times 2144, giving 3216 ft-lbs as the number which the Durst drive's rating must beat.

According the rating sheet, the Durst 2PD10 is only rated for 1996 ft-lbs, which is less than the nominal torque from the engine, let alone the torque modified by the Durst service factor. The 2PD10 is the biggest two-pad pump drive in the Durst product line. It would appear that the Durst pump drives on the MWI drive units were severely undersized.

More floodgate pump news

First, this is the first of three new posts I've put up this weekend, along with an update to my Monday London Avenue post. So keep scrolling when you get to the end of this, because there's a lot more information to see.

The Times-Picayune put out a big article last Tuesday, February 6th, about the extra pumps going in at the 17th Street and London Avenue floodgates. I've been reporting on these pumps for quite a while, and much of the article contained previously reported information that's appeared here and in the paper, so I'll dispense with that and skip right to the new information.

1) The $85 million (that's a lot!) design-build contract for the construction and installation of the pump platforms, discharge pipes and other structural matters, as well as the installation of the pumps themselves, has been issued to Weston Solutions of West Chester, PA. The press release on Weston's website gives more details.

According to that press release, Weston is partnering with M.R. Pittman of Harahan, LA and Dynamic Indistries of New Iberia, LA. M.R. Pittman also won the $52.5 million contract to procure the 19 extra pumps, as I reported on January 18th about this contract. In that post, I also reported there appears to be a $15.5 million gap between the money going to Pittman and the money going to the two pump manuafacturers. Now Pittman has gotten another huge chunk of work.

However, what is really interesting about the Weston contract is who is not mentioned: Boh Brothers, the current installation contractor at 17th Street. From this press release, it would appear at first blush that they are OUT at 17th Street. Here's the exact verbiage from the press release:

"In order to perform this fast-track project, [Weston] partnered with two Louisiana-based construction contractors, M.R. Pittman Group, LLC and Dynamic Industries, Inc. LLC, who will perform construction services on the London Avenue and 17th Street Canals."

So it sure looks like the Corps is dropping Boh at 17th Street and replacing them with Dynamic.

2) Also very interesting is that the contract was issued through the Air Force Center for Environmental Excellence, or AFCEE. I don't know what to make about an Army contract getting issued by the Air Force, but it would appear to be the Corps actually thinking outside the box. One wonders why they haven't done this before?

3) Engineering firms on the job will be BBG&S Engineering Consultants of Birmingham, AL (apparently very small, with no webpage) and Gresham, Smith & Partners (apparently very large, webpage here). According to the Louisiana Secretary of State's website, BBG&S just registered to do business in Louisiana on January 4, 2007. Considering that bids for this contract were due late last year, that means that the Corps accepted a bid from a company that wasn't licensed to do business in Lousiana at the time of the bid. Interesting.

Weston appears to have at least four standing contracts with the AFCEE. Also, many of their executives have spent significant time with the Corps.

4) On his way out the door, retiring top Corps civilian Dan Hitchings took a peevish final swipe at New Orleans' citizens and local governments. He thinks we should be far more thankful for the Corps' efforts, no matter how late their results are:

"The goal is to have the last of the 19 new pumps in place and fully functioning by late July or mid-August. But corps representatives say one of the lessons of the past year of delays, setbacks and underestimated challenges is not to guarantee a date of delivery.

'We've learned not to speculate on schedules and not to give optimistic schedules,' said Dan Hitchings, who was the corps' civilian director of Katrina rebuilding until he retired Friday. 'It hurts your credibility because the dates are all people remember, no matter how much you're doing to try and get there.

'So until we get the contractors' schedules, the July-August window is our best estimate for getting the increased capacity,' he said."

Poor Dan. Perhaps he thinks the Corps should get a medal of appreciation for all their hard work, no matter how few the results. Oh wait! They already got one - from themselves!



We're supposed to take time out and reflect on how much the Corps has done every time they miss a deadline? Yeah, right. What chutzpah.

5) The Corps has finally recognized that they don't have enough pumps at the floodgates, even with this extra infusion of cash:

"The higher numbers are still considerably shy of the maximum rated capacity of Sewerage & Water Board pumps before floodgates were installed: 10,500 cfs in the 17th, 3,400 cfs in the Orleans and 8,000 cfs in the London. To reach those rated capacities, which are the top under ideal conditions, the corps would have to add another 3,000 cfs or so at both the 17th Street and the London Avenue canals. At Orleans, which drains far less water, corps officials have no plans to increase its pumping capacity above 2,200 cfs.

Corps officials said they are investigating whether they can equal the maximum provided by the S&WB.

'We've been working to match what the drainage systems are designed to handle -- a 10-year rain event -- and that's where the 7,600 and 4,800 numbers came from,' Hitchings said.

'But the Sewerage & Water Board says they can pump more than that, and they're right,' he said. 'They can pump that much if it's ponded, but what is the sustained pumping capacity? That's what we're re-examining.

'That's the guidance I've just given to Col. Bedey,' Hitchings said during his last week on the job before retiring. 'We'll re-evaluate and put in as much flow as they can sustain.'

It's amazing to see how slow their minds turn. We are a year and a half out from the storm, and they are only now realizing that their pumping scheme comes up woefully short?

By the way, the number for Orleans Avenue is an error on the part of the Times-Picayune. The actual capacity of station 7, which is at the entrance of the Orleans Avenue canal, is 2690 cfs. There are two 1000 cfs pumps, a 550 cfs pump, and two 70 cfs pumps.

6) Finally, a local paper in Pennsylvania did a write-up on Weston's contract award. It mostly includes information from the Weston press release. However, it also includes this gem from a Weston spokesman about the MWI Corp. pumps that are out there right now, and which are not going anywhere:

"'The pumps that were there didn’t work all that well,' said David Campbell, spokesman for Weston Solutions."

Lovely.

Pumps shut down because of weak levees on London Ave canal Dec. 30

Updated February 11th. Originally posted February 5, 2007. See below.
Also updated 7/28/07. See bottom of post.

Earlier this week, after speaking with multiple extremely reliable sources, I learned the following jaw-dropping news...

During the December 30, 2006 rainstorm in New Orleans, one of the very large 1000 cubic feet per second (cfs) pumps was shut down at Drainage Pumping Station 3 on the London Avenue canal for approximately thirty minutes. This reduced the pumping capacity at the station by nearly 25%. The reason for this was that the depth in the London Avenue canal - specifically the depth at its weakest section just south of the Mirabeau Avenue bridge - was too close to the Safe Water Level of four feet. Therefore, a pump had to be turned off in Station 3 to keep the water from rising any further.

Specifically, near the end of the December 30th storm, which was in the evening, when all the pumps were running at Station 3, the level in the weakened canal section (also known as a "reach") rose to 4 feet, 4 inches (Update: this number is according to the Times-Picayunes' February 8, 2007 article). The Sewerage & Water Board then called Jim St. Germain, the Corps' pumping guy. Mr. St. Germain came out to Station 3, agreed that the Safe Water Level was equalled or exceeded, and recommended the shutdown of at least one of the station's pumps. One of the large 1000 cfs pumps was shut down (there are three 1000 cfs units in station 3) for about a half hour, and the level in the canal dropped far enough that the pump could be turned back on.

Incidentally, this marks the second time in a two week span that the 4 foot Safe Water Level in the London Avenue canal was approached or exceeded.

This is shocking. This is huge. It means that my concerns about permanent reductions in New Orleans' pumping capacity due to weakened canal levees and walls are legitimate, and that those worries have already been realized at least once already. The true rainy season in New Orleans (outside of hurricane season) doesn't start until April and lasts for over two months.

What it also means is that personnel in the Corps' New Orleans District have failed to report this information publicly, even when given the explicit opportunity to do so on at least two different occasions when they have been asked directly about the strength of the London Avenue canal walls in public forums.

1) On January 8th, 2007, Corps senior project manager Jim St. Germain and project manager Daniel Bolinger appeared before the New Orleans City Council's Public Works Committee to speak about drainage matters. Mr. St. Germain was asked by Councilperson Stacy Head about the then-recent revelations that water depth had reached 3.6 feet in the London Avenue canal during the December 21, 2006 storm. He did not mention the pumping shutdown on December 30, despite his presence in the pump station that evening. I have the entire January 8th hearing on tape.

2) On January 26th, 2007, senior members of the Corps' New Orleans District office and the Hurricane Protection Office appeared before the board of the newly constituted Southeast Louisiana Flood Protection Authority-East. Those individuals included Colonel Richard Wagenaar (commander of the New Orleans District) and Colonel Jeff Bedey (head of the Hurricane Protection Office). When asked directly by the board about the London Avenue canal walls, they did not mention the December 30th pumping shutdown. They said they were aware of the problem and were working on it. Their slide presentation from the meeting says only this about the London Avenue canal, "Future tests may raise elevation."

These lies of omission have gotten to be too much. Citizens of New Orleans are now suffering directly for the Corps' inaction in attending to the weak canal walls and levees. On December 30th, it was 30 minutes. How long will the pumping shutdown be the next time? How long must New Orleans be trampled by all the kings' horses and all the kings' men before enough is enough?

I contacted the Corps for comment before posting this. They didn't reply.

Update, 2/11/07
As I mentioned above, the Times-Picayune caught up with this story three days after this post appeared. Their article had some of the information I had above, along with some very bland, uninformative reassurances from Jim St. Germain:

"In addition, St. Germain said the corps already has a private contractor working to determine whether 4 feet remains an appropriate safe water level in London Avenue Canal, which runs through the 7th Ward from Gentilly to the Lakefront.

And even before the Dec. 30 event, he said, the corps was looking at ways to bolster the walls so the canal could safely hold more water, in the event the level cannot be raised above 4 feet."

How long will they look before they finally do something? This attitude is exactly what rankles people when the Corps claims they have the same urgency to act as all the citizens of this imperiled city. Frankly, we've been hearing this same claptrap about plans for the London Avenue canal for five months now. On the September 7, 2006 edition of WLAE's "Road to Recovery," Colonel Wagenaar said almost the same thing as Jim St. Germain said above.

On that evening, Col. Wagenaar talked about how the Safe Water Level on London Avenue was to drop to four feet, and that the Corps was looking at things to do to raise the level, including reinforcing the walls or lining the interior banks of the canal. I posted that information to a number of New Orleans Yahoo discussion groups the next day. You can see my post to the Gentilly_after_Katrina group here.

The loss of pumping capacity due to the Corps' inaction has been demonstrated. The time for study is now over, period. It's time to get off the pot and do something.

[Update 7/28/07]
See my June 11, 2007 post, "Testing, testing," for the most up to date information on the London Avenue canal testing. The Engineering News-Record articles linked there are far better than the Times-Picayune's pathetic coverage. [end update]

Corps paying close attention to London Ave canal

I just happened to be out at the Mirabeau Avenue bridge over the London Avenue canal on Friday afternoon. For the second time in less than two weeks, I observed Corps personnel out in boats looking at the inside of the walls along the canal south of Mirabeau.

On Friday February 9, there were three white Chevy trucks with US Government license plates and a fourth with Oklahoma plates parked at the bridge. I'm sure the Government vehicles belong to the Corps. The access gate on the west side of the canal was wide open, and a flatboat could be seen south of the bridge. The bow of the boat was pointed at the eastern wall the entire time.

On Wednesday, January 31, there were two white Government vehicles parked next to the same gate. This time there were two flatboats in the canal. One seemed to be on the shore of the eastern bank, in the same area as the boat on February 9. The other boat was left at the bridge after the gentlemen (undoubtedly Corps folks) closed the gate and left in the Government trucks.

For the locals in the audience, the area they seemed to be most concerned with is located just north of the large cell phone tower on the Dillard University campus. You can see that area on this satellite photo. From these observations, I'm guessing that it is the canal's east wall that is causing the most consternation, and which is also governing the Safe Water Level of four feet.

And the leaks go on

More pump testing at the floodgates means more oil leaks.

As I last reported in November, the Corps and its contractors (specifically the pump manuafacturer, MWI Corp.) have spilled over 700 gallons of hydraulic oil into the three outfall canals during testing or demonstrations of pumps.

The spills keep coming. On the morning of January 25th, an unnamed MWI employee called in another spill at the 17th Street canal site:

January 25, 2007

Quantity released: 20 gallons
Description of incident: The caller is reporting a release of materials into the 17th Street canal from a water pump due to equipment failure on the pipe.

This is the sixth reported spill at the 17th Street site. Overall, we know of a total of ten spills at all three floodgate sites, nine of which have been reported to the National Response Center, as required by law for any oil spill which produces a sheen on the water or sludge in the soil. A September 15, 2006 spill at the London Avenue canal floodgates, which got reported by the local media the following day, did not get reported to the National Response Center. This is an obvious breach of Federal regulations which has yet to be addressed.