Fix the pumps

Saturday, May 12, 2007


Updated quite often. See below at various spots.

In light of some very cloying, cowardly coverage by the local media...

a) WWL-TV does a report on London Avenue canal testing on May 10, 2006, and only interviews Corps personnel, and then shows the wrong spot on the canal where the test will take place [they showed the repaired breaches at Robert E. Lee Ave, which is over a mile north of the area where the tests will really take place - just south of Mirabeau Ave.]!

[Update, 6/1/07: I have issued a correction for the above paragraph, which is erroneous. The correction can be found here.]

b) WWL-TV does a special series of reports titled "Closing the Gaps," from April 30th to May 3, 2007, supposedly about all the problems we still face with levees and pumping, problems brought about by the Corps. So who do they interview as the main source of information... the Corps! Part 1 here (video here), part 2 here, part 3 here, I couldn't find the print version of part 4, but the video is here.

c) On May 9, 2007, the Times Picayune republishes a Corps press release about the new head of the New Orleans District as an "article" without mentioning that the current commander - Colonel Wagenaar - is retiring a year before his usual 3 year tour is up.

d) This one reaches back a bit, but on Sunday March 25, 2007 the Times-Picayune published what was portrayed as a "big" front page article, a real love letter to the Corps, detailing how ready the Corps was to save us all. In reality, it was just a huge rewrite of the Floodgates Operating Manual, a document which has been publicly available here - due to my requests - since last August. Also the article included only quotes from Corps of Engineers personnel (a hallmark of a lot of Times-Picayune coverage of the Corps these days), including a tremendously cheap shot by Wagenaar:

"...And although Wagenaar applauds the 'heroic' work of local emergency responders to the 17th Street Canal breach -- especially West Jefferson Levee District employees who engaged in road building -- he said he welcomes the streamlined chain of command achieved through levee district consolidation. 'What is not streamlined is dealing with the Sewerage & Water Board. They don't seem to work for anybody.' Wagenaar said."

Of course, the article's author leaves that comment - which is the height of hubris from an agency that seems to answer to no one, including Congress - unrebutted. She apparently didn't even bother to call the S&WB for comment. Also missed in all the glorification of the Corps' communications plan is this actual quote from the Manual:

"One Corps of Engineers radio will be furnished to the S&WB for communications in addition to having the Canal Captains stationed at their respective S&WB pump stations."

Yep, that's right: the agency in charge of pumping gets all the benefits of a single radio to communicate with the Corps.

[Update, 6/7/07: The radio has been addressed as part of the kerfluffle over control of Drainage Pumping Station #6 on the 17th Street Canal. See my June 7th post here.]

... So in light of all that (and much, much more) crappy coverage of what the Corps has done and isn't doing over the last two weeks, allow me to let you in on some real stories that anyone with a modicum of investigative sense would have ferreted out by now:

1) The hydraulic fluid reservoirs on MWI's engine skids are undersized by a factor of 3. MWI's own catalog calls for a 900 gallon reservoir. FPI's 60 inch pumps use a 900 gallon reservoir. The MWI models at the floodgates have 300 gallon reservoir. This decreases the capacity to get rid of heat generated when the pressure of the hydraulic oil is raised up to 3000 psi (theoretically - though it's probably closer to 2600 psi), and then shares some of that energy with the Rineer hydraulic motor in the pump unit. If there's too little room in the reservoir, the fluid doesn't have enough time to cool, meaning the oil stays thinner than it needs to be. Less viscosity means less ability to transfer energy means less ability to pump water.

Why did MWI do this? Maybe to save money on hydraulic fluid, which they were responsible for supplying for startup and commissioning. Such a savings could have been in the range of $50,000 to $150,000, money that goes straight to MWI instead of some supplier.

According to my latest batch of testing reports, received via FOIA request (and held back by New Orleans since January), during testing MWI actually was pointing fans at the drive units to keep them cool until the Corps QA's told them to stop it. Overheating of many components - not just the hydraulic system, but also the Durst gearboxes - on the drive units is still a big, unaddressed concern.

2) The Corps has yet to issue three Orleans Parish pump station repair contracts, over 20 months since Katrina. These contracts are part of the original $40 million set aside for pump station repairs. The three contracts - Electrical-Mechanical Repairs, Building/Structural Repairs, and cleanout of the PS#17 Discharge line and the inlet basins of three or four (it's unclear if Elaine Street's basin cleanout is included in this contract) other stations - appear to be on the very slowest track possible, since the Corps has had the money since January of 2006. The following is a quote from a February email sent to me by one of the people working on the Discharge Line solicitation (W912P8-07-R-0027), which has been indefinitely postponed since February 12, 2007:

"The reason we suspended the solicitation was due to the Right-of-Entry agreements we are working on getting from the Port of New Orleans and Project Permits from the Orleans Levee District and DOTD. We were under the assumption that we could do the work under the ROE and CA with the New Orleans S&WB. We are doing this so that everyone knows what is going on and we don't have any delays once the project is awarded. The ROE from the Port of New Orleans has been obtained and we are working on getting the two project permits."

Sure sounds like urgency to me!

[Update, 7/9/07] And the ugency continues on this one. The PS-17 Discharge Line/PS-6 & PS-7 Basin Cleanout solicitation was revived on June 5th, and has since gone through numerous changes and delays, still without award of a contract. The latest modification (number Seven), issued July 9th, changes the response date to July 11.

Also, one of the other remaining Orleans Parish repair contracts was finally issued a couple of weeks ago. According to this Corps contract award announcement, a $3.25 million contract for Electrical/Mechanical Repairs at the S&WB drainage pump stations has been awarded to Healtheon, Inc on June 19, 2007. These repairs will happen at nearly every one of the 23 stations.

While the award announcement gives a local addess (Suite 2500 in the downtown skyscraper at 201 St. Charles Ave.), the Louisiana Corporations database shows that Healtheon's principal office is in Birmingham, Alabama. A Google Maps search of that address shows it to be a private house. Its webpage shows its focus to be in "healthcare construction," but it appears to have a division which just happens to deal in "disaster recovery." Healtheon does not appear to be registered as a contractor in Louisiana, according to the Louisiana Licensing Board for Contractors website.

Healtheon also received the contract for Electrical-Mechanical repairs to pump stations in St. Bernard Parish, as noted in this January 28, 2007 award announcement. The value of that contract was about $786,000. How is it there was only one company anywhere that could handle these two contracts? Why wasn't this open to competitive bidding?

The Orleans Parish E-M repair contract, like the St. Bernard Parish E-M repair contract, was yet another sole source 8(a) set-aside. Sole source 8(a) awards have become the pattern for the Orleans Parish pump station repair contracts. Almost all of them, except the motor rewinding contracts, have been sole source set-asides. There's no competition required, and I've yet to find where one can even find solicitations for them. They stink to high heaven. And what the heck took so long to issue this latest contract? I'll tell you - no urgency.

Speaking of which, two other Corps pump station repair contracts still remain unissued. They are the Building & Structural Repairs contract, and the Elaine Street Station Reconstruction contract. The Elaine Street Station reconstruction contract is somewhat new, having appeared on the docket within the first few months of 2007.

[Update, 7/24/07]
As I surmised above, the PS-17 Discharge Line/PS-6 & PS-7 Basin Cleanout, the Building & Structural Repairs, and Elaine Street Reconstruction contracts are still unawarded, according to the July 19, 2007 schedule of Orleans Parish Pump Station Repairs from the Corps.

Also extremely disappointing embarrassing is the fact that the repairs are moving at a snail's pace. As we approach the two year anniversary of the storm - and as we just passed the 18-month anniversary of the signing of the bill which appropriated the money for these contracts (the 3rd Katrina supplemental was signed December 30, 2005) - three contracts remain unawarded and fewer than half of the 14 contracts have been completed. In addition, those that have been awarded are now taking even longer than previously scheduled.

For example, contract OPS-6A, (awarded to IPS of Louisiana and then subcontracted to Conhagen; Corps award notice from 12/1/2006 here), intended to replace bearings in vertical pumps at five pump stations, was previously scheduled to take 243 days, according to the Corps' November 1, 2006 repair schedule. The latest schedule shows that work now taking 290 days and dragging into the end of September, 2007. And if one takes a close look at the percent completion (42%) versus the percent of time passed since the contract started (75% of the anticipated 290 scheduled days) it appears that the work on this contract is even further behind than indicated. I doubt the contractor will be able to finish 58% of the work in 25% of the time. The delays seem to be concentrated in the three New Orleans East stations (14 and 16 on the lakefront, and 15 way out east near the end of the world), where construction progress is respectively 19%, 12%, and 26%.

Looking at the big picture, the scheduled completion date for all of the repair work is February 9, 2008. That's 34 months after the storm hit!

They really don't care, do they?

3) [Update, 5/16/07] Has anyone asked about the Corps' plans to evacuate workers from the floodgates for anything over category 2 (see the Floodgates Operating Manual)?

Also see this 5/16/07 article from the Times-Picayune:

"The other three members of each corps gate team will stay at the gate for storms up to Category 2 hurricane strength. For Category 3 storms, they'll relocate to a safer structure nearby. For Category 4 and 5 storms, considered worst-case hurricanes, one three-man team will retreat to a bunker at the corps headquarters complex on Leake Avenue at Prytania Street, while the other two teams will evacuate the area until the storm passes, Wagenaar said."

According to the manual, that nearby safer structure is the Louisiana Dep't of Transportation & Development building at 7252 Lakeshore Drive, shown here. It's about a quarter mile from the 17th Street gates, which makes it quite far from the gates at Orleans and London.

There's no way to remotely control (other than on/off) or adjust the floodgate pumps; that has to be done at the drive skid. The pumps will very likely need manual adjustments - which will be impossible with no one there. This was buried at the bottom of the T-P March 25 article, written by a different, less rigorous author than the May 16th article, and made to sound not very important:

"If the area is threatened by a Category 4 or 5 storm, the men probably would be pulled off site once the gates were down and the pumps running. They would return as soon as possible after the storm passed, Accardo said. 'Why have them in harm's way? These aren't really safe houses in the sense that Jefferson Parish is building safe houses,' he said. 'These are equipment houses designed to handle a hurricane, but not a major storm surge.'"

Well, what if there's a problem with the fuel lines from the 20,000 gallon diesel fuel tanks? By the way, the Corps could have gotten dual drive units, with electrical motors as well as diesel ones - MWI offers them in their catalog, which is the basis for the Corps specs after all. Or they could have specified a drive system that could actually be controlled remotely, rather than one that can barely be turned on and off by remote. But why bother going for backup power or a truly robust control design? They were in a hurry!

In any case, the Corps knows leaving the pumps alone is a problem, but they just skirt over it in their own manual. From "Step 8 (T - 12 hours)" on page 10 (Adobe page 14):

"Due consideration must be given to what storm conditions are expected to be after the gates are installed and the pumps are initialized to permit the personnel that are part of the closure operation will be out of harm’s way as they travel to local safe locations or are being evacuated to remote safe locations."

That is, the Corps plans on "initializing" the pumps, i.e. turning them on and hopefully maximizing their flow, and then leaving them. What happens if they shut down and can't be restarted remotely (a very likely circumstance if one reads how to start them in the first place- which definitely requires someone being physically in front of the drive unit)? Well, I guess then we're out of luck. The Corps says, "c'est la vie."

I wonder if they plan on rehearsing leaving the pumps unattended the entire time they're running in their drills for this hurricane season? I sincerely doubt it.

4) Does the entire city know and understand that we're depending on a contract to a diving firm to guarantee the gates actually work? When the gates are lowered, they seat in a trench, which a firm called H.J. Merrihue is responsible for keeping clear during operations. If the gates don't seat in that trench, they'll fail. I'd like to know exactly how the diving firms do that clearing while a tropical storm is coming into the lake, and the city is pumping rainwater out along the canals. Plus, the Corps says the diving firm is responsible for evacuating their own personnel after the gates are lowered, despite the Corps getting the Corps folks out on helicopters under a separate contract. Again, see the Floodgates Operating Manual for the details on this one.

These are the same trenches that the Corps' contractors filled with grout during the height of hurricane season last summer without realizing it. They had to chip the stuff out mechanically at 17th Street and Orleans Avenue, meaning had they tried to lower the gates before the trenches were clear, they wouldn't have seated, and the city would have been defenseless against storm surge. I broke this story last August and posted about it to a number of local newsgroups. You can find my postings here, here, and here. I also posted some post-game analysis, including catching the Corps in some lies about the importance of the trenches, here. My concerns about the gates seating still exist, and have been echoed in National Geographic's latest feature, from which comes this nugget about the floodgates:

"A Dutch engineer recently visited some of the new floodgates and pumps installed at the mouths of the city's three main drainage canals. His verdict: They may be 'doomed to fail' in the next big storm. The engineer, who asked not to be named because he sometimes collaborates with the corps, notes that the gates have no mechanism to remove sediment and other debris that might keep them from closing as a storm approaches. Instead, the corps says it will rely on divers to check for obstructions and clear them away."

This is very troubling. Supposedly, the Corps was supposed to construct some metal pieces designed to fit in the trenches to keep them clear until they're needed, but of course we never heard whether that happened, or if it did, if that idea was successful. Plus, I'm pretty sure the trenches themselves are pretty beat up from all the jackhammering last summer, so I doubt they could build pieces that would completely seal against silt, rocks, and other debris building up. It's just another example of how what appears to be robust could in fact be very weak.

[Update, 7/24/07]
The grout in the trench problem was apparently very serious. It was noted in a PowerPoint slideshow given June 27, 2007 by the consulting engineers on the floodgates job at last months' SAME Infrastructure conference. The main conference webpage is here, and the presentation on the gates - given by guys from Linfield Hunter Junius and URS - is here.

Here's the exact text regarding the grout problem, from slide 43. Note the final bullet point.
"- Soil Mixing
• Soil Improved to an Average Shear Strength of 1,000 PSF minimum
• Allowed Fewer Piles
• Critical Item in Construction Schedule
• Soil Mix Residue Issues – Grout Silted in Gate Foundation and Pump Sumps – Major effort to remove soil mix residue with divers"

There are also pictures of the grout that flowed all over the site and is still there (slides 73 and 74). The grout went everywhere. It looks like someone stepped on a giant tube of toothpaste.

The presentation is fascinating, since it contains information never before revealed, like alternatives considered to the final needle gate design. One choice was to sink barges!

The presentation appears to have been thrown together fairly hastily; the pictures of the grout from 17th Street are actually dated two days before the slideshow.

And of course, they are still putting lies about the pumps out there. There is a mention of them on slide 47: "Hydraulic Pumps – Only Type of Pump Available in Allotted Time." This fails to mention that the pump specification was taken straight out of MWI's catalog. It also fails to mention all the problems before and since the installation of the pumps.

One other thing that isn't mentioned: the sill at London Avenue was completely screwed up and delayed startup of that floodgate for over a month. The sill problems were mentioned in Times-Picayune in a May 12, 2006 article (back when the T-P actually cared what was going on around them). There were more details in this July 3, 2006 article from something called "Construction News:"
"Complications with the sill prevented M.R. Pittman from completing the London Avenue Closure by the June 1 deadline. 'We expected to have the sill complete, but we ran into difficulties with the sheetpiling,' said Leroy Smith of the USACE. 'Some split due to hard driving. When compressing sheetpiling, we hit hard material at elevations of minus 58 or minus 60.' Smith says the piling started racking and fell out of alignment. The sheets, separated at the bottom, had to be pulled up, and Pittman had to start all over.

Ranjit Gujja, USACE project engineer for the London Avenue Closure, said the same strength of sheetpile was used when driving the second time around. 'We used a double template instead of a single template to give more balance and alignment.' Sheetpiling was driven between the two templates the second time around, to prevent racking.

The sill, a crucial element in the design of the canal, held the job up. On June 1, divers were cutting down the sheetpile on the west side of the sill, which was complete, and Pittman was in the process of completing the east side. The closure is expected to be complete by July 1. The cells were expected to be completed within a week. Discharge pipes were set, and the pumps were scheduled for a test run the next week."

[end update]

There's tons of other questions which must be asked, and which are not. This is just a sample.


  • Hey, I think you are doing a great job and providing a wonderful service to all residents.

    Two questions...

    1) did you see the tv (wwl? I don't remember) coverage of the corps' Vitter response - after they say everything is fine, the tv station shows video footage of one worker DUCT TAPING some (hydraulic, I think?) hoses together while a 2nd worker holds them together for him? What a joke!

    2) why isn't everyone in the city/state/etc outraged about what is going on, and what can we do? No one seems accountable or seems to care about the outright lies/criminal activity of the corps!

    I am angry,depressed,scared and feel as if we cannot make a difference. God help the city when the next big storm comes along.

    PS Is wagenar retiring, OR forced out?

    By Anonymous Anonymous, at May 12, 2007 12:20 AM  

  • Any updates on Eustis engineering's liability for their part in the original failure? I haven't heard anymore since a brief mention in the news many months ago.

    By Anonymous Anonymous, at May 12, 2007 12:30 AM  

  • Matt:

    Great work! This is the observation that just stunned me:

    during testing MWI actually was pointing fans at the drive units to keep them cool until the Corps QA's told them to stop it

    I imagine the conversation might have gone like this. (Note: This is not a real conversation...)

    Corps QA: What's the fan for?

    MWI: Um, well we're hot and the air conditioning isn't working.

    Corps QA: Please.

    MWI: Er. The night shift left it on?

    Corps QA: Keep trying.

    MWI: When the building caught fire during the welding the regular fans for ventilation broke and we need these to get rid of fumes.

    Corps QA: Better, but still not good enough.

    MWI: We made the reserve tank smaller to save money on fluid. The idiots in New Orleans won't know the difference so we can save $50,000 to $150,000 on hydraulic fluid, money that goes straight to our bottom line. Besides, the pump will only be used once every 20 years and by then we will all be dead and gone.

    Corps QA: Now THAT sounds about right.

    I guess it would be amusing it the failure of these pump wasn't you know CRITICAL.

    Keep on them Matt.

    By Blogger spocko, at May 12, 2007 8:51 PM  

Post a Comment

<< Home

Go to older posts Go to newer posts