Pumps ... fixed?

[Updated June 4, 2011, see bottom of post]

When we last looked in on the pumps at the Corps' West Closure Complex (WCC), they were all getting yanked out because all their bearings were overheating after just a few minutes' run time. How'd that work out?

Apparently pretty well. The Southeast Louisiana Flood Protection Authority - West has published their latest progress report on the WCC. It covers most of May, 2011 and it includes an update on the pumping situation:

"PUMP TRAIN:
During the past two (2) weeks, Pumps 6, 7, 8, 9, 10, 11, 12 and 13 were reassembled, realigned, break-in tested and wet tested for four (4) hours. There were some minor issues regarding fuel delivery and engine speed control but all issues were corrected and the four (4) hour tests were completed as specified. Test results have been submitted and are being reviewed, but preliminary vibration testing indicates that most recorded vibrations were in the 0.060 inches per second range and the allowable vibration according to Hydraulics Institute Standards ANSI/HI 9.64 is 0.28 inches per second RMS for this pump speed. All pump bearing temperatures stabilized well below the 200 deg. F maximum limit with 134 deg. F being the maximum bearing temperature recorded on a non-flooded upper enclosing tube bearing.
This brings the pump count to eight (8) that have been officially wet tested as required for the 01 June 2011 Interim Protection deadline. The Contractor intends to complete an additional pump next week and two (2) others in the next several weeks."

Sounds like good news.

"the four hour [wet] tests were completed as specified." Let's look at the details of those tests. We looked at the specifications of the wet tests earlier. Here they are again:

"Each pump unit shall be given a test with water under load, at or near normal operating conditions, for at least 4 hours or as directed by the Contracting Officer. The test shall be conducted by the Contractor and will be witnessed by the Government. All supplies and equipment required to conduct the test shall be provided by the Contractor. During the test the operation of the pumps will be observed and measurements of sound, vibration and bearing temperatures shall be taken and recorded. Without additional costs to the Government, the Contractor shall make all changes and correct any errors for which the Contractor is responsible. The Contracting Officer may waive or postpone the test if sufficient water is not available. Appropriate changes will then be made to the contract.

Vibration and bearing temperature records shall be kept by the Contractor and turned over to the Contracting Officer at the conclusion of the contract."

So there are four basic parameters involved here (besides the four hour duration), though I'm sure there's more detailed monitoring going on for the test. The basics are:

1) Sound
2) Vibration measurements
3) Bearing temperature measurements
4) "Normal operating conditions"

The first three are to be measured at the fourth. Thanks to this latest SLFPA-W update, we know a little about the last three of these parameters. The update is (ahem) silent on the sound measurements.

Vibration is supposed to fall within limits set by ANSI/HI 9.6.4, "Rotodynamic Pumps for Vibration Measurement and Allowable Values" for a given speed. It's good to see this standard number in print, because it is the latest standard available (unlike the list of standards in the actual specification, which was obviously not checked before it was issued). It's from 2009.

Bearing temperatures are apparently not supposed to top out above 200 degrees F, and during the tests never exceeded 134 degrees F.

The other parameter mentioned in the wet test specifications is "normal conditions." That refers to two numbers: the height of the incoming water and the speed at which the pumps spin. The Corps doesn't control how much water is at the inlets to these pumps. It can range anywhere from 0 feet to 7 feet, and is dependent on the tides and if there was rainfall. But they can change the speed, which directly affects the flowrate they are shooting for - around 1740 cubic feet per second (cfs) per pump. The exact speed-flowrate correlation was determined during model testing by the manufacturer, Fairbanks-Morse.

I have learned from the author of the SLFPA-W update, Mr. Danny Caluda, that the 5400 horsepower C280-12 Caterpillar pump engines are programmed to spin at two speeds: 990 RPM and 1010 RPM. After the power is transmitted from the engine through the Lufkin Industries gearbox, the two engine speeds correspond to a low pump speed of 171.7 RPM and a high pump speed of 175.2 RPM. The lower speed is for lower inlet water elevations (anything below 2 feet). The higher speed is for the higher elevations (2 feet and above). The WCC Water Control Plan and the more detailed Interim Standing Instructions call for the inlet height to be kept below 4 feet, with hopes that it never gets to 7 feet. If water does get that high, that probably means the local pump stations feeding the WCC will be shutting down in order to prevent levee breaches, but causing neighborhood flooding. More details regarding this scenario, which is more likely with the 8 pumps currently in service as opposed to the full complement of 11 pumps, can be found here and here.

Note the rated - and specified maximum - speed of the engines is 1000 RPM, but the top speed in practice is 1010 RPM. Mr. Caluda tells me that at the 1010 RPM speed, the engines do not exceed their continuous duty horsepower rating, which is good. The increase above the original specification came about when the number of pumps was reduced from 13 to 11 in a cost-cutting move. In order to keep the total flowrate of the 11-pump arrangement equal or greater to that of the 13-pump arrangement, it was decided to run the pumps a little faster at higher inlet elevations. The speed increase from 990 to 1010 RPM allows for an additional 45 cubic feet per second (cfs) of flow for each pump, or an overall increase of 495 cfs, something which will be useful when the water is higher during a storm.

Since the water was so low in the canals during these May tests (around 0), the pumps were only tested at the lower speed, 990 RPM (as noted on the photos below). The difference between the two settings, 20 rpm on the engine or 3.5 rpm on the pump, is not particularly substantial, so running these tests at 990 RPM would appear to satisfy the specification requirements for testing "at or near normal operating conditions." In addition, Mr. Caluda informs me that when the water does get above 2 feet (after a rainstorm causes stormwater to be pumped into the canals upstream of the WCC), the Corps and SLFPA-W hope to have tests done at the higher speed.

Mr. Caluda included pictures of from the tests of four pumps - 6, 8, 11, and 13. There are no pictures from the tests of pumps 7, 9, 10, or 12.

On May 18, 2011, pump 11 was tested:


Pumps 13 and 8 were tested on May 21st:




Pump 8 was tested again on May 23:


Pump 6 was tested May 25th:



The only press video I can find around the pump testing was a video shot by the Times-Picayune newspaper on May 24th:

The date of the video, May 24th, is confirmed by this Corps photo:


What's interesting about this video, besides the fact that it appears to be the only one of the pump testing, is that it shows another test of pump number 8. That would make at least three different days pump 8 was turned on for testing - May 21st, May 23rd, and May 24th.

I've learned that the actual final 4 hour test of pump 8 did not occur until May 31st. Apparently, there was an issue with the Caterpillar engine speed control during the intial tests. An engine control module was replaced and the pump was retested, but the problem persisted. After replacing a wire from the engine control panel to the governor, the problem was repaired and the pump made it through its four hour test on the 31st. These kind of hiccups are not unusual during testing, though it is notable that the engine had previously been tested by Caterpillar last summer before installation.

However, this does bring into question Corps public statements about the testing on May 24th. Take a look at this screenshot of another Corps photo taken that day:


There are different flavors of "success." While the pump may have pumped the correct amount of water that day, its engine was still not meeting specifications, and the Corps knew that. It's a small thing, admittedly, but would it have really hurt them to exclude the word "successful" from the caption on this photo (and this one, and this one)? Such slipperiness with the truth does not help the Corps' case.

As noted in the SLFPA-W update, a ninth WCC pump is due to be tested within the week, and the remaining two pumps are due to be tested by the end of June. I will be sure to pass along the results of that testing.

[Update, June 4, 2011]

With the four hour wet tests apparently out of the way, the Corps felt confident enough to have the cameras out for a photo op, which they portrayed as a test.

WWL-TV was there:


As was Fox 8:


Both pretty much regurgitated Corps press release materials ("can fill a swimming pool in blah blah seconds," etc).

The Times-Picayune was also there, and provided a little more depth, mentioning the bearing failures in April, along with excuse-making by the Corps:

"A bearing on one of the pumps overheated and cracked, prompting the contractor to remove all of the pumps and modify the bearings, said Kevin Wagner, a corps senior project manager who said the need for adjustments was not surprising.

'These are custom pumps,' he said. 'It’s not like we can put them in a lab and test them. They had to be adjusted in the field.'"

"Adjusting" is tweaking the engine speed controls like they did on pump number 8. Pulling every pump out a month before startup and remilling every bearing on them, after spending hundreds of thousands of dollars to rush to get most of them installed before the building's roof was put on? That's systemwide screwup, either by Fairbanks Morse in the bearing specifications and drawings, Bollinger Quick Repair in the assembly, or some subcontractor in the manufacture. Here's the list of those subcontractors, courtesy of Fairbanks-Morse:

Advance Manufacturing, Cleveland, OH
Advantage Machining, Aurora, IL
Baumann Coatings, Bessemer, AL
Bay Cast, Inc, Bay City, MI
Bollinger Quick Repair, Harvey, LA
Decatur Foundry, Decatur, IL
Girard Machine Co, Girard, OH
Hardie Tynes Co., Birmingham, AL
J&J Precision Machine Co, Cuyahoga Falls, OH
Louisiana Machinery, New Iberia, LA
Lubrication Systems Co, Baton Rouge, LA
Lufkin Industries, Lufkin, TX
Monarch Corp, Milwaukee, MI
Profile Finishing Systems, Waukesha, WI
RH Fastener Supply, Riverside, MO
RoMac Freight, Parkville, MO
St. Mary's Foundry, St. Mary's, OH
WJW Freight, Syracuse, NY

And as far as the pumps being "custom," it's not like Fairbanks Morse is new to this business. Their website, right next to a link about these very pumps, says:

"For over a century, Fairbanks Morse has been manufacturing a wide range of pumps for applications in municipal and industrial installations."

It's hard to believe anyone at the Corps New Orleans District still tries to gild the lily, but I guess old habits die hard.