Here we go again

The last time we took a look at the pumps at the new West Closure Complex (WCC) on the west bank, across from New Orleans, we found that only 8 of the 11 pumps would be ready for June 1. No reason was given for the shortfall. In addition, we found there were very loose terms on the testing of the pumps. That testing was due to start in April.

We know the Southeast Louisiana Flood Protection Authority - West (SLFPA-W) has inspectors watching all the Corps' jobs on the West Bank. From their inspection reports, we were able to learn about systemwide problems with excessive debris being placed in the levees all over the west bank (Part 1, Part 2, Part 3). SLFPA-W's reports on the West Closure Complex have been similarly detailed, including descriptions of the first pump testing, which did occur last month. It did not go well.

The May 3, 2011 SLFPA-W report on their April 29, 2011 inspection visit tells the tale (note, while the report indicates an inspection on April 29th, it actually recounts events that happened over a multi-day span dating back to mid-April):

"During this reporting period the dry testing of pump no. 13’s engine, gear and pump began. There were several issues regarding gear oil pressure alarms and engine fuel delivery, but once those items were corrected the engine was started and the pump rotated with a dry impeller."

[Note: while there are currently 11 pumps at the WCC, the original design called for 13. It appears the original numbering has survived into the final design, thus the descriptions of pumps 12 and 13 in a station with only 11 pumps.]

So a few hiccups, but they got it going for the dry test. The terms of that test are laid out in the WCC specifications. Here's an excerpt:

"After pumping unit has been completely assembled, including all rotating elements and lubrication system, operate at full rated speed for a period of 4 hours or until the bearing temperatures have stabilized, whichever is longer, to assure proper alignment and satisfactory operation."

The dry test specs also call for monitoring of vibration in horizontal and vertical axes. They then describe exactly what is meant by stabilization of the bearing temperatures:

"Pumping unit shall be operated at rated speed and the bearing temperatures recorded at 5 minute intervals until the temperature rate of rise has stabilized for all bearings. Bearings' temperature shall be considered stabilized when the rate of rise does not exceed 1 degree Fahrenheit in five minutes."

If that doesn't happen?

"Dry test run shall be repeated if it is necessary to interrupt the test before all bearing temperatures have become stable. If after a run of reasonable duration the temperature rate of rise for any bearing has not stabilized, test shall be terminated until the cause of overheating is determined and corrections made. Then dry test run shall be repeated."

This is all pretty reasonable. And if they have problems that are the fault of the manufacturer or the pump assembler?

"Should tests reveal that there is a design deficiency or a manufacturing error in pumping unit components, the problem shall be promptly corrected by and at the expense of Contractor."

Good. The public shouldn't be on the hook for a manufacturer not meeting the specs. By the way, the manufacturer of these pumps is Fairbanks-Morse. They were assembled by local company Bollinger Quick Repair.

So how did all this work in practice? Back to the SLFPA-W report:

"The engine was brought up to operating speed and the test concluded with a failure of the upper bearing in the pump."

That's not good.

"Upon observation it was determined that the bronze sleeve type bearing had cracked and was seized to the shaft sleeve. Further analysis indicated that the bearing was installed such that it was not plumb with the shaft and the shaft loading on the bearing caused the failure."

SLFPA-W included pictures of the cracked upper bearing on pump 13:


Date stamps on these and other pictures of pump 13 in the report indicate the bearing failure probably occurred between April 17th and April 19th.

After a failure like that, the prudent thing to do would be to check the other pumps for similar defects. That's what was done, according to the SLFPA-W inspection:

"The remaining aligned pumps were checked and minor adjustments were made."

With the lack of detail on this checking procedure, it's difficult to ascertain how comprehensive the procedure was.

The next step was to try another pump:

"Next pump no. 12 was prepared for a dry pump test and the operation of that unit was discontinued when the lower pump bearing temperatures became elevated above the safe operating range."

Again, not good.

"The decision to test pump no. 11 under wet conditions was made and the pump produced flow for approximately five (5) minutes before the lower pump bearing RTD indicated excessive bearing temperatures."

This decision certainly raised my eyebrows. It's not told how the Corps jumped from dry testing to wet testing, which is a big leap. While the specifications do not explicitly call for the wet test to follow the dry test, it would certainly seem prudent to do so.

Anyhow, at this point it was clear to them there was a systemic problem with the bearings:

"With that issue it was concluded that the bearing clearance tolerances were too tight and that all bearings in each pump would be removed and re-bored to achieve a total bearing clearance of 0.025 inches."

Whoa! After spending months installing all these pumps (and showing them off "victory lap"-style to various groups, multiple bunches of fellow Corps employees, and assorted Pentagon bigwigs before anything had even run), they now had to yank all 11 of them out and completely disassemble them. Maybe the tours should have waited until they had a functioning facility?

So they started pulling them apart:

"The removal of the three (3) bearings in each pump began with pulling of pump no. 3. Once the bearings were removed from pump no. 3, they were delivered to Bollinger Marine for boring. While boring operations were ongoing, pump no. 12 was removed from its intake bay setting and the pump was disassembled."

A passage later in the report gives an idea of what was involved:

"The majority of the mechanical work performed for this period revolved around the testing and ultimate disassembly of pumps 3, 12 & 13. That work included removal of the intake air and exhaust piping, disconnecting the gear oil and coolant piping from the right angle gear, disconnecting engine to gear drive shafts, disconnecting gear to pump shaft couplings, removal of the right angle gear, removal of the right angle gear pedestal and pulling of the three pumps that had stop-logs already installed in the intake bays."

Here's pump 3 after it was lifted out but before it was taken apart:

And here it is after disassembly:


That is a ton of work, and they're going to do this on ten more pumps (all eleven pumps were already installed, as noted in the previous SLFPA-W report). I assume the Corps is having the contractors segregate the expenses associated with this work so the taxpayer doesn't end up on the hook.

The pump removal didn't exactly go as smoothly as the above paragraph would indicate. Returning to the earlier passage in the report:

"At the time of this report, pump no. 13 was being prepared for removal but there was some difficulty in removing the lower half of the pump to gear coupling. Nitrogen was being used to cool the upper pump shaft and the pump coupling was being heated with rose bud torches to provide the expansion needed to release the coupling from its interference fit."

Nitrogen! They had to bring in nitrogen to decouple the pump shaft!


So how did all this wind up? From the SLFPA-W report:

"Currently the bearings from pump no. 3 are being installed in pump no. 12 and the current scheduling indicates a possible test of that pump by 07 May 2011."

Presumably, the next SLFPA-W report will detail what happened during that test.