Debris, Part 1

Part 2
Part 3
Part 4
Part 5
Part 6
Part 7
Part 8
Part 9
Part 10
Part 11
Part 12
Part 13
Part 14

Over the next couple of months, the Corps of Engineers will be trumpeting the June 1, 2011 "completion" of their 100 year protection across the greater New Orleans area, made up of raised levees, big gate complexes, pump stations, and lots of federal dollars. But there's a problem lurking beneath the surface.

At Corps 100 year projects across West Bank of the Mississippi, the dirt being placed on levees is loaded with stuff that isn't dirt. Excessive organic debris and other flotsam and jetsam keeps coming out of the borrow pits with whom the Corps has contracted. And then when it's poured on the levees, it's not picked out by contractors. Also, it appears some Corps personnel are less than diligent about identifying it as well. If there's too much debris in the dirt, it can undermine the integrity of the levee. The Corps specifications for the soil say this:

"All fill materials shall be free from masses of organic matter, sticks, branches, roots, and other debris including hazardous and regulated solid wastes. As earth from the designated excavation areas may contain excessive amounts of wood, isolated pieces of wood will not be considered objectionable in the embankment provided their length does not exceed 1 foot, their cross-sectional area is less than 4 square inches, and they are distributed throughout the fill. Not more than 1 percent (by volume) of objectionable material shall be contained in the earth material placed in each cubic yard of the levee section. Pockets and/or zones of wood shall not be placed in the embankment."

It's the 1% number that is the problem. It looks like some projects might be exceeding it. This debris problem certainly has the attention of the West Bank flood protection authority (officially the Southeast Louisiana Flood Protection Authority - West, or SLFPA-W), which sends inspectors out to the various contruction sites in its jurisdiction nearly daily. Fortunately for the public, SLFPA-W posts the reports of those inspections nearly instantly, so we have a real-time history of this problem.

This post is made mostly of excerpts from the SLFPA-W inspection reports. There are mounds of such reports going back two years. They represent the best record of construction of the west bank portion of the new hurricane defenses the Corps is throwing up around New Orleans and environs.

One other thing: they are loaded with pictures.

We'll start at the center of concern right now, project WBV-14c.2. That's the Corps' project code. "WBV" stands for "West Bank & Vicinity," the official name given to the string of levees, floodwalls, gates and other defenses running from St Charles Parish all the way over to Plaquemines Parish south of New Orleans.

WBV-14c.2 is a levee project that goes for about 3.5 miles from the New Westwego Pump Station to a point about 1 mile east of the Westminster Pump Station, all in Jefferson Parish. The Corps is raising the levee height most of that length, and also installing sheet pile in some spots. Here's where it's located on the Corps' April 15, 2011 construction progress map of the entire hurricane protection system:


Raising the levee mostly involves putting dirt down. It's more complicated than that, but that's the basic idea. The dirt is drawn from borrow pits all over the metro area. These pits were supposed to put out quality material, but the one feeding this project obviously has not. WBV-14c.2 has had serious debris problems with the dirt pretty much right from the start, yet the Corps and their prime contractor - Phylway - just continue to put the bad dirt down.

Before I go any further, I should mention that this isn't exactly breaking news. The WBV-14c.2 inspection reports are online for anyone to read. Also, this particular project's debris problems have already been called out publicly by SLFPA-W's regional director, Giuseppe Miserendino. At the April 15, 2011 Tulane Engineer's Forum, attended by over 600 engineers and other professionals, Mr. Miserendino gave a presentation on the hurricane protection efforts occurring on the West Bank, and what the SLFPA-W views as successes and worries (h/t to Clay at Nola-dishu).

In a section of the presentation titled, "What's Not Working?" he devoted two slides to project WBV-14c.2, including explicit references to the debris problem:



So all this post does is fill in the details...

The first hints of problems came in the October 15, 2010 report:

"Issue: Unsuitable material needs to be picked up."

Then again in the October 29, 2010 report:

"Issue: Unsuitable material found on protected side of levee."

And once again about a week later, in the November 9, 2010 report:

"Issue: Unsuitable material found on protected side of levee."

These were small pieces of debris that hardly represented a systemic problem at the time. But it kept cropping up in every report - November 22, 2010; November 30, 2010.

Then in the December 13, 2010 report, the SLFPA-W inspectors noticed the debris coming in the dump trucks as they were pouring the dirt:

"Issue: 12/10/2010 - During site visit SLFPA-W representatives watched several tandem trucks unloading along approximate station 88+50 to 78+50 and noticed debris in one of the trucks. Inspected area as dozer was spreading material and found some debris."

Notice these are the pictures in one of Mr. Miserendino's slides from the April 15, 2011 TEF presentation:


Moving on, the inspectors also found debris in an area where dirt was being processed before getting poured:

"Issue: Debris found in processing area. Unsuitables need to be picked out of material."

The report notes that this area was later cleaned of most debris. But there was also unsuitable material all over the site:

"Issue: Unsuitable material found on protected side and flood side of levee from Sta.120+00 to Sta.182+15."

Yes, that's a tire. Sigh.

This sort of thing continued on through the December 28, 2010 report and the January 10, 2011 report.

Things finally broke with an inspection conducted January 19, 2011. We read in the January 21, 2011 report:

"Arrived on site at 7:12am and met with USACE representative (Audie Watson) discussed concerns on project.
- Debris on west end of project coming from the River Birch pit.
- Debris on east end of project from degrading the existing levee.
- USACE representative has previously marked debris with flags and paint.

SLFPA-W and USACE representatives agreed that debris localized in two areas along the flood side. First on the east [sic, should be "west"] end of the project between approximate C/L stations 0+44 and 56+75. The second area is along the degrading operation between approximate C/L stations 119+02 and 129+02 (1000 ft.).

Debris concerns are a high priority ranking. SLFPA-W request a detailed work plan on removal of existing debris within 48 hours or prior to additional lifts. Also, please provide a method to assure that future lifts are constructed without unsuitable materials."

That last paragraph is the line in the sand for the SLFPA-W. To put an exclamation mark on that, the report is filled with 14 pages of debris pictures. Unfortunately, after that, things seem to get worse.

About a week later, some extraordinary work took place. On January 27th, an exercise to determine how much debris existed within the existing levee was conducted. This levee had previously been worked on by the Corps after Katrina, and even further back than that as well. The exercise was to find out how much of the existing levee was debris filled. This was important because some of the dirt scraped, or "degraded," from this existing levee had quite a bit of debris in it. This degraded material was being used as fill on the new levee elsewhere on the project, per the specifications and drawings. For clarification, here's the what the project drawings call for in the area of concern:


In the January 27th exercise, trenches were dug from the top of the existing levee back down the protected side to the toe of the levee. The flood side wasn't excavated. They dug trenches every 500 feet along the levee until they stopped seeing debris. This diagram from the internal Corps report on this activity shows the locations of some of the trenches:


Here's the overview of the January 27th trenching from the January 31, 2011 SLFPA-W report:

"Excavating between stations 180+00 and 125+00. Debris was heavily localized between stations 180+00 and 170+00. After station 170+00 debris in the previously topped material greatly decrease at station 165+00 and debris in the core decreased at station 160+00. At station 125+00 some debris was found through out excavation but not above a manageable amount. All investigation excavation was performed along the protected side slope through the crown section of the existing levee. The flood side slope extending into the new levee slope was not investigated."

The resolution was:

"- Material unsuitable for fill between stations 180+00 and 170+00
- Contractor to remove unsuitable material placed that was degraded between stations 180+00 and 170+00. Material will be stock piled and later removed from site (contractor will wait on written directions from USACE)."

That is, 1000 feet (found by subtracting 170+00 from 180+00) of the existing levee, which had been sitting there for years, had been discovered to be too full of debris. The contractor had used the material degraded from that existing levee as fill on the new job, per the specifications. They would now have to scrape it back off and send it off site.

Curiously, the Corps' internal report on the January 27th activities doesn't recommend removing the debris-laden material at all. In fact, it's silent on what to do with that material. It only addresses what to do with the area of the degraded existing levee. They say to put 2 feet of fresh clay over the area. Frankly, the Corps report and the SLFPA-W report are so different, it's tough to reconcile them. For your convenience, here's the Corps report. The original report was in .docx format, so I resaved in .doc format. I didn't change it in any other way. The original is here.

The SLFPA-W January 27th report is filled with over 100 pictures of debris like this:


However, all this activity was nowhere near the end of the problems. The next report was about nothing but debris. This was a large contrast to the early reports from fall, 2010, when debris was confined to a single comment very far down in the text. Reading between the lines, one can sense the deep concern that this job was going off the rails. From the March 21, 2011 report, which documented a March 17th inspection and included mention of a meeting between the Corps and Phylway about "debris being hauled in fill material:"

"USACE inspector stated that the majority of the debris was localized at both ends of the project. SLFPA-W focused reviewed on the flood side of project, debris was noted as stated by the USACE inspector.
- Area between stations 0+40 to 55+00 contractor was disking to dry out material.
- Review began at station 0+40, amount of debris began heavy and lessened toward station 55+00.
- Area between stations 65+00 to 107+00 (flood side) has a small amount of debris.
- East of pump station crew was noted picking up debris at approximate station 138+00 and working toward station 124+00 (flood side).
SLFPA-W reviewed several areas between stations 124+00 to 175+00 and found wood, brick and concrete debris (reviewed focused on flood side)"

Reading that carefully, I get the impression that - while they agreed during the January 27th exercise the debris was localized on both ends - the SLFPA-W didn't buy it now, two months later. Indeed, the SLFPA-W found debris across the entire project this day. Here's a sample:


There's 75 more pictures like this, just in this report.

The resolution to this was:

"At the meeting between USACE and Phylway handling of debris issue was outlined as follows;
- Fill would not be compacted or tested before USACE representative inspected for debris.
- Contractor to step up debris removal at pit and on site, no testing to be performed until USACE inspector is confident that the contractor's efforts to remove debris are substantial."

Lord knows how much bad material had been placed between the previous inspection in late January and this one nearly two months later. Also, reading report after report filled with promises to address the debris problem while not seeing any progress leads one to believe Phylway and the Corps were perhaps not serious.

Perhaps with that thought in mind, the next inspection took place just a week later, kicking off weekly reinspections of WBV-14c.2 by SLFPA-W. The March 25th report documents a pair of visits on March 22nd and March 24th by SLFPA-W. It is exclusively about debris, and explicitly mentions the concerns about exceeding the 1% organic criterion in the soil specifications:

"Site visit was concentrated on the floodside of the levee from Sta.0+40 to Sta.55+00. Material is being added to the floodside of the levee from Sta.55+00 to Sta.45+00. The material being dumped in this area is filled with debris and will need to be picked out. An abundant amount of small wood chips (within spec size) are being found in the levee section from Sta.55+00 to Sta.25+00. SLFPAW representatives are concerned that the amount of wood chips in this area would be beyond the 1% of organics allowed in the material, but would like USACE Geotech to determine this."

"The material being dumped in this area is filled with debris" - so much for the contractor's promise to step up removal at the pit:


As for the small wood chips:


This would get worse. There would be no resolution of this particular problem for at least two weeks, during which presumably similar soil was being dumped.

The SLFPAW concerns were enough to convince the Corps to send someone to the borrow pit putting out this material:

"SLFPAW representatives met with Wade Wright (USACE) at the WBV-14e.2 site visit later that day. He informed SLFPAW that he will be making a visit to the WBV-14c.2 jobsite today (3/24/2011), and also to the River Birch pit to investigate the debris problem."

And yes, the "River Birch" mentioned here is the same River Birch landfill that is the center of a wide-ranging federal investigation into corruption within Jefferson Parish politics. That investigation has already led to the resignation of three top officials within Jefferson Parish, including the parish president, Aaron Broussard. The Corps is approved to get borrow material from four River Birch-owned pits on the west bank of Jefferson Parish. We don't know which one is supplying the dirt for this proect. What's odd is that earlier SLFPA-W inspections for this project say that the borrow was coming from the Willow Bend borrow pit in Donaldsonville, LA. It is unclear whether WBV-14c.2's contractor Phylway has switched borrow sources for this project, or if they are using two pits, or what.

Anyhow, here's how the SLFPA-W representatives stated their conclusions:

"SLFPAW representatives expressed their concerns about the debris in the material in the levee, and the material that is being hauled in to Jeremy George (USACE). The current lift is being placed on top of a questionable amount of debris before the problem has been resolved; this may raise concerns of acceptability."

That's a gentle way of the SLFPA-W saying, "We think this levee might be crap, and we probably won't accept it from the Corps until it's fixed." This is quite serious.

Just a few days later, SLFPA-W was out to WBV-14c.2 again. The April 1, 2011 report documents a visit on March 28, 2011. They noted there was still debris coming into the site:

"- Site visit was concentrated on the floodside of the levee from Sta 0+40 to Sta.55+00. Debris in the material that is being hauled in from Sta 55+00 to Sta.25+00. Area has not been picked yet but USACE assured SLFPAW representatives that this area would be picked before the next levee lift was placed."

And despite those reassurances, SLFPA-W reiterated their concerns, not just to the local Corps guy, but also to a Major who was on site (a sign of increased attention by the Corps):

"SLFPAW representatives expressed their concerns about the debris in the material in the levee, and the material that is being hauled in to Jeremy George (USACE) and Maj. Kim Giles (USACE). These areas are being picked before the next levee lifts are being placed."

Those pickers would have had to dig pretty hard, as the pictures from this report show the debris had been pressed into the dirt by bulldozers already:


I imagine there's also similar stuff below the surface.

The frequent visits to WBV-14c.2 have continued, with three four more inspections since March 28th. The April 7th visit documented in the April 8th report shows things continue to deteriorate, including a rather alarming expansion of the wood chip problem mentioned in the March 25th report:

"Site visit began on the floodside of the levee between Sta.0+44 to Sta 55+00. The material that is being hauled in contains a considerable amount of debris and is being picked out. Pickers were at approx. Sta.15+00 during site visit. Levee lifts are being placed on top of an abundant amount of small wood chips (within spec size) in this area. Phylway reps informed SLFPAW that USACE stopped hauling of material to this area until the problem is further investigated. Currently, material is being hauled to the south east end of the job for levee lifts."

There's lots of pictures of the areas with the "abundant amount of small wood chips:"


Frankly, it looks like more wood than dirt in some spots there. Who knows what that area looks like below the surface, especially after this problem had been identified two weeks earlier?

The following week SLFPA-W was out at WBV-14c.2 twice - on April 12th and April 13th, as documented in the April 18th report. Things were still bad:

"Site visit began on the east end of the job (Sta.119+00 to Sta.182+00). A considerable amount of large debris, stones, and smaller wood chips was found on the levee section in this area. Contractor informed SLFPAW reps that the area had been sealed due to expecting rains earlier in the week. This area needs to be picked before more material is placed."

Here's why it needs to be picked:


There's over two dozen more pictures just like this in the report.

The area with all the wood chips remained shut down:

"Work remains on hold from Sta.0+44 to Sta.55+00 to investigate problem with wood chips and smaller debris."

And SLFPA-W continued to make their concerns known:

"All issues and concerns were discussed at the progress meeting held that day. Sta.0+44 to Sta.55+00 remains on hold to investigate problem with smaller wood chips. Concerns with larger debris on the east end of the job (Sta.119+00 to Sta.182+00) were brought up to USACE and Phylway representatives."

That brings us to the latest next to latest inspection report we have, the one from April 21, 2011. It describes an effort to characterize the extensive debris problem during an April 20th inspection:

"On 4/20/2011 a geotech evaluation of unsuitable material by volume was held. Samples were taken from 6 areas ranging from the floodside berm to the crown of the levee. (3 from Westwego levee, 3 from Westminister East/West Levee) Results of the test will be available after 1 and a half weeks."

Here's pictures of that sampling:


But other sections of the job continue as if there's no problem, despite ample evidence to the contrary:

"On the east end of the job (Sta.119+00 to Sta.182+00) work continues on enlarging and filling in the protected side berm in accordance with the contract modification to eliminate seepage problems. A considerable amount of large debris, stones, and smaller wood chips was found in this area. Pickers were at approx. Sta.120+00 during the site visit, picking the levee and the flood side berm."

And so on. There's 7 pages of pictures like these, just in this report.

Finally, the most recent inspection report shows the Corps just keeps putting debris-laden material down, as if there's no problems. From the April 29th report of the April 28th inspection:

"Overall review of site visit:
- West end of job (Sta.0+44 to Sta.55+00). Material has been hauled in to this area earlier in the week. A considerable amount of large debris, stones, and smaller wood chips was found in this area. Pickers were at approx Sta.45+00 during site visit.
- At approx Sta.55+00, the levee is being degraded to elevation +3.0' where geotextile will be installed.
- A considerable amount of debris is being found from Sta.68+00 to Sta.112+00 (north/south levee). Material is being added to the protected side berm in this area at approx. Sta.110+00, debris is also being found in this material.
- All sheet piling on the west side of Westminister Pump Station has been driven. Hpiles are currently being driven at monolith F.
- On the east end of the job (Sta.119+00 to Sta.182+00) debris is being found within the levee section and on the protected side berm.

So there's still debris pretty much through the entire job.

After months of what is appears to be very poor quality dirt getting poured on this levee, as well SLFPA-W raising loud alarms over the same period, it still doesn't appear the Corps is doing the smart thing. The smart thing would be to shut the job down completely until the problem is totally characterized, eventually ensuring the levee is being built properly. Instead, they are just pushing ahead so they can meet their vaunted June 1st deadline. Thank goodness for the SLFPA-W inspectors.

It should probably come as no surprise that this is not the only project having these problems. In the next part, we'll start to see how widespread it is.

17th St and Orleans Ave SWE = 0-1 feet

[Despite repeated attempts to obtain comment, the Corps has refused to offer any information about this article.]

SWE = Safe Water Elevation, the term of art the Corps of Engineers has applied indicating the maximum amount of water canal walls and levees can see before failure.

I have obtained the October, 2010 iterations of the 17th Street canal and Orleans Avenue canal Safe Water Elevation reports, including all their appendices:

17th Street: just the report and everything including appendices
Orleans Avenue: just the report and everything including appendices

And while I was stunned to find out that at least 7 sections of the London Avenue canal actually have Safe Water Elevations below the publicly disseminated 5 foot level, the findings for the 17th Street and Orleans Avenue canals are even more startling.

Since 2007, the Corps has claimed the SWE for 17th Street was 6 feet; they said it was 8 feet for Orleans Avenue. They have made operational decisions to close or not to close the lakefront gates during high water events based on these numbers, specifically during Gustav and Ike in 2008. The 17th Street gates were closed during both those events, and the Orleans Avenue gates were not. In fact, the Orleans Avenue gates have never been closed during a storm event.

As it turns out, the city may have just gotten lucky since 2005. Because these reports reveal there are sections along both canals which are so weak, they cannot even stand normal tidal motions throughout the year, never mind storm surge of 6 or 8 feet.

The reports provide handy charts indicating the overall SWE for each bank of each canal. First, the east bank of the Orleans Avenue canal:


And the west bank of Orleans:


Yes, there are sections along both banks with Safe Water Elevations of 1 foot (Actually, their SWE's are so low, they've just been assigned "1 foot" to keep things simple). Overall, there are 10 sections, or "reaches" with SWE's below the "official" 8 foot level.

Over at 17th Street, the picture's just as bad. First the east (Orleans Parish) bank:


And the west (Jefferson Parish) bank:


The reaches marked in red (12 of them) are those below the currently publicized SWE of 6 feet. As along the Orleans Avenue canal, there are also reaches with SWE's of 1 foot at 17th Street. The reaches in purple (7 of them) are the ones above 6 feet but below the target SWE of 8 feet, which is what the current remediation project is supposed to deliver. All 19 of these reaches should be included in the remediation project, theoretically (the Corps is running remediation projects along all three outfall canals currently, with the stated objective to raise the SWE to 8 feet along all stretches of all three canals).

So let's look at that. Here's the SWE's from the report overlaid on a map of the remediation project along the 17th Street canal (map taken from a March 31, 2011 Corps presentation included in the April monthly update from the Orleans Levee District, found at the website of the Southeast Louisiana Flood Protection Authority - East here):


The Corps is remediating nearly all the troubled stretches. I can't speak to the quality or propriety of their chosen method of remediation (mostly deep soil mixing, combined in some streches with increases in the height of the levee embankment), but I can speak to the location of that remediation.

18 of the 19 reaches of concern are included in the 17th Street remediation project. But one stretch on the east bank, Reach 30, is not included, despite its published safe water elevation of 7 feet, one foot below the target SWE of 8 feet. Reach 30 extends 736 feet south from the Vets bridge, not quite all the way to the I-10 bridges. The 7 foot SWE in this reach comes from inadequate "global stability," meaning Reach 30 is susceptable to sliding of the entire levee and floodwall horizontally at water levels greater than 7 feet. I've gone through the report and can't really find any caveats around the global stability analysis of Reach 30 to explain why it was left out of the remediation project. The Corps should tell the public what's going on.

Over at Orleans Avenue, reaches have also been left out of the remediation project for that canal:


You can see there are 8 (!) reaches with SWE's below 8 feet which are not part of the remediation efforts along the Orleans Avenue canal. This includes facing stretches of the canal just north of the Robert E Lee Blvd bridge which have SWE's so low, they cannot tolerate normal tides. These reaches - 10B and 20B - are bare levee and do not have floodwalls.

I'm no geotechnical engineer, but leaving stretches of canal bank unremediated when they have a Safe Water Elevation of 1 foot deserves an explanation. Never mind the other 6 stretches with sub-8 foot SWE's which are also left out of the project.

The latest on lakefront pump repairs

This entry was updated July 13, 2011 to bring it in line with revisions made elsewhere on this blog. These revisions came about due to a large pump repair document release in June and July, 2011 in response to FOIA requests. As such, the tense is an odd mix of tone, mixing voices from both the original date of composition on March 23, 2011 and the later revision date of July 13, 2011.

The last portion of story of the repairs to the rusty 60" hydraulic pumps at the three New Orleans lakefront gate complexes left off in June of 2010. The Corps continued to pull pumps since then. A brief overview of the rusty pump story can be found at this earlier post. One of the more detailed posts, "Worse than previously known," traces the first few task orders of the contract described below, which is the second of three repair contracts given to small business Healtheon, with pump rebuild shop Conhagen doing the actual repairs.

It took over a year of badgering the Corps of Engineers to release contract documents and reports on these repairs, but they finally did so July 8, 2011. Thankfully, we didn't have to wait all that time, because there was plentiful evidence on the ground of what was happening.

In sum, there were six more pump pulls between June, 2010 and March, 2011 involving 7 pumps across all three lakefront sites. Two of those pumps - E1 and E5 on the east side of the London Avenue canal site - came out the week of March 23, 2011.

However, that still leaves about 18 pumps remaining totally untouched, with another 5 incompletely repaired in 2009 and requiring the full overhaul treatment. The lack of urgency continues unabated. Meanwhile, pumps continue to rust in the brackish waters of Lake Pontchartrain.

June 14, 2010
Healtheon contract W912P8-10-D-0054, Task Order #4, $186,793.42

The FPDS-NG description on this task order reads, "Removal, reconditioning, and reinstallation of Pumps at London Avenue Canal, 17th Street Canal Interim Closure Structures per scope of work dtd 09 June 2010."

The dollar amount of this task order indicated the repair of a single pump, but that appeared to be contradicted by the plain text of the description, which mentions two sites. Upon receipt of the contract documentation, the confusion was cleared up.

There was a spill of oil from 17th Street pump W1 on June 2, 2010. A Corps contracting officer contacted QRI that day and told them to clean it up. Remarkably, the Corps also reported the spill to the National Response Center, as required under federal law.

A task order covering the cleanup work - task order #9 - was drafted the next day. The text of the task reads:

"Clean up of hydraulic oil spill at Pump #12 at 17th Street Canal. The spill is estimated at 15 gallons and is within scope of a small spill on water. Verbal authorization to begin work was given on 6/2/2010 by a Contracting Officer."

The description of "Pump #12" in the QRI cleanup task order - which got propagated to FPDS-NG - is a typo. The normal nomenclature used by the Corps to label the pumps is [Number][Direction]. The number - "1" - was correct, but someone's finger slipped and typed "2" instead of "W" for the direction. We know this from subsequent documentation described below.

As with so many of the other pump repairs, the Corps was reactive rather than proactive, and only pulled this pump out after it sprung an oil leak. They issued task order #4 on the second Healtheon contract officially on June 15, 2010, though the enclosed scope of work in the task order is dated June 9, 2010.

The pump was pulled out sometime in the first half of June and sent to Conhagen's shop. Courtesy of the Conhagen repair report for the 17th Street pump W1 work, we have pictures of the pump in the shop.

The leak could have come from anywhere that was corroded. Maybe it was at this nearly unrecognizable corroded pipe fitting at the discharge of one of the oil coolers:


or the Rineer motor, or through the steel seal flanges:


or anywhere else, because the single pair of zinc anodes originally installed on the pump to prevent corrosion certainly hadn't been doing its job for a very long time:



Wherever the leak occurred, it doesn't really matter, because this pump got nearly the full rebuilding treatment. All the internal and external carbon steel pipes, including the oil coolers, were changed out for stainless steel. All the seals, bearings, and associated hardware were completely replaced. The pump housing was hydroblasted, and any chunks missing from corrosion were weld repaired. Effectively, the pump was completely rebuilt, except perhaps for the Rineer hydraulic motor, which isn't mentioned as getting replaced.

Based on the dates on the Conhagen report, this pump probably went back in the water around July 10, 2010. A reconciling of of the rebuild costs was recorded as modification 1 to task order #4, issued on April 6, 2011. $2,619.84 was subtracted, bringing the final cost of the 17th Street pump W1 rebuild to $184,173.58.

July 14, 2010
Healtheon contract W912P8-10-D-0054, Task Order #5, $153,560.66

The FPDS-NG description of this task order reads, "Remove, recondition & reinstall Pump #1W at London Ave. Canal."

It is unclear why pump W1 at the London Avenue site was pulled. There's no record of an oil spill preceding its removal, but that doesn't necessarily mean anything, since the Corps is absolutley awful about following the federal law requiring immediate reporting of all oil spills. It is unusual for the Corps to pull single pumps instead of pairs, as noted in the scope of work below, so it seems likely there was an unreported spill.

Anyhow, this pump got the expanded rebuild treatment that the previous few pumps got, including new bearings and seals, except for one aspect: like 17th Street pump W1, apparently its Rineer motor was not replaced. Here's the scope of work from the task order:

"The Scope of Work for this task order is for a single pump in lieu of a pair of pumps as noted in Schedule B of the contract.
The Scope of Work for this task order shall also include the following additional work:

Repair Mechanical Seals at subcontractor shop 1 Each
Remove Pump Shaft Bearings
Replace Radial bearings Impeller End (1 Each)
Replace Radial Bearings Motor End (1 Each)
Replace Thrust Bearings (1 Set)
Replace bearing Spacers (1 Each)
Inspect Shaft And Housing
Mount New Bearings
Clean All Parts For Assembly
Reinstall Pump Shafts"

It's unclear why the Corps wouldn't have the Rineer motor replaced on this unit and 17th Street pump W1.

The pictures in the Conhagen repair report for London Avenue pump W1 don't show as much as in other reports. But they do show one part that got pretty beat up:



The seal flange is a piece of metal meant to help keep oil from leaking out and water from leaking in to the enclosed section of the pump which contains the shaft bearings. Looking at these pictures, it seems unlikely it was performing its task. So, as with all the other pumps, a new seal flange was remanufactured by Conhagen.

Based on the dates on the Conhagen report, this pump probably went back in the water around August 15, 2010.

October 8, 2010
Healtheon contract W912P8-10-D-0054, Task Order #6, $169,746.50

The task order description in FPDS-NG reads, "Removal, Reconditioning, and Reinstallation of Pump 4W at Orleans Avenue Canal, Interim Closure Structure."

Again, like the last pump, we don't know exactly why this pump was pulled out. There's no spill report, and it remains unusual for the Corps to pull a single pump rather than a pair (a fact again noted within the scope of work in the task order for repair of this pump). Like pump W1 at the London Avenue site, this is probably also a case of an unreported oil spill.

Steve Beatty of The Lens got a number of photos of this pump removal on the day it happened, October 12, 2010. This is my personal favorite:


This is a Corps employee making sure to document who's taking pictures. That's pretty chilling. I wonder what a FOIA request for the picture this guy took - along with all other pictures taken by Corps employees and contractors on the pump repair job - would turn up?

Here's some more shots. The pump was already pulled out of the water when Mr. Beatty arrived. However, there was plenty of other work caught on camera

Here the elbow from atop the pump is replaced above the now empty pump slot:



The inspection hatch is what the Corps opens up to see if there's oil leaking, which is a primary criterion for pump removal. Whether such a spill gets properly reported or not is a separate question (answer: mostly "not").

Here's a nice clear shot of the elbow without the pump while the pump was in Conhagen's shop:


I believe pump elbows are now being replaced when pumps are removed for only one reason: to camoflauge pump removals from the prying eyes of the public, especially after I noted the easy way to determine if pumps were out. That is, the Corps used to leave the elbows on the levee next to the gates:



Now they put the elbows back after they pull pumps. It is money being spent for no good reason other than effectively stifling public information.

The next step after the elbow was replaced was to prepare to lift the pump from the deck...


[The pump is still connected to its suction bell and vortex suppressor here. They would be transported to Conhagen's shop separately.]

...And place it on temporary braces for transport:


Then it is lifted to a flatbed truck from the prime repair contractor, Conhagen:




The rusting and buildup of marine growth is obvious in these photos.

The scope of work for this pump, like the prior two, included the extended rebuild package (new bearings and seals, in addition to new stainless piping and stainless oil coolers and weld repairs). And also like the prior two, seemingly did not include work on the Rineer motor. Here's the scope of work from the Orleans Ave pump W4 rebuild task order:

"1. Line Item 1001CC (Revised - See Note): Removal, reconditioning and reinstallation of pump #4W at Orleans Ave. Canal Interim Closure Structure according to the Scope of Work in Section C-1-Summary of Work in the subject contract.

Note: This scope is for a single pump in lieu of the scope for a quantity of two pumps in line item 1001CC, schedule B of the contract. The scope of work for the single pump shall be revised to include the following additional work:

Repair Mechanical Seals at subcontractor shop 1 Each
Remove Pump Shaft Bearings
Replace Radial bearings Impeller End (1 Each)
Replace Radial Bearings Motor End (1 Each)
Replace Thrust Bearings (1 Set)
Replace bearing Spacers (1 Each)
Inspect Shaft and Housing
Mount New Bearings
Clean All Parts For Assembly
Reinstall Pump Shafts"

There's no mention of the Rineer motor, which is strange, especially in light of modification #2 to task order 3 on this contract, which called for the rebuilding of 12 Rineer motors, 8 which were to be kept as spares. That modificiation was issued in July, 2010, and was scheduled to be completed in August, two months before this pump was pulled out.

Anyhow, the Conhagen report of the rebuilding of Orleans Ave. pump W4 includes pictures of its particularly rusty innards. Here's the Rineer motor and all the pipes leading into and out of it, with a considerable coating of rust:


And here is the suction bell of the pump (the suction bell is the flared section at the bottom of the pump where the water enters):


Note the substantial chunks missing due to corrosion.

After rebuilding, this pump was placed back in service November 20, 2010.

Upon review of the contractors' invoices, this task order was closed out with modification no. 1 on June 15, 2011. $4,150.46 was subtracted, bringing the final cost for the Orleans Avenue pump W4 rebuild to $165,596.04.

October 15, 2010
Solicitation W912P8-10-R-0066 issued

The Corps opened the pump repairs up for public bid on this day, with a solicitation for "Removal, Reconditioning, and Reinsitallation" of the pumps at all three canal sites. The solicitation has since disappeared from the FedBizOpps site, but it and all five of its amendments can be found here.

The solicitation follows the same general scope as the repairs to date. That is, pumps are to be pulled, they're to be hosed down, all their carbon steel guts are to be replaced with stainless steel, zinc anodes are to be added, and any other bits and pieces that are messed up are to be repaired before they're put back in.

The difference on this solicitation from the second Healtheon contract, issued in March, 2010, is that the Rineer hydraulic motors at the heart of each pump unit are paid more attention. In the scope of work, this paragraph has been added:

"Remove the Rineer hydraulic motors from the pump housing and transport to a hydraulic facility approved by the [Contracting Officer's Representative (COR)]. Disassemble, clean and inspect. Provide written report of conditions found to the COR. Provide a written recommendation for any necessary repairs to be performed under the “Reimbursable Expenses” contract line item."

This likely ties in to the purchase of 12 Rineer motors in September from a Rineer distributor in San Antonio. There's much more detail about those Rineers in an update I embedded in this earlier post. More than likely, the Rineer purchases were in anticipation of replacement in case repairs didn't work out. That's probably wise, considering the Corps had already replaced a bunch earlier in the repair effort (also noted in that earlier post).

This solicitation led a later contract, number W912P8-11-D-0015 issued February 7, 2011, again to Healtheon. Details on that contract can be found below in the section titled "March 22, 2011."

December 9, 2010
Healtheon contract W912P8-10-D-0054, Task Order #7, $282,709.39

The FPDS-NG description reads "ITEM 1001BC - Pumps 2E and 6E - Remove, Recondition, and Reinstall Pumps at London Avenue Canal."

Unlike the earlier 2010 removals and repairs of London Avenue pump W1 and Orleans Avenue pump W4, this pair of pump pulls was accompanied by sudden uptick in oil spill cleanup operations. There was a flurry of task orders and modifications to QRI in December, 2010 in conjunction with this Healtheon task order. The first was issuance of a task order for a large oil spill on December, 8, 2010. It was task order #10 on the QRI oil spill response contract, with a value of $944.06. The description within the task order reads:

"Clean up of hydraulic oil spill at London Avenue Canal. The spill was from pump #2E and #6E, spread from pump booms over within the south and north booms of the canal due to the high water level. The spill is estimated at 100 gallons."

"Verbal authorization was given by the Contracting Officer on 12/08/2010 at 1:50 p.m."

The jumbled verbiage ("spread from pump booms over within the south and north booms of the canal due to the high water level") refers to the fact that the oil spread from the primary containment which immediately surrounds the pumps, and was out in the canal, between the booms to the south and the north of the site.

A few days later, the Corps added more funds:

December 15, 2010
QRI contract W912P8-09-D-0022, Task Order #10, modification 1, added $3,439.64

The FPDS-NG description reads, "Mod to Adjust Funds." The description on the task order is more detailed:

"The work under this task order w as completed on 8 December 2010. Actual totals were recieved and funding for this task order is increased by $3,439.64. Total task order amount is now $4,383.70. Task order completion date has been changed to 14 December 2010 so that the modification could be processed."

This makes sense. If the spill was 100 gallons, it would cost a lot more than $944 to clean it up, based on previous QRI task orders for spills of similar magnitude.

A couple of days later, they added even more funds, seemingly in response to a second spill:

December 21, 2010
QRI contract W912P8-09-D-0022, Task Order #10, modification 2, added $3,979.89

The FPDS-NG description reads, "Mod for an addition oil spill of approximately 25 gallons."

The task order reads,

"The above numbered task order for a hydraulic oil spill at London Avenue Canal is hereby modified to provide for additional oil clean up at pump 2E. The oil spill is approximately 25 gallons and will use contract CLIN 1002AA. The modification will increase funds in the amount of $3,979.89. Total task order amount is now $8,363.59. The completion date has been extended until 21 December 2010 as a result of this modification."

The effective date of the task order is December 17th. I'm not sure what to make of this mention of a second spill when I would think the pump would be in the shop.

Making it difficult to suss out is the fact none of this oil spill activity appears at the website of the National Response Center. These apparent multiple Clean Water Act violations are of a pattern with the Corps' activity around these pump repairs, as documented above and in Corps of Oil, Part 3.

The task order for the repair of these pumps was issued on December 9, 2010, the day after the oil spill. At first blush, this would appear to be typical of the Corps' reactive - rather than proactive - approach to maintenance. However, there's something hinky about the dates of work on these pumps.

The Conhagen report on the rebuild of London Ave. pumps E2 and E6 provides the oddity. Its cover page mentions that their work on these two pumps began November 29, 2010, over a week before the issuance of the task order for that work:


That's somewhat unusual for these types of task orders.

In addition, the date of completion on the task order is December, 24, 2010. That would theoretically give only 16 days for Conhagen to rebuild two pumps, when they've averaged 4 to 8 weeks on previous pairs of pumps. A December 24th completion date argues for an earlier start date, like the November 29th date shown on the front of the Conhagen report.

Putting two and two together indicates that the first 100 gallon spill occurred during the pull of these pumps, not before it. For some reason, that pull - along with the issuance of the task order to Healtheon for it - had been delayed from the last week of November until December 8th. And then, when Conhagen went to pull the pumps, they dumped 100 gallons of hydraulic oil in the canal. If National Response Center records are to be believed, the Corps didn't bother to report that spill, or another spill that occurred a few days later. If it is true that oil was spilled during the pumps being pulled (as we know it has in many, many other instances), it will be quite important, as I'll describe in a future entry.

But enough about the oil spills. As to the rebuild work on the pumps themselves, they got what had become - at that point - a standard scope of work for rebuilding. See the above entries on repairs to other pumps over the summer and fall of 2010 for details on that scope. The salient part is that neither the scope of work in task order #7, nor that in the Conhagen repair report, mentions rebuilding or repalcement of the Rineer motor. This makes five pumps in 2010 where the repair of the Rineer motor was not explicitly addressed, which is troubling, because the motor is the very heart of the pump unit. If all the other parts of the pumps were rusted to bits and required replacement, it's very likely the Rineers required it as well.

The Conhagen report on the rebuilding of London Avenue pumps E2 and E6 contains a number of pictures of the pumps. They did not look good when received at Conhagen's shop in Kenner, LA.

Here's the rusty Rineer hydraulic motor from pump E2, along with rusty pipes:


Pump E6's Rineer motor and attached pipes looked just as bad:


This is the bottom section of pump E2, which was removed for transport:


It's made up of three parts: a vortex suppressor, the suction bell, and the liner.

The vortex suppressors were added to the pumps at London and Orleans Avenues after installation. They are supposed to cut down on the formation of small whirlpools called vortexes (or vortices), brought on by what boils down to poor design of the pumps and their facilities. In essence, the pumps don't see enough water when the lake is at elevation 0. This is part of the reason the Corps only operates the pumps for extended periods when the canal elevation at the closure structures is above 2 feet.

The suction bells and liners are original parts of the pumps. They guide the incoming water to the impeller.

As one can see from the photo above, there was significant corrosion happening on this pump, as evidenced by the severe depletion of the single pair of zinc anodes, as well as the considerable marine growth.

That marine growth appears to have caused the paint on the liner to delaminate, leading to corrosion of the base carbon steel underneath:


The carbon steel piping of the oil coolers and pipe extensions on E2 never stood a chance in Lake Pontchartrain's brackish water:


Conhagen replaced the piping above, and repainted and weld repaired the suction bell, vortex suppressor, liner, and whatever else was missing metal due to rust and marine growth. They also put in new bearings, seals, gland plates, and other stuff to completely rebuild the pumps.

Since the start of the work appears to have been delayed, it was no surprise that modification 1 to the task order for this work extended the delivery date to January 10, 2011. Even so, the work took the usual six weeks for a pair of pumps, and they went back in on January 24, 2011.

However, the story was not complete with these pumps. You can read the unsurprising surprise conclusion in a subsequent post.

March 22, 2011
In the first spill report to the National Response Center since 17th Street pump W1 leaked oil on June 2, 2010 (despite the copious pump removal work in the interim documented above), the Corps reported this:

"CALLER IS REPORTING THAT DURING TWO PREVENTATIVE PUMP MAINTENANCE PULLS OVER THE PAST TWO DAYS THERE WAS A RELEASE OF ABOUT 1 CUP OF HYDRAULIC OIL INTO THE BOOM THAT WAS ALREADY IN PLACE. THE MATERIAL NEVER LEFT THE CONFINES OF THE CANAL STRUCTURE OR THE BOOM."

Regulations stipulate that these spills are supposed to be reported when they happen, not two days later. Also, the quantity released is unbelievable in the extreme, especially given the well documented history of tens of gallons per spill when pumps are pulled. In fact one only has to go back to the above section on the repairs last December to find spills (unreported of course) of "Over 50 to 100 Gallons" and "approximately 25 gallons."

The London Avenue pumps removed during this activity were E1 and E5, the last two hydraulic pumps on the east side of the site not to be repaired. They went back in sometime around the third or fourth week of April, as seen in this subsequent post.

This work is part of a third contract to Healtheon for pump repairs. On February 7, 2011, Healtheon was awarded contract W912P8-11-D-0015, in response to solicitation W912P8-10-R-0066 (original FBO page, complete solicitation), which was circulated in late 2010. It is a four year contract (a base year and three option years) which has a maximum value of $6.75 million.

That third option year is interesting, because that places the possible work for this contract in 2015. Yet the Corps has said in its recent announcement of an award for the permanent pump stations that the new stations are due to be finished in 2014. Could this be a hint that the schedule is actually slipping once again?

[Note: The schedule for the permanent pumps did slip, as a result of multiple bid protests after the contract award in April, 2011. Details are available at the subsequent post, "Permanent pumps slip to Katrina 10th anniversary."]

Anyhow, on March 14, 2011, the new Healtheon contract had task order #1 issued on it for a value of $598,101.68, calling for pulling of four pumps: E1 and E5 at London Avenue and W4 and W7 at 17th Street. We'll concentrate on the London Avenue pumps for now.

I have pictures of the empty slots where London's E1 and E5 usually reside. First, E1 (with the elbow back in, as noted above):


I know this picture's a little busy. You can tell that E1 is out because you can't see its piping extensions, like you can on E2:


Noteworthy is the yellow boom near the E1 slot. It's been left there by oil spill contractor QRI after the pump removal, and got tossed onto the supporting steel by the tides or bad weather:


Here's a picture showing E5 out of its slot:


And again, more yellow temporary boom placed by QRI when the pump was removed:


The rest of the work on London pumps E1 and E5, as well as other work under the the third Healtheon/Conhagen contract, is covered in subsequent posts, starting with "The 2011 pump rebuild scramble." There you can find links to more contract documents and Conhagen repair reports, as well as Conhagen pictures of the pumps pulled out under that new contract.

Summary
At the time of this post, details on many 17th Street pumps were unknown. All that was known for certain for certain was that pump W1 came out for repairs between June, 2010 and late March, 2011. So the 17th Street pump status looked like this:


while the Orleans and London statuses were:




Since E1 and E5 were still out at London Avenue, they were not counted as "repaired" in the tallies. See this subsequent post for confirmation of their replacement, and further details on their repair.

Karen Gadbois and Steve Beatty of The Lens contributed to this report.