Waive

My last post was about the severe pumping shortfall the West Bank is facing this summer due to schedule slippage at the Corps' West Closure Complex, or WCC. There will a public meeting devoted to the shortfall and other matters on March 16th. Details about the meeting are also at the previous post.

The shortfall comes about from 3 of the 11 WCC pumps not being ready on June 1. Supposedly, there's going to be testing of remaining 8 pumps this spring. The Corps claims the other 3 will be tested and will come on line before the height of hurricane season, generally acknowledged as August 15 through September 30.

So what will that testing consist of?

The complete specifications for the West Closure Complex tell us - specifically, Section 22 10 00.00 10, "Vertical Pumps, Axial-flow and Mixed-flow Impeller-type."

The first testing was comprehensive model testing of a scaled-down version of the pumps. This model test at the factory - not a test of the real pumps at the site - was to be used to verify the pumps' performance fell within the specifications. Four and a half pages of the specs are devoted to this testing. Supposedly, there's a complete report about the model testing out there somewhere; the requirements for it are called out in detail within the specs.

Then, with the real pumps installed in the station, they'll do two types of testing.

First, they'll run them "in the dry." This dry testing (that is, no water flowing) is pretty much to make sure everything spins right, that the bearings don't heat up too much, that instrumentation works okay, that the pump doesn't vibrate too much, etc. Each pump is supposed to be run for 4 hours or until the bearing temperatures stabilize.

And then, finally, comes the run test with water flowing. Here's the verbiage from the specs:

"Each pump unit shall be given a test with water under load, at or near normal operating conditions, for at least 4 hours or as directed by the Contracting Officer. The test shall be conducted by the Contractor and will be witnessed by the Government. All supplies and equipment required to conduct the test shall be provided by the Contractor. During the test the operation of the pumps will be observed and measurements of sound, vibration and bearing temperatures shall be taken and recorded. Without additional costs to the Government, the Contractor shall make all changes and correct any errors for which the Contractor is responsible. The Contracting Officer may waive or postpone the test if sufficient water is not available. Appropriate changes will then be made to the contract.

Vibration and bearing temperature records shall be kept by the Contractor and turned over to the Contracting Officer at the conclusion of the contract."

It's a remarkably short passage, especially compared to the densely detailed 4.5 pages describing the model test. There's no real requirements for the wet testing, or even requirements to develop requirements. The pumps just have to be run and stuff is measured. What range those measurements should fall within, the format of the reporting, what standards should be referenced, and a myriad of other questions are simply left unmentioned. Clearly, these field tests will be little more than show; otherwise, there'd be a lot more than two paragraphs about them in the specs.

I've highlighted two relevant parts. First,

"Each pump unit shall be given a test with water under load, at or near normal operating conditions, for at least 4 hours or as directed by the Contracting Officer."

"Near" normal operating conditions is not defined. That's not surprising, since flow measurement is not mentioned anywhere - except for future press releases from the Corps and the manufacturer, where sufficiently big flowrate numbers can be made up without fear of contradiction.

But more importantly, 4 hours is it for the field test duration. Sound familiar? This is the same thing that happened with the hydraulic pumps at the outfall canals on the east bank in 2006 and 2007, except that testing of those was knocked down to 3 hours at first, and then finally to 2 hours. And some of them could not even do that.

It's a good thing pumps don't ever, ever have to run longer than 4 hours during a hurricane around New Orleans.

Next, though, is the scarier part:

"The Contracting Officer may waive or postpone the test if sufficient water is not available."

The thought that these pumps could simply not have any water put through them before the contractor is paid is in-freaking-conceivable. Yet there it is, smack in the middle of the specs.

So where does such an idea come from?

The Corps has its engineering contractors use generic specifications for its projects, which are then modified to suit the particular project. The specifications are officially known as the "Unified Facilities Guide Specifications," or UFGS, and are made available by the National Institute of Building Sciences through their "Whole Building Design Guide" website. Other branches of the military use them too.

From the WBDG page, we can look at the unmodified version of the spec I quoted from, Section 22 10 00.00 10 [Note: the "10" at the end of the spec number indicates it is only used by the Corps]. In the generic paragraph on wet testing we find much the same verbiage as in the WCC version:

"Each pump unit shall be given a test under load, at or near normal operating conditions, for at least [_____] hours or as directed by the Contracting Officer; the test will be witnessed by the Government. Provide all supplies and equipment required to conduct the test. During the test the operation of the pumps will be observed and measurements of[ sound,] vibration and bearing temperatures shall be taken and recorded. Without additional costs to the Government, make all changes and correct any errors for which the Contractor is responsible. The Contracting Officer may waive or postpone the test if sufficient water is not available. Appropriate changes will then be made to the contract."

The stuff in brackets is where the engineer needs to make a decision and either fill in a blank, or choose to keep a particular word or phrase. In this case, one of those bracketed bits is the duration of the test.

Above that paragraph in the generic UFGS specification is this paragraph intended as guidance to the specifying engineer:

"NOTE: The longest period should be used if water for testing will be available. The estimated water available and the number of pumps requiring tests should be considered when specifying length of tests. Sound testing, if required would only establish a base line for future reference. Consult HI 9.1-9.9."

So, from the fact that the blank got filled in with a "4," the Corps (or more likely their consultants) may believe that water will only be available for 4 hours before it gets too low in the canals. I personally doubt that's the case if they run the tests one pump at a time, though who knows? The Corps releases everything with an eye dropper. Hopefully some reporter will remember to ask about the test duration this spring in between taking down ridiculous Corps quotes about how many swimming pools could be emptied really fast by the pumps.

However, the fact the sentence, "The Contracting Officer may waive or postpone the test if sufficient water is not available," made it from the generic version of the specification all the way to the final, as-purchased version is insane. I'm not sure there's any more critical test of the hurricane protection system on the West Bank than the field wet test of the pumps at the billion-dollar-plus WCC, yet the Corps just blithely wrote the ability of its own Contracting Officer to cancel the test at a whim into the contract. That sentence should read, "The test may be postponed by the Contracting Officer if sufficient water is not available, but it may not be waived."

Now we know 8 pumps will be tested this spring, with cameras pointing at them (though not likely for four hours) and invited press corralled off to the side. But what of the other 3 pumps? Will the Corps invite the press out for those tests? No. Will we ever hear about them? No. Will we know if they even took place, or if they were waived due to low water? Probably not. But this is meant to put the Corps on notice: people are watching.

West Bank facing pumping shortfalls in 2011

UPPEST-DATE - August 28, 2012 - All 11 pumps were run simultaneously on June 27, 2012, according to the latest West Closure Complex inspection report published by the SLFPA-W. All the WCC inspection reports are available here.

[UPPER-ER-DATE - February 23, 2011 - Public meeting in March about Water Control Plan announced]
Yesterday - February 22nd - the Corps announced a public meeting to discuss the WCC Water Control Plan. The press release is here.

Details:
Date: Wednesday, Mar. 16, 2011
Informational Open House 6 to 6:30 p.m.
Presentation and discussion 6:30 p.m.
Location: Visitation of Our Lady School
3520 Ames Blvd., Marrero, LA 70072

In conjunction with the meeting announcement, they put out a summary of the Water Control Plan. Remarkably, it mentions the pump shortfall detailed below. However, it mentions none of the consequences of that shortfall (*cough*local pump stations shut down*cough*), it leaves out a lot of essential detail, and doesn't even include the basics like a map of the WCC and the local pump stations. The much more useful complete draft plan is available here, and the far less-detailed summary is here.

The slides for this meeting are here. The slides for an unpublicized "stakeholder" meeting the day before are here. "Stakeholders" is Corps-code for all the behind-the-scenes folks that are not the public, e.g. contractors, bureaucrats, connected government officials, shipping industry bigwigs, businessmen with interests that intersect with the Corps, etc.

And, unsurprisingly, neither set of slides mentions the pumping shortfall.

[UPPERDATE - January 28, 2011 - with Times-Picayune coverage; see bottom of post]
[UPDATE - January 24, 2011 - with video; see bottom of post]

The Corps of Engineers is building a giant storm barrier/gate complex/pump station on the West Bank, across from Orleans Parish. It is designed to be the front line of hurricane defense for the vast majority of the West Bank. It is called the West Closure Complex, or WCC for short. Here's where it is:


The pump station part of the WCC is meant to drain the rainwater that will be pumped into the Harvey and Algiers canals during a tropical storm or hurricane, because the WCC's storm surge gates (along with gates at the opposite, northern ends of each canal) will be closed during such an event, turning the canals into big bathtubs. There are 9 local pump stations that will put rainwater into the two canals. They drain the following area, covering about 70 square miles of the West Bank:


The WCC pumps - there are 11 of them - are supposed to exceed the capacity of the local pumps putting rainwater into the canals. And if all 11 WCC pumps are working, they theoretically would do that.

However, that won't be the case come hurricane season 2011. That's because according to the Corps' own documents for operating the WCC, only 8 of those pumps will be functioning then.

Naturally, this creates a shortfall. According to those Corps documents, the walls surrounding the Algiers and Harvey canals would overtop in 17 hours with all 9 local pump stations going and only 8 pumps going at the WCC. Their solution is to tell the local pump operators to stop pumping and thus flood residences and businesses.

Not helping the situation is the fact that the communications and control systems for the WCC will not be finished before the 2011 hurricane season either, which will definitely hamper coordination during storms among the Corps (running the WCC), Jefferson Parish (which runs 5 of the local stations), Plaquemines Parish (which runs 2) and New Orleans' Sewerage & Water Board (which runs the other 2). It's not hard to imagine the clusterblank that will develop when three different parishes are told by the Corps to stop pumping rainwater out during a hurricane.

Here's some excerpts from those Corps documents, which together form the interim "Water Control Plan" for the WCC (dated December 14, 2010):

"The pumping capacity deficiency during the interim period could result in higher water levels within the Algiers Canal and lower Harvey Canal (below the Lapalco sector gate) during rainfall events that occur when the gates are closed. With 8 pumps in service, the WCC capacity of 14,000 cfs trails the peak capacity of the 9 interior pump stations by about 5,500 cfs. This 5,500 cfs differential, if it remains constant, would overtop the detention basin levees in about 17 hours. Rather than allow that to happen, in such an extreme event the interior pump station managers would be asked to scale back their pumping, which would result in flooding within the interior basins. Precautionary measures include making the interior pump station managers aware of the situation, and maintaining close communications during extreme rainfall events."
-from Interim Water Control Plan for WCC, main text, dated Dec. 14, 2010

They've got their fingers crossed that the WCC gates are open part of the time the heavy rain is falling and getting pumped into the canals:

"It is unlikely that a rainfall event with such intensity and areal coverage to require all 9 pump stations in the Algiers/Harvey Basin to run at full speed for many hours while a hurricane storm surge is threatening the area. As Plates A-24 and A-25 illustrate, the length of time that interior pumping rates exceed the Interim WCC pumping capacity is 12 hours for the 10-year rainfall event, and 16 hours for the 100-year event. It is most likely that some part of that 12 or 16 hours will fall outside the time when the gates have to be closed."
- from Standing Instructions, Appendix A to WCC Water Control Plan, dated Dec. 14, 2010

I'm glad we can hang our hat on "most likely."

Part of the problem arises from the plan itself, which calls for closing the WCC gates 24 hours ahead of excessive surge (+4.0 ft) or wind conditions (greater than 50 mph):

"This regulating plan, shutting the gates 24 hours in advance of a surge, creates the potential situation where the interior pumping is greater than the capacity of the WCC pump station. This is especially true during the Interim Period [the Interim Period is the months or years when the station only has 8 pumps instead of 11], when the pump station is operating at reduced capacity. This condition will not be problematic most of the time because the canals have sufficient storage to absorb the excess interior pumping. It is very important for the WCC project manager to pump the water levels as low as possible as soon as the gates are closed to maximize this storage potential."
- from Standing Instructions, Appendix A to WCC Water Control Plan, dated Dec. 14, 2010

This exposes a further vulnerabilty: there is no spare capacity built into the WCC pump station. It is designed for 19,305 cubic feet per second (cfs), which is the calculated outflowrate for a ten-year rain storm at the WCC location (please note: a ten year rainfall is completely different than a 100 year surge. For more details on what a "ten year" rainstorm is, please see this earlier post). But it is only when all 11 pumps are running that the WCC system achieves that rate. There are no spare pumps built into the station to allow for mechanical breakdowns or maintenance during hurricane season. The locally-owned-and-built pump stations were constructed with spare capacity, despite the extra cost. Because things break.

The Corps doesn't say how long it will be for the levees to be overtopped or for local pump stations to shut down if just one of their 8 pumps in operation this summer fails during a storm, but we can assume something substantially less than 17 hours - probably closer to 9 or 10 hours. They also don't say which neighborhoods get flooded in case local pump stations get shut down to avoid overtopping.

If this all sounds familiar, that's because it is. The east bank of Orleans Parish and Jefferson Parish went through this same rigamarole in 2006 and 2007 with the construction of the pump systems and gates at the three outfall canals. The Corps constantly made promises and set deadlines of increased pumping capacities, and constantly broke those promises and busted those deadlines. And of course, we know now that the pumps they installed would never have done the relatively minor job they were advertised for in those years. Even now, those outfall canal pumps' capacity appears to be far below their nameplates, according to the Corps' own SCADA data for those stations. But that's a matter for future posts.

This WCC delay appears to be somewhat recent, or perhaps it was known all along. And it's happening despite the full force of Pentagon contracting being thrown at it. Here's excerpts from the special website about the WCC pumps set up by their manufacturer, Fairbanks-Morse:

"DO-N7 Rating

The West Closure Complex work comes with a DO-N7 Rating. This rating is part of the Defense Priorities and Allocations System (DPAS). A DO-N7 rating applies to critical infrastructure protection and restoration. A DO-N7 requires congressional approval. With this rating – and the priority that the pump station be online by June of 2011 -- it was necessary to ensure that all vendors selected to work on the project agreed to (by law, with acceptance of any contractual agreements) to commit to doing whatever it takes to make forward progress.

Accelerated Schedule

In the middle of the project we were asked to accelerate the schedule. Fairbanks worked with its vendors to establish a new schedule in order to accommodate contractor needs. However, even though the schedule was accelerated, our end date never changed: To have flood control by June 2011.

So despite all that government prioritization, having flood control by June 2011 now appears to be impossible.

So when will the other 3 pumps be ready? The Water Control Plan only says, "Installation and testing of the three remaining pumps will continue during the hurricane season." If you have a problem with that, there's a bunch of people's names and phone numbers on pages 28 and 30 of this document.

[UPDATE - January 24, 2011]

The Water Control Plan for the West Closure Complex, found linked and explained above, is scheduled to be was discussed by the West Bank Flood Protection Authority (official name, Southeast Louisiana Flood Protection Authority - West) at 3 PM, Monday, January 24 at their building in the 7000 block of River Road in Marrero. The agenda for the meeting is here. The Water Control Plan is item number 8. The meeting minutes are here. They reveal nothing of the discussion, or if there even was one.

In addition, Fox 8's John Snell got this out there during their January 21, 2010 broadcast:


Despite the Corps' own document saying exactly what will happen, the Corps spokesperson (Rene Poche) pooh-poohed any concern: "That's a lot of speculation there, and I don't want to really venture into a speculative-type question as to how it may or may not work."

That's reassuring. And I'm sure the hundreds of thousands, if not millions of dollars spent on hydraulic modeling the system down to the gnat's ass mean absolutely nothing too.

Let me put that another way... Poche's ridiculous, patronizing hand waving of his own agency's written concerns in a governing operational document is the exact opposite of how risk communication is supposed to work among adults.]

[UPPERDATE - January 28, 2011]

The Times-Picayune attended the West Bank authority meeting on January 24th and published an article about it four days later.

Most of the article deals with problems of cooling coils on the pumps. They will be mounted in the discharges of the pumps, getting pummelled the entire time the pumps are on. The authority is understandably skeptical of the Corps' assurance everything will be fine. That assurance seems to be based solely on a Corps static analysis of the coils and a promise to monitor stuff after the system is started up, which is, pardon the pun, very shaky indeed when talking about vibrations. The Board is demanding a dynamic analysis, which should have been done in the first place.

They made the same promises about the east bank hydraulic pumps in 2006, which displayed vibration problems among the cornucopia of flaws in their spring 2006 factory testing. When the vibration problems - among many others - manifested themselves in a huge way (after zero previous investigation) nearly immediately after installation, all those pumps had to get yanked out and didn't go back in the water for nine months. They still have numerous design flaws in them, including vibrations, as stated by Corps engineer Ray Newman in January, 2008 to an interviewer from PentagonTV (click here for Molly Peterson's KPCC "Pumps Under Pressure" special report. Scroll to the right in the timeline until you come to January, 2008 and click on note, "Engineer to PentagonTV: 'Pumps vibrate'").

So based on the east bank pumps experience, why should the leopard change its spots this time around?

Anyhow, the article also mentioned the pumping shortfall as well, finally giving a vague deadline:

"Matt McBride, a mechanical engineer and blogger, said the fact that just eight of the 11 pumps are expected to be online by the start of the hurricane season June 1 leaves a large portion of the West Bank more vulnerable to flooding.

He noted that the corps' interim plan for operating the West Closure Complex indicates the floodwalls along the two canals would overtop in 17 hours if the floodgate were closed during a 10-year rainstorm.

Corps officials said that the odds of such an event occurring early in the hurricane season are extremely low and that all 11 pumps are expected to be online before the peak of hurricane season.

Levee authority commissioners said they are satisfied with that assurance, but McBride said he is not.

'The corps provided similar assurances for all of 2006 and most of 2007 for the pumps at the east bank outfall canals, and almost all proved exaggerated and ultimately untrue," he said in a written statement. "So until all 11 custom designed and fabricated pumps have been fully tested with independent witnesses, ... it is best to withhold judgement on any corps predictions of the performance of their equipment.'"

Here's the full quote I sent the T-P:

"The Corps provided similar assurances for all of 2006 and most of 2007 for the pumps at the east bank outfall canals, and almost all proved exagerated and ultimately untrue. So until until all eleven custom designed and fabricated pumps have been fully tested with independent witnesses, and the complete results of those tests shared with the public, it is best to withhold judgement on any Corps predictions of the performance of their equipment."

It's unclear why they took out "and the complete results of those tests shared with the public," but I'm just glad they covered the problem at all.

And as far as "all 11 pumps are expected to be online before the peak of hurricane season," I have two things to say:

1) What exactly is the cause for the delay, and why so reluctant to tell the public?
After all, they're going to have to explain it come this April or May, when they turn the pumps on in front of the cameras and there's only 8 waterfalls coming out of the station where there should be 11. So why not just tell us now what is going on? Last I checked, coyness is not an element of risk communication.

2) Once again the goalposts start moving again. Plus ca change...

Imperative

After reading about all the problems with the gates, it's fair to say they're not the most robust systems possible.

But in addition to all the technical pieces of the gates, there's other parts: the Corps' procedures and the supervisors and operating crews charged with following them. And here too, the possibility of risk is introduced.

The procedure for lowering the gates

In addition to preparing the sites before storm season, including the annual demonstration for the press in late May, the operators and supervisors are responsible for a number of tasks immediately in advance of a storm. They have to:

a) Remove the dogging pins which are supporting the gates in their stored position
b) Lower the gates into the trench via the hydraulic winches
c) Attempt to reinsert the dogging pins to determine if the gates are properly seated in the trench
d) If the dogging pins won't go in, the diving contractor Independent Divers (aka H.J. Merrihue) is supposed to remove the obstructions from the trench
e) Once the obstructions are removed, they then must lower the gates and try again to insert the dogging pins to secure the gates.
f) If the obstructions can't be removed, large sandbags are to be dropped in front of the gates (also called needles) that are not completely seated.

[Pictures and more details on the pins are available at my earlier entry, "The Pins."]

It is the insertion of the dogging pins that lets the operators and supervisors know the needles are properly placed to resist storm surge. The Corps Operations & Maintenance Manual for the gates says,

"If the dogging pins can be inserted when the gate is down, this will confirm that the notch in the sill is free from obstructions."

So if the dogging pins are not inserted completely, that would indicate an obstruction in the trench (or "notch" in Corps-ese) at that gate section. That would mean the gates are not contacting the inner surface of the trench fully - an unacceptable condition, as emphasized in the Corps' Manual :

"Because the notches in the sill structure provide the reaction for the bottom of the gates it is imperative that these notches be free of all foreign material and obstructions."

That is, the structures are designed to have surge force distributed in a certain way between the trench and the structure:



If the bottom of the gate is not in contact with the back of the trench, almost all the surge force is then going through the structure, with a large chunk falling against the unreinforced guide beams.

These forces from storm surge are huge. If they weren't, the sill under the trench at the London and Orleans sites wouldn't have sheet piling extending down 48 feet below the bottom of the canal and the structure wouldn't have pilings going down 100 feet below the canal bottom.

In sum, if the dogging pins are not completely inserted and locked, there's a problem in the trenches that has not been solved. That means the gates are not fully seated, and the city is placed at risk.

The procedure in practice - September 12, 2009

So how has this procedure worked during real events?

Gates have been closed for storm events on four occasions since the structures began construciton in 2006:

1) Hurricane Gustav in September, 2008 - 17th Street and London Avenue gates were closed
2) Hurricane Ike in September, 2008 - 17th Street and London Avenue gates were closed
3) September, 2009 - London Avenue gates were closed
4) Hurricane/Tropical Storm Ida in November, 2009 - London Avenue gates were closed

A fifth event, on December 12-13, 2009, should have triggered gate closures at London and 17th Street, but did not, likely due to a systemwide SCADA failure four days earlier during a very heavy rainstorm. Details are here. The Corps continues to refuse to release FOIA-requested emails from before, during, and after this event.

Pins unpinned

It's very difficult to get contemporaneous accounts of what happened at the sites during these events. The Corps can put out press releases, and the media can do one- or two-minute stand-up reports standing hundreds of feet away, but actually knowing what happened in detail is tough. In addition to my FOIA request for the December, 2009 emails, the Corps is continuing to refuse release of emails relating to the November, 2009 London Avenue closure during Ida.

Fortunately, the Corps themselves dispatched someone to take pictures during at least one of these events. On September 12, 2009, heavy rains hit the New Orleans area. As a result of coincident rising lake levels, the gates were closed starting around 6 PM.

There weren't a lot of pictures from that Corps photographer, but one only needs a single good one.

Here is that one:


This photo was taken on September 12, 2009, after the gates were closed. It's the area inside the red square that should concern everyone in the New Orleans metro. Let's zoom in:


This shows that Corps operators and supervisors apparently left at least four dogging pins uninserted into the gates, a major violation of procedure. Those pins appear to be on needles 4, 5, 7, and 8. There are 11 needles in total.

The Times-Picayune also sent a photographer, Scott Threlkeld, out to London Avenue that afternoon to photograph the gate closing process. Mr. Threlkeld snapped two photos whch might partially confirm the pins' status.

The caption on the first (linked here) reads, "Randy Faherty of the U.S. Army Corps of Engineers shoves in a steel pin to lock a lowered flood gate in place Saturday, September 12, 2009, at the London Avenue Outfall Canal structure."

The caption on the second (linked here) reads, "Randy Faherty of the U.S. Army Corps of Engineers checks the water level after dropping the flood gates Saturday, September 12, 2009, at the London Avenue Outfall Canal structure."

I believe the captions on these photos are likely wrong. I do not believe the operator is checking the water level or shoving in a pin. I believe he is trying to figure out why a pin won't go in. And I believe that pin - on needle #8 - is the same one visible out of place in the deep background inside the blue frame in the Corps photo above. It appears Corps supervisors allowed the needles to remain unsecured.

Divers undived

So multiple gates were very likely not seated properly, due to obstructions in the trench. In that case, the Corps was supposed to have called Independent Divers (who are supposed to be on standby during storm events) to clear the obstructions. Let's take a look at the task orders given to Independent Divers (aka H.J. Merrihue) around September 12, 2009:

Task order 60
August 24, 2009

"For an emergency Dive required on 22 Aug 2009 due to a seal blow-out on the dewatering needles. Divers will be directed as needed."

The location of this task order's work is unknown. However, it is definitely not the outfall canals, since there are no dewatering needles at those sites. It was likely one of the Corps' locks.

Task order 61
August 31, 2009

"4 Man dive team required at Port Allen Lock for rewatering Process."

The Port Allen lock is across the Mississippi River from Baton Rouge.

London Avenue gates closed
September 12-14, 2009

Task order 62
September 24, 2009

"Dive contractor is required to perform magnetometer surveys to locate unknown sunken objects."

Even without a location, this is clearly not for trench cleanouts.

London Avenue gates closed
Hurricane/Tropical Storm Ida
November 9-11, 2009

Task order 63
November 18, 2009

"Four Man Dive team to investigate loud noise at gates at Algiers Lock.

The Algiers Lock is across the Mississippi River from St Bernard Parish.

As we can see, at least according to the contract record, Independent Divers was not summoned by Corps supervisors for trench cleaning during the time of the September London Avenue gate closure, even through photos seem to show an inability to close multiple gate sections. It also doesn't appear they made it out any time between that September event and the November London Avenue gate closure. This appears to be another violation of the Corps' own procedure.

Frankly, it appears there was little diving activity at any of the outfall canals through the 2009 hurricane season. Here's all the confirmed task orders for those sites during 2009:

Task order 54
May 18, 2009

"PROVIDE DIVING SERVICES TO CLEAR DEBRIS FROM GATE SILLS WITH THE USE OF A WATER JET PUMP, WITH GENERAL GATE INSPECTIONS REQUIRED AT EACH OF THE THREE LOCATIONS.
LOCATION: 17TH ST. CANAL, ORLEANS AVE. CANAL, & LONDON AVE. CANAL FLOODGATES."

This is the annual cleanout of the trenches before hurricane season

Task order 56
June 3, 2009

"Inspect the bridge pumps 1 and 6 and their foundations and remove any existing obstructions.
The dive work is estimated to take approximately five (5) hours.
LOCATION: 17th Street Canal
WORK TO START AT HRS: 0800 HOURS, 04 JUNE 2009."

This was diving associated with the bridge pump repair work at 17th Street.

Task order 59
August 17, 2009

"Emergency Dive at 17th St Canal Floodgate to clear debris from gates sills with the use of a water jet pump, with general gate inspections required at each of the two locations. Verbal NTP was issued on Aug 17, 2009."

This would appear to be preparation associated with Tropical Storm Claudette. Claudette was a minor threat that formed west of Tampa on August 16th and dissipated the next morning as its remnants came ashore in Alabama. By the time the Corps issued the verbal notice to proceed ("NTP") on the 17th, the storm was already past. Tides ran about a foot above normal.

Since the gates at the London Avenue canal, with its lower Safe Water Level, would be closed before 17th Street, it seems unlikely this task order only applied to 17th Street. The mention of "two locations" at the end of the description hints at the possibility that divers were supposed to also visit London, but it's impossible to know whether that occurred. It could simply refer to obstructions under two separate gate sections at 17th Street. In any case, the inability of the operators to completely secure multiple gate sections at London Avenue on September 12th makes the question moot. The London Avenue trenches do not appear to have been clear on the 12th for a real storm event.

Those three task orders are it for 2009. It does not appear the divers were regularly cleaning out the trenches at all during 2009, despite the Corps Operations & Maintenance Manual calling for the gates to be tested every three weeks during hurricane season:

"Once hurricane season has started, certain activities should be performed at regular intervals so that issues that could potentially interfere with the operation of the gates can be identified and addressed without the threat of an approaching storm.

Operation of the gates should be performed every three (3) weeks. Gate operation should be performed in the same manner as described in paragraph 3.2.1.2 for the check of gate operation prior to the hurricane season. A diver should be brought in to inspect and assist in removing obstructions if they are encountered."

It seems unlikely, if the gates are being tested every three weeks, that there would be no call for removal of obstructions on any of the gates across all three sites. City pumps are run often during the rainy New Orleans summer, sending all sorts of debris down the canals. Some of that debris would inevitably get caught in the trenches. This near complete lack of diving activity brings into question if the gates are truly being tested every three weeks during hurricane season, and if not, why?

It would have been Corps supervisory personnel in the Operations Division making these decisions regarding divers.

Sandbags unplaced

So we've got gates not being secured and divers not being called in to clean the trenches. What else happened - or didn't happen - during that September 2009 event?

The 2009 operations manual says if the obstructions in the trenches cannot be dislodged by the divers, massive sandbags are supposed to be dropped by the on-site rental crane in front of the needle where the obstruction lies. This is supposed to close the gap along the trench. Just for reference, here's the sandbags at the London Avenue canal earlier in 2010:


I previously established that this Google satellite shot was from the spring of 2010.

Zooming into the area inside the red box:


On the ground shots around the same time as the Google shot:



I can't speak to whether those sandbags were placed or not, or if they were even on site. However, the lack of photographic evidence both on September 12th (from two independent sources) as well as no Corps photos of sandbags getting removed from the water on September 14th certainly leads one to believe that no sandbags were placed. That's a third violation of procedure.

1) Gates unpinned
2) Divers not summoned
3) Sandbags not placed

The Corps "responds"

I repeatedly asked for comment on this story from the Corps' New Orleans District Public Affairs office. Here's what they sent me:

"Mr. McBride -- The gates do not have to be pinned to be sealed. Everything worked as it should.

V/R
Ken [Holder, Chief, Public Affairs, Corps New Orleans District]"

This semi-confirms the gates were unpinned last September, and certainly doesn't deny it. The mention of them being sealed is not relevant, because the seals at London Avenure are just for show. The importance of pinning is in effective transfer of surge forces to the foundation, not sealing, which is made abundantly clear in the Corps' own Operations & Maintenance Manual for the gates.

Summing up

So, it looks pretty bad. Compared to other storm events, the September, 2009 closure was a minor event. The lake never got above about 3 feet, and that happened just two hours after the gates were closed on September 12th. Such an event would act as a real world drill, so you'd think the Corps supervisors would try to get everything right with a bit of the pressure off. That way, when the Big One comes with its Big Pressure, they'd be able to handle it. Instead, it appears they viewed this as a nothing event when key parts of procedure could be skipped, which makes one wonder what's happening the rest of the time, and what will happen when the Big One hits.

Next up: How the pumps performed during this event.

Karen Gadbois of The Lens, and Molly Peterson of Southern California Public Radio contributed to this report.

The gates story so far

In examining the Interim Closure Structures, we've looked at the trench the gates sit in when lowered, the gates themselves, the guide columns, the seals, the pins that are meant to lock the gates in place, the winches that raise and lower the gates, and the SCADA system that controls and monitors the whole thing.

Along the way, we've learned about these unresolved problems:

- The Corps relies on divers to clean out the trenches in which the gates sit, rather than an automated cleaning system. Over the years, the use of those divers has been somewhat spotty, as I'll show in my next post.

- During construction in 2006, the trenches at Orleans Avenue and 17th Street were filled and covered with jet grout to a depth of at least three feet. All that had to be jackhammered out only after the Corps discovered the problem in advance of a tropical system. One of the needles at the site was lowered in the trench at the time they found this problem. Here's a picture of that needle after it was freed:


Two years later, they were still chiseling the stuff out. Inserts meant to keep silt from clogging the trenches, promised in 2006, were never installed. The upshot is the public does not know what condition the trenches are in currently, though smart money suggests "not good."

- The gates themselves are not built in accordance with the Corps' own design standards. Among the weaknesses are vertical framing, unreinforced guide columns, and point concentration of storm surge forces into a handful of locations on the main frames of the structures.

- The gates are lowered through the guide beams with no means to ensure they (the gates) don't rattle around as water flows around them. This increases the potential of damage to the gates and the guides, as well as creating the possibility of the gates binding up. The Corps' Operation and Maintenance Manual for the gates recognizes this as a possibility in the instructions for lowering the gates:

"If during this process it is observed that as the gate is lowered it does not continue at a constant rate, the top elevation of the gate where the interruption occurred shall be noted and the gate shall go through the operation procedure a second time. If an interruption occurs again at the noted location then an inspection of the guide columns along their full height, as well as the portion of the gate within the guide columns shall be performed to determine what is causing the gate to hang up."

The Corps' engineering manual and their own consultant's report calls for a series of rollers on the guides to ensure against these problems. There are no rollers.

And it's not like the Corps doesn't know rollers are needed. They are building ten of them into each of the two vertical lift gates at the Seabrook closure just a couple of miles from the outfal canals, also along the Lake Pontchartrain lakefront. That design also includes two pours of concrete around the guide beams, horizontal reinforcement for the gate segments, and basically everything the current outfall canal gates do not. Frankly, the Seabrook design amply demonstrates how flimsy the outfall canal gates are.

- At 17th Street, the gates are sealed only on one side, and at London and Orleans Avenues, there is no sealing (a cosmetic seal on the lake side of the structures at those two sites is purely decorative). Thus, the gates are actually - as built - leaky.

Here's the location of the "seals" on the front of the London Avenue structure, away from the sealing surfaces between the needles and the guide beams (photo here):


A detail from the applicable London Avenue as-built drawing shows the seals from above:


This design includes a 4.5" gap between the needles and the structure:



Not good.

- At all three sites, the hydraulic power units and control boxes for the winches - which lower and raise the gates - are placed directly in the line of fire of storm-driven waves and overwash.



At 17th Street, the units are actually closer to the water than the gates themselves:


These winch control units are not enclosed in storm-hardened enclosures and remain completely exposed to waves and storm-thrown debris. This arrangement places the equipment and the operators - and by extension, the city - at risk.

- The SCADA control system at all three sites appears to be extremely vulnerable to weather induced failures, demonstrated by the five day outage in December, 2009. During this outage, water levels in both the 17th Street and London Avenue canals exceeded the criteria for closure of the gates on each canal, but neither set of gates was dropped. On the London Avenue canal, the water depth exceeded the Safe Water Elevation during this outage as well.

In the next part, we'll take a look at how some of these moving parts interact during a real event. In this case, it'll be the closure of the London Avenue gates on September 12-14, 2009

SCADA

I've been working on entries about the September 12-14, 2009 closure of the gates along the London Avenue canal. That particular closure is instructive since it arose not from a tropical storm, but from a heavy non-tropical rainfall event. The gates closed just days after the Corps issued new procedures specific to London Avenue which called for dropping the gates when the water got to just 2.5 feet, down from the previous 4 feet. I'll bring that information out soon.

When I say it is instructive, I mean that it should have set the tone for future events. Indeed, when Hurricane Ida approached New Orleans in early November, 2009, the London Avenue gates dropped again when water got to 2.5 feet.

But then came the very serious rains in mid-December, 2009, and something very odd happened. Despite water getting close to, and then eventually exceeding the London Avenue Safe Water Elevation of 5 feet, the gates did not drop. Indeed, despite numerous, repeated instances of waters exeeding the 2.5 foot criterion in the London Avenue canal over a seven day period from December 8th through the 15th, never did the gates drop.

This perplexed me. For a while, I figured it was because the Corps was scared of their own rusty pumps at the gates, none of which had been fully repaired at that point (they still haven't). But that didn't make sense, because they knew the condition of those pumps in September and November, yet they still proceeded with gate closures and pump operations during those events.

It wasn't until I started reviewing the actual SCADA data for all three events that I found a clue to why the December, 2009 rainfall was treated differently from the September and November events.

First, "SCADA" stands for Supervisory Control and Data Acquisition. It is the computerized control system that alows the Corps to monitor and control many aspects of operations at the gates, including water levels within the canals, starting and stopping of their pumps, and even weather (just the monitoring on that one).

Along the London Avenue canal, the Corps has placed 10 level gauges which are connected to the SCADA system and provide near-real-time feedback on the water level within the entire length of the canal. Four of the gauges are at the gates, while the other six are placed along the canal, at the bridge crossings and Sewerage & Water Board drainage pump stations. Water levels are recorded every ten seconds from these gauges and are transmitted either by submerged fiberoptic cable or by microwave radio to the SCADA system. There are display terminals for the system at the gate structures, in the Sewerage & Water Board pump stations, and at the Corps' building in Leake Avenue.

The level gauges also have two backup systems in their individual control boxes. The secondary system polls the gauges a set number of times in an hour (maybe four or six) and takes the highest reading for the hour. That number is transmitted via satellite to a website the Corps maintains called Rivergauges, which is publicly available. A tertiary system involves satellite phone lines.

Obviously, due to its near-real-time basis, the Corps relies on the primary system for gauge readouts along the canal, especially during storms. The SCADA system also allows the Corps to run their pumps remotely, and monitor them for problems. Presumably, the status of gate segments (i.e. open/closed) is also included in the SCADA system. All these things can be done manually by crew at the gate sites, but the SCADA system provides a great deal more visibility at a glance.

Early on the morning of December 8, 2009, it appears the Corps' SCADA system suffered a systemwide failure across all three canals. Level gauges stopped sending updated readings to the system, and it is likely the Corps lost the ability to remotely start, stop, and monitor all their pumps. This occurred during a very heavy rainstorm that caused widespread street flooding, as the Sewerage & Water Board (S&WB) was pumping water into the canals. For some reason, the secondary level gauge reporting system continued to work, reporting canal water levels throughout.

This SCADA system failure appears to have continued for five days, until the afternoon of December 13th when the level gauges started reliably reporting water depths again. In between, New Orleans saw massive rainfall. On the evening of December 12th, as torrents lashed the city, water levels in the London Avenue canal rose above the 5 foot safe water elevation. Remember that Corps protocol called for gates to be dropped when water hit just 2.5 feet. The Corps knew this was serious, because their London Avenue canal captain showed up in the S&WB pumping station at the south end of the canal, Drainage Pumping Station #3, at around 6:30 that evening.

Meanwhile, a couple of miles west at the 17th Street canal, waters were also rising. Along that canal, the safe water level is six feet, and the criteria for closure is a level of five feet. As S&WB personnel turned on nearly every pump in Drainage Pumping Station #6 at the south end of the canal, the water level rose past the closure criteria along that canal as well. That is according to the Rivergauges readings coming from the backup level signal system, since the primary SCADA system was still out along all three canals.

On the 17th Street canal, the water appears to have stayed at or above the five foot closure criterion for at least 3 hours on the evening of the 12th. There's no evidence of Corps presence in DPS#6 that evening.

In neither of these cases were the gates dropped and the Corps pumps started, as called for in their own procedures. At 17th Street, water was within inches of the safe water level for hours. At London Avenue, it exceeded the safe water level for approxmately six hours, and was above the closure criteria of 2.5 feet for the better part of 10 to 16 hours, well into the morning of the 13th.

The denoument of this tale came on the 16th, when the Corps issued a $57,388.07 task order to their SCADA system maintenance contractor, Prime Controls (the same firm that installed the SCADA system) for maintenance of the system along all three canals. I believe it is a fair assumption that task order was for emergency work Prime performed after the system went down on the 8th. I have placed a FOIA request for the complete contract, including the December 16th task order.

I noted these events when they happened, but it is only recently that I have gotten the complete SCADA record for all three canals, which allowed me to see exactly what was happening minute by minute. However, what would truly allow clarity would be the emails of all the principal Corps personnel during the period around the December events.

I placed a FOIA request for those emails, as well as any after-action reports, almost nine months ago. It remains unfilfilled, along with about a half dozen other months-old requests. It is understandable that the Corps would be reluctant to discuss what happened that week in December, but it doesn't stop them from spinning it.

In multiple recent public relations documents, starting in March of this year, they actually point to the walls seeing more water than they should have in December as a good thing. From their March, 2010 "Teammate Update:"

"Since their initial construction in the mid 1990s, the outfall canal walls have performed well in removing rainwater from the city, including the record rainfall events in May 1995 and December 2009."

That's chutzpah!

When (if?) I get those emails, I'm sure I'll have a better perspective on events. However, I do not believe it is a coincidence that of the three major rainfall or surge events to occur in New Orleans in the last third of 2009, the only one when the gates should have been dropped - but weren't - was the one when the Corps' entire control system appears to have been out of commission for nearly a week. Add in the over $57,000 in repairs to that system just days after the event, and there appears to be a scary story there.

As I mentioned at the beginning, I've also got stuff on the September, 2009 event that I'll be bringing out, but this seemed too large to hold back until I get every single bit of information. After all, I've been waiting nine months for about a week's worth of emails confined to a very small section of the Corps' business, and I don't feel like holding back any more.

The winches

In the previous four parts of this series on the gates, we've looked at the trench the gates sit in when lowered, the gates themselves, the guide columns, the seals, and the pins that are meant to lock the gates in place. This part covers the equipment that lifts and lowers the gates - the winches.

The Basics

When the gate structures were first built in 2006, they were not automated. The individual segments, known as needles, were lowered and raised by crane. For example, here's the Orleans Avenue gates in October, 2006:


And 17th Street in July, 2006:


This presented a number of problems. Cranes couldn't operate in winds over 30 mph - a problem if there's a tropical storm or hurricane on the way. Also, it took many hours to lower the needles into place. The gate closure process laid out in the 2006 operating manual is actually 5 days long.

In mid-2006, or perhaps earlier, the Corps recognized this problem and started designing and installing an automated winching system at all three sites. With winches, there would be no need to worry about the wind speed. The final system would allow the gates to be dropped into place in less than an hour. That would allow the Corps to keep them open for much longer, avoiding or minimizing use of the suspect pumps installed at all three sites.

By the 2007 hurricane season, the guide columns at all three sites had been extended vertically and every needle had a winch installed above it. 17th Street got them first, as seen here:


The winches are manufactured by Pullmaster division of TWG (formerly Tulsa Winch Group), which is itself part of the Dover Corporation, a multinational that focuses on commerical and indutrial equipment. Pullmaster is in Surrey, British Columbia.

The Corps went with the model R7 from Pullmaster (their general sales bulletin is here). And so that the winch is not pulling the entire weight of each needle, the design firms called for multipart lines between the needles and the winches. At 17th Street, the line travels through two pulleys before attaching to the needles, while at London/Orleans it goes through three pulleys:

17th Street:


And from the drawings attached to modification P00037 to Boh Brothers' construction contract (obtained via FOIA by Molly Peterson):



London (Orleans is similar) (via Flickr here):


Zooming in for more detail:



The Problems

The winches are powered by hydraulic power units (HPU), with two units at 17th Street, and one each at Orleans Avenue and London Avenue. Each HPU has a control cabinet adjacent to it which allows the operators to raise and lower the winches.

Not much has been written about the winches, other than the Corps pointing them out when they were installed. However, in December, 2006, the local paper in Tulsa, Oklahoma, the Tulsa World, wrote an article about the hydraulic power unit and control cabinet that was installed at the London Avenue site. It's got a good picture of the unit, and it's worth reading.

Now the problems... Despite the front line importance of the gates and the ability to lower and raise them, the winch hydraulic power units and control boxes are not located in a secure locations far away from storm surge. This is different from the controls and most of the power units for the hydraulic pumps, which are in elevated blockhouses on the perimeter of the site. In fact, the winch controls and power units are directly in the line of fire.

At London (gates shown raised) (2nd photo via Flickr here):



Orleans (gates shown raised):



and 17th Street (pics here):


That is the winch hydraulic power unit on the west side of the site (note drive unit for bridge pump placed next to it). A matching winch HPU is on the east side of the structure:


We know the current gates' tops are 16 feet above sea level. The winch power units and controls also rest on concrete or steel (depending on the site), exposed at the same 16 foot elevation. We also know that is two feet shorter than what the Corps' own design guidelines call for.

What if a wave washes over the gates and the concrete cells next to them, as is likely? At 17th Street, the Corps' 100 year storm calculations call for wavetops at 17.4 feet. And what if a 200 year storm comes rolling toward the city?

Or what about flying debris? Can we be sure that all those light poles on the structures will hold up in 100 mph winds? The controls and HPU's aren't exactly housed in hardened boxes. There's nothing around them at 17th Street, and at London and Orleans Avenues, the only thing protecting them is a canvas tarp:


Better than most, this picture (via Flickr here) shows the delicate positioning of the winch power unit and controls when the gates are lowered.

My point is that the controls and power units for the gates themselves are placed in about as vulnerable a spot as the Corps could find. At 17th Street, they're actualy in front of the gates. The only worse spot would be a barge out in the lake. It doesn't take a genius to think of bad news scenarios resulting from this vulnerability.

The nightmare situation is some kind of combined human factor/technological failure of the system during a storm which causes one of the gates to raise with surge against it, pouring water into the canals again. I can't speak to the exact controls architecture on this matter, but is it really that inconceivable?

The solution to part of this problem is simple - use electric winches. The New Orleans' District's mechanical love affair with all things hydraulic makes no sense when placing those things in a hurricane. From pumps to winches, simpler is better when one is talking about systems meant to protect hundreds of thousands of lives and the property belonging to those people. Instead, the Corps has put hydraulic pumps and hydraulic winches with tons of moving parts, pressurized oil, and which have critical pieces totally exposed to hurricane storm conditions. And because the distance from the winches' hydraulic power units to the winches themselves is limited by the fluid nature of the system, they had to place the power units close to the winches, i.e. on the deck. It's stupid.

Electric winches would have eliminated the hydraulic power units and allowed placement of the winch controls in the blockhouses, protecting both the operators and the equipment during a storm. Black & Veatch also noted this deficiency in their evaluation of the Interim Closure Structures, pointing out "Hoisting equipment is exposed to hurricane blown debris" and "Remote operation of gates is needed to protect personnel during storm events."

There is a larger problem with placing the winch controls in such a vulnerable position. If they go out during one storm, they will likely not be available for a few weeks while repairs are effected (lots of moving parts, remember). That's not a big deal when raising the gates after a storm has passed - the cranes can do that. But what happens if another storm comes in on the heels of that first one? It's not a theoretical question. Rita followed Katrina in 2005 by just 3 weeks. Ike was only 2.5 weeks after Gustav in 2008. In an active season, there can be up to four storms brewing in the Atlantic basin.

If the winches are not available to lower the gates, then the Corps is left with the system in place during the 2006 season - rental cranes. And those things can't be used in winds over 30 mph, and take much, much longer to lower the gates. Plus those cranes were "pre-owned" when the Corps first started renting them in 2006. Now they're four years older, and the Corps has already had to pay Scott Equipment over $8000 for damaging them.

In sum, the Corps could have done a lot better going with electric winches. However, since they insisted on hydraulic winches, and also insisted on placing the hydraulic power units on the gate structures in the line of fire for storm surge, the least they could have done is place them inside a hardened, vented steel box. That goes for the winches themselves as well. But not even that small bit of protection was implemented.

Karen Gadbois of The Lens and Molly Peterson of Southern California Public Radio contributed to this report.

The pins

So we've covered the trench the gates sit in when lowered, the gates themselves, the guide columns, and the seals. Next up - the dogging pins.

As detailed in my post "The Seals," at 17th Street there are provisions to keep the gates in place horizontally. Namely, mating wedges on the needles and the guide columns which force the gates against the back (city) face of the guides (diagram adapted from 17th Street as-built drawing here):


There is no similar system at the London or Orleans gates, meaning the gates can rattle around in the guides when lowered or raised, as well as when they are seated in the trenches.

But what keeps the gates in place vertically? The answer is dogging pins. Dogging pins are large cylinders slid into circular slots on the needles to restrain them from moving. There are two for every gate section (pictures from USACE Flickr stream here and here):



At all three sites, the dogging pins slide into slots near the top of each needle when the needle is placed in the trench (picture here):


This serves to "lock" the needle in place, as well as letting the operators know that the needle is firmly seated in the trench. The pins are very similar to the lock on a double-hung window; that lock pushes the sashes into the framework of the window, ensuring a tight fit.

There is a minor difference between the 17th Street and London/Orleans designs in how the gates are stored when they are not lowered. At 17th Street, the needles hang off a second set of dogging pins placed at the top of the main framework (from this drawing)...



Here's a picture of the upper pins and their frameworks at 17th Street. This picture, from the 2007 SAME presentation I cited in my earlier post on the trench, shows the pins before the framework for the winches was added. This would place this photo in the mid-2006 timeframe:


At London Ave. and Orleans Ave., the dogging pins at the platform level do double duty, supporting the needles in a second set of circular slots (from London Ave. bid drawings):


As mentioned above, the pins signify to the Corps personnel that the gates are firmly seated in the trench. Remember that the Corps is supposed to be having their diving contractor clearing those trenches regularly, as well as immediately before any closure of the gates during a storm. So if the pins don't go in the slots on the needles, that means the trench is not clean, which is bad.

Here's what the 2009 Operating Manual for the gates says about the pins:

"To close a gate, it will need to be lifted slightly to allow the dogging pins to be removed. A come-along may be needed to remove the dogging pin. Once the dogging pins are removed the winch can lower the gate until the lifting cables go slack. Once the cables are slack the dogging pins should be inserted into the holes near the top of the gate that are meant for locking the gate in the closed position. If the dogging pins can be inserted when the gate is down, this will confirm that the notch in the sill is free from obstructions. The eye bolt may need to be removed from the dogging pin and a sledge hammer used to move the pin into place. This process for closing the gates is identical to that described in subparagraph 3.2.1.2. If gates do not fully close and conditions are no longer suitable for divers, pre-positioned sandbags within reach of the crane shall be utilized to seal gaps."

One can see those pins getting pulled before lowering the gates during the May 27, 2009 media availability in this Times-Picayune video, starting at 0:43:

If one looks at the dogging pins closely, and then considers the lack of horizontal restraint on the gates at London and Orleans, one might wonder, "If the gates are rattling around during a storm, what keeps them from knocking the dogging pins out?"

The answer is a second set of pins. The Corps had the contractors add cotter pins which slide through the eyebolts on the ends of the much bigger dogging pins. These cotter pins are bent pieces of threaded rod which screw into welded nuts. Here's a few pictures (from here, here, and here):




A similar set of second pins is also installed on the dogging pins at 17th Street.

Effectively, the threaded rods are the last line of defense for the gates. That is, the threaded rods restrain the dogging pins, and the dogging pins restrain the gates, and the gates restrain the storm surge. Of course, the gates are pretty heavy, but so were the wall sections that collapsed along the canals. Water can do amazing, scary things.