SCADA
I've been working on entries about the September 12-14, 2009 closure of the gates along the London Avenue canal. That particular closure is instructive since it arose not from a tropical storm, but from a heavy non-tropical rainfall event. The gates closed just days after the Corps issued new procedures specific to London Avenue which called for dropping the gates when the water got to just 2.5 feet, down from the previous 4 feet. I'll bring that information out soon.
When I say it is instructive, I mean that it should have set the tone for future events. Indeed, when Hurricane Ida approached New Orleans in early November, 2009, the London Avenue gates dropped again when water got to 2.5 feet.
But then came the very serious rains in mid-December, 2009, and something very odd happened. Despite water getting close to, and then eventually exceeding the London Avenue Safe Water Elevation of 5 feet, the gates did not drop. Indeed, despite numerous, repeated instances of waters exeeding the 2.5 foot criterion in the London Avenue canal over a seven day period from December 8th through the 15th, never did the gates drop.
This perplexed me. For a while, I figured it was because the Corps was scared of their own rusty pumps at the gates, none of which had been fully repaired at that point (they still haven't). But that didn't make sense, because they knew the condition of those pumps in September and November, yet they still proceeded with gate closures and pump operations during those events.
It wasn't until I started reviewing the actual SCADA data for all three events that I found a clue to why the December, 2009 rainfall was treated differently from the September and November events.
First, "SCADA" stands for Supervisory Control and Data Acquisition. It is the computerized control system that alows the Corps to monitor and control many aspects of operations at the gates, including water levels within the canals, starting and stopping of their pumps, and even weather (just the monitoring on that one).
Along the London Avenue canal, the Corps has placed 10 level gauges which are connected to the SCADA system and provide near-real-time feedback on the water level within the entire length of the canal. Four of the gauges are at the gates, while the other six are placed along the canal, at the bridge crossings and Sewerage & Water Board drainage pump stations. Water levels are recorded every ten seconds from these gauges and are transmitted either by submerged fiberoptic cable or by microwave radio to the SCADA system. There are display terminals for the system at the gate structures, in the Sewerage & Water Board pump stations, and at the Corps' building in Leake Avenue.
The level gauges also have two backup systems in their individual control boxes. The secondary system polls the gauges a set number of times in an hour (maybe four or six) and takes the highest reading for the hour. That number is transmitted via satellite to a website the Corps maintains called Rivergauges, which is publicly available. A tertiary system involves satellite phone lines.
Obviously, due to its near-real-time basis, the Corps relies on the primary system for gauge readouts along the canal, especially during storms. The SCADA system also allows the Corps to run their pumps remotely, and monitor them for problems. Presumably, the status of gate segments (i.e. open/closed) is also included in the SCADA system. All these things can be done manually by crew at the gate sites, but the SCADA system provides a great deal more visibility at a glance.
Early on the morning of December 8, 2009, it appears the Corps' SCADA system suffered a systemwide failure across all three canals. Level gauges stopped sending updated readings to the system, and it is likely the Corps lost the ability to remotely start, stop, and monitor all their pumps. This occurred during a very heavy rainstorm that caused widespread street flooding, as the Sewerage & Water Board (S&WB) was pumping water into the canals. For some reason, the secondary level gauge reporting system continued to work, reporting canal water levels throughout.
This SCADA system failure appears to have continued for five days, until the afternoon of December 13th when the level gauges started reliably reporting water depths again. In between, New Orleans saw massive rainfall. On the evening of December 12th, as torrents lashed the city, water levels in the London Avenue canal rose above the 5 foot safe water elevation. Remember that Corps protocol called for gates to be dropped when water hit just 2.5 feet. The Corps knew this was serious, because their London Avenue canal captain showed up in the S&WB pumping station at the south end of the canal, Drainage Pumping Station #3, at around 6:30 that evening.
Meanwhile, a couple of miles west at the 17th Street canal, waters were also rising. Along that canal, the safe water level is six feet, and the criteria for closure is a level of five feet. As S&WB personnel turned on nearly every pump in Drainage Pumping Station #6 at the south end of the canal, the water level rose past the closure criteria along that canal as well. That is according to the Rivergauges readings coming from the backup level signal system, since the primary SCADA system was still out along all three canals.
On the 17th Street canal, the water appears to have stayed at or above the five foot closure criterion for at least 3 hours on the evening of the 12th. There's no evidence of Corps presence in DPS#6 that evening.
In neither of these cases were the gates dropped and the Corps pumps started, as called for in their own procedures. At 17th Street, water was within inches of the safe water level for hours. At London Avenue, it exceeded the safe water level for approxmately six hours, and was above the closure criteria of 2.5 feet for the better part of 10 to 16 hours, well into the morning of the 13th.
The denoument of this tale came on the 16th, when the Corps issued a $57,388.07 task order to their SCADA system maintenance contractor, Prime Controls (the same firm that installed the SCADA system) for maintenance of the system along all three canals. I believe it is a fair assumption that task order was for emergency work Prime performed after the system went down on the 8th. I have placed a FOIA request for the complete contract, including the December 16th task order.
I noted these events when they happened, but it is only recently that I have gotten the complete SCADA record for all three canals, which allowed me to see exactly what was happening minute by minute. However, what would truly allow clarity would be the emails of all the principal Corps personnel during the period around the December events.
I placed a FOIA request for those emails, as well as any after-action reports, almost nine months ago. It remains unfilfilled, along with about a half dozen other months-old requests. It is understandable that the Corps would be reluctant to discuss what happened that week in December, but it doesn't stop them from spinning it.
In multiple recent public relations documents, starting in March of this year, they actually point to the walls seeing more water than they should have in December as a good thing. From their March, 2010 "Teammate Update:"
That's chutzpah!
When (if?) I get those emails, I'm sure I'll have a better perspective on events. However, I do not believe it is a coincidence that of the three major rainfall or surge events to occur in New Orleans in the last third of 2009, the only one when the gates should have been dropped - but weren't - was the one when the Corps' entire control system appears to have been out of commission for nearly a week. Add in the over $57,000 in repairs to that system just days after the event, and there appears to be a scary story there.
As I mentioned at the beginning, I've also got stuff on the September, 2009 event that I'll be bringing out, but this seemed too large to hold back until I get every single bit of information. After all, I've been waiting nine months for about a week's worth of emails confined to a very small section of the Corps' business, and I don't feel like holding back any more.
When I say it is instructive, I mean that it should have set the tone for future events. Indeed, when Hurricane Ida approached New Orleans in early November, 2009, the London Avenue gates dropped again when water got to 2.5 feet.
But then came the very serious rains in mid-December, 2009, and something very odd happened. Despite water getting close to, and then eventually exceeding the London Avenue Safe Water Elevation of 5 feet, the gates did not drop. Indeed, despite numerous, repeated instances of waters exeeding the 2.5 foot criterion in the London Avenue canal over a seven day period from December 8th through the 15th, never did the gates drop.
This perplexed me. For a while, I figured it was because the Corps was scared of their own rusty pumps at the gates, none of which had been fully repaired at that point (they still haven't). But that didn't make sense, because they knew the condition of those pumps in September and November, yet they still proceeded with gate closures and pump operations during those events.
It wasn't until I started reviewing the actual SCADA data for all three events that I found a clue to why the December, 2009 rainfall was treated differently from the September and November events.
First, "SCADA" stands for Supervisory Control and Data Acquisition. It is the computerized control system that alows the Corps to monitor and control many aspects of operations at the gates, including water levels within the canals, starting and stopping of their pumps, and even weather (just the monitoring on that one).
Along the London Avenue canal, the Corps has placed 10 level gauges which are connected to the SCADA system and provide near-real-time feedback on the water level within the entire length of the canal. Four of the gauges are at the gates, while the other six are placed along the canal, at the bridge crossings and Sewerage & Water Board drainage pump stations. Water levels are recorded every ten seconds from these gauges and are transmitted either by submerged fiberoptic cable or by microwave radio to the SCADA system. There are display terminals for the system at the gate structures, in the Sewerage & Water Board pump stations, and at the Corps' building in Leake Avenue.
The level gauges also have two backup systems in their individual control boxes. The secondary system polls the gauges a set number of times in an hour (maybe four or six) and takes the highest reading for the hour. That number is transmitted via satellite to a website the Corps maintains called Rivergauges, which is publicly available. A tertiary system involves satellite phone lines.
Obviously, due to its near-real-time basis, the Corps relies on the primary system for gauge readouts along the canal, especially during storms. The SCADA system also allows the Corps to run their pumps remotely, and monitor them for problems. Presumably, the status of gate segments (i.e. open/closed) is also included in the SCADA system. All these things can be done manually by crew at the gate sites, but the SCADA system provides a great deal more visibility at a glance.
Early on the morning of December 8, 2009, it appears the Corps' SCADA system suffered a systemwide failure across all three canals. Level gauges stopped sending updated readings to the system, and it is likely the Corps lost the ability to remotely start, stop, and monitor all their pumps. This occurred during a very heavy rainstorm that caused widespread street flooding, as the Sewerage & Water Board (S&WB) was pumping water into the canals. For some reason, the secondary level gauge reporting system continued to work, reporting canal water levels throughout.
This SCADA system failure appears to have continued for five days, until the afternoon of December 13th when the level gauges started reliably reporting water depths again. In between, New Orleans saw massive rainfall. On the evening of December 12th, as torrents lashed the city, water levels in the London Avenue canal rose above the 5 foot safe water elevation. Remember that Corps protocol called for gates to be dropped when water hit just 2.5 feet. The Corps knew this was serious, because their London Avenue canal captain showed up in the S&WB pumping station at the south end of the canal, Drainage Pumping Station #3, at around 6:30 that evening.
Meanwhile, a couple of miles west at the 17th Street canal, waters were also rising. Along that canal, the safe water level is six feet, and the criteria for closure is a level of five feet. As S&WB personnel turned on nearly every pump in Drainage Pumping Station #6 at the south end of the canal, the water level rose past the closure criteria along that canal as well. That is according to the Rivergauges readings coming from the backup level signal system, since the primary SCADA system was still out along all three canals.
On the 17th Street canal, the water appears to have stayed at or above the five foot closure criterion for at least 3 hours on the evening of the 12th. There's no evidence of Corps presence in DPS#6 that evening.
In neither of these cases were the gates dropped and the Corps pumps started, as called for in their own procedures. At 17th Street, water was within inches of the safe water level for hours. At London Avenue, it exceeded the safe water level for approxmately six hours, and was above the closure criteria of 2.5 feet for the better part of 10 to 16 hours, well into the morning of the 13th.
The denoument of this tale came on the 16th, when the Corps issued a $57,388.07 task order to their SCADA system maintenance contractor, Prime Controls (the same firm that installed the SCADA system) for maintenance of the system along all three canals. I believe it is a fair assumption that task order was for emergency work Prime performed after the system went down on the 8th. I have placed a FOIA request for the complete contract, including the December 16th task order.
I noted these events when they happened, but it is only recently that I have gotten the complete SCADA record for all three canals, which allowed me to see exactly what was happening minute by minute. However, what would truly allow clarity would be the emails of all the principal Corps personnel during the period around the December events.
I placed a FOIA request for those emails, as well as any after-action reports, almost nine months ago. It remains unfilfilled, along with about a half dozen other months-old requests. It is understandable that the Corps would be reluctant to discuss what happened that week in December, but it doesn't stop them from spinning it.
In multiple recent public relations documents, starting in March of this year, they actually point to the walls seeing more water than they should have in December as a good thing. From their March, 2010 "Teammate Update:"
"Since their initial construction in the mid 1990s, the outfall canal walls have performed well in removing rainwater from the city, including the record rainfall events in May 1995 and December 2009."
That's chutzpah!
When (if?) I get those emails, I'm sure I'll have a better perspective on events. However, I do not believe it is a coincidence that of the three major rainfall or surge events to occur in New Orleans in the last third of 2009, the only one when the gates should have been dropped - but weren't - was the one when the Corps' entire control system appears to have been out of commission for nearly a week. Add in the over $57,000 in repairs to that system just days after the event, and there appears to be a scary story there.
As I mentioned at the beginning, I've also got stuff on the September, 2009 event that I'll be bringing out, but this seemed too large to hold back until I get every single bit of information. After all, I've been waiting nine months for about a week's worth of emails confined to a very small section of the Corps' business, and I don't feel like holding back any more.
posted by mcbrid35 at
9:29 AM
0 Comments:
Post a Comment
<< Home