Sid - N1MVM writes:
I am sending the following link along as a cautionary tale.
Although it focuses on Verizon and Virginia, it notes that many of the underlying systems and issues are not unique to company and area. And since Verizon is considered in many instances to be superior to AT&T, it begs the question of how reliable are our systems under stress.
LINK
www.washingtonpost.com/local/crime/veriz...a6ac36a08_story.html

73,
Sid Gale, N1MVM
CT ARES, Emergency Coordinator - Guilford
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Verizon 911 failures began long before derecho
By Mary Pat Flaherty and Joe Stephens, Published: 12/01/12

The Washington region’s 911 emergency network has suffered widespread systemic failures over the past two years, a rash of outages that started long before June’s derecho storm, according to interviews and records obtained by The Washington Post.

There have been at least 11 outages since July 2010 in Maryland and Virginia, some of which left panicked 911 callers listening to busy signals as they tried to report emergencies, the records and interviews show.

Failures have occurred across the region, from the Chesapeake Bay to Virginia horse country. Some outages blocked all calls in a particular area; others restricted the number of calls or deprived authorities of location data and call-back numbers.

The troubles occurred in a system operated by Verizon, whose lines handle every 911 call made in Washington’s immediate suburbs. Verizon routes 911 calls to 1,800 government-run call centers in 12 states, making it one of the largest such carriers in the nation.

The 911 networks are specifically designed to be fail-safe and to continue operating when other critical infrastructure, such as power lines, are knocked out.

But as failures quietly mounted, officials in the Washington area and elsewhere began asking whether the incidents were symptoms of deeper flaws permeating the nation’s emergency-response system, perhaps requiring federal action.

“At the time you’re in most dire need of assistance, it would be nice if someone could hear your call for help,” said Carol Henn of Rockville, whose husband was killed by lightning at a picnic in 2010. Calls to 911 seeking help for Carl Henn met with busy signals. It is unclear why the calls did not go through, but 911 troubles that day are being reviewed by Maryland regulators.

Experts said the outages are especially troubling because they hobbled some of the most advanced and best funded emergency call centers in the nation, such as those in Fairfax and Montgomery counties.

No single factor

The outages were caused by various problems and could not be traced to a single factor, The Post’s review found. The problems included struggles to maintain equipment, technical glitches and automatic alarms going unheeded.

Verizon officials said they took all of the problems seriously, while emphasizing that they do not consider all of the failures equally significant.

“If we step back, our overall reliability is very strong,” said Maureen Davis, Verizon’s vice president of network operations for the Mid-Atlantic region. “Having said that, we care about each one of these incidents.”

The Washington area outages have national implications, experts said.

“Everyone is waiting to understand exactly what happened” with the derecho, said Steve Marzolf, an information technology official for Virginia and an emergency call center specialist. “The network in Northern Virginia . . . is pretty much the architecture used nationally.”

Verizon provides the backbone of the region’s emergency service system. No matter what telephone company a person uses to call 911 in the Washington suburbs, the call passes through Verizon’s system, which operates like a high-tech traffic cop and directs requests for help to the closest dispatch center.

Verizon routes calls from all devices, whether land line, mobile phone or Internet-based network. In the District, where there is a different system, Verizon does not route all calls, and each telephone service delivers its own calls to the 911 center.

“It is extremely important that we get this right. This is 911,” said Anthony A. Lewis, Verizon’s regional vice president for government affairs. “Every call should go through.”

D.C. area thunderstorms knock out power across region: Homes and businesses across Washington, D.C., Maryland and Virginia lost power June 29 as one of the most powerful and punishing storms in months swept across the region.

Davis said equipment that failed in June was repaired or replaced and worked properly during Hurricane Sandy in October. “I think we passed,” she said.

Many inquiries

But not everyone is convinced, and among emergency response directors, concerns appear to have come to a head with the cascade of investigations.

In July, the Federal Communications Commission took the unusual step of opening an official inquiry into the aftermath of the June 29 derecho, including what it called Verizon’s “systemic” 911 failures in Northern Virginia. In September, members of Congress devoted much of a Capitol Hill hearing to exploring the derecho outages in Northern Virginia.

And in Connecticut, state officials recently cited Verizon’s Northern Virginia outages while arguing for stricter reliability standards there for Verizon and other 911 providers. Requiring telephone companies to provide more details on each outage is one of the changes being considered.

“We look to the Virginia . . . situation as an example of the potential harm that can come if we don’t have adequate regulation and standards in place, for both reliability and restoration,” said Elin Swanson Katz, the Connecticut consumer counsel. “It’s a cautionary tale.”

Virginia utility regulators are looking into whether the June outages are evidence of a “systematic deficiency.”

“We have a high level of frustration,” said Jeffrey A. Horwitz of Arlington County’s Emergency Communications Center, which was crucial in the response to the 2001 terrorist strike on the Pentagon. “Worst-case scenarios is what they’re supposed to be prepared for.”

The frustration is shared in Montgomery County, where officials investigated outages that occurred during a January 2011 snowstorm, remembered for its horrendous traffic congestion and epic commutes. Nearly 10,000 calls to Montgomery and Prince George’s counties did not get through, but Verizon remained oblivious to the failures until the counties complained, according to interviews. Federal regulators described that failure as “truly alarming.”

Verizon said the problem has been corrected. But after the June 29 storm, Montgomery officials said, some 911 lines shut down for a week, and again, Verizon did not notify them. Officials said they discovered the outage days later while reviewing calling data. It took days to restore service to the affected lines, Verizon’s Davis told The Post, “because we just missed” the repair request in the crush of problems after the derecho storm.

“We were not happy about that,” said Bill Ferretti, a Montgomery County 911 official. “We should have known.”

The outage did not affect all of Verizon’s lines into the county call center, so residents’ calls to 911 went through. But Montgomery officials said the incident represented “a notification failure on the part of Verizon,” a complaint made by other regional 911 center managers.

For most of the 11 outages,Verizon has not made public information that establishes how many calls did not reach 911 centers. In other cases, company data supplied to regulators or counties show hundreds of calls backing up when 911 service was limited or out completely.

D.C. area thunderstorms knock out power across region: Homes and businesses across Washington, D.C., Maryland and Virginia lost power June 29 as one of the most powerful and punishing storms in months swept across the region.

Maryland’s Office of People’s Counsel, which represents consumer interests, has proposed fining the company at least $1 million for 911 outages, a penalty that Verizon’s Lewis said was not needed. “We are the good guys here,” Lewis said. “We jump all over these issues. . . . There is no corporate incentive to us not to fix these.”

To investigate the 911 failures, The Post interviewed dozens of experts and reviewed company filings, along with e-mails, letters and other documents submitted to local, state and federal officials. The Post obtained records through public information requests. Officials declined to release others, citing corporate privacy.

The records and interviews indicate that in a number of incidents, repair teams struggled to maintain failing equipment, working under the supervision of managers who reacted slowly because they misunderstood the scope of problems. Backup systems did not kick in. At least one alarm was out of service for a week and was discovered only after a 911 line failed.

Vital equipment that remotely monitors 911 problems in several states shut down during a blackout because it had less than an hour’s worth of backup battery power. Time and again, Verizon did not discover outages until notified by government officials.

Some experts go so far as to suggest that flaws in 911 systems in the Washington area and elsewhere could undermine the multibillion-dollar investment that U.S. taxpayers have made in homeland security programs since Sept. 11, 2001.

“It’s all fine and well to say, ‘If you see something, say something.’ But if you can’t say it to 911, the system breaks down,” homeland security consultant Michael Hopmeier said.

Busy signals

Success for 911 service is measured in seconds, not minutes or hours. Carl Henn’s friends and family know all too well the anguish of hearing a busy signal when a loved one is in need.

In July 2010, a storm came on fast over King Farm Park in upper Montgomery County, where friends had gathered to honor Henn, a well-known environmentalist, for helping to create a community garden.

Other picnickers made it to their cars. Henn did not, and when the rain lifted minutes later, John Burke of Rockville found him on the ground, felled by lightning.

“I tried to call 911,” Burke said, but his wireless call didn’t go through. The crowd tried different phones, different carriers, but got the same busy signals in dozens of attempts, Burke and others said. The group gave up, flagged a passing sport-utility vehicle and rushed Henn, 48, to a hospital. He died two days later.

Burke and Henn’s wife, Carol, aren’t convinced that Henn could have survived the lightning strike. And it’s impossible to determine whether the calls failed because of a Verizon outage or another, unidentified reason.

Montgomery officials initially assumed that the sudden storm prompted a barrage of calls and overloaded their system. But a similar outage six months later sent county officials back to their records, and they concluded that both failures probably stemmed from problems in Verizon’s system.

D.C. area thunderstorms knock out power across region: Homes and businesses across Washington, D.C., Maryland and Virginia lost power June 29 as one of the most powerful and punishing storms in months swept across the region.

Verizon says it does not have records from July 2010, but agreed that a flaw could have blocked some 911 calls during both events.

Verizon later told the FCC that the company had noticed in 2010 that its lines had automatically shut down when hit with a deluge of 911 calls. But Verizon did not diagnose and fix a problem at the root of the outages until after the FCC asked for explanations in early 2011, Verizon and FCC correspondence shows.

“After September 11th, you would think we would have the best system in the country, and it’s pretty clear we don’t,” Burke said.

A summer storm

The Fairfax County 911 center is a large operation that has invested heavily to deliver on the promise “always there, always ready . . . 24/7/365.”

It is among the 10 largest 911 centers in the country, and Fairfax pays $3 milliona year for Verizon 911 service alone, county records show. It pays an additional $18,000 a month to have a Verizon technician on-site to troubleshoot problems, a rare precaution among the nation’s nearly 6,000 emergency 911 centers, experts said.

But when Verizon’s service failed in the derecho, cutting all calls to the 911 center for seven hours, the county resorted to telling residents to go to a firehouse or a police station or to wave down passing emergency workers.

“You don’t want something like this to happen, but believe me, because this happened in the Washington area, this has gotten the attention of 911 centers across the nation,” said Steve Souder, Fairfax’s emergency services director.

As the storm hit, shortly before 11 p.m. June 29, the lights inside Fairfax’s call center briefly flickered before its backup power kicked in.

“We kept on trucking,” Souder said.

Verizon didn’t.

Fairfax, Prince William County, Manassas and Manassas Park lost all 911 service for hours. Those jurisdictions and 21 others in Virginia — extending to the Roanoke and Richmond areas — had intermittent problems with 911 calls that continued at some locations until July 4, state regulators said.

During the first 29 hours, about 1,900 calls were not relayed by Verizon to Fairfax County’s 911 center, according to company data.

When power failed, most of Verizon’s generators for Northern Virginia either did not start or quickly failed, and, later, backup batteries went dead at company facilities. Calls could not go through, and Verizon was unaware of the problem because the system that monitors alarms for 911 outages throughout the Mid-Atlantic also lost battery power.

Verizon technicians did not know how to restart some equipment. One worker checked only part of a Verizon facility, saw that lights were on there and assumed that everything was fine. That assumption delayed 911 repair for several hours.

Alarms indicated that backup batteries were running out of juice, a warning that should have made correcting that situation a priority, according to Verizon’s policy. Verizon had confronted the issue before, telling Maryland regulators last year that it had run tests and done training to be certain that alarms would get a rapid response, state records show.

Fairfax sent teams to its backup emergency center. “We had people going through every office picking up phones on desks” looking for working 10-digit land-line numbers to use for emergency calls until 911 service returned, Souder said. They found two.

D.C. area thunderstorms knock out power across region: Homes and businesses across Washington, D.C., Maryland and Virginia lost power June 29 as one of the most powerful and punishing storms in months swept across the region.

Andrew Duke, a deputy fire chief, worked many of the outage hours, relying on radios to communicate. “I never thought I would come to work and nothing worked.”

At Duke’s fire station in Merrifield, a man drove in to report a utility pole fire, but he was unsure of the street’s name.

“We told him to lead us there,” Duke said, “but drive calm and normal, like nothing is on fire and there weren’t firetrucks behind him.”

Corrective action

Verizon says it has corrected some of the problems brought to light by the June storm, including replacing batteries, repairing generators and clarifying procedures for technicians.

Company officials are trying to ensure that lines meant to be redundant are truly independent of those they back up within Verizon’s network.

Verizon’s regional staff has met with the area’s 911 directors to discuss specific troubles each center faced. The company plans to run blackout simulations and to redesign the system that monitors alarms for the region’s 911 centers to improve its power supply and add auxiliary sites.

The company also pledged to do a better job of staying in touch with 911 centers during outages.

Local government officials said they appreciate those changes but remain skeptical.

Nearly two years ago, Verizon executives told Maryland regulators that they aimed to notify 911 centers within 15 minutes if a 911 line went out, case files show.

“We’ve heard loud and clear” that communication with 911 centers “is extremely important,” Davis told regulators in March 2011. Verizon has “devoted a considerable amount of effort and energy and imagination to how we deliver on that.”

Yet in June, notices about Virginia outages were sent nearly two hours after batteries had drained.

Fairfax County said in an FCC filing, “The string of 9-1-1 failures over the past several years shows that Verizon cannot be relied upon to diagnose and cure its own problems unassisted.”

Unlike with electric utilities, there are nopublicly available benchmarks for comparing the reliability of 911 service providers.

The FCC requires companies to report large outages that hit specific thresholds, including those lasting at least 30 minutes. The causes and remedies also have to be spelled out. But the FCC keeps those filings confidential, partly because of security concerns but also to protect companies’ competitive business information.

The Northern Virginia 911 outages have renewed calls for that information to be shared with state regulators.

Helen Mickiewicz, assistant general counsel at the California Public Utilities Commission, said sharing FCC reports would enhance public safety. “Some service providers really do view themselves as good citizens and follow best practices,” she said, “and others couldn’t care less.”

Verizon has told the FCC that it might be open to sharing reports with states but that the reports should not be used for “measuring service quality or consumer protection.”

FCC Commissioner Jessica Rosenworcel said the federal inquiry into June’s outage is critically important. “You might make this call once, but the old saying goes: It’s the most important call you can ever make.”