NSA risking electrical overload
August 6, 2006
Siobhan Gorman
WASHINGTON -- The National Security Agency is running out of juice.
The demand for electricity to operate its expanding intelligence systems has left the high-tech eavesdropping agency on the verge of exceeding its power supply, the lifeblood of its sprawling 350-acre Fort Meade headquarters, according to current and former intelligence officials.
Agency officials anticipated the problem nearly a decade ago as they looked ahead at the technology needs of the agency, sources said, but it was never made a priority, and now the agency's ability to keep its operations going is threatened. The NSA is already unable to install some costly and sophisticated new equipment, including two new supercomputers, for fear of blowing out the electrical infrastructure, they said.
At minimum, the problem could produce disruptions leading to outages and power surges at the Fort Meade headquarters, hampering the work of intelligence analysts and damaging equipment, they said. At worst, it could force a virtual shutdown of the agency, paralyzing the intelligence operation, erasing crucial intelligence data and causing irreparable damage to computer systems -- all detrimental to the fight against terrorism.
Estimates on how long the agency has to stave off such an overload vary from just two months to less than two years. NSA officials "claim they will not be able to operate more than a month or two longer unless something is done," said a former senior NSA official familiar with the problem, who spoke on condition of anonymity.
Agency leaders, meanwhile, are scrambling for stopgap measures to buy time while they develop a sustainable plan. Limitations of the electrical infrastructure in the main NSA complex and the substation serving the agency, along with growing demand in the region, prevent an immediate fix, according to current and former government officials.
"If there's a major power failure out there, any backup systems would be inadequate to power the whole facility," said Michael Jacobs, who headed the NSA's information assurance division until 2002.
"It's obviously worrisome, particularly on days like today," he said in an interview during last week's barrage of triple-digit temperatures.
William Nolte, a former NSA executive who spent decades with the agency, said power disruptions would severely hamper the agency.
"You've got an awfully big computer plant and a lot of precision equipment, and I don't think they would handle power surges and the like really well," he said. "Even re-calibrating equipment would be really time consuming -- with lost opportunities and lost up-time."
Power surges can also wipe out analysts' hard drives, said Matthew Aid, a former NSA analyst who is writing a multivolume history of the agency. The information on those hard drives is so valuable that many NSA employees remove them from their computers and lock them in a safe when they leave each day, he said.
A half-dozen current and former government officials knowledgeable about the energy problem discussed it with The Sun on condition of anonymity because of the sensitivity of the issue.
NSA spokesman Don Weber declined to comment on specifics about the NSA's power needs or what is being done to address them, saying that even private companies consider such information proprietary.
In a statement to The Sun, he said that "as new technologies become available, the demand for power increases and NSA must determine the best and most economical way to use our existing power and bring on additional capacity."
Biggest BGE customer
The NSA is Baltimore Gas & Electric's largest customer, using as much electricity as the city of Annapolis, according to James Bamford, an intelligence expert and author of two comprehensive books on the agency.
BGE spokeswoman Linda Foy acknowledged a power company project to deal with the rising energy demand at the NSA, but she referred questions about it to the NSA.
The agency got a taste of the potential for trouble Jan. 24, 2000, when an information overload, rather than a power shortage, caused the NSA's first-ever network crash. It took the agency 3 1/2 days to resume operations, but with a power outage it could take considerably longer to get the NSA humming again.
The 2000 shutdown rendered the agency's headquarters "brain-dead," as then-NSA Director Gen. Michael V. Hayden told CBS's 60 Minutes in 2002.
"I don't want to trivialize this. This was really bad," Hayden said. "We were dark. Our ability to process information was gone."
As an immediate fallback measure, the NSA sent its incoming data to its counterpart in Great Britain, which stepped up efforts to process the NSA's information along with its own, said Bamford.
The agency came under intense criticism from members of Congress after the crash, and the incident rapidly accelerated efforts to modernize the agency.
One former NSA official familiar with the electricity problem noted a sense of deja vu six years later.
"To think that this was not a priority probably tells you more about the extent to which NSA has actually transformed," the former official said. "In the end, if you don't have power, you can't do [anything]."
Already some equipment is not being sufficiently cooled, and agency leaders have forgone plugging in some new machinery, current and former government officials said. The power shortage will also delay the installation of two new, multimillion-dollar supercomputers, they said.
To begin to alleviate pressure on the electrical grid, the NSA is considering buying additional generators and shutting down so-called "legacy" computer systems that are decades old and not considered crucial to the agency's operations, said three current and former government officials familiar with the situation.
"It's a temporary fix," one former senior NSA official said.
On Wednesday, the same day that The Sun inquired about the power issue with the NSA's public affairs office, the agency sent word to Capitol Hill about its energy conservation efforts.
"They have told us they have been shutting down all non-essential uses of power to help out BG&E;," said one congressional aide, adding that the NSA is also raising the temperature in its buildings two degrees to conserve.
The information was presented in the context that the NSA was making these changes "to be a good corporate citizen," the aide said.
Contractors on at least one high-priority, power-intensive NSA project that is located off the headquarters campus, have upgraded their electrical infrastructure to ensure power for their project, according to two former agency officials. That lone upgrade, a fraction of the agency's total demand, took four months.
Longer-term solutions being considered would move some operations to off-campus facilities with more electrical capacity, current and former officials said.
Adding more capacity to the substation feeding NSA is an obvious answer, but constraints on that particular facility make an expansion difficult, they said. BGE's Foy declined to discuss specifics about the substation. She said it takes 1 1/2 to 2 1/2 years to design, procure equipment, obtain permits, and build a new one.
Post-9/11 needs
Since the 2001 terrorist attacks, the NSA has ramped up its operations, and the electricity needed to sustain major projects -- such as the warrantless surveillance program and technology modernization programs -- has increased sharply.
The computer systems supporting these programs demand far more wattage per square foot than their predecessors and still more energy to cool them.
Area development like the Arundel Mills mall has contributed to the problem by putting additional strain on the local electrical grid, according to two sources familiar with the issue. Joe Bunch, BGE's director of strategic customer engineering, said, however, that the mall's demand "was fairly easily accommodated."
Demand in the Baltimore-Washington region has been growing, and the regional operator for Maryland and 12 other states has been studying the installation of up to $10 billion in new power lines to deliver more and cheaper electricity to this region.
"We've seen a lot of growth in Anne Arundel County as a whole but particularly in the north and northwest area of the county," said Bunch, who agreed to talk about trends in the area but not the NSA's specific demand. Much of that growth is because of the surge of high-tech jobs in the area from the NSA and government contractors, he said.
He said BGE is working to meet the demand by building new substations in the area. One was built about a year ago, and another is scheduled to be built in two to three years, he said.
"We have adequate capacity" now, he said, but upgrades like the new substation are being planned to stave off future strains on the electrical grid.
The NSA's problem was identified in the late 1990s and could have been fixed by now -- and for much less money -- had keeping the lights on been a priority, current and former officials said.
"It fits into a long, long pattern of crisis-of-the-day management as opposed to investing in the future," said one former government official familiar with the NSA's electricity shortfall.
Electrical infrastructure maintenance and upgrades have been a casualty of the fight against terrorism, according to unclassified budget documents.
Upgrades delayed
Even as the NSA's budget has ballooned after 9/11, the agency has put off basic utility upgrades such as a $4 million computer system to manage the allocation of power at the NSA -- a sliver of the NSA's estimated $8 billion budget.
"Due to budget constraint [sic] and other development [sic] in the fight against terrorism," a 2007 budget document reads, the system was never fully implemented.
Without this system, the document stated, the NSA "may experience difficulties in meeting its power requirement to support critical war fighting missions."
Neglect of infrastructure at the NSA has been a chronic problem, often fraught with bureaucratic politics, former agency officials said.
Fort Meade is not the only NSA outpost facing limitations on its ability to upgrade electrical infrastructure. Listening posts around the world, such as Menwith Hill in Britain and Bad Aibling in Germany, are ailing.
The NSA's largest listening station, Menwith Hill, has an "aging infrastructure that cannot support the people or equipment" there, according to a budget document for 2007.
It is faced with "concrete foundations that are crumbling," an "electrical infrastructure that is not in compliance with current codes," and a weakened infrastructure that poses a safety hazard, the document said.
Identical language appeared in the previous year's budget documents.
With agency operations facing an imminent threat, facilities issues are front and center. "It's a big deal," said one former senior NSA official. "They're all talking about it, anyway. That's progress."
siobhan.gorman@baltsun.com
Sun reporter Paul Adams contributed to this article.