System Error
The NSA has spent six years and hundreds of millions of dollars trying to kick-start a program, intended to help protect the United States against terrorism, that many experts say was doomed from the start.
A Sun Special Report
January 29, 2006
By SIOBHAN GORMAN
A program that was supposed to help the National Security Agency pluck out electronic data crucial to the nation's safety is not up and running more than six years and $1.2 billion after it was launched, according to current and former government officials.
The classified project, code-named Trailblazer, was promoted as the NSA's state-of-the art tool for sifting through an ocean of modern-day digital communications and uncovering key nuggets to protect the nation against an ever-changing collection of enemies.
Its main goal when it was launched in 1999 was to enable NSA analysts to connect the 2 million bits of data the agency ingests every hour - a task that has grown increasingly complex with the advent of the Internet, cell phones, and instant messaging - and enable analysts to quickly pick out the most important information.
The stakes could scarcely be higher.
A major failure leading up to Sept. 11, 2001, involved communications intelligence, investigators found. More than 30 hints of the impending attack had been collected in the previous three years but had sat, unnoted, in the NSA's databases, according to a joint congressional inquiry into pre-Sept. 11 intelligence operations.
The NSA initiative, which was designed to spot and analyze such hints, has resulted in little more than detailed schematic drawings filling almost an entire wall, according to intelligence experts familiar with the program. After an estimated $1.2 billion in development costs, only a few isolated analytical and technical tools have been produced, said an intelligence expert with extensive knowledge of the program.
Trailblazer is "the biggest boondoggle going on now in the intelligence community," said Matthew Aid, who has advised three recent federal commissions and panels that investigated the Sept. 11 intelligence failures.
Complex from the start - the initial Trailblazer plan called for more than 1,000 priority items - the project ballooned as it was passed through three separate NSA divisions, each with its own priorities, former intelligence officials said. And, they said, Trailblazer's overseers lacked either the influence or the time to clearly define their goals and keep the project on track.
When the agency's inspector general looked at the NSA's handling of the project in its first three years, it found in a 2003 report "inadequate management and oversight" of private contractors and overpayment for the work that was done, according to a recently declassified version of the report obtained by The Sun through a Freedom of Information Act request.
Meanwhile, Science Applications International Corp., the lead contractor on the project, did not provide enough people with the technical or management skills to produce such a sophisticated system, according to industry and NSA experts familiar with Trailblazer. And, they said, the company did not say no when the NSA made unrealistic demands.
The company was initially awarded $280 million in 2002 to begin construction.
SAIC spokesman Jared Adams declined to comment, saying, "We have been asked to defer all comment regarding the NSA Trailblazer contract to the NSA."
The reporting in this article includes interviews conducted over the past three months with 25 intelligence professionals, 13 of whom worked on or had oversight of Trailblazer. Because the program is classified, most would not allow their names to be used.
Although the Bush administration spent much of the past week defending the NSA's eavesdropping work as vital to keeping Americans safe from terrorism, virtually no attention has been paid to the agency's failure to deliver the system the NSA said was key to fulfilling that mission.
That means the government has been standing by while the agency has been gradually "going deaf" as unimportant communications drown out key pieces of information, a government official with extensive knowledge of Trailblazer told The Sun.
NSA spokesman Don Weber said the agency would have no response to requests for comment.
Listening in
Based at Fort Meade in Anne Arundel County and with field offices around the world, the NSA harvests virtually every form of electronic communication - including phone calls, e-mails, video links and bank transactions - through a vast array of satellites, clandestine posts at U.S. embassies, ground-based listening stations, and military airplanes, ships and submarines.
The information collected and culled by the agency's approximately 40,000 employees accounts for an estimated 75 percent of the president's daily intelligence briefing, said Aid, an intelligence consultant who is writing a multivolume history of the NSA.
But there are huge holes in the agency's information filter. As a result, a congressional report on 9/11 intelligence failures found, "potentially vital" information is lost, particularly with regard to terrorist groups.
That is what Trailblazer was designed to fix.
All digital communications trapped by the NSA are transmitted to the agency's offices in computer codes of zeroes and ones. The sheer volume of data being gathered is overwhelming the NSA's ability to digest it. And that volume is growing every day with the advent of text messages, hand-held computers such as the BlackBerry and phone conversations over Internet lines.
The result is akin to looking for a needle in a haystack that doubles in size every few months, said Aid, who has written extensively on intelligence issues.
The agency has only blunt tools - largely based on the information's origin or keywords linked to items of interest - to use in making decisions about whether to keep captured data or discard it.
Intelligence experts familiar with the system said it is like deciding whether to keep a piece of mail or throw it out based only on what is on the outside of the envelope.
An estimated 95 percent of the information gathered is discarded without being translated into an understandable form, said an intelligence expert who has tracked the system for years. The remaining 5 percent, still in the form of zeroes and ones, is turned into plain text or voice recordings and routed to the appropriate division for analysis.
In each division, it might be run through software programs to identify patterns or links with other data. But that is not guaranteed. Nor is there a guarantee that a communication sent to a division dealing with Latin America, for example, will ever be seen by an analyst tracking a terrorist group that finances its activities through Latin American drug smuggling.
NSA officials knew they needed to make changes in how they handled the deluge of digital data and spent a year developing a broad concept for how to do so.
As initially envisioned, said four intelligence experts with extensive knowledge of the project, Trailblazer would have translated all of the digital computer language (the zeroes and ones) into plain text or voice. The data would have been analyzed to identify new patterns of activity or connections among people whose communications are intercepted, and then been stored in an easily searchable database. Key communications automatically would have been forwarded to the appropriate analysts, who for the first time could have followed up with their own searches of the database.
To implement Trailblazer, the NSA would have vastly expanded its computing hardware and software, and made revolutionary changes in the way huge amounts of data are stored and retrieved.
But years after the initiative was launched, there was still no unanimity within the agency on how to achieve those goals, or even on whether all of them were necessary or possible, interviews and records show.
A December 2002 report by the House and Senate intelligence committees investigating pre-Sept. 11 intelligence failures found that although Trailblazer was "frequently cited" as the solution to many of the NSA's information management problems, "implementation of those solutions is three to five years away and confusion still exists as to what will actually be provided by the program."
The unclassified report also noted that without Trailblazer the NSA employees with whom committee investigators spoke knew of no "near-term efforts to alleviate their current system's technical limitations."
Another division of the NSA had been working on a separate, less expensive program, code-named Thinthread. In development before Trailblazer was launched, Thinthread tried to accomplish a similar goal of separating the important communications from the junk.
A classified report from the Pentagon in 2004 found that Thinthread was more promising than Trailblazer and could be put to use faster, said an intelligence expert who was briefed on its contents.
NSA managers disagreed with the Pentagon report's conclusions and canceled Thinthread, said the expert briefed on the report's contents.
As a result, nearly 4 1/2 years after the Sept. 11 attacks, the NSA still lacks a system to comprehensively evaluate all of the communications collected by its vast networks of high-tech ears.
Blazing trails
Trailblazer began as a signature program of Gen. Michael V. Hayden, who was the NSA's director from March 1999 until last spring. Early on, former officials familiar with the program said, it became clear to Hayden that the agency, with its rich history of developing cutting-edge technology, was falling behind the technology curve. He cast Trailblazer as the agency's future.
A cerebral Air Force general with close-cropped hair and wire-rimmed glasses, Hayden saw his tenure as a key opportunity to turn the agency around. In November 1999, he made Trailblazer a centerpiece of his "100 Days of Change" agenda.
Presented nearly two years before the 9/11 attacks, former colleagues noted, Hayden was prescient.
"It was going to structure us to handle the digital revolution," said a former intelligence official. And, the official recalled, it would start by building on the agency's existing computer systems.
But two months after Trailblazer was launched, the agency's computers had a 3 1/2 -day meltdown. Hayden later described the episode in a 60 Minutes II interview as the agency's headquarters going "brain dead."
"It doesn't take a rocket scientist to know that an [information technology] infrastructure that crashes ... is not going to be able to handle Trailblazer applications," said a former intelligence official familiar with the program.
Hayden declined, through a spokeswoman, to comment for this article and referred questions to the NSA.
Several former intelligence and government oversight officials contend that Trailblazer was doomed almost from its inception. The program "kicked off with not a real great definition of what it was trying to achieve," said a government oversight official, recalling an initial briefing in December 1999.
Trailblazer began with such a burst of energy that it skipped some crucial first steps, said current and former government officials close to the program.
For example, to make sense of the communications it pulled in, Trailblazer needed a standard format for all data so that it could sort it properly, much like the standards Google uses so that it can search different kinds of information on the Internet. But intelligence officials said those standards were never defined.
The agency also boxed itself in by underestimating how long it should keep old data, a former national security official said. As a result, the system was designed to discard information that could later prove useful, particularly in an open-ended war on terror.
Such early errors were exacerbated by the Sept. 11 attacks, which prompted Hayden to push for faster implementation, eliminating time for review and corrections, a former intelligence official said. And Congress began throwing money at Trailblazer, discouraging a more disciplined approach, said a former government official with extensive knowledge of the program.
Monitoring project
While internal and external warnings that Trailblazer was going off course were sounded, the extent of its problems gained little public attention because the program was so secret and technical.
Since 1999, for example, more than 10 unclassified congressional reports have pointed to "deficiencies" in NSA modernization efforts, including two that specifically pointed to problems with Trailblazer.
A 2003 NSA inspector general's report obtained by The Sun found that the spy agency was unable to monitor the progress or the results of its early Trailblazer contractors. Moreover, Inspector General Joel Brenner said his office could find no evidence of the program's specific priorities, could not track the ways all of the money was being spent and found that the NSA had overpaid some contract employees. The contracts showed no limits on labor costs.
"These conditions are directly related to inadequate management and oversight," Brenner's team said.
An intelligence expert who was briefed on a December 2004 report conducted by the Pentagon's inspector general said the report found that Trailblazer was not producing the system that had been promised and was unlikely to produce it. The intelligence expert said the report suggested agencywide management problems and recommended further investigations of the NSA's overall acquisition and financial systems, at least one of which has begun.
That report remains classified.
Shortly after the inspector general's review was completed, the NSA hired IBM to take the lead on the project, said intelligence sources familiar with the program.
Despite such warnings of problems, the Government Accountability Office, the investigative arm of Congress that specializes in assessing program management and government waste, has not looked into Trailblazer. Randolph Hite, the GAO's director of information technology, said no one in Congress has asked it to.
Former Sen. Bob Graham, a Florida Democrat who chaired the Senate Intelligence Committee during and after the Sept. 11 attacks, said Congress has failed to provide adequate oversight of the NSA.
"Most of the members don't have the background or the expertise to understand very well an organization like the NSA," Graham said.
It did not help that Trailblazer, then less than two years old, was faltering just as the country and Congress were attempting to cope with the deaths of an estimated 3,000 people in the Sept. 11 attacks. Congress was not interested in cutting any program related to fighting terrorism for fear that it would be blamed if terrorists struck again.
From that point, "our overwhelming focus was on trying to understand that tragedy and the role the intelligence agencies had played," Graham said. "Then, in 2002, summer and fall was the run-up to the war in Iraq, so our attention was diverted."
That meant that on Capitol Hill, much of the oversight was left to the few congressional staff members who understood the program. In July 2003, they persuaded their bosses to send the NSA a no-confidence message about the agency's ability to manage complex programs such as Trailblazer, said congressional aides.
Congress took away the NSA's authority to sign big-ticket contracts without getting permission from the Department of Defense. The Defense Department continues to be responsible for approving the NSA's proposals to pursue and pay for large programs.
But that didn't stop or fix the project. At the NSA, Trailblazer continued to stumble along.
Rep. C.A. Dutch Ruppersberger, a Democrat from Maryland who sits on the House Intelligence Committee and whose district includes NSA headquarters, said abandoning the concept behind Trailblazer is not an option because "our national security depends on it."
"There was congressional oversight, and that's one of the reasons this program has been red-flagged as a program that needs work," he said. "The conclusion we all had was there were mistakes made, but the concept has to move forward for the sake of our national security."
Sen. John D. Rockefeller IV of West Virginia, who succeeded Graham as the senior Democrat on the Senate Intelligence Committee, said the committee has "worried about the direction of the Trailblazer program specifically and NSA acquisition practices generally" over the past few years.
"We and the Armed Services Committee required detailed reports, withheld money and ultimately removed the NSA's acquisition authority," Rockefeller said. "I expect that [NSA Director Lt. Gen. Keith B.] Alexander will have a major restructuring plan to present to the committee as part of the request for fiscal year 2007."
Turf battles
To understand where Trailblazer ended up, it is helpful to understand the internal politics at the time of its launch.
The beginning of the Trailblazer program in 1999 coincided with a major NSA reorganization. Deep inside the spy agency, knowledgeable bureaucrats rushed to get their programs redefined as part of a project favored by the director and presumably immune from budget cuts, two former intelligence officials said.
As a result, they said, Trailblazer's scope mushroomed in the first few years.
Meanwhile, the project was passed among several divisions, including the new Transformation Office, which was shut down after a year; the Signals Intelligence Directorate, which housed many of the analysts; and the Information Technology Directorate, which builds technology systems.
Each time Trailblazer was moved, its new leaders altered its design.
"Every year or so, their story would be somewhat different about what it is, what it's going to accomplish and how it is going to be implemented," said a congressional aide who works with intelligence programs.
The program landed in the lap of William B. Black Jr., the agency's deputy director.
Black had spent four decades at the NSA before leaving it in 1997 to join SAIC, a San Diego-based contractor with strong ties to the agency. In 2000, Hayden called him back to become his top deputy and to take charge of Trailblazer.
Two years later, the NSA awarded the prime contract to build Trailblazer to SAIC, Black's former employer.
A careful bureaucrat who shies away from the media, Black was an expert at navigating the agency's many fiefdoms and insisted that he make all key decisions about Trailblazer, said intelligence officials with extensive knowledge of the program. But they said Black had too much on his plate to pay close attention to the program.
NSA spokesman Weber said Black was not available for an interview.
Lax internal oversight and shifting priorities quickly sent Trailblazer's costs skyrocketing. In April 2005 Hayden testified before Congress that the program, with publicly announced contracts then worth $500 million, was "a couple to several hundred million" dollars over budget and behind schedule.
Although he didn't provide details of the program's troubles, he acknowledged in his testimony that getting the program off the ground "was far more difficult than anyone anticipated."
New direction?
Five months into his tenure as NSA director, Alexander has been reviewing Trailblazer, and he recently decided that he will try to revamp it rather than scrap it, according to three intelligence experts familiar with the program.
These officials and others knowledgeable about the program's history said they were skeptical that Trailblazer could be fixed without starting from scratch.
"Trailblazer is completely beyond fixing," said a former government official who has tracked the program carefully. "Everybody who reviewed Trailblazer after the first few months [of the program's launch] said it was doomed or it should be scrapped."
Bobby Ray Inman, a former NSA director and a retired admiral, said there needs to be some tolerance for altering the course of ambitious projects.
Several projects he considered "successes," he said, were scaled-back versions of an initial vision that "took us forward from where we were, but it really didn't meet the aspirations of what we would have liked to have had."
Alexander plans to hire a new executive to run the NSA's technology programs, and Trailblazer will be one of this executive's top priorities, said an intelligence consultant.
Alexander declined to comment for this article, but in August he told The Sun that he would look to shift the agency's approach away from large programs such as Trailblazer and toward smaller programs that build on each other.
"I think the way to do it efficiently is smaller steps, more rapidly done, rather than try to take one big jump and make it all the way across," he said.
Those steps would involve significant changes in the way the NSA manages data, including, he said, "how you handle data, how you visualize that data and how we jump from industrial age analysis to the information age analysis that our country needs."
Intelligence experts with extensive knowledge of the program said Alexander is likely to salvage what he can from Trailblazer and largely start over, casting it as a kind of "Trailblazer 2.0."
The country's new spymaster, Director of National Intelligence John D. Negroponte, is taking on the job of connecting the technology systems of all 15 intelligence agencies, and former intelligence officials said Trailblazer's troubles should serve as a cautionary tale.
If Negroponte wants to learn the details, he won't have to go far. Since last spring, his top deputy has been Hayden, the former NSA chief.