(Undated), Panorama: NSA: XKeyscore rules


NSA targets the privacy-conscious

von J. Appelbaum, A. Gibson, J. Goetz, V. Kabisch, L. Kampf, L. Ryge

The investigation discloses the following:

* Two servers in Germany - in Berlin and Nuremberg - are under surveillance by the NSA.

* Merely searching the web for the privacy-enhancing software tools outlined in the XKeyscore rules causes the NSA to mark and track the IP address of the person doing the search. Not only are German privacy software users tracked, but the source code shows that privacy software users worldwide are tracked by the NSA.

* Among the NSA's targets is the Tor network funded primarily by the US government to aid democracy advocates in authoritarian states.

* The XKeyscore rules reveal that the NSA tracks all connections to a server that hosts part of an anonymous email service at the MIT Computer Science and Artificial Intelligence Laboratory (CSAIL) in Cambridge, Massachusetts. It also records details about visits to a popular internet journal for Linux operating system users called "the Linux Journal - the Original Magazine of the Linux Community", and calls it an "extremist forum".

Three authors of this investigation have personal and professional ties to the Tor Project, an American company mentioned within the following investigation. Jacob Appelbaum is a paid employee of the Tor Project, Aaron Gibson is a paid contractor for the Tor Project, and Leif Ryge is a volunteer contributor to various Tor-related software projects. Their research in this story is wholly independent from the Tor Project and does not reflect the views of the Tor Project in any way. During the course of the investigation, it was further discovered that an additional computer system run by Jacob Appelbaum for his volunteer work with helping to run part of the Tor network was targeted by the NSA. Moreover, all members of this team are Tor users and appear to be have been targets of the mass surveillance described in the investigation.
It is a small server that looks like any of the other dozens in the same row. It is in a large room devoted to computers and computer storage, just like every other room in this industrial park building on Am Tower Street just outside the city of Nuremberg. That the grey building is surrounded by barbed wire seems to indicate that the servers' provider is working hard to secure their customers' data.

Yet despite these efforts, one of the servers is targeted by the NSA.

The IP address is explicitly specified in the rules of the powerful and invasive spy software program XKeyscore. The code is published here exclusively for the first time.

After a year of NSA revelations based on documents that focus on program names and high-level Powerpoint presentations, NDR and WDR are revealing NSA source code that shows how these programs function and how they are implemented in Germany and around the world.

Months of investigation by the German public television broadcasters NDR and WDR, drawing on exclusive access to top secret NSA source code, interviews with former NSA employees, and the review of secret documents of the German government reveal that not only is the server in Nuremberg under observation by the NSA, but so is virtually anyone who has taken an interest in several well-known privacy software systems.

The NSA program XKeyscore is a collection and analysis tool and "a computer network exploitation system", as described in an NSA presentation. It is one of the agency's most ambitious programs devoted to gathering "nearly everything a user does on the internet." The source code contains several rules that enable agents using XKeyscore to surveil privacy-conscious internet users around the world. The rules published here are specifically directed at the infrastructure and the users of the Tor Network, the Tails operating system, and other privacy-related software.

Tor, also known as The Onion Router, is a network of several thousand volunteer-operated servers, or nodes, that work in concert to conceal Tor users' IP addresses and thus keep them anonymous while online.

Tails is a privacy-focused GNU/Linux-based operating system that runs entirely from an external storage device such as a USB stick or CD. It comes with Tor and other privacy tools pre-installed and configured, and each time it reboots it automatically wipes everything that is not saved on an encrypted persistent storage medium.

Normally a user's online traffic - such as emails, instant messages, searches, or visits to websites - can be attributed to the IP address assigned to them by their internet service provider. When a user goes online over the Tor Network, their connections are relayed through a number of Tor nodes using another layer of encryption between each server such that the first server cannot see where the last server is located and vice-versa.

Tor is used by private individuals who want to conceal their online activity, human rights activists in oppressive regimes such as China and Iran, journalists who want to protect their sources, and even by the U.S. Drug Enforcement Agency in their efforts to infiltrate criminal groups without revealing their identity. The Tor Project is a non-profit charity based in Massachusetts and is primarily funded by government agencies. Thus it is ironic that the Tor Network has become such a high-priority target in the NSA's worldwide surveillance system.

As revealed by the British newspaper The Guardian, there have been repeated efforts to crack the Tor Network and de-anonymize its users. The top secret presentations published in October last year show that Tor is anathema to the NSA. In one presentation, agents refer to the network as "the king of high-secure, low-latency internet anonymity". Another is titled "Tor Stinks". Despite the snide remarks, the agents admit, "We will never be able to de-anonymize all Tor users all the time".

The former NSA director General Keith Alexander stated that all those communicating with encryption will be regarded as terror suspects and will be monitored and stored as a method of prevention, as quoted by the Frankfurter Allgemeine Zeitung in August last year. The top secret source code published here indicates that the NSA is making a concerted effort to combat any and all anonymous spaces that remain on the internet. Merely visiting privacy-related websites is enough for a user's IP address to be logged into an NSA database.

An examination of the XKeyscore rules published here goes beyond the slide presentation and provides a window into the actual instructions given to NSA computers. The code was deployed recently and former NSA employees and experts are convinced that the same code or similar code is still in use today. The XKeyscore rules include elements known as "appids", "fingerprints", and "microplugins". Each connection a user makes online - to a search engine, for example - can be assigned a single appid and any number of fingerprints.

Appids are unique identifiers for a connection in XKeyscore. Appid rules have weights assigned to them.  When multiple appids match a given connection, the one with the highest weight is chosen. Microplugins may contain software written in general-purpose programming languages, such as C++, which can extract and store specific types of data. The rules specifically target the Tor Project's email and web infrastructure, as well as servers operated by key volunteers in Germany, the United States, Sweden, Austria, and the Netherlands. Beyond being ethically questionable, the attacks on Tor also raise legal concerns.  The IP addresses of Tor servers in the United States are amongst the targets, which could violate the fourth amendment of the US constitution.

The German attorney Thomas Stadler, who specializes in IT law, commented: "The fact that a German citizen is specifically traced by the NSA, in my opinion, justifies the reasonable suspicion of the NSA carrying out secret service activities in Germany. For this reason, the German Federal Public Prosecutor should look into this matter and initiate preliminary proceedings."

One of NSA's German targets is  The string of numbers is an IP address assigned to Sebastian Hahn, a computer science student at the University of Erlangen. Hahn operates the server out of a grey high-security building a few kilometers from where he lives. Hahn, 28 years old and sporting a red beard, volunteers for the Tor Project in his free time. He is especially trusted by the Tor community, as his server is not just a node, it is a so-called Directory Authority. There are nine of these worldwide, and they are central to the Tor Network, as they contain an index of all Tor nodes. A user's traffic is automatically directed to one of the directory authorities to download the newest list of Tor relays generated each hour.

"anonymizer/tor/node/authority" fingerprint.

Hahn's predecessor named the server Gabelmoo, or Fork Man, the nickname of a local statue of Poseidon. After a look at the NSA source code, Hahn quickly found his server's name listed in the XKeyscore rules. "Yes, I recognize the IP address of my Tor server called 'gabelmoo'." he said. "Millions of people use it to stay safe online, and by watching the server and collecting metadata about its users, those people are put at risk." The rule shown to Hahn, published below, has a fingerprint called 'anonymizer/tor/node/authority'. The fingerprint targets users who connect to Gabelmoo and other Tor Directory Authority servers. In Germany, the Tor Directory Authorities like Gabelmoo that are specifically targeted by XKeyscore rules are in Berlin and Nuremberg. Additional targets are located in Austria, Sweden, the United States, and the Netherlands.

Fragments of XKeyscore rules targetting Tor directory authorities.

The expression below performs essentially the same function, but it specifies the Tor directory authorities located in Five Eyes countries (Australia, Canada, New Zealand, the United Kingdom and the United States) separately from those in other countries. As the comment explains, the "goal is to find potential Tor clients connecting to the Tor directory servers."

Another rule catalogs users connecting to known Tor relays. This is not difficult, because the addresses of all normal Tor relays are published by the directory authorities so that the Tor software on users' computers can select its own path through the network. In addition to the public relays, connections characterized as Tor based on protocol identifiers are also cataloged.

Not only Metadata

Internet service providers in countries with strong censorship such as China and Iran frequently block connections to known Tor relays. To avoid this blocking, The Tor Project maintains a list of non-public relays called "bridges" to allow users to avoid this type of blocking. Bridges are run by volunteers and they share the details with the Tor Project to help censored users reach the internet.

Microplugin which extracts bridge addresses from the full text of Tor Project emails.

Users can request a bridge address via email or on the web. The following fingerprints show two ways that XKeyscore attempts to track Tor bridge users. First, the fingerprint "anonymizer/tor/bridge/tls" records connections to the server. Second, in order obtain the actual bridge addresses for the purpose of tracking connections to them in the future, the "microplugin" fingerprint called "anonymizer/tor/bridge/email" extracts data from the body of the emails that the Tor Project sends to its users.

This code demonstrates the ease with which an XKeyscore rule can analyze the full content of intercepted connections. The fingerprint first checks every message using the "email_address" function to see if the message is to or from "". Next, if the address matched, it uses the "email_body" function to search the full content of the email for a particular piece of text - in this case, "". If the "email_body" function finds what it is looking for, it passes the full email text to a C++ program which extracts the bridge addresses and stores them in a database.

The full content of the email must already be intercepted before this code can analyze it. XKeyscore also keeps track of people who are not using Tor, but who are merely visiting The Tor Project's website (, as this rule demonstrates:

Fingerprint to identify visitors to the Tor Project website.

The full content of the email must already be intercepted before this code can analyze it. XKeyscore also keeps track of people who are not using Tor, but who are merely visiting The Tor Project's website (, as this rule demonstrates:

Rules targeting people viewing the Tails or Linux Journal websites, or performing Tails-related web searches.

It is interesting to note that this rule specifically avoids fingerprinting users believed to be located in Five Eyes countries, while other rules make no such distinction. For instance, the following fingerprint targets users visiting the Tails and Linux Journal websites, or performing certain web searches related to Tails, and makes no distinction about the country of the user.

The comment in the  source code above describes Tails as "a comsec mechanism advocated by extremists on extremist forums". In actuality, the software is used by journalists, human rights activists, and hundreds of thousands of ordinary people who merely wish to protect their privacy.

The rules related to Tails clearly demonstrate how easily web searches and website visits can be spied on by XKeyscore. On June 25, 2014, the United States Supreme Court noted how sensitive this type of information is in their Riley v. California decision against warrantless searches of mobile phones: "An Internet search and browsing history [...] could reveal an individual's private interests or concerns - perhaps a search for certain symptoms of disease, coupled with frequent visits to WebMD."

C++ program which searches "raw traffic" for .onion addresses.

In addition to anonymous internet access, Tor also provides a mechanism for hosting anonymous internet services called "Hidden Services". These sites' URLs contain a domain name in the pseudo-top-level-domain ".onion" which is only accessible using Tor. The code shown below finds and catalogs URLs for these sites which XKeyscore sees in "raw traffic", creating a unique fingerprint for each onion address. Each .onion address found in raw traffic is extracted and stored in an NSA database:

"anonymizer/mailer/mixminion" appid matching all connections to

There are also rules that target users of numerous other privacy-focused internet services, including HotSpotShield, FreeNet, Centurian,, MegaProxy, and an anonymous email service called MixMinion as well as its predecessor MixMaster. The appid rule for MixMinion is extremely broad as it matches all traffic to or from the IP address, a server located on the MIT campus.

That server is operated by the Tor Project's leader Roger Dingledine, an MIT alumnus. The machine at this IP address provides many services besides MixMinion, and it is also one of the above-mentioned Tor directory authorities. Dingledine said "That computer hosts many websites, ranging from open source gaming libraries to the Privacy Enhancing Technologies Symposium website."

Sebastian Hahn, the Tor volunteer who runs Gabelmoo, was stunned to learn that his hobby could interest the NSA: "This shows that Tor is working well enough that Tor has become a target for the intelligence services. For me this means that I will definitely go ahead with the project."

When asked for a reaction to the findings, the Tor Project's Roger Dingledine stated the following: "We've been thinking of state surveillance for years because of our work in places where journalists are threatened. Tor's anonymity is based on distributed trust, so observing traffic at one place in the Tor network, even a directory authority, isn't enough to break it. Tor has gone mainstream in the past few years, and its wide diversity of users - from civic-minded individuals and ordinary consumers to activists, law enforcement, and companies - is part of its security. Just learning that somebody visited the Tor or Tails website doesn't tell you whether that person is a journalist source, someone concerned that her Internet Service Provider will learn about her health conditions, or just someone irked that cat videos are blocked in her location. Trying to make a list of Tor's millions of daily users certainly counts as wide scale collection. Their attack on the bridge address distribution service shows their 'collect all the things' mentality - it's worth emphasizing that we designed bridges for users in countries like China and Iran, and here we are finding out about attacks by our own country. Does reading the contents of those mails violate the wiretap act? Now I understand how the Google engineers felt when they learned about the attacks on their infrastructure."

NDR and WDR wanted to know from the NSA how it justified attacking a service funded by the U.S. government, under what legal authority Tor Network users are monitored, and whether the German government has any knowledge of the targeting of servers in Germany. Instead of adressing the questions repeatedly posed to them, the NSA provided the following statement: "In carrying out its mission, NSA collects only what it is authorized by law to collect for valid foreign intelligence purposes - regardless of the technical means used by foreign intelligence targets. The communications of people who are not foreign intelligence targets are of no use to the agency. In January, President Obama issued U.S. Presidential Policy Directive 28, which affirms that all persons - regardless of nationality - have legitimate privacy interests in the handling of their personal information, and that privacy and civil liberties shall be integral considerations in the planning of U.S. signals intelligence activities. The president's  directive also makes clear that the United States does not collect signals intelligence for the purpose of suppressing or burdening criticism or dissent, or for disadvantaging persons based on their ethnicity, race, gender, sexual orientation, or religion. XKeyscore is an analytic tool that is used as a part of NSA's lawful foreign signals intelligence collection system. Such tools have stringent oversight and compliance mechanisms built in at several levels. The use of XKeyscore allows the agency to help defend the nation and protect U.S. and allied troops abroad. All of NSA's operations are conducted in strict accordance with the rule of law, including the President's new directive."

However, the research contradicts the United States' promise to Germany that German citizens are not surveiled without suspicion. Using Tor in Germany does not justify targeting someone, the German attorney Thomas Stadler states: "Tor users do not breach any laws, it is absolutely legitimate to act anonymously on the internet. There are many good reasons to remain anonymous."
What is deep packet inspection?

Deep Packet Inspection, or DPI, refers to the class of technology which examines the content of data packets as they travel across a network. A packet is the fundamental unit of transfer in packet switched networks like the internet. While DPI is commonly used by organizations to monitor their own networks, its use on public networks for censorship and surveillance has been widely condemned by privacy advocates and the United States government alike.

In 2012, the head of the U.S. Delegation to the World Conference on International Telecommunications, U.S. Ambassador Terry Kramer, said "some companies have used deep packet inspection technologies to not look at aggregate customer information, traffic information, et cetera, but to look at individual customer information. So looking at individuals and what sites they're on and how much capacity they're using, et cetera, as you can imagine, we're very much opposed to that because we feel that's a violation of people's privacy and gets into, obviously, censorship, et cetera".

Despite its public political condemnations of invasive DPI use, the United States "Intelligence Community" and its "Five Eyes" partners (Australia, Canada, New Zealand, and the United Kingdom) operate massive internet-scale DPI systems themselves, including XKeyscore. The use of XKeyscore is not limited to these partners, however. The software has been shared with the German BND and BfV, as well as the Swedish FRA, amongst others.

Active vs Passive

XKeyscore and the systems that feed it are considered "passive", meaning that they silently listen but do not transmit anything on the networks that they are targeting. However, through a process known as "tipping", data from these programs can trigger other systems which perform "active" attacks.

Quantum is a family of such programs, including Quantuminsert, Quantumhand, Quantumtheory, Quantumbot, and Quantumcopper, which are used for offensive computer intrusion. Turmoil, Quantum, and other components of the Turbulence architecture are running at so-called "defensive sites" including the Ramstein Air Force base in Germany, Yokota Air Force base in Japan, and numerous military and non-military locations within the United States.

Both Turmoil and XKeyscore feed selected data to real-time "tipping" programs, such as Trafficthief, which can both alert NSA analysts when their targets are communicating and trigger other software programs. Selected data is "promoted" from the local XKeyscore data store to the NSA's so-called "corporate repositories" for long term storage, analysis and exploitation.

More information about XKeyscore

In 2013, the British newspaper The Guardian revealed that by 2008 more than 150 internet surveillance facilities around the world were running the XKeyscore Deep Packet Inspection software. All of the internet traffic observed by XKeyscore, both metadata and full content, is analyzed and stored temporarily at the collection sites for periods ranging from days to weeks, while selected data is forwarded on to other locations for long-term storage.

The storage, indexing, and querying functions are performed at or near the collection sites because the volume of data being collected is too large to forward everything back to facilities in other countries. Analysts working from various locations around the world may search specific XKeyscore sites, or send their queries to a collection of sites.

XKeyscore provides a modular architecture in which tens of thousands of small computer programs, or rules, written in XKeyscore's specialized programming languages called Genesis and XKScript as well as general-purpose languages such as C++ and Python, are run against all traffic to categorize it and extract data. This indexing of the "full take" allows analysts to query the temporary storage stored at the XKeyscore site, effectively sifting through already pilfered communications which occurred before they had deemed them interesting for a specific reason. XKeyscore can be fed by several different programs, including Wealthycluster and Turmoil.

These programs "sessionize" the data, which means that individual connections, such as a request for a web page, are reconstructed from the stream of intercepted packets.

Locations where the NSA runs XKeyscore include Special Source Operations (SSO) sites, typically found at or near major telecommunication providers' infrastructure; Special Collection Service (SCS) sites, usually located inside diplomatic facilities like embassies and consulates; and FORNSAT sites where satellite communications are intercepted. All of these types of sites are known to exist in Germany.

Other "Five Eyes" partners also operate XKeyscore installations. The United Kingdom's Tempora program runs the largest instance of XKeyscore. Both the software itself and limited access to NSA databases have been shared with so-called "3rd party" partners including Germany. The German foreign intelligence agency BND and the domestic intelligence agency BfV are testing the Software.