Boundless Informant: the NSA's secret tool to track global surveillance data

[AdamSmith]

Boundless Informant: the NSA's secret tool to track global surveillance data

Revealed: The NSA's powerful tool for cataloguing global surveillance data – including figures on US collection

• Boundless Informant: mission outlined in four slides
• Read the NSA's frequently asked questions document

• Glenn Greenwald and Ewen MacAskill
• guardian.co.uk, Sunday 9 June 2013 10.08 EDT

The color scheme ranges from green (least subjected to surveillance) through yellow and orange to red (most surveillance). Note the '2007' date in the image relates to the document from which the interactive map derives its top secret classification, not to the map itself.

The National Security Agency has developed a powerful tool for recording and analysing where its intelligence comes from, raising questions about its repeated assurances to Congress that it cannot keep track of all the surveillance it performs on American communications.

The Guardian has acquired top-secret documents about the NSA datamining tool, called Boundless Informant, that details and even maps by country the voluminous amount of information it collects from computer and telephone networks.

The focus of the internal NSA tool is on counting and categorizing the records of communications, known as metadata, rather than the content of an email or instant message.

The Boundless Informant documents show the agency collecting almost 3 billion pieces of intelligence from US computer networks over a 30-day period ending in March 2013. One document says it is designed to give NSA officials answers to questions like, "What type of coverage do we have on country X" in "near real-time by asking the SIGINT [signals intelligence] infrastructure."

An NSA factsheet about the program, acquired by the Guardian, says: "The tool allows users to select a country on a map and view the metadata volume and select details about the collections against that country."

Under the heading "Sample use cases", the factsheet also states the tool shows information including: "How many records (and what type) are collected against a particular country."

A snapshot of the Boundless Informant data, contained in a top secret NSA "global heat map" seen by the Guardian, shows that in March 2013 the agency collected 97bn pieces of intelligence from computer networks worldwide.

The heat map reveals how much data is being collected from around the world. Note the '2007' date in the image relates to the document from which the interactive map derives its top secret classification, not to the map itself.

Iran was the country where the largest amount of intelligence was gathered, with more than 14bn reports in that period, followed by 13.5bn from Pakistan. Jordan, one of America's closest Arab allies, came third with 12.7bn, Egypt fourth with 7.6bn and India fifth with 6.3bn.

The heatmap gives each nation a color code based on how extensively it is subjected to NSA surveillance. The color scheme ranges from green (least subjected to surveillance) through yellow and orange to red (most surveillance).

The disclosure of the internal Boundless Informant system comes amid a struggle between the NSA and its overseers in the Senate over whether it can track the intelligence it collects on American communications. The NSA's position is that it is not technologically feasible to do so.

At a hearing of the Senate intelligence committee In March this year, Democratic senator Ron Wyden asked James Clapper, the director of national intelligence: "Does the NSA collect any type of data at all on millions or hundreds of millions of Americans?"

"No sir," replied Clapper.

Judith Emmel, an NSA spokeswoman, told the Guardian in a response to the latest disclosures: "NSA has consistently reported – including to Congress – that we do not have the ability to determine with certainty the identity or location of all communicants within a given communication. That remains the case."

Other documents seen by the Guardian further demonstrate that the NSA does in fact break down its surveillance intercepts which could allow the agency to determine how many of them are from the US. The level of detail includes individual IP addresses.

IP address is not a perfect proxy for someone's physical location but it is rather close, said Chris Soghoian, the principal technologist with the Speech Privacy and Technology Project of the American Civil Liberties Union. "If you don't take steps to hide it, the IP address provided by your internet provider will certainly tell you what country, state and, typically, city you are in," Soghoian said.

That approximation has implications for the ongoing oversight battle between the intelligence agencies and Congress.

On Friday, in his first public response to the Guardian's disclosures this week on NSA surveillance, Barack Obama said that that congressional oversight was the American peoples' best guarantee that they were not being spied on.

"These are the folks you all vote for as your representatives in Congress and they are being fully briefed on these programs," he said. Obama also insisted that any surveillance was "very narrowly circumscribed".

Senators have expressed their frustration at the NSA's refusal to supply statistics. In a letter to NSA director General Keith Alexander in October last year, senator Wyden and his Democratic colleague on the Senate intelligence committee, Mark Udall, noted that "the intelligence community has stated repeatedly that it is not possible to provide even a rough estimate of how many American communications have been collected under the Fisa Amendments Act, and has even declined to estimate the scale of this collection."

At a congressional hearing in March last year, Alexander denied point-blank that the agency had the figures on how many Americans had their electronic communications collected or reviewed. Asked if he had the capability to get them, Alexander said: "No. No. We do not have the technical insights in the United States." He added that "nor do we do have the equipment in the United States to actually collect that kind of information".

Soon after, the NSA, through the inspector general of the overall US intelligence community, told the senators that making such a determination would jeopardize US intelligence operations – and might itself violate Americans' privacy.

"All that senator Udall and I are asking for is a ballpark estimate of how many Americans have been monitored under this law, and it is disappointing that the inspectors general cannot provide it," Wyden told Wired magazine at the time.

The documents show that the team responsible for Boundless Informant assured its bosses that the tool is on track for upgrades.

The team will "accept user requests for additional functionality or enhancements," according to the FAQ acquired by the Guardian. "Users are also allowed to vote on which functionality or enhancements are most important to them (as well as add comments). The BOUNDLESSINFORMANT team will periodically review all requests and triage according to level of effort (Easy, Medium, Hard) and mission impact (High, Medium, Low)."

Emmel, the NSA spokeswoman, told the Guardian: "Current technology simply does not permit us to positively identify all of the persons or locations associated with a given communication (for example, it may be possible to say with certainty that a communication traversed a particular path within the internet. It is harder to know the ultimate source or destination, or more particularly the identity of the person represented by the TO:, FROM: or CC: field of an e-mail address or the abstraction of an IP address).

"Thus, we apply rigorous training and technological advancements to combine both our automated and manual (human) processes to characterize communications – ensuring protection of the privacy rights of the American people. This is not just our judgment, but that of the relevant inspectors general, who have also reported this."

She added: "The continued publication of these allegations about highly classified issues, and other information taken out of context, makes it impossible to conduct a reasonable discussion on the merits of these programs."

Additional reporting: James Ball in New York and Spencer Ackerman in Washington

http://www.guardian.co.uk/world/2013/jun/08/nsa-boundless-informant-global-datamining?guni=Network%20front:network-front%20full-width-1%20bento-box:Bento%20box:Position1:sublinks

[RA1]

Perfidy in all its' forms is amazing, isn't it?

I still contend that a well placed spy could provide more useful information than all our technology. Besides, computers don't (yet) write best sellers after his or her career is over.

Best regards,

RA1

[AdamSmith]

Here is one commentator making exactly your point: HUMINT over SIGINT

The writer himself does not seem too brilliant but he points to some interesting data.

[RA1]

You are describing me perfectly.

Best regards,

RA1

[Guest hitoallusa]

Who me? I often don't point to some interesting data.

"The writer himself does not seem too brilliant but he points to some interesting data."

I sometimes wonder what Greenwald's intention is writing for this kind of article. So most people in the intelligence community don't respect privacy and all the scholars and liberal college professors who consult programs like these are little Hitlers and anti-constitutional? I don't think so. The leak itself is a bit strange, though. It seems something is wrong in the intelligence community and it is so severe that it can't solve internal issues like itself without a leak. Is there communication break down in the system?

I am not too worried about this kind of program. I'm actually for it since it could be developed into something that can be beneficial. At least let them study and research what they could do. What I am worried about from these reports is that there is something wrong or a serious flaw in the intelligence community that it can't cope with. That is what makes me worry.

[RA1]

The so called intelligence community is not at fault. They should be "controlled" by the Congress if not the administration. I am VERY WORRIED about this kind of program.

Yes, all the liberal college professors do not respect the Constitution and yes they are all little Hitlers, but that is beside the point.

Even the "conservative" professors, if you can find them, are little Hitlers.

Best regards,

RA1

[Guest hitoallusa]

I'm not so sure why you are so concerned about it. If a medical researcher uses a set of metadata to identify the source of an illness, I don't think no one will be against it. I don't know why Greenwald can have that kind of perspective on this. What the NSA is trying to do isn't like something we see in the movie, The Enemy of the State. Well I have to confess Will Smith looks a bit like young Obama in that clip... Why does hot ones always have to be straight.

[AdamSmith]

I'm not so sure why you are so concerned about it.

hito, here is one reason to be concerned (re-posting this from the other day):

As lookin said, even if you like and trust Obama with these tools and capabilities, we have no gaurantee that "good guys" will always be in charge of government.

[Guest hitoallusa]

If you apply that logic then the government should not make nuclear bombs since we don't know whether an evil guy will be our next president. I think there was a communication break down between the leaker and his colleagues. I see it as the bigger problem than the program itself. It seems the environment doesn't promote free expression of concerns and issues so that the leaker had to take a dramatic measure. What the leaker exposed was the breakdown in the communication system and it needs to be addressed so that this doesn't happen again. Instead, Greenwald focus on making the NSA this evil organization without any respect for one's privacy. I don't think it's a constructive story at all. Who exactly in the government or the NSA is he talking about? Can you say The NSA = the government ?

It is very romantic, ideal and pure not to sign prenuptial agreements. But people do. It is ideal not to have CCTVs and cameras at stores. I don't want to be recorded on my bad hair day. But store owners do. That is the reality. We don't protest against CCTVs and in the future our perspective on the program will change too and accept it as we do CCTVs. The real issue hear is that, the leaker was so frustrated but couldn't get the frustration resolved or get his points addressed within his work place. That needs to be addressed and resolved to make the intelligence community function better.

[lookin]

This isn't, of course, the first time the NSA has overreached. I may have posted this link a few years ago, probably over at Daddy's, to the 2007 Frontline program, Spying on the Homefront, after the NSA tapped into AT&T's internet lines at their Folsom Street building in San Francisco. Yes, that Folsom Street.

You can watch the program on-line, and click on the "Interviews" tab for some interesting reading.

From the interview with Mark Klein, a technician who got access not only to the secret NSA equipment, but also to some documents that were carelessly left lying around:

How do you know that it wasn't just some kind of newfangled AT&T thing that was going beyond what had already been established for its security purposes elsewhere?

First of all, they wouldn't need the NSA for that purpose, and we would be allowed in by union contract to service the equipment as we always were. So anyway, there's that question right off the bat: Why NSA?

Now, in October, while I was working and learning the Internet room, I came across these three documents, which were documents that the technicians had that were given to the technicians so they would know how to install things like the splitter cabinet in particular, because it tells how things are wired up. Those documents were left lying around. Some of the technicians still had them. One of them was just left lying on top of a router. I picked it up, and I looked at it, and I brought them back to my desk, and when I started looking at it, I looked at it more, and I looked at it more, and finally it dawned on me sort of all at once, and I almost fell out of my chair, because this showed, first of all, what they had done, that they had taken working circuits, which had nothing to do with a splitter cabinet, and they had taken in particular what are called peering links which connect AT&T's network with the other networks. It's how you get the Internet, right? One network connects with another. So they took 16 high-speed peering links which go to places like Qwest [Communications] and Palo Alto Internet Exchange [PAIX] and places like that. … These circuits were working at one point, and the documents indicated in February 2003 they had cut into these circuits so that they could insert the splitter so that they can get the data flow from these circuits to go to the secret room.

So this data flow meant that they were getting not only AT&T customers' data flow; they were getting everybody else's data flow, whoever else might happen to be communicating into the AT&T network from other networks. So it was turning out to be like a large chunk of the network, of the Internet.

[Guest hitoallusa]

Oh my Folsom is dirtier than I expected.. Well this whole fiasco in Las Vegas started because of a mistake by analysts. It was a human error that misinterpreted whatever data they had. They thought they identified a threat even though it turned out a wrong analysis later. I think it was better to look into it rather than do nothing and be sorry later. It's unfortunate, in the process, they violated people's privacy. I think the NSA had to do what they had to do at that time. During a battle, you have to unfortunately kill an opponent who family and loved ones waiting at home. I find that tragic but we have no choice. I wish there were no wars or terrors in the world. But that's just my wishful thinking. I always wonder what would I do if I get drafted and am driven into a battle. Will I kill someone who is equally valuable as a human being shooting at me or not? Someone is doing that job for me so I don't have to kill someone with my own hands or face a major attack that might take lives of people around me.

Instead of labeling these agencies evil villains, what about changing our perspective and work with them. Give them the benefit of the doubt that they seek out our best interest. Most of all, they are human beings with families and loved ones who want to serve the country to make it a safer place.

[AdamSmith]

hito, a simple question:

What is the meaning and significance of the phrase "the consent of the governed"?

[lookin]

I can't believe we won a free trip.

Wow! I've never been to Chicago.

That Mr. Hormel's such a nice man.

[Guest hitoallusa]

“If you harness all the capabilities of our nation, you could have a better understanding of foreign threats,” he said. “But what makes it hard is that everyone has an opinion. There’s very little appreciation for the threat, and there are so many special interests, particularly civil liberty groups with privacy concerns. That mix keeps us from getting to the crux of the national issue.”

- Mike McConnell, a former head of the N.S.A. and director of national intelligence

http://www.nytimes.com/2013/06/10/us/former-cia-worker-says-he-leaked-surveillance-data.html?pagewanted=2&_r=0

[RA1]

lookin-

Unfortunately those about to be slaughtered are the citizens of the US when it ought to be those who first and foremost feed at the public trough, namely pols but, there are others. I suppose we all do to some extent but I feel much more like a little piggy rather than a BIG hog. But, that is just my opinion.

Best regards,

RA1