I remember during the Boston bombing investigation, Tim Clemente, a former FBI counter-terrorism agent, told Erin Burnett on CNN that they could go back and get access to the content of the calls between the deceased bomber and his wife, Katherine Russell.
After some Googling, I found a partial transcript of the CNN interview...
"Almost immediately Erin Burnett, the host of CNN's Outfront, wanted to know how the government knew. Aren't phone calls supposed to be private? She interviewed Tim Clemente, a former FBI counter-terrorism agent on May 1, asking:"
Is there any way … they [the federal investigators] can
try to get the phone companies to give that up … It’s not
a voice mail. It's just a conversation. There’s no way
they can actually find out what [was said on the call],
right, unless she tells them?
Clemente: There is a way. We certainly have ways in
national security investigations to find out exactly what
was said in that conversation. It's not necessarily
something that the FBI is going to want to present in
court, but it may help lead the investigation … we
certainly can find that out.
Burnett: So they can actually get that? … that is
incredible.
Clemente: Welcome to America. All of that stuff is being
captured as we speak, whether we know it or like it, or
not.
When I initially saw this on CNN, my first thought was, do they also have access to all photos and videos that are taken and transmitted online?
And if so, couldn't they stitch together a multi-angle montage from all the photos and videos taken at the scene of the Boston bombing (like http://photosynth.net), rather than asking everyone to manually scour through their personal footage?
I am assuming that if something is technically possible, they're doing it. So, they're tracking telecom metadata and storing content because they can.
In fact I believe that the only reason our physical mail was mostly not read had nothing to do with the 4th Amendment, and everything to do with that it was logistically impossible and still have a working postal service, and difficult to keep secret.
With modern mail sorting machinery, I'm sure there is metadata being collected on physical mail.
>So, they're tracking telecom metadata and storing content because they can.
Correct me if I'm wrong, but to theoretically record every single phone conversation in America would require unbelievable amounts of data storage, would it not? Even if it's stored in a highly compressed format.
The metadata I can believe, but I find it kind of unlikely they have the means to store the contents of every call.
"Brewster Kahle, a computer engineer who founded the Internet Archive, has vast experience storing large amounts of data. He created a spreadsheet this week estimating that the cost to store all domestic phone calls a year in cloud storage for data-mining purposes would be about $27 million per year, not counting the cost of extra security for a top-secret program and security clearances for the people involved."
Storage is cheap, and only getting cheaper. You could archive things older than a certain date onto cheaper media. Or consider converting audio into text-transcripts after a period of time, and you can dramatically reduce your storage requirements. Either way, $27 million isn't really much.
Others on HN have done the math on this. It's doable, at bad but reasonable quality, with 2% of the storage capacity of NSA's new datacenter corresponding to roughly one year of all American voice traffic.
> Correct me if I'm wrong, but to theoretically record every single phone conversation in America would require unbelievable amounts of data storage, would it not?
GSM voice codec is 9600bps = 1200 Bytes/s
x60 sec, x20 min per day, x365 days per year = about 500MB per person per year.
times however many people there are in the US, that's not a whole lot of storage and definitely a mere fraction of the NSA's data-centres.
Whatever they're technically capable of doing, I assume they're doing it. If they can't store everything yet, I assume they're working on it. I also assume they're somewhat ahead of whatever we commonly think is possible.
Maybe not everything, maybe not now, but close to it and soon.
According to this article (https://sumanrs.wordpress.com/2012/04/14/youtube-yearly-cost...), about 76 petabytes is stored to Youtube per year. (calculated from the released info, that "one hour of video is uploaded to YouTube every second."). So Petabytes/day is probably doable for them.
Dan Carlin pointed this out on his podcast days after it happened. Not surprising to those paying attention, yet amazing how these guys can slip up and only a few notice.
After some Googling, I found a partial transcript of the CNN interview...
"Almost immediately Erin Burnett, the host of CNN's Outfront, wanted to know how the government knew. Aren't phone calls supposed to be private? She interviewed Tim Clemente, a former FBI counter-terrorism agent on May 1, asking:"
Source: http://www.thenewamerican.com/usnews/crime/item/15340-boston...When I initially saw this on CNN, my first thought was, do they also have access to all photos and videos that are taken and transmitted online?
And if so, couldn't they stitch together a multi-angle montage from all the photos and videos taken at the scene of the Boston bombing (like http://photosynth.net), rather than asking everyone to manually scour through their personal footage?
UPDATE:
CNN Interview Clip (http://www.youtube.com/watch?v=pPHZrVPt4-U)
CNN Follow-Up Interview (http://www.youtube.com/watch?v=vt9kRLrmrjc)