Black Hat USA 2009: Realistic probabilities in modern signals intelligence[1] attempts to provide a more realistic prediction of what could be possible. The paper is structured in a similar way to how Physics and Technology for Future Presidents[2] arrives at the physical limits of remote sensing satellites.
comScore qSearch reports that US Internet search traffic to the top 5 search engines (Google, Yahoo, Bing, Ask, AOL) is on the order of 20 billion per month or 7,600 searches per second averaged across the day[3].
You'd only need a few cheap off-the-shelf servers to process and store all of that information for eternity. See [4] for example performance figures for inserting data to a PostgreSQL database on a single server (not even using fancy sharding techniques). The C10K problem[5] is also worth noting in this context.
The reason I use search traffic as an example is that it is very high value -- it doesn't just capture what people are reading or doing, it captures what they're seeking to read and do (motivations, desires). It's also a very compact data stream and requires little or no processing and analysis. DNS query traffic, registers of source and destination phone numbers, etc also provide cheap, valuable data.
Whether or not the NSA is looking at this kind of data is of little concern when compared to what the commercial world could use this data for. The NSA has a strong incentive not to use their power too widely or they will tend to lose any advantage gained. The commercial world has a similar incentive to act with restraint -- they want to avoid being regulated by new privacy laws. The ACLU's stance fails to make consideration towards these incentives for restraint.
> Whether or not the NSA is looking at this kind of data is of little concern when compared to what the commercial world could use this data for.
And don't most privacy laws protect citizens against spying by the government, but not against spying by the 3rd parties (companies, other citizens). So isn't there an obvious loophole -- the govt just relies on commercial companies to provide it all the captured information. They wouldn't even need special "patriot act" laws for that. Isn't that what ChoicePoint is doing?
Comments
Black Hat USA 2009: Realistic probabilities in modern signals intelligence[1] attempts to provide a more realistic prediction of what could be possible. The paper is structured in a similar way to how Physics and Technology for Future Presidents[2] arrives at the physical limits of remote sensing satellites.
comScore qSearch reports that US Internet search traffic to the top 5 search engines (Google, Yahoo, Bing, Ask, AOL) is on the order of 20 billion per month or 7,600 searches per second averaged across the day[3].
You'd only need a few cheap off-the-shelf servers to process and store all of that information for eternity. See [4] for example performance figures for inserting data to a PostgreSQL database on a single server (not even using fancy sharding techniques). The C10K problem[5] is also worth noting in this context.
The reason I use search traffic as an example is that it is very high value -- it doesn't just capture what people are reading or doing, it captures what they're seeking to read and do (motivations, desires). It's also a very compact data stream and requires little or no processing and analysis. DNS query traffic, registers of source and destination phone numbers, etc also provide cheap, valuable data.
Whether or not the NSA is looking at this kind of data is of little concern when compared to what the commercial world could use this data for. The NSA has a strong incentive not to use their power too widely or they will tend to lose any advantage gained. The commercial world has a similar incentive to act with restraint -- they want to avoid being regulated by new privacy laws. The ACLU's stance fails to make consideration towards these incentives for restraint.
[1] http://www.blackhat.com/presentations/bh-usa-09/TOPLETZ/BHUS...
[2] http://books.google.com/books/about/Physics_and_Technology_f...
[3] http://www.dailydisruption.com/2012/02/google-still-king-com...
[4] http://archives.postgresql.org/pgsql-admin/2011-07/msg00107....
[5] http://www.kegel.com/c10k.html
> Whether or not the NSA is looking at this kind of data is of little concern when compared to what the commercial world could use this data for.
And don't most privacy laws protect citizens against spying by the government, but not against spying by the 3rd parties (companies, other citizens). So isn't there an obvious loophole -- the govt just relies on commercial companies to provide it all the captured information. They wouldn't even need special "patriot act" laws for that. Isn't that what ChoicePoint is doing?