Similarly, Google+ userIDs were assigned as 21-character numeric strings, beginning with '10' or '11', but otherwise appearing to be randomly assigened.
A full listing was available through the site's robots.txt sitemaps file, or rather, a listing to the listing of 50,000 user profile sitemap files, with about 44k profiles per file. This worked out to 25 GB of profile listings alone.
Rather than download the full set (though I eventually did), I picked an arbitrary file from near the middle of the listing, and ran some spot checks on the profiles, which seemed to be reasonably randomly distributed by age, location, and other characteristics. With as few as 100 profile page downloads, it was clearly evident that active posting to G+ was limited to about 8-11% ofall profiles. The full 50k profile sample, and a third party's independent (and more robustly randomised) 500k profile sample eventually showed this to be 9.7%.
(And yes, if I was being more rigorous I could have done much more testing or work, but I was mostly addressing personal curiosity and an online disagreement with someone.)
An interesting proof of the power of random sampling.
Larger samples do allow for clearer views of rare phenomena -- such as dialing in on the fraction of 1% of G+ users highly active on the site. Or when I later looked at Communities characteristics, the properties of the very largest (about 50 > 1 million members) of the 8 million total. In that case, I eventually got access (also via a third-party) to a comprehensive summary dataset.
The userID hashing also made approaches such as exhaustively searching the ID space for user pages nonviable. The search space was trillions pf times larger than the target space.
There's a zillion things you can estimate this way. A lot of sites use sequential cookies, user IDs, etc. Until about a decade ago UPS tracking numbers were sequential for each shipper which made it trivial to estimate output for online shops. Apple invoice numbers used to be dense and sequential and you only needed the number to retrieve the invoice. The IMEI is actually just about the worst way to have estimated iPhone sales in 2008; at that time you could literally have crawled Apple's website for every invoice whether sold online or in stores.
Comments
FWIW, this is part of why Amazon's product identifiers (ASINs) are obfuscated the way they are
Similarly, Google+ userIDs were assigned as 21-character numeric strings, beginning with '10' or '11', but otherwise appearing to be randomly assigened.
A full listing was available through the site's robots.txt sitemaps file, or rather, a listing to the listing of 50,000 user profile sitemap files, with about 44k profiles per file. This worked out to 25 GB of profile listings alone.
Rather than download the full set (though I eventually did), I picked an arbitrary file from near the middle of the listing, and ran some spot checks on the profiles, which seemed to be reasonably randomly distributed by age, location, and other characteristics. With as few as 100 profile page downloads, it was clearly evident that active posting to G+ was limited to about 8-11% ofall profiles. The full 50k profile sample, and a third party's independent (and more robustly randomised) 500k profile sample eventually showed this to be 9.7%.
(And yes, if I was being more rigorous I could have done much more testing or work, but I was mostly addressing personal curiosity and an online disagreement with someone.)
An interesting proof of the power of random sampling.
Larger samples do allow for clearer views of rare phenomena -- such as dialing in on the fraction of 1% of G+ users highly active on the site. Or when I later looked at Communities characteristics, the properties of the very largest (about 50 > 1 million members) of the 8 million total. In that case, I eventually got access (also via a third-party) to a comprehensive summary dataset.
The userID hashing also made approaches such as exhaustively searching the ID space for user pages nonviable. The search space was trillions pf times larger than the target space.
People have used a similar strategy to estimate iPhone production [0].
[0] https://www.theguardian.com/technology/blog/2008/oct/08/ipho...
There's a zillion things you can estimate this way. A lot of sites use sequential cookies, user IDs, etc. Until about a decade ago UPS tracking numbers were sequential for each shipper which made it trivial to estimate output for online shops. Apple invoice numbers used to be dense and sequential and you only needed the number to retrieve the invoice. The IMEI is actually just about the worst way to have estimated iPhone sales in 2008; at that time you could literally have crawled Apple's website for every invoice whether sold online or in stores.
See also: https://hashids.org/