If the replication factor for blobs is calculated from the 'hotness' of the data (by looking at the age for example), I'm wondering how CDNs come into play.
Increasing the replication factor would allow for faster reads, but there usually wouldn't be a need for that if you offload reads of hot data to a CDN service (could be an internal CDN...).
There is a just-before-storage cache (through which the CDN edge requests) which reduces requests per object, but it does not affect the number of hot objects per disk. With Haystack each object is fully available with sane performance in degraded state. This means 1/3 I/O per disk, whereas F4 only has 1 easily available full copy, which means 100% load on one disk per object stored.
You can use 1-3 petabyte per rack as a rough yardstick (before you start counting RAID overhead etc.)
It can be more/less than this depending on your rack configuration and how big drives you use, and how dense you pack them (e.g. "standard" off the shelf servers typically peak at 24 3.5" drives per 4U, while top-loaded specialised storage enclosures reach about 50 x 3.5" drives per 4U; full racks vary between 42U and around 47U etc.)
But you could if you wanted to get this much storage into about a 5m x 5m server room....
Of course you wouldn't, since you'd want multi-datacentre redundancy etc. Often there's also no point using the biggest drives available because you need higher throughput. Or no point packing them as tight as possible because you need more processing power per drive.
Comments
I found this part interesting: "F4 currently stores over 65PB of logical data and saves over 53PB of storage."
It looks like most of the savings here is due to optimization of the replication level of storage, according to the usage patterns of the data.
Also found it interesting that it doesn't talk about CDN usage.
What would you have liked to know about CDN usage?
(I work on the team that builds and runs the Facebook CDN infrastructure.)
If the replication factor for blobs is calculated from the 'hotness' of the data (by looking at the age for example), I'm wondering how CDNs come into play.
Increasing the replication factor would allow for faster reads, but there usually wouldn't be a need for that if you offload reads of hot data to a CDN service (could be an internal CDN...).
There is a just-before-storage cache (through which the CDN edge requests) which reduces requests per object, but it does not affect the number of hot objects per disk. With Haystack each object is fully available with sane performance in degraded state. This means 1/3 I/O per disk, whereas F4 only has 1 easily available full copy, which means 100% load on one disk per object stored.
Hope that makes sense - let me know if not.
"An Analysis of Facebook Photo Caching" [1] discusses CDN usage and complements the original Haystack paper [2] [3] well.
[1] http://www.cs.cornell.edu/~qhuang/papers/sosp_fbanalysis.pdf
[2] https://www.usenix.org/legacy/event/osdi10/tech/full_papers/...
[3] http://muratbuffalo.blogspot.com/2010/12/finding-needle-in-h...
What kind of footprint would a system of this nature require I wonder? Geographically speaking.
You can use 1-3 petabyte per rack as a rough yardstick (before you start counting RAID overhead etc.)
It can be more/less than this depending on your rack configuration and how big drives you use, and how dense you pack them (e.g. "standard" off the shelf servers typically peak at 24 3.5" drives per 4U, while top-loaded specialised storage enclosures reach about 50 x 3.5" drives per 4U; full racks vary between 42U and around 47U etc.)
But you could if you wanted to get this much storage into about a 5m x 5m server room....
Of course you wouldn't, since you'd want multi-datacentre redundancy etc. Often there's also no point using the biggest drives available because you need higher throughput. Or no point packing them as tight as possible because you need more processing power per drive.