For a fleet of "scale-down" servers to be practical the first problem is that the admin cost per server must be very low. With, ohhh guesstimate 70,000+ servers, Facebook presumably has that metric under control.

Next, certain types of workloads don't make sense. Anything CPU-bound or poorly scalable (eg. traditional database workloads). Again FB should have plenty of work that scales out relatively effortlessly (note the relatively) and have moderate memory and bandwidth requirements per process. Though in the aggregate you'd expect very large usage of both!

For suitable workloads, working back of the envelope, ARM will hopefully lead to some highly competitive, if not new record, scores on metrics like requests/joule or requests/$. Enery consumption and server cost being important at that scale..

Now for the in-memory caching or database workloads, which want either more memory or faster CPUs, Flash can be used to address capacity at the cost of a couple of orders of magnitude extra latency - albeit a couple less than hitting disk. Back of the envelope, anything that looks too much like a traditional database workload I'd leave on grunty x86 machines. Ditto for any CPU-bound.

So, lets speculate on how to build a machine based on ARM for the types of workloads we care about. Lets assume we don't design a new core but work with a vendor on a System-on-Chip using ARM hard macros. This is very back of the envelope and we'd need to break out spreadsheets to get this nailed down right.. but lets have some fun..

Our hypothetical SoC would be

. Cache-coherent quad-core Cortex A-9 @ 2GHz . PCIe interface on chip . 1Gb ethernet interface on chip . SATA ports on chip . Memory controller

connected to 4GB ECC memory per server. I'll get back to storage.

Now, this should be small and quite low power server. Within a 1U sled we should be able to pack say 6 or perhaps 8 of these, along with dual power supplies (for the entire sled of machines). If possible I'd have distributed power redundancy by including a battery in the sled rather than hooking up to external UPS.

I'd use an internal 1Gb switch which itself is connected to the top of rack switch. We get local cheap communication between the servers, plus we make cabling significantly easier and keep the cost of the top-of-rack switch down. A more whacky alternative would be to use short-range radio. Fewer wires, potentially more bandwidth, but something I'd like to hammer on in the lab before going anywhere near the datacenter.

Now, for the pesky 4GB per server memory limit and storage. We have a few interesting options. We can add flash per server, or, given the $/GB perhaps one machine in the sled gets it and acts as a local memcached server with the ability to fall back to accessing remote ones. We could also have a local file server with, for example, 1TB of storage via 2 flash-augmented disks (eg. seagate momentus xt disks). With good staging of data, we could even make this sled a good building block for throughput-oriented data-intensive work (eg. mapreduce type work). We have lower IO bandwidth but we've also kept the CPU performance down to levels where we have a fighting chance of feeding them.

Obviously, you need your software to run there. Chalk one up for relatively easily ported open source code without being dependent on a slow-moving vendor.

The above is rampant speculation and there are many interesting design points - it's great to see someone trying new things and taking advantage of changing hardware ratios to profit.