A bit unfair, esp. when Microsoft budgeted $2000+ on SSDs alone.
I think the paper has a good point: scale-up should be definitely considered. I see systems that support 1.5 TB of RAM now... do you really expect your dataset to grow beyond that?
And as long as the dataset fits within 1.5 TB of RAM, scale up will be more scalable than scale out.
Yeah, ignore my numbers and I agree that scale-up should be considered and it's worthwhile to read a paper where somebody reality-checks the conventional wisdom.
> I see systems that support 1.5 TB of RAM now... do you really expect your dataset to grow beyond that?
According to Figure 1, about 20% of their jobs are already beyond that. If you spend your entire hardware budget on a single server and all your analytic processing jobs end up being coded with the (gasp) single threaded assumption, what are you going to do next?
Most of us have been there before because it's where you end up by accident ("Do you think if we max out the RAM on the database server it will return to acceptable performance?") This is why people love parallel map-reduce based systems.
Certainly, I don't mean to contradict the conventional wisdom :-p. Scaling out with ~$1500 2U servers makes sense a lot of the times.
I'm more concerned about another problem. There are people out there who think that they can scale out with Hadoop cluster built on top of Atoms (or more recently... ARMs). And while yes... there are some applications where that is a decent scaling strategy, I doubt it works in most cases.
But yeah, I know that wasn't your argument. I'm just musing a bit on random stuff.
I admit to being ARM/Atom M/R curious. Sometimes I wonder about Hadoop for a few seconds until I remember it's heavily JVM-based. Poking around on the web you find folks who sound serious about porting it to Android.
Curious is good. Things definitely move quickly in the tech industry, and I'm willing to bet that within 5 years things can change. And I'd support building up the skills / infrastructure needed to hedge against the uprising of ARM.
But at the moment, the evidence is solidly in favor of the typical 2U Xeon. (by nature of ARM servers barely exist right now. Maybe next year ARM will have something ready)
Atom looks mostly dead however, with Intel beginning to advertize 16W Xeons. Can two 8W Centrino Atoms compete against a 16W Xeon? I doubt it. Intel only has the resources to choose one "hero" chipset in server-space, and it looks like it will be the Xeon again.
Someday somebody is going to realize they are in possession of a trash barge full of 5 million discarded smartphones and break into the top supercomputer list with them.
That idea is a non-starter. Consider the cost of transportation, the cost of connecting all those ridiculously slow CPUs and incompatible CPUs together, powering them, programming for them. Building a supercomputer using standard x86_64 CPUs will probably be cheaper.
Here's an example: http://www.androidauthority.com/samsung-galaxy-s3-sales-20-m... Within a few years there will be 20 million Samsung Galaxy S III's with dead batteries and/or cracked screens. That's a dual-core 1 GHz ARM with 1 GB of RAM, 8 GB of SSD, and a hardware GPU.
Right now, there are professional 1st-world developers developing apps for all those "incompatible CPUs". So it's possible and even cost effective under the right circumstances.
Yet someone is currently being paid to haul away this stuff as electronic recycling.
Xeon just blows the ARM chip out of the water in Performance per Watt used. A modern day Xeon gives you somewhere on the order of 20x more performance per watt compared to a Snapdragon S4 Pro. Otherwise, all the electricity that it takes to run your supercomputer built out of crap chips will make your supercomputer more costly to run in the long term.
I don't know, but the Snapdragon has 1MB of Cache, while the Xeon has 20MB of Cache. Furthermore, the Xeon actually has a multi-socket connector... so two Xeons can talk to each other at a rate of 25.6 GB/s through QPI (QuickPath Interconnect).
So my bet on memory is that Xeon wins that too.
On the multi-processor communications issue... I doubt that the Snapdragon would have PCIe connections... let alone a high-bandwidth low-latency connection like the Xeon's QPI.
There's a reason why supercomputers stick with Intel or AMD. They've got the processor power, the performance per watts, and I/O girth to communicate to other processors. A small embedded chip (like Snapdragon), just doesn't have the features to make a supercomputer out of.
ARM does have promise though, its just not in Snapdragons. As noted, the primary issue in Supercomputers is bandwidth (so that the processors can talk to each other).
Calxeda for example has spent a lot of time trying to solve this interconnect issue. Each Calxeda card is composed of 4 CPUs. Every CPU has 8 10Gb (aka: 1GB) ethernet links, 2 8xPCI connections and more.
Its a relatively young project, but they're moving towards the right direction. There are rumors that Facebook is deploying a Calxeda server.
If 20% of your jobs are beyond 1.5TB of RAM, ideally you want a heterogeneous environment. Some scale-up machines , and some scale-out machines. Mix and match as appropriate, and don't give in to the siren song of scale-out where it isn't appropriate.
Comments
A bit unfair, esp. when Microsoft budgeted $2000+ on SSDs alone.
I think the paper has a good point: scale-up should be definitely considered. I see systems that support 1.5 TB of RAM now... do you really expect your dataset to grow beyond that?
And as long as the dataset fits within 1.5 TB of RAM, scale up will be more scalable than scale out.
Yeah, ignore my numbers and I agree that scale-up should be considered and it's worthwhile to read a paper where somebody reality-checks the conventional wisdom.
> I see systems that support 1.5 TB of RAM now... do you really expect your dataset to grow beyond that?
According to Figure 1, about 20% of their jobs are already beyond that. If you spend your entire hardware budget on a single server and all your analytic processing jobs end up being coded with the (gasp) single threaded assumption, what are you going to do next?
Most of us have been there before because it's where you end up by accident ("Do you think if we max out the RAM on the database server it will return to acceptable performance?") This is why people love parallel map-reduce based systems.
Certainly, I don't mean to contradict the conventional wisdom :-p. Scaling out with ~$1500 2U servers makes sense a lot of the times.
I'm more concerned about another problem. There are people out there who think that they can scale out with Hadoop cluster built on top of Atoms (or more recently... ARMs). And while yes... there are some applications where that is a decent scaling strategy, I doubt it works in most cases.
But yeah, I know that wasn't your argument. I'm just musing a bit on random stuff.
I admit to being ARM/Atom M/R curious. Sometimes I wonder about Hadoop for a few seconds until I remember it's heavily JVM-based. Poking around on the web you find folks who sound serious about porting it to Android.
Curious is good. Things definitely move quickly in the tech industry, and I'm willing to bet that within 5 years things can change. And I'd support building up the skills / infrastructure needed to hedge against the uprising of ARM.
But at the moment, the evidence is solidly in favor of the typical 2U Xeon. (by nature of ARM servers barely exist right now. Maybe next year ARM will have something ready)
Atom looks mostly dead however, with Intel beginning to advertize 16W Xeons. Can two 8W Centrino Atoms compete against a 16W Xeon? I doubt it. Intel only has the resources to choose one "hero" chipset in server-space, and it looks like it will be the Xeon again.
Someday somebody is going to realize they are in possession of a trash barge full of 5 million discarded smartphones and break into the top supercomputer list with them.
That idea is a non-starter. Consider the cost of transportation, the cost of connecting all those ridiculously slow CPUs and incompatible CPUs together, powering them, programming for them. Building a supercomputer using standard x86_64 CPUs will probably be cheaper.
Here's an example: http://www.androidauthority.com/samsung-galaxy-s3-sales-20-m... Within a few years there will be 20 million Samsung Galaxy S III's with dead batteries and/or cracked screens. That's a dual-core 1 GHz ARM with 1 GB of RAM, 8 GB of SSD, and a hardware GPU.
Right now, there are professional 1st-world developers developing apps for all those "incompatible CPUs". So it's possible and even cost effective under the right circumstances.
Yet someone is currently being paid to haul away this stuff as electronic recycling.
A great example of why Xeons are superior. :-)
Snapdragon S4 completes the Linpack benchmark with 460 MegaFlops: http://www.androidauthority.com/snapdragon-s4-pro-vs-exynos-.... Snapdragon is rumored to use ~3 to 5 Watts of power.
Consider a Xeon E5-2690: which gets you 347,000 MegaFlops of performance in 135 Watts. http://www.intel.com/content/www/us/en/benchmarks/server/xeo...
Xeon just blows the ARM chip out of the water in Performance per Watt used. A modern day Xeon gives you somewhere on the order of 20x more performance per watt compared to a Snapdragon S4 Pro. Otherwise, all the electricity that it takes to run your supercomputer built out of crap chips will make your supercomputer more costly to run in the long term.
Does Linpack even touch the memory bus?
I don't know, but the Snapdragon has 1MB of Cache, while the Xeon has 20MB of Cache. Furthermore, the Xeon actually has a multi-socket connector... so two Xeons can talk to each other at a rate of 25.6 GB/s through QPI (QuickPath Interconnect).
So my bet on memory is that Xeon wins that too.
On the multi-processor communications issue... I doubt that the Snapdragon would have PCIe connections... let alone a high-bandwidth low-latency connection like the Xeon's QPI.
There's a reason why supercomputers stick with Intel or AMD. They've got the processor power, the performance per watts, and I/O girth to communicate to other processors. A small embedded chip (like Snapdragon), just doesn't have the features to make a supercomputer out of.
OK, I'm sold.
ARM does have promise though, its just not in Snapdragons. As noted, the primary issue in Supercomputers is bandwidth (so that the processors can talk to each other).
Calxeda for example has spent a lot of time trying to solve this interconnect issue. Each Calxeda card is composed of 4 CPUs. Every CPU has 8 10Gb (aka: 1GB) ethernet links, 2 8xPCI connections and more.
Its a relatively young project, but they're moving towards the right direction. There are rumors that Facebook is deploying a Calxeda server.
If 20% of your jobs are beyond 1.5TB of RAM, ideally you want a heterogeneous environment. Some scale-up machines , and some scale-out machines. Mix and match as appropriate, and don't give in to the siren song of scale-out where it isn't appropriate.