Nice try, but a few thoughts from someone who now spends a lot of time in Hadoop/MapReduce. I will admit it took me a while to warm up to the whole concept, but I'm now so familiar with it that I'm not able to think of compute before Hadoop.
I'm able to fire off pretty intensive MapReduce jobs on an Amazon Elastic MapReduce cluster with many nodes for a fraction of the price mentioned in the post (less than $100).
While I can imagine I could repurpose all my MapReduce/Hadoop code to run on a single box - especially since Amazon does offer several high-memory instances today - I would be loathe to.
The MapReduce framework provides a really nice framework that lets me horizontally scale out compute, rather than vertically, and that is really handy at terabyte-data volumes (data warehousing and large-data analytics.)
MapReduce is handy at terabyte-data volumes, but how often do we run jobs with input size > 1 TB? According to the paper:
at least two analytics production clusters (at Microsoft and
Yahoo) have median job input sizes under 14 GB, and 90%
of jobs on a Facebook cluster have input sizes under 100 GB
The other point the authors make is that DRAM costs are following Moore's law and terabyte-data workloads should "soon" be cost-feasible on single servers with DRAM.
The thing is that MapReduce is still a very good programming paradigm for a single box.
Machines have 8-64 cores these days -- you don't want to write multi-threaded code every time you want to do an analysis. So you can write MapReduce, but use a multicore framework instead of a cluster framework (hadoop).
The unfortunate thing is that there is no popular open source implementation of a multicore mapreduce, so people use Hadoop, which is wasteful on small data sets, as mentioned.
But the great part about it is that you will use the same application code for both. I fully expect in 5 years or so that people will be running multicore mapreduce jobs on 100 or 1000 core boxes.
There is Disco[1], which is a MR framework written in Python and Erlang. It's open source and pretty awesome, and if I'm not mistaken it will leverage multi-core processors, no need for a cluster.
Speaking of Erlang, I've got my eye on Riak pipes. Now that the Basho folks have a large object store, I wonder if more in that department is a natural path.
> The unfortunate thing is that there is no popular open source implementation of a multicore mapreduce
Not popular, I'd agree, but I have had a lot of success with one-off Akka projects. My mappers and reducers are usually under 10 lines of Scala (more if I'm stuck writing Java, obviously).
It sounds like you're using Hadoop correctly, which is fine. But a lot of people are using "big data" that isn't very big (<1TB) and crunching it with small clusters that are less powerful than a single server due to the massive overhead of Hadoop.
It's less about the number of bytes than the number of records produced by the map step(s). Sometimes 10gb input data will produce many billions of records to reduce (if you're looking at combinations of things).
Basically, if your computation would never exceed memory on a single machine, then it is more processor efficient to code a more simple multi processing method and run on a single box than to code a map reduce on a cluster.
But what if you're not sure of the input data size? Processors are cheap. Engineers are expensive. Code the thing once for map reduce and you don't have to worry about making the transition later.
I agree that you should code your analytics once. I think the lesson from this work is that the market could benefit from a Hadoop-compatible but non-clustered runtime which should be easier to run and 10X faster.
It's a tradeoff - clear simplicity for limited logic. I.e. not every problem that requires distributed computation fits well with map reduce, yet many attempt to fit the problem into it to begin with, instead of trying to shape a right distributed solution for particular problem.
Comments
Nice try, but a few thoughts from someone who now spends a lot of time in Hadoop/MapReduce. I will admit it took me a while to warm up to the whole concept, but I'm now so familiar with it that I'm not able to think of compute before Hadoop.
I'm able to fire off pretty intensive MapReduce jobs on an Amazon Elastic MapReduce cluster with many nodes for a fraction of the price mentioned in the post (less than $100).
While I can imagine I could repurpose all my MapReduce/Hadoop code to run on a single box - especially since Amazon does offer several high-memory instances today - I would be loathe to.
The MapReduce framework provides a really nice framework that lets me horizontally scale out compute, rather than vertically, and that is really handy at terabyte-data volumes (data warehousing and large-data analytics.)
MapReduce is handy at terabyte-data volumes, but how often do we run jobs with input size > 1 TB? According to the paper:
at least two analytics production clusters (at Microsoft and Yahoo) have median job input sizes under 14 GB, and 90% of jobs on a Facebook cluster have input sizes under 100 GB
The other point the authors make is that DRAM costs are following Moore's law and terabyte-data workloads should "soon" be cost-feasible on single servers with DRAM.
The thing is that MapReduce is still a very good programming paradigm for a single box.
Machines have 8-64 cores these days -- you don't want to write multi-threaded code every time you want to do an analysis. So you can write MapReduce, but use a multicore framework instead of a cluster framework (hadoop).
The unfortunate thing is that there is no popular open source implementation of a multicore mapreduce, so people use Hadoop, which is wasteful on small data sets, as mentioned.
But the great part about it is that you will use the same application code for both. I fully expect in 5 years or so that people will be running multicore mapreduce jobs on 100 or 1000 core boxes.
Not all algorithms map well to MapReduce, which is the authors' main point. They explain one such example in the paper (section 3).
There is Disco[1], which is a MR framework written in Python and Erlang. It's open source and pretty awesome, and if I'm not mistaken it will leverage multi-core processors, no need for a cluster.
[1] http://discoproject.org/about
Speaking of Erlang, I've got my eye on Riak pipes. Now that the Basho folks have a large object store, I wonder if more in that department is a natural path.
> The unfortunate thing is that there is no popular open source implementation of a multicore mapreduce
Not popular, I'd agree, but I have had a lot of success with one-off Akka projects. My mappers and reducers are usually under 10 lines of Scala (more if I'm stuck writing Java, obviously).
It sounds like you're using Hadoop correctly, which is fine. But a lot of people are using "big data" that isn't very big (<1TB) and crunching it with small clusters that are less powerful than a single server due to the massive overhead of Hadoop.
It's less about the number of bytes than the number of records produced by the map step(s). Sometimes 10gb input data will produce many billions of records to reduce (if you're looking at combinations of things).
Basically, if your computation would never exceed memory on a single machine, then it is more processor efficient to code a more simple multi processing method and run on a single box than to code a map reduce on a cluster.
But what if you're not sure of the input data size? Processors are cheap. Engineers are expensive. Code the thing once for map reduce and you don't have to worry about making the transition later.
I agree that you should code your analytics once. I think the lesson from this work is that the market could benefit from a Hadoop-compatible but non-clustered runtime which should be easier to run and 10X faster.
What happens if you run Hadoop on that single machine?
It works but it's fairly slow because of things like HDFS that you don't need.
It's a tradeoff - clear simplicity for limited logic. I.e. not every problem that requires distributed computation fits well with map reduce, yet many attempt to fit the problem into it to begin with, instead of trying to shape a right distributed solution for particular problem.
Do you have any website, tutorial, pdf, etc. for me to reduce the latency on Hadoop?