It sounds like you're using Hadoop correctly, which is fine. But a lot of people are using "big data" that isn't very big (<1TB) and crunching it with small clusters that are less powerful than a single server due to the massive overhead of Hadoop.
It's less about the number of bytes than the number of records produced by the map step(s). Sometimes 10gb input data will produce many billions of records to reduce (if you're looking at combinations of things).
Basically, if your computation would never exceed memory on a single machine, then it is more processor efficient to code a more simple multi processing method and run on a single box than to code a map reduce on a cluster.
But what if you're not sure of the input data size? Processors are cheap. Engineers are expensive. Code the thing once for map reduce and you don't have to worry about making the transition later.
I agree that you should code your analytics once. I think the lesson from this work is that the market could benefit from a Hadoop-compatible but non-clustered runtime which should be easier to run and 10X faster.
Comments
It sounds like you're using Hadoop correctly, which is fine. But a lot of people are using "big data" that isn't very big (<1TB) and crunching it with small clusters that are less powerful than a single server due to the massive overhead of Hadoop.
It's less about the number of bytes than the number of records produced by the map step(s). Sometimes 10gb input data will produce many billions of records to reduce (if you're looking at combinations of things).
Basically, if your computation would never exceed memory on a single machine, then it is more processor efficient to code a more simple multi processing method and run on a single box than to code a map reduce on a cluster.
But what if you're not sure of the input data size? Processors are cheap. Engineers are expensive. Code the thing once for map reduce and you don't have to worry about making the transition later.
I agree that you should code your analytics once. I think the lesson from this work is that the market could benefit from a Hadoop-compatible but non-clustered runtime which should be easier to run and 10X faster.
What happens if you run Hadoop on that single machine?
It works but it's fairly slow because of things like HDFS that you don't need.