In many cases distributed computation is needed when there is BIG data, which won't fit on a laptop like in that example. I.e. distribution is needed not only because computation can't be done on one node (i.e. it would take too long), but because data can't possibly fit on any one node.
How big are the jobs being done? I remember reading a statistic[1] that is a couple of years old that 50% of the big data jobs were less than 10 GB. 90% was less than 100 GB and only something like 1% was bigger than a TB. You can fit multiple terabytes on a single machine. I wonder if anyone has new statistics about this.
A previous project I worked on was persisting more than 10 TB of data per week, data that had to be analyzed, the requirement being to extract meaningful user profiles from it. We also went back to analyze older data with new queries / algorithms quite a lot and playing with a petabyte is fun. It was a startup, it was dissolved at some point, so you've never heard of it.
I don't like the 90% argument. 90% of apps are just dumb frontends to a MySQL database, so you don't need Rust, or Spark. As you can see, that doesn't tell you anything meaningful about the other 10%.
And the point being made (and with which I agree) is there's probably fewer than a few hundred of such deployments in the world, and their problems are very likely not your problems.
I have good context on this from seeing the analytics warehouse for a very popular social network. Even with a multi-terabyte hot database, the analytics dataset fit on a single 2U server full of drives. I have not seen a deployment of Hadoop yet that makes sense. Usually I see things like a 2 or 3 TB log spread across 60 nodes and a job to count entries matching a regular expression (because Hadoop becomes the hammer hunting for nails), which can easily be done in a VM on a Mac in minutes.
That was the thrust of the original post, and I agree wholeheartedly.
If the data already lives on 60 nodes, then transferring all this data to a vm + performing work may be longer (and clog the pipes) than just doing the work where the data already is and aggregating results.
> because Hadoop becomes the hammer hunting for nails
Yes, I agree that it's not an uncommon situation. But regardless whether legitimately big deployments are common or not, I never really appreciated some negligence with performance in using Java for such things. Just because it scales, it doesn't mean that overhead should be ignored, because the cost of the overhead scales too.
So going back to my original point, having such kind of systems made with Rust would be really much better.
Many parallel algorithms become slower if you make them use too many cores, as they spend most of the time in communication, and very little in actual computations. Maybe the parallel systems would have been faster on a smaller number of cores than 128?
Also GNU utils http://aadrake.com/command-line-tools-can-be-235x-faster-tha...
The above experiment (which has an interesting github repo) is somewhat over (and real world unusable), but still is eye opening. Hadoop and Spark bring so much complexity that looking for simpler solutions is something worth considering.
GNU parallel is one of the most underrated projects in the whole GNU universe. Sure it doesn't work for everything, but for all cases where it does work, spinning up 50 ec2 spot instances and just pointing parallel at them is by far the quickest and easiest way to do distributed computing.
Comments
Little secret... with Rust you can just do big data processing on a single core => http://www.frankmcsherry.org/graph/scalability/cost/2015/01/...
In many cases distributed computation is needed when there is BIG data, which won't fit on a laptop like in that example. I.e. distribution is needed not only because computation can't be done on one node (i.e. it would take too long), but because data can't possibly fit on any one node.
See his next post: http://www.frankmcsherry.org/graph/scalability/cost/2015/02/...
How big are the jobs being done? I remember reading a statistic[1] that is a couple of years old that 50% of the big data jobs were less than 10 GB. 90% was less than 100 GB and only something like 1% was bigger than a TB. You can fit multiple terabytes on a single machine. I wonder if anyone has new statistics about this.
[1]: http://www.msr-waypoint.com/pubs/204499/a20-appuswamy.pdf
A previous project I worked on was persisting more than 10 TB of data per week, data that had to be analyzed, the requirement being to extract meaningful user profiles from it. We also went back to analyze older data with new queries / algorithms quite a lot and playing with a petabyte is fun. It was a startup, it was dissolved at some point, so you've never heard of it.
I don't like the 90% argument. 90% of apps are just dumb frontends to a MySQL database, so you don't need Rust, or Spark. As you can see, that doesn't tell you anything meaningful about the other 10%.
I'm talking about really big data, which requires hundreds of nodes just to store it and etc.
And the point being made (and with which I agree) is there's probably fewer than a few hundred of such deployments in the world, and their problems are very likely not your problems.
I have good context on this from seeing the analytics warehouse for a very popular social network. Even with a multi-terabyte hot database, the analytics dataset fit on a single 2U server full of drives. I have not seen a deployment of Hadoop yet that makes sense. Usually I see things like a 2 or 3 TB log spread across 60 nodes and a job to count entries matching a regular expression (because Hadoop becomes the hammer hunting for nails), which can easily be done in a VM on a Mac in minutes.
That was the thrust of the original post, and I agree wholeheartedly.
If the data already lives on 60 nodes, then transferring all this data to a vm + performing work may be longer (and clog the pipes) than just doing the work where the data already is and aggregating results.
> because Hadoop becomes the hammer hunting for nails
Yes, I agree that it's not an uncommon situation. But regardless whether legitimately big deployments are common or not, I never really appreciated some negligence with performance in using Java for such things. Just because it scales, it doesn't mean that overhead should be ignored, because the cost of the overhead scales too.
So going back to my original point, having such kind of systems made with Rust would be really much better.
Many parallel algorithms become slower if you make them use too many cores, as they spend most of the time in communication, and very little in actual computations. Maybe the parallel systems would have been faster on a smaller number of cores than 128?
That's a pretty sad result to get from 128 cores. I've seen amateur Beowulf clusters get better results.
Also GNU utils http://aadrake.com/command-line-tools-can-be-235x-faster-tha... The above experiment (which has an interesting github repo) is somewhat over (and real world unusable), but still is eye opening. Hadoop and Spark bring so much complexity that looking for simpler solutions is something worth considering.
Let's also not forget GNU parallel (https://www.gnu.org/software/parallel/)
GNU parallel is one of the most underrated projects in the whole GNU universe. Sure it doesn't work for everything, but for all cases where it does work, spinning up 50 ec2 spot instances and just pointing parallel at them is by far the quickest and easiest way to do distributed computing.
Thanks for the tip!
Excuse me, I guess that post had no related github repo. I was probably recalling this repo: https://github.com/erikfrey/bashreduce
Oh my. That's a treat. Did someone forget to tell these people that the performance from more nodes is supposed to scale in the other direction? Lol.