Also GNU utils http://aadrake.com/command-line-tools-can-be-235x-faster-tha...
The above experiment (which has an interesting github repo) is somewhat over (and real world unusable), but still is eye opening. Hadoop and Spark bring so much complexity that looking for simpler solutions is something worth considering.
GNU parallel is one of the most underrated projects in the whole GNU universe. Sure it doesn't work for everything, but for all cases where it does work, spinning up 50 ec2 spot instances and just pointing parallel at them is by far the quickest and easiest way to do distributed computing.
Comments
Also GNU utils http://aadrake.com/command-line-tools-can-be-235x-faster-tha... The above experiment (which has an interesting github repo) is somewhat over (and real world unusable), but still is eye opening. Hadoop and Spark bring so much complexity that looking for simpler solutions is something worth considering.
Let's also not forget GNU parallel (https://www.gnu.org/software/parallel/)
GNU parallel is one of the most underrated projects in the whole GNU universe. Sure it doesn't work for everything, but for all cases where it does work, spinning up 50 ec2 spot instances and just pointing parallel at them is by far the quickest and easiest way to do distributed computing.
Thanks for the tip!
Excuse me, I guess that post had no related github repo. I was probably recalling this repo: https://github.com/erikfrey/bashreduce
Oh my. That's a treat. Did someone forget to tell these people that the performance from more nodes is supposed to scale in the other direction? Lol.