I may have written about this before on HN, but once I wrote a simple Perl script that could run the daily trade reconciliation for an entire US primary exchange. It could run on my laptop and complete the process in under 20 minutes. Ten years later, I watched a team spending days setting up a Spark cluster to handle a comparable amount of data in a somewhat simpler business domain.
Comments
I may have written about this before on HN, but once I wrote a simple Perl script that could run the daily trade reconciliation for an entire US primary exchange. It could run on my laptop and complete the process in under 20 minutes. Ten years later, I watched a team spending days setting up a Spark cluster to handle a comparable amount of data in a somewhat simpler business domain.