A lot of the nitty-gritty data munging and processing often gets discarded after a project or never included in the project repo in a meaningful way. I like Drake [0] because we used it a lot at Factual and it really made data generation and formatting very repeatable and easy.
I really think the packaging system the author is going for would be best built on top of Drake or a similar workflow management program. Instead of following their laundry list of configuration steps one could manage that automatically with a source-controlled workflow. Drake does have the advantages of non-linear and async workflows being pretty easy to build, maintain, and update.
What I would love to see is a data package manager that downloads the raw data and processing workflow, updates any software packages needed to run the workflow, and then spits out the data in the form you need it, whether CSV/TSV/JSON/etc. I don't know much about dat yet, but it looks like it would be a good end-point for serving the data as well.
Hey, thanks for your thoughts. I like the idea of a data specific packaging platform. In our case, we're using the Django and Python system because it's one we're familar with, but I'm open to considering other options and would love to learn more about them.
Comments
A lot of the nitty-gritty data munging and processing often gets discarded after a project or never included in the project repo in a meaningful way. I like Drake [0] because we used it a lot at Factual and it really made data generation and formatting very repeatable and easy.
I really think the packaging system the author is going for would be best built on top of Drake or a similar workflow management program. Instead of following their laundry list of configuration steps one could manage that automatically with a source-controlled workflow. Drake does have the advantages of non-linear and async workflows being pretty easy to build, maintain, and update.
What I would love to see is a data package manager that downloads the raw data and processing workflow, updates any software packages needed to run the workflow, and then spits out the data in the form you need it, whether CSV/TSV/JSON/etc. I don't know much about dat yet, but it looks like it would be a good end-point for serving the data as well.
0 - https://github.com/Factual/drake
Hey, thanks for your thoughts. I like the idea of a data specific packaging platform. In our case, we're using the Django and Python system because it's one we're familar with, but I'm open to considering other options and would love to learn more about them.