Sequence read archive has been such a boon for developing reproducible biological pipelines without having to worry about data. A paper references a dataset by ID and I can use it as input for my pipelines and keep the raw data locally only for as long as its needed to generate analysis within the running pipeline. I can even set threshold levels of how much local or cloud compute resources should be used at a time if I didn't want to exhaust my systems with one job.
Comments
Sequence read archive has been such a boon for developing reproducible biological pipelines without having to worry about data. A paper references a dataset by ID and I can use it as input for my pipelines and keep the raw data locally only for as long as its needed to generate analysis within the running pipeline. I can even set threshold levels of how much local or cloud compute resources should be used at a time if I didn't want to exhaust my systems with one job.