Are there advantages over storing data in HDF? I've been working with a few hundred gigabytes of financial data this summer and I'm finding that python's data-oriented libraries (h5py, numpy, scipy, matplotlib, scikits.learn) cover my needs.
depends on your usage. The rest of the toolchain is all python, numpy, scipy, matplotlib especially.
This might be poorly titled: it's not just about the storage as it is about the aggregation of disparate sensor data into coherent, continuous data streams.
if your data can fit into arrays, then there's no advantage in terms of the types of aggregations.
however mongo allows you to store complex structures, think nested dictionaries/lists, and query on those nested structures, even allowing you to reach inside of nested structures to do the querying.
I guess you could do nested structured arrays in numpy, I've never done that before.
I use h5py's datasets (which are organized hierarchically and stored in compressed chunks) to do basic filtering and then load a fraction of my data into memory as numpy arrays.
Comments
Are there advantages over storing data in HDF? I've been working with a few hundred gigabytes of financial data this summer and I'm finding that python's data-oriented libraries (h5py, numpy, scipy, matplotlib, scikits.learn) cover my needs.
depends on your usage. The rest of the toolchain is all python, numpy, scipy, matplotlib especially.
This might be poorly titled: it's not just about the storage as it is about the aggregation of disparate sensor data into coherent, continuous data streams.
I'm totally ignorant of mongodb: what does it do for you (in the way of data aggregation) that's not easy in numpy?
if your data can fit into arrays, then there's no advantage in terms of the types of aggregations.
however mongo allows you to store complex structures, think nested dictionaries/lists, and query on those nested structures, even allowing you to reach inside of nested structures to do the querying.
I guess you could do nested structured arrays in numpy, I've never done that before.
I use h5py's datasets (which are organized hierarchically and stored in compressed chunks) to do basic filtering and then load a fraction of my data into memory as numpy arrays.