Blosc is an excellent choice if speed is what you are after. Give or take 5 years ago I had to use a compression to transport a lot of data over zmq and blosc ran in circles over all other compressions.
Yes it's apparently so fast that in some scenarios it's even usable for compressing RAM. A framework I'm using does that to be able to process much bigger data sets than what would fit in RAM otherwise.
It's a framework called OpenVDB [1], which we use to represent and manipulate volumetric data (level sets). It stores the data as a sparse hierarchical grid with (from a practical perspective) infinite dimensions, and allows very efficient iteration and local manipulations of the grid.
I'm not an expert on how it is implemted exactly, but I believe the way it uses Blosc is by saving leaves of the VDB grids in blosc-compressed chunks, which are loaded into memory directly and only decompressed on-demand when the data is accessed, then re-compressed when the leaves are processed.
Comments
Blosc is an excellent choice if speed is what you are after. Give or take 5 years ago I had to use a compression to transport a lot of data over zmq and blosc ran in circles over all other compressions.
Yes it's apparently so fast that in some scenarios it's even usable for compressing RAM. A framework I'm using does that to be able to process much bigger data sets than what would fit in RAM otherwise.
Can you be more specific around the framework and data type and access patterns ?
It's a framework called OpenVDB [1], which we use to represent and manipulate volumetric data (level sets). It stores the data as a sparse hierarchical grid with (from a practical perspective) infinite dimensions, and allows very efficient iteration and local manipulations of the grid.
I'm not an expert on how it is implemted exactly, but I believe the way it uses Blosc is by saving leaves of the VDB grids in blosc-compressed chunks, which are loaded into memory directly and only decompressed on-demand when the data is accessed, then re-compressed when the leaves are processed.
[1] https://www.openvdb.org