If you combine mmap with "filesystem with transparent compression", and want the efficiency of mmap, mmap will only see the compressed data.

If you want your mmap to magically see the uncompressed data, your file system will have to do decompress the data, and that doesn't come for free.

I would try and aim for compression in the application, as data size likely will be the bottleneck in reading and writing such files. If your data isn't very sparse, you could delta encode the indices of the non-zero columns, and use some variable-length encoding for it. Compressing each row of deltas may help after the delta encoding (especially if it is reasonably dense, because you expect the deltas to be small).

Once you go down that route, you have sacrificed simplicity, so you might just as well encode your floats, too.