For this use-case, you can squeeze out even more performance by using the SHA-1 implementation in Intel ISA-L Crypto [1]. The SHA-1 implementation there allows for multi-buffer hashes, giving you the ability to calculate the hashes for multiple chunks in parallel on a single core. Given that that is basically your usecase, it might be worth considering. I doubt it'll provide much speedup if you're already I/O bound here though.
I came across this repo recently and it looks great. It's a pity there doesn't seem to be an official Ubuntu package for it though. There is one for the Intelligent Storage Acceleration Library though.
Comments
For this use-case, you can squeeze out even more performance by using the SHA-1 implementation in Intel ISA-L Crypto [1]. The SHA-1 implementation there allows for multi-buffer hashes, giving you the ability to calculate the hashes for multiple chunks in parallel on a single core. Given that that is basically your usecase, it might be worth considering. I doubt it'll provide much speedup if you're already I/O bound here though.
[1]: https://github.com/intel/isa-l_crypto
Thank you, I will definitely have a look, and update the article if there’s any interesting finding
I came across this repo recently and it looks great. It's a pity there doesn't seem to be an official Ubuntu package for it though. There is one for the Intelligent Storage Acceleration Library though.