However, many engineers, including kernel hackers AFAIK, still subscribe to the more-is-better model, making Fortuna an obvious choice. But they also still believe they can accurately and _reliably_ estimate entropy[1], which Fortuna moved away from. And Fortuna is probably slower than they'd like. They recently moved away from their homegrown hashes to DJB's ChaCha20 stream cipher, with much improved single-core and multi-core performance.
[1] It's been awhile since I researched this, but long ago, in a land far, far way, there was published a Usenix paper (IIRC) that carefully examined and estimated entropy from disk latencies. These figures were hardcoded into many entropy estimators, including the Linux kernel's, over many cycles of hard drive evolution. IOW, even assuming the paper accurately measured entropy of then-current hard drive technology, it was totally irrelevant over the subsequent decade (and maybe more) it was relied upon. No matter how conservatively you discounted the estimate, the baseline was effectively arbitrary in the context of newer technology. And AFAIK nobody even tried carefully re-validating the original paper. At best there was just more hand waving, fudging of constants, and equivocations of spinning rust.
Which is one reason why, IMHO, entropy estimation is a bad idea, especially when you allow it to dictate the design of your PRNG. It's fundamentally all a bunch of hand waving outside the context of purpose built RNG hardware, and even if you got it right, the next model of a product could destroy your proof. One of the designers of the new PRNG in the Linux kernel put a lot of effort into entropy estimation, including comprehensive software instrumentation to validate some of his assumptions. That's laudable work, but even if correct and accurate, unless you lock him in a room and feed him every new model of CPU, memory controller, disk controller, etc, coming out, it's not reliable.
From a security posture, you never want to make decisions based on the assumption that you're at least as clever as a hacker; that you thought of _all_ the corner cases that might be exploitable. That's what entropy estimation is--trying to discover all the hidden variables and then, when you've exhausted your patience, assuming there are no more, at least none your attacker is likely to discover.
Rather, like with ciphers, you want your proof to be capable of being distilled down to a small number of very clear variables and premises (e.g. key length, cipher construction), which you can then easily keep an eye on for evidence of invalidation. Entropy estimation is really far removed from that model of risk analysis.
Libgcrypt was probably considered a top tier implementation in its time (I'm guessing), but this man page[1] brings up a lot of concerns along the lines of the things you mention.
You've got:
- Multiple "Random quality levels", defined as "weak random", "strong random", and "very strong random". How does a any CSPRNG sensibly differentiate?
- "Weak random" is described as for use in "all functions", with a specific list of exceptions. In other words, it's the default.
- A random function that doesn't "drain the precious entropy pool", which is described as producing "unpredictable bytes"
- A "secure" variant of the gcry_random_bytes function, which still takes a "level" as an input, including "weak"
If I picture myself as a developer using this, there's a horrible amount of options, and a feeling that every recommended option is a place I'm likely to make a fatal error.
Purpose-built RNG hardware is not a panacea --- in fact, it can be treacherous. Hardware RNGs can fail transiently, and you have to detect and compensate for those failures.
You're absolutely right. I didn't mean to imply that it was. But purpose built hardware has design specifications that can be validated. Good hardware is designed with failure modes in mind. Which means you have some hope of a decent risk management strategy.
And, exactly as you said, using and relying on a hardware RNG isn't a panacea--it still comes with difficult problems--which only emphasizes how hopeless, futile, and misleading entropy estimation is.
I'm not arguing that trying to indirectly capture entropy is a bad idea, even if you also have a proper hardware RNG. My point is just that entropy estimation is a poor idea, particularly of hardware that hasn't been specifically analyzed. By pretending like we can reliably quantify the entropy, we're giving users a false sense of security. That's not justifiable.
Collecting entropy indirectly is best effort. You can't know, except in pathological cases, whether it's working or not. You can only know when it's failing spectacularly, which would be rare when collecting from many different sources.
All those magic, per-device constants could likely be replaced with a single heuristic--collect everything for N seconds, then move on. If it worked, it worked; if not, well then you're not any worse off than you were when the entropy estimators gave a false sense of confidence. You can still add hacks to try to collect more entropy, and to do it faster, to improve effective security; just don't pretend like the kernel is accurately quantifying it.
Comments
One good reason would be:
However, many engineers, including kernel hackers AFAIK, still subscribe to the more-is-better model, making Fortuna an obvious choice. But they also still believe they can accurately and _reliably_ estimate entropy[1], which Fortuna moved away from. And Fortuna is probably slower than they'd like. They recently moved away from their homegrown hashes to DJB's ChaCha20 stream cipher, with much improved single-core and multi-core performance.[1] It's been awhile since I researched this, but long ago, in a land far, far way, there was published a Usenix paper (IIRC) that carefully examined and estimated entropy from disk latencies. These figures were hardcoded into many entropy estimators, including the Linux kernel's, over many cycles of hard drive evolution. IOW, even assuming the paper accurately measured entropy of then-current hard drive technology, it was totally irrelevant over the subsequent decade (and maybe more) it was relied upon. No matter how conservatively you discounted the estimate, the baseline was effectively arbitrary in the context of newer technology. And AFAIK nobody even tried carefully re-validating the original paper. At best there was just more hand waving, fudging of constants, and equivocations of spinning rust.
Which is one reason why, IMHO, entropy estimation is a bad idea, especially when you allow it to dictate the design of your PRNG. It's fundamentally all a bunch of hand waving outside the context of purpose built RNG hardware, and even if you got it right, the next model of a product could destroy your proof. One of the designers of the new PRNG in the Linux kernel put a lot of effort into entropy estimation, including comprehensive software instrumentation to validate some of his assumptions. That's laudable work, but even if correct and accurate, unless you lock him in a room and feed him every new model of CPU, memory controller, disk controller, etc, coming out, it's not reliable.
From a security posture, you never want to make decisions based on the assumption that you're at least as clever as a hacker; that you thought of _all_ the corner cases that might be exploitable. That's what entropy estimation is--trying to discover all the hidden variables and then, when you've exhausted your patience, assuming there are no more, at least none your attacker is likely to discover.
Rather, like with ciphers, you want your proof to be capable of being distilled down to a small number of very clear variables and premises (e.g. key length, cipher construction), which you can then easily keep an eye on for evidence of invalidation. Entropy estimation is really far removed from that model of risk analysis.
Libgcrypt was probably considered a top tier implementation in its time (I'm guessing), but this man page[1] brings up a lot of concerns along the lines of the things you mention.
You've got:
- Multiple "Random quality levels", defined as "weak random", "strong random", and "very strong random". How does a any CSPRNG sensibly differentiate?
- "Weak random" is described as for use in "all functions", with a specific list of exceptions. In other words, it's the default.
- A random function that doesn't "drain the precious entropy pool", which is described as producing "unpredictable bytes"
- A "secure" variant of the gcry_random_bytes function, which still takes a "level" as an input, including "weak"
If I picture myself as a developer using this, there's a horrible amount of options, and a feeling that every recommended option is a place I'm likely to make a fatal error.
[1] https://www.gnupg.org/documentation/manuals/gcrypt/Retrievin...
Sadly it's still used by even recent software projects, notably systemd.
Purpose-built RNG hardware is not a panacea --- in fact, it can be treacherous. Hardware RNGs can fail transiently, and you have to detect and compensate for those failures.
You're absolutely right. I didn't mean to imply that it was. But purpose built hardware has design specifications that can be validated. Good hardware is designed with failure modes in mind. Which means you have some hope of a decent risk management strategy.
And, exactly as you said, using and relying on a hardware RNG isn't a panacea--it still comes with difficult problems--which only emphasizes how hopeless, futile, and misleading entropy estimation is.
I'm not arguing that trying to indirectly capture entropy is a bad idea, even if you also have a proper hardware RNG. My point is just that entropy estimation is a poor idea, particularly of hardware that hasn't been specifically analyzed. By pretending like we can reliably quantify the entropy, we're giving users a false sense of security. That's not justifiable.
Collecting entropy indirectly is best effort. You can't know, except in pathological cases, whether it's working or not. You can only know when it's failing spectacularly, which would be rare when collecting from many different sources.
All those magic, per-device constants could likely be replaced with a single heuristic--collect everything for N seconds, then move on. If it worked, it worked; if not, well then you're not any worse off than you were when the entropy estimators gave a false sense of confidence. You can still add hacks to try to collect more entropy, and to do it faster, to improve effective security; just don't pretend like the kernel is accurately quantifying it.