Be careful. Tptacek may show up soon to explain in more detail why this is a dangerous direction to go, but the fact that PBKDF2 is a reasonably well-studied key stretching algorithm is the only reason I'd be comfortable iterating SHA1.
To put it simply, there's just so many incredibly subtle ways to introduce weaknesses into crypto that even if you really do understand everything that's going on, you'll still break things by building it yourself.
I'm not tpatcek, but yes there's a reason why PBKDF2 doesn't do it that way. Because SHA1(SHA1(SHA1(SHA1(...)))) is broken. Approximately 0.8*log_2(iteration count) bits of entropy are lost with that construction.
the whole point is in the "to explain". As far it is a matter of explaining everything is fine and I and others can improve. The problem is when you see 8000 tweets just saying "use bcrypt" without having a clue about what is wrong about another approach.
Note that I also trust that we should try to follow well established and proven standard, simply not as drones.
Marshray started getting dead comments for some reason 4 days ago, but his reply to my top comment is extremely relevant:
> I'm not tpatcek, but yes there's a reason why PBKDF2 doesn't do it that way. Because SHA1(SHA1(SHA1(SHA1(...)))) is broken. Approximately 0.8*log_2(iteration count) bits of entropy are lost with that construction.
http://eprint.iacr.org/2010/384
It's not a horrifying game-over break to go from 160 bits of entropy to 154 with 1,000 iterations, but you're not getting the security you thought you were; and there's no amount of personal investigation into the problem that would have helped--this is the kind of thing that takes an ecosystem of very smart people working very hard for a long time.
antirez, while I really appreciate the sentiment that one should not simply follow dogmas but be curious/question/understand them, there is a fine balance. In your case, if it were a learning exercise, it's perfectly fine to invest time and understand why other approaches are bad.
But if you're writing a software that other people depend on, perhaps it's best if you follow widely accepted principles, but do the questioning and curiosity _later_. On a meta level, depending on a person's level of curiosity, it's very difficult to answer why certain things are the way they are. Sometimes, you just have to take the word for it. :)
I am not pointing fingers at you specifically, but it's just a general observation. I do not know if Lamer News was just an experiment, or a serious effort that others can use.
I agree with you. The problem comes when we are so focused on proven stuff that we end allowing only one possibility at all, since the implementation I suggested was well established practice with an RFC.
> The problem comes when we are so focused on proven stuff...
I sympathize with your overall point, but there's a really good reason why the focus should be on proven stuff: because it's been proven.
You are building an application which will be responsible for safeguarding some of its users' information. You should typically want to do that using a proven approach. If nothing else, it's a cover-your-ass situation: if it turns out that the proven approach has a flaw, you can at least say that you followed standard practices. If your invention has a flaw, you hold all the responsibility.
I hate to fall back to argument-by-analogy (but I have to, since I'm not a cryptographer), but you're somewhat in the situation of installing a fire suppression system for someone, and you're saying, "Well, I don't want to use halon for this, even though it's what all the experts say I should use, because I don't want to deal with the company selling the halon equipment. So let's just use lots and lots of kitchen fire extinguishers instead. They do the same job, and if I use lots of them, it will work just as well."
Maybe. Maybe it will. But, if it doesn't, you will be responsible for having made a decision that flies in the face of lots of advice from lots of very smart people, and the consequences of that decision will affect not just you but also anyone that uses your system.
edit: On the more technical side of things, maybe nobody has yet pointed you to Bruce Schneier's 2005 article on SHA1 weaknesses (http://www.schneier.com/blog/archives/2005/02/cryptanalysis_...). There are several very good points in there about the dangers of using broken cryptographic algorithms. By stretching a broken hash, you could be -- and probably are -- compounding its weaknesses. i.e., you might be making it easier to find a collision, not harder. And, that article was written almost 7 years ago; since then, newer low-cost hardware has been made available which can compute 33 billion MD5 hashes per second (http://blog.zorinaq.com/?e=42), and that was almost a year ago. I swear I read this year about some Russian kids doing a lot better than that, and that ignores things like EC2 which make it economical to do heavy-duty distributed hash breaking.
Comments
Be careful. Tptacek may show up soon to explain in more detail why this is a dangerous direction to go, but the fact that PBKDF2 is a reasonably well-studied key stretching algorithm is the only reason I'd be comfortable iterating SHA1.
To put it simply, there's just so many incredibly subtle ways to introduce weaknesses into crypto that even if you really do understand everything that's going on, you'll still break things by building it yourself.
I'm not tpatcek, but yes there's a reason why PBKDF2 doesn't do it that way. Because SHA1(SHA1(SHA1(SHA1(...)))) is broken. Approximately 0.8*log_2(iteration count) bits of entropy are lost with that construction.
http://eprint.iacr.org/2010/384
the whole point is in the "to explain". As far it is a matter of explaining everything is fine and I and others can improve. The problem is when you see 8000 tweets just saying "use bcrypt" without having a clue about what is wrong about another approach.
Note that I also trust that we should try to follow well established and proven standard, simply not as drones.
Marshray started getting dead comments for some reason 4 days ago, but his reply to my top comment is extremely relevant:
> I'm not tpatcek, but yes there's a reason why PBKDF2 doesn't do it that way. Because SHA1(SHA1(SHA1(SHA1(...)))) is broken. Approximately 0.8*log_2(iteration count) bits of entropy are lost with that construction. http://eprint.iacr.org/2010/384
It's not a horrifying game-over break to go from 160 bits of entropy to 154 with 1,000 iterations, but you're not getting the security you thought you were; and there's no amount of personal investigation into the problem that would have helped--this is the kind of thing that takes an ecosystem of very smart people working very hard for a long time.
antirez, while I really appreciate the sentiment that one should not simply follow dogmas but be curious/question/understand them, there is a fine balance. In your case, if it were a learning exercise, it's perfectly fine to invest time and understand why other approaches are bad.
But if you're writing a software that other people depend on, perhaps it's best if you follow widely accepted principles, but do the questioning and curiosity _later_. On a meta level, depending on a person's level of curiosity, it's very difficult to answer why certain things are the way they are. Sometimes, you just have to take the word for it. :)
I am not pointing fingers at you specifically, but it's just a general observation. I do not know if Lamer News was just an experiment, or a serious effort that others can use.
Cheers,
I agree with you. The problem comes when we are so focused on proven stuff that we end allowing only one possibility at all, since the implementation I suggested was well established practice with an RFC.
> The problem comes when we are so focused on proven stuff...
I sympathize with your overall point, but there's a really good reason why the focus should be on proven stuff: because it's been proven.
You are building an application which will be responsible for safeguarding some of its users' information. You should typically want to do that using a proven approach. If nothing else, it's a cover-your-ass situation: if it turns out that the proven approach has a flaw, you can at least say that you followed standard practices. If your invention has a flaw, you hold all the responsibility.
I hate to fall back to argument-by-analogy (but I have to, since I'm not a cryptographer), but you're somewhat in the situation of installing a fire suppression system for someone, and you're saying, "Well, I don't want to use halon for this, even though it's what all the experts say I should use, because I don't want to deal with the company selling the halon equipment. So let's just use lots and lots of kitchen fire extinguishers instead. They do the same job, and if I use lots of them, it will work just as well."
Maybe. Maybe it will. But, if it doesn't, you will be responsible for having made a decision that flies in the face of lots of advice from lots of very smart people, and the consequences of that decision will affect not just you but also anyone that uses your system.
edit: On the more technical side of things, maybe nobody has yet pointed you to Bruce Schneier's 2005 article on SHA1 weaknesses (http://www.schneier.com/blog/archives/2005/02/cryptanalysis_...). There are several very good points in there about the dangers of using broken cryptographic algorithms. By stretching a broken hash, you could be -- and probably are -- compounding its weaknesses. i.e., you might be making it easier to find a collision, not harder. And, that article was written almost 7 years ago; since then, newer low-cost hardware has been made available which can compute 33 billion MD5 hashes per second (http://blog.zorinaq.com/?e=42), and that was almost a year ago. I swear I read this year about some Russian kids doing a lot better than that, and that ignores things like EC2 which make it economical to do heavy-duty distributed hash breaking.
Interesting points. Another way to convince oneself, is that even experienced professionals are prone to make mistakes.
This is an interesting set of slides that captures the same thoughts:
Ask and you shall receive:
http://codahale.com/how-to-safely-store-a-password/
(also explains _why_)
I'll be succinct: if I found a client iterating SHA1 to build a password hash, I would not write them up for it.