I usually recommend an 8-byte length suffix for each piece instead of a separator.* That way you don't need to make any assumptions about what bytes are legal. Putting the length in suffix position means you can still stream input into your hash state without knowing how long it's going to be in advance, and 2^64 bytes is enough for any practical application. (The maximum input size of SHA-256 is less than this.)
I'm less sanguine than the authors about using Protobuf or CBOR for "canonical" serialization like this. I think it tends to "work until it doesn't". It's not what these formats were designed for, and you have to ask awkward questions like "is the order of struct fields guaranteed?" and "do integers always use the smallest possible representation?". This is more obvious for JSON, as the post points out, but every common serialization format I'm aware of has problems like this. I think we need a dedicated standard to do a good job of this, but I'm not aware of anything widespread. It's a surprisingly hard problem.
* Technically you only need the suffix for each variable-length piece, and you can omit the first one. But it's more complicated if the number of pieces is variable. (If you have two adjacent fixed-length pieces, and then you combine them, does the hash change?) This sort of penny-pinching is interesting to think about in a design that's going to come with a giant set of test vectors, but it's asking for trouble in an application doing something custom. This is another reason I'd like to have a standard here.
It must be a length prefix or you're still vulnerable to length extension attacks. Suppose you had encoded "foo3"; the attacker could easily extend that to "foo3bar7".
This is confusing two different questions, "How do we hash multiple things at once without breaking the collision property?" and "How do we MAC messages in a way an attacker can't forge?" Length suffixes are answering the first question, not the second.
I agree that this approach works in the use case you're describing, but there are a couple reasons I wouldn't want to teach it broadly. Mainly, a caveat like "remember to only use this with UTF-8 strings and not with arbitrary bytes" is exactly the sort of thing applications routinely get wrong in the wild. Also, focusing on wacky edge cases, do we really know that our UTF-8 strings are valid UTF-8? Maybe we're comfortable staking our security on that in a language like Rust, where invalid strings are literally undefined behavior anyway. But what about in Go for example, where this snippet writes 0xff to stdout with no warnings whatsoever?
func main() {
s := string([]byte{0xff})
fmt.Println(s)
}
do we really know that our UTF-8 strings are valid UTF-8?
I’m explicitly talking about cases where you UTF-8-encode right where you’re hashing.
With the length approach, you also have to take care you’re using the correct byte length, for example, and not forget the final length (i.e. it’s a suffix and not just a separator). And for user input like passwords, you’d probably want to NFC-canonicalize before hashing. There’s always things you have to pay attention to.
You can encapsulate it in a function that hashes a list of character strings passed as its (typed) argument. Then you have a safe and reusable function.
I’m explicitly talking about cases where you UTF-8-encode right where you’re hashing.
Totally, I get that. I think what you're pointing out is that in a language like Python for example, the scenario I'm trying to describe is meaningless. You can't make an "invalid string" in Python (as far as I know, without resorting to FFI), because it checks things like that during string decoding, and it'll just crash.
But languages like C/C++/Rust/Go work differently. As these languages are commonly used, the string -> UTF-8 step is actually a no-op, because the assumption is that strings are already UTF-8 in memory. (In C or C++ this is usually in the programmer's head rather than in the types, but it's a common choice.) In these languages it's possible for the result of that no-op "encoding" to be invalid, if the input string was invalid somehow. This is a pretty weird edge case and almost certainly a bug that the application needs to fix fix anyway, but if we're noodling about cryptography best practices, it might be nice to limit the "blast radius" of a bug like that.
But languages like C/C++/Rust/Go work differently. As these languages are commonly used, the string -> UTF-8 step is actually a no-op, because the assumption is that strings are already UTF-8 in memory.
No.
Rust's string types are explicitly UTF-8 text. If what you've got isn't UTF-8 text, it's not a Rust string type. Here's the signature of the conversion you say is "actually a no-op".
That says if you've got a growable array of bytes and you claim it's UTF-8 text, you can have a String back if you're correct about that. If you were wrong you get a FromUtf8Error, which is a wrapper around that growable array and some diagnostic information.
Edited to add:
The reason I was looking at this thread is because of course Rust for its own purposes does exactly what layer8 describes - it emits a single 0xFF byte to separate strings because Rust's strings are guaranteed UTF-8.
why are you hashing "username–password"? Isn't username a key index for the lookup of hashed(password), or do you select all hashed("username–password") and see if any match and if so, authenticate?
Does that mean the random suffix is effectively a secret? Otherwise an attacker could construct inputs with the separator, achieving the same thing as a simple separator, right?
It's not a random suffix, but a length suffix. So for example putting "hello" and "world!" together looks like "hello" | 5 | "world!" | 6. Concretely, you probably represent 5 and 6 as eight little-endian bytes, i.e. `n.to_bytes(8, 'little')` in Python. If you want to stop attackers from guessing/constructing hashes, you need to get a secret key involved, which is what the "YoloMAC" section of the post is about.
bencode was designed to solve the canonicality problem. so is asn.1 cer. but it is sufficient to have an unambiguous encoding to prevent the vulnerabilities in this post; nonunique encodings will only produce interoperability problems, not security holes
Comments
I usually recommend an 8-byte length suffix for each piece instead of a separator.* That way you don't need to make any assumptions about what bytes are legal. Putting the length in suffix position means you can still stream input into your hash state without knowing how long it's going to be in advance, and 2^64 bytes is enough for any practical application. (The maximum input size of SHA-256 is less than this.)
I'm less sanguine than the authors about using Protobuf or CBOR for "canonical" serialization like this. I think it tends to "work until it doesn't". It's not what these formats were designed for, and you have to ask awkward questions like "is the order of struct fields guaranteed?" and "do integers always use the smallest possible representation?". This is more obvious for JSON, as the post points out, but every common serialization format I'm aware of has problems like this. I think we need a dedicated standard to do a good job of this, but I'm not aware of anything widespread. It's a surprisingly hard problem.
* Technically you only need the suffix for each variable-length piece, and you can omit the first one. But it's more complicated if the number of pieces is variable. (If you have two adjacent fixed-length pieces, and then you combine them, does the hash change?) This sort of penny-pinching is interesting to think about in a design that's going to come with a giant set of test vectors, but it's asking for trouble in an application doing something custom. This is another reason I'd like to have a standard here.
It must be a length prefix or you're still vulnerable to length extension attacks. Suppose you had encoded "foo3"; the attacker could easily extend that to "foo3bar7".
This is confusing two different questions, "How do we hash multiple things at once without breaking the collision property?" and "How do we MAC messages in a way an attacker can't forge?" Length suffixes are answering the first question, not the second.
With sha2 (non truncated) but not with all hash functions eg sha3.
It depends on the use case. If, for example, you want to hash a username–password pair, I find
to be most straightforward and parsimonious, and the assumption is maximally local.I agree that this approach works in the use case you're describing, but there are a couple reasons I wouldn't want to teach it broadly. Mainly, a caveat like "remember to only use this with UTF-8 strings and not with arbitrary bytes" is exactly the sort of thing applications routinely get wrong in the wild. Also, focusing on wacky edge cases, do we really know that our UTF-8 strings are valid UTF-8? Maybe we're comfortable staking our security on that in a language like Rust, where invalid strings are literally undefined behavior anyway. But what about in Go for example, where this snippet writes 0xff to stdout with no warnings whatsoever?
I’m explicitly talking about cases where you UTF-8-encode right where you’re hashing.
With the length approach, you also have to take care you’re using the correct byte length, for example, and not forget the final length (i.e. it’s a suffix and not just a separator). And for user input like passwords, you’d probably want to NFC-canonicalize before hashing. There’s always things you have to pay attention to.
You can encapsulate it in a function that hashes a list of character strings passed as its (typed) argument. Then you have a safe and reusable function.
Totally, I get that. I think what you're pointing out is that in a language like Python for example, the scenario I'm trying to describe is meaningless. You can't make an "invalid string" in Python (as far as I know, without resorting to FFI), because it checks things like that during string decoding, and it'll just crash.
But languages like C/C++/Rust/Go work differently. As these languages are commonly used, the string -> UTF-8 step is actually a no-op, because the assumption is that strings are already UTF-8 in memory. (In C or C++ this is usually in the programmer's head rather than in the types, but it's a common choice.) In these languages it's possible for the result of that no-op "encoding" to be invalid, if the input string was invalid somehow. This is a pretty weird edge case and almost certainly a bug that the application needs to fix fix anyway, but if we're noodling about cryptography best practices, it might be nice to limit the "blast radius" of a bug like that.
No.
Rust's string types are explicitly UTF-8 text. If what you've got isn't UTF-8 text, it's not a Rust string type. Here's the signature of the conversion you say is "actually a no-op".
pub fn from_utf8(vec: Vec<u8>) -> Result<String, FromUtf8Error>
That says if you've got a growable array of bytes and you claim it's UTF-8 text, you can have a String back if you're correct about that. If you were wrong you get a FromUtf8Error, which is a wrapper around that growable array and some diagnostic information.
Edited to add:
The reason I was looking at this thread is because of course Rust for its own purposes does exactly what layer8 describes - it emits a single 0xFF byte to separate strings because Rust's strings are guaranteed UTF-8.
why are you hashing "username–password"? Isn't username a key index for the lookup of hashed(password), or do you select all hashed("username–password") and see if any match and if so, authenticate?
Does that mean the random suffix is effectively a secret? Otherwise an attacker could construct inputs with the separator, achieving the same thing as a simple separator, right?
It's not a random suffix, but a length suffix. So for example putting "hello" and "world!" together looks like "hello" | 5 | "world!" | 6. Concretely, you probably represent 5 and 6 as eight little-endian bytes, i.e. `n.to_bytes(8, 'little')` in Python. If you want to stop attackers from guessing/constructing hashes, you need to get a secret key involved, which is what the "YoloMAC" section of the post is about.
bencode was designed to solve the canonicality problem. so is asn.1 cer. but it is sufficient to have an unambiguous encoding to prevent the vulnerabilities in this post; nonunique encodings will only produce interoperability problems, not security holes