For anyone else reading this and rolling their own crypto (hint: don't! No really, just don't!):
a lax PKCS#1 v1.5 RSA signature verifier that failed to enforce the total encoded length (256 bytes) or pin the digest to its correct length for the claimed hash, allowing an attacker to forge a valid-looking signature without the private key
the correct way to do this is encode-then-memcmp(). You can't get it wrong that way because a memcmp() only has two outcomes, match or no match, not a whole range of "seems to work OK on the tests we ran it on".
Comments
For anyone else reading this and rolling their own crypto (hint: don't! No really, just don't!):
the correct way to do this is encode-then-memcmp(). You can't get it wrong that way because a memcmp() only has two outcomes, match or no match, not a whole range of "seems to work OK on the tests we ran it on".