I had serious WFIO case a few years ago. We were building a crypto solution on J2ME phones.
The solution was days away from roll-out with our first big corporate client. Late on a Friday afternoon we were busy with final field testing - paying students a few $ to use their phones to test the app.
As the test data came in we realised we had a major problem: a small % of the phones weren't returning the correct test vectors for hashing algo.
After checking for obvious user error we came to the conclusion that something big was broken. Specific firmware sets didn't execute the crypto part of the code correctly. The entire value-proposition was that it works on every phone that can run an app, so it was a pretty big deal. I thought we were totally fucked.
We didn't sleep for two days and finally found the bug in the way the phones implement a bit-shift operation (doesn't carry a bit about 1/10000000 times). Then had to figure out a workaround that was still fast enough.
We shipped a fixed version before the end of the weekend, but I wouldn't wish that kind of stress on anyone.
I ran into something similar a few months ago. There's a bug in Oracle JDK 7u45 that causes SSL handshakes to fail ~5% of the time due to a bug in that version's Diffie-Hellman cypher suite.
Comments
I had serious WFIO case a few years ago. We were building a crypto solution on J2ME phones.
The solution was days away from roll-out with our first big corporate client. Late on a Friday afternoon we were busy with final field testing - paying students a few $ to use their phones to test the app.
As the test data came in we realised we had a major problem: a small % of the phones weren't returning the correct test vectors for hashing algo.
After checking for obvious user error we came to the conclusion that something big was broken. Specific firmware sets didn't execute the crypto part of the code correctly. The entire value-proposition was that it works on every phone that can run an app, so it was a pretty big deal. I thought we were totally fucked.
We didn't sleep for two days and finally found the bug in the way the phones implement a bit-shift operation (doesn't carry a bit about 1/10000000 times). Then had to figure out a workaround that was still fast enough.
We shipped a fixed version before the end of the weekend, but I wouldn't wish that kind of stress on anyone.
I ran into something similar a few months ago. There's a bug in Oracle JDK 7u45 that causes SSL handshakes to fail ~5% of the time due to a bug in that version's Diffie-Hellman cypher suite.
Tracking that bug down was a fun three days.