This fragmentation is already causing issues for myself and many other developers. I work at a small startup doing gov't work and we develop Android applications that talk to massive server based backends. Yay cloud!
On 2.2 we were seeing issues with the GC not firing enough, especially after shutting down a thread and no longer requiring the data allocated therein and thus we would run out of memory (heap size being set to 24 MB), on our 2.3 devices we were not seeing that issue because 1, the heap size was set to 32 MB, but 2, because our app used much less memory, from what we could see when a thread went away the GC properly cleaned up the memory associated with it, not only that but the same functionality had a difference of almost 5 MB of heap size. The 2.3 VM was more efficient compared to the 2.2 VM.
We have also found many issues with BouncyCastle, the default Android encryption/decryption library that we couldn't reproduce with Java JCE, we filed bugs with BouncyCastle and they said they were features/that we were using it wrong/that they wouldn't fix it. We ended up writing our own CTR block mode (no, we didn't rewrite AES) to fix one of the many issues we found with BouncyCastle.
Also, the SQLite version on the 2.x line of Android OS allows certain constructs that are technically not legal in the versions it claims to be but it accepts and ignores that bad input. The developer had taken output from MySQL workbench and put it in as SQL directly into the SQLite stuff on Android no realising some of his stuff was being ignored because it wasn't valid, no errors were thrown though. As soon as we ran our APK on an Android 3.0 tablet the app would crash because the SQLite there DOES throw errors. Yes it was a simple fix, but it shouldn't have been allowed in the first place!
We also found a whole range of issues with various different keyboards that you can download from the market. Some were causing our app to crash, others would cause the text entry field to show random characters yet when reading back the text from the field in code just the user inputted stuff was there. We'd have our software designed to have a button in a certain location but with certain keyboards up the button was no longer a clickable target and you had to first exit the keyboard. Looking at our bug tracker keyboard related issues are HUGE and there are plenty. It is even worse because the same keyboard on one device may work just fine but move to another device from a different manufacturer and it may be completely broken making it hard to verify bugs do exist.
We now have almost 20 test devices that we have to manually test our software on to make sure it looks right, that nothing is overlapping, and that it works correctly. Designing for multiple different sizes of screen, and then for landscape mode on those screens is absolutely horrible. On some screens elements get so stretched in landscape it just looks terrible and on others everything is so squashed together in portrait mode that it makes it hard for the user to accurately hit a target.
Fragmentation is driving me personally insane. I wonder how much of my work related stress is from having to deal with that kind of crap.
The CTR mode in bouncy castle does not allow one to do partial block encryption. So if you want to encrypt 4 bytes you need to pad it to the full 16 bytes. It will always want to do a full block.
This also means that you can't do a partial offset into the CTR. You can increment the counter correctly, but you can't start encrypting from within a partial block.
Lets say you need to encrypt 17 bytes and write them to a file, which later can be opened in append mode to append more data to it using AES-128/CTR-BE mode:
1234567890ABCDEF\0
C-style terminated string.
That is 2 blocks in CTR mode (1 full block used, 1 partial block (1 byte out of a 16 byte block)).
The next time you open the file you want to append data to it, what you would do is this:
1. Load your previous key, and counter into AES-128/CTR
2. Increment the counter by the full blocks already used
3. Update the location into the current block by encrypting a random byte
4. Provide the rest of your plaintext which will now be correctly encrypted for appending to an already encrypted segment
Now you can read your file back using the following:
1. the AES key, counter for the start of the file
2. Use CTR to "decrypt" the content from start to finish
So that would look something like this in Java code with JCE:
Cipher c = Cipher.getInstance("AES/CTR/NoPadding");
SecretKeySpec keySpec = new SecretKeySpec(aeskey, "AES");
IvParameterSpec ivSpec = new IvParameterSpec(iv);
c.init(Cipher.ENCRYPT_MODE, keySpec, ivSpec);
I've left out some stuff like creating the iv (really the counter), so now if we wanted to advance into the first block what we could do (and this works with JCE) is this:
c.update(new byte[count]);
Where count contains the amount we want to offset into the next block.
On JCE this does exactly what I described above, it moves forward "count" characters into the next block and then you can do:
CipherOutputStream cipher_out = new CipherOutputStream(output, c);
where "output" is DataOutputStream(new FileOutputStream("filename", true)) which is a file opened in append mode. Now you can write stuff to the file using the normal functions used by an OutputStream. This will then correctly append your new data to the end of the file so that if you start reading at the beginning of the file you can read through the end and get valid data. (We are using this to encrypt log files that are opened in Append only mode).
Where BouncyCastle breaks down is that you can not use c.update() to move forward into the block, thus appending is not possible because if you don't end on a block boundary your next write is going to have overlap. The only thing you can do in c.update() is provide it the amount of bytes that is the same as the block size. So you have to pad all input into c.update() to 16 bytes. Basically instead of being a stream cipher that allows seeking it has become yet another block cipher.
Comments
This fragmentation is already causing issues for myself and many other developers. I work at a small startup doing gov't work and we develop Android applications that talk to massive server based backends. Yay cloud!
On 2.2 we were seeing issues with the GC not firing enough, especially after shutting down a thread and no longer requiring the data allocated therein and thus we would run out of memory (heap size being set to 24 MB), on our 2.3 devices we were not seeing that issue because 1, the heap size was set to 32 MB, but 2, because our app used much less memory, from what we could see when a thread went away the GC properly cleaned up the memory associated with it, not only that but the same functionality had a difference of almost 5 MB of heap size. The 2.3 VM was more efficient compared to the 2.2 VM.
We have also found many issues with BouncyCastle, the default Android encryption/decryption library that we couldn't reproduce with Java JCE, we filed bugs with BouncyCastle and they said they were features/that we were using it wrong/that they wouldn't fix it. We ended up writing our own CTR block mode (no, we didn't rewrite AES) to fix one of the many issues we found with BouncyCastle.
Also, the SQLite version on the 2.x line of Android OS allows certain constructs that are technically not legal in the versions it claims to be but it accepts and ignores that bad input. The developer had taken output from MySQL workbench and put it in as SQL directly into the SQLite stuff on Android no realising some of his stuff was being ignored because it wasn't valid, no errors were thrown though. As soon as we ran our APK on an Android 3.0 tablet the app would crash because the SQLite there DOES throw errors. Yes it was a simple fix, but it shouldn't have been allowed in the first place!
We also found a whole range of issues with various different keyboards that you can download from the market. Some were causing our app to crash, others would cause the text entry field to show random characters yet when reading back the text from the field in code just the user inputted stuff was there. We'd have our software designed to have a button in a certain location but with certain keyboards up the button was no longer a clickable target and you had to first exit the keyboard. Looking at our bug tracker keyboard related issues are HUGE and there are plenty. It is even worse because the same keyboard on one device may work just fine but move to another device from a different manufacturer and it may be completely broken making it hard to verify bugs do exist.
We now have almost 20 test devices that we have to manually test our software on to make sure it looks right, that nothing is overlapping, and that it works correctly. Designing for multiple different sizes of screen, and then for landscape mode on those screens is absolutely horrible. On some screens elements get so stretched in landscape it just looks terrible and on others everything is so squashed together in portrait mode that it makes it hard for the user to accurately hit a target.
Fragmentation is driving me personally insane. I wonder how much of my work related stress is from having to deal with that kind of crap.
Just out of curiosity, how was CTR mode broken? Were you using the JCE provider or "direct" API?
The CTR mode in bouncy castle does not allow one to do partial block encryption. So if you want to encrypt 4 bytes you need to pad it to the full 16 bytes. It will always want to do a full block.
This also means that you can't do a partial offset into the CTR. You can increment the counter correctly, but you can't start encrypting from within a partial block.
Lets say you need to encrypt 17 bytes and write them to a file, which later can be opened in append mode to append more data to it using AES-128/CTR-BE mode:
C-style terminated string.That is 2 blocks in CTR mode (1 full block used, 1 partial block (1 byte out of a 16 byte block)).
The next time you open the file you want to append data to it, what you would do is this:
Now you can read your file back using the following: So that would look something like this in Java code with JCE: I've left out some stuff like creating the iv (really the counter), so now if we wanted to advance into the first block what we could do (and this works with JCE) is this: Where count contains the amount we want to offset into the next block.On JCE this does exactly what I described above, it moves forward "count" characters into the next block and then you can do:
where "output" is DataOutputStream(new FileOutputStream("filename", true)) which is a file opened in append mode. Now you can write stuff to the file using the normal functions used by an OutputStream. This will then correctly append your new data to the end of the file so that if you start reading at the beginning of the file you can read through the end and get valid data. (We are using this to encrypt log files that are opened in Append only mode).Where BouncyCastle breaks down is that you can not use c.update() to move forward into the block, thus appending is not possible because if you don't end on a block boundary your next write is going to have overlap. The only thing you can do in c.update() is provide it the amount of bytes that is the same as the block size. So you have to pad all input into c.update() to 16 bytes. Basically instead of being a stream cipher that allows seeking it has become yet another block cipher.
I've written code for this before, I have it over at Github:
https://gist.github.com/fd98541dd158d7e7be9e
So if anyone wants a full example they can run and play with themselves. Your example stuff looks like it was pulled from the code I wrote above...
Yes, I found your stuff through a Stackoverflow post you created.