Neither of those make much sense as the cause anyway. Both sacrifice database consistency if the hosting process crashes or os crashes or reboots due to power failure, but with something like a coverage database, neither is particularly worrying. Worst case the users nukes the db, and reruns the tests.
Even if removing those pragmas fixed things, it feels likely that would just be hiding the issue instead of fixing it.
Disk io errors are not supposed to happen in sqlite except in cases like: accessing a corrupted database, drives failing or running out of disk space, bad permissions on some folder sqlite needs to write to.
So unless the test case involves a python process crashing or getting killed, I question if this is actually a bug in your your code.
I feel like this being some bug in either sqlite, or in the file systems used by docker (specifically by behaving in some manner not anticipated by sqlite).
With those optimizations, the only remaining sqlite disk IO should be the database itself, and the temp store. Since I stronly suspect the error you are getting is coming from sqlite, it would be helpful to know which of those two disk io cases it is.
Perhaps you can try the pragma to force the temp store to be in memory? The results of that might help us know if the disk io error is in reading/writing the database file itself, or if it is in reading/writing temp files.
Thanks, this was a good idea. I tried adding "pragma temp_store = 2", and it did not fix the problem, though I'm not sure I wasn't already using memory temp storage. The interaction of compile-time macros and pragmas is hard to decipher from the Python bindings.
For some reason Docker is struggling to download the image right now (lots of interested parties? :), I just wanted to trap writes to the database file and look at what's actually getting written there.
Is there any possibility e.g. of an FD reuse issue in the code somewhere? It might even be possible to spot it with strace (although you need to start the container with the right caps to try that). An example where this can happen is programs closing 'stderr' fd 2, only for e.g. SQLite to open its database reusing fd 2, then some library code calls fprintf(stderr, ..) etc.
Would love a copy of the DB 'sometime before' and after to look at. There are tools around for inspecting SQLite databases on a page-by-page basis
Comments
I have tried removing the two pragmas, and the problem still happens.
Neither of those make much sense as the cause anyway. Both sacrifice database consistency if the hosting process crashes or os crashes or reboots due to power failure, but with something like a coverage database, neither is particularly worrying. Worst case the users nukes the db, and reruns the tests.
Even if removing those pragmas fixed things, it feels likely that would just be hiding the issue instead of fixing it.
Disk io errors are not supposed to happen in sqlite except in cases like: accessing a corrupted database, drives failing or running out of disk space, bad permissions on some folder sqlite needs to write to.
So unless the test case involves a python process crashing or getting killed, I question if this is actually a bug in your your code.
I feel like this being some bug in either sqlite, or in the file systems used by docker (specifically by behaving in some manner not anticipated by sqlite).
With those optimizations, the only remaining sqlite disk IO should be the database itself, and the temp store. Since I stronly suspect the error you are getting is coming from sqlite, it would be helpful to know which of those two disk io cases it is.
Perhaps you can try the pragma to force the temp store to be in memory? The results of that might help us know if the disk io error is in reading/writing the database file itself, or if it is in reading/writing temp files.
Knowing that could narrow things down a bit.
Thanks, this was a good idea. I tried adding "pragma temp_store = 2", and it did not fix the problem, though I'm not sure I wasn't already using memory temp storage. The interaction of compile-time macros and pragmas is hard to decipher from the Python bindings.
For some reason Docker is struggling to download the image right now (lots of interested parties? :), I just wanted to trap writes to the database file and look at what's actually getting written there.
Is there any possibility e.g. of an FD reuse issue in the code somewhere? It might even be possible to spot it with strace (although you need to start the container with the right caps to try that). An example where this can happen is programs closing 'stderr' fd 2, only for e.g. SQLite to open its database reusing fd 2, then some library code calls fprintf(stderr, ..) etc.
Would love a copy of the DB 'sometime before' and after to look at. There are tools around for inspecting SQLite databases on a page-by-page basis