There’s also https://testcontainers.com/ not sure about the speed difference but testcontainers has never felt slow to me when using it from node js unittests.
Ding ding ding. Testcontainers is a fanatstic way to write the most important tests (arguably) for your app. Don't test E2E with a mock, just use a real database.
If that feels hard to you (to set up your app pointing to another DB, run a single E2E-testable part of your app with it's own DB connection, etc), fix that.
If you feel you need E2E tests, that means that you have implementation details leaking across different areas of your application. That's what you really need to fix. Bandaids may be found useful, but even better is not cut yourself in the first place.
Normally you would write tests at the user boundaries (like a set of APIs) for the purposes of documenting the behavioural intent for the user as to not leave unspecified behaviour. While you could theoretically do that in Word instead, testing, as we call it, offers a couple of advantages:
1. It is written in code, which is what coders want to read. There is nothing worse than having to read documentation in English (or insert your natural language of choice).
2. It is able to automatically prove that what is documented is true. Programmers tend to have a penchant for lying if you let them, so this is a surprisingly big advantage.
You might not need them per se, but unless you are trying to be an asshole, it is socially expected that you will be kind to your users by filling them in on important details; to not leave them guessing.
Now, if your application only has one user boundary (a.k.a. spaghetti code) then any tests you have will technically end up being E2E tests by there being only one end (although calling that E2E would be rather silly), that is true, but if that's what your code looks like then you're going to have trouble connecting to a test database anyway, and per the earlier comment you need to fix that for the sake of E2E tests. But if you reach that point, you may as well go all the way and fix it properly.
It's remarkably easy to integrate. We have a tests/db module with a single conftest.py file that does everything. Every test suite run spins up a fresh container (no state), runs all migrations against it to prepare it for duty, and then the test suite continues. You could do this on a per-test basis for ultra isolation, but this works for us. That change would simply be to modify the `scope` value.
Also a convenient `conn` fixture at the end which allows any test in the module to get a new db connection.
The last GH Actions jobs with SQLite and Python 3.9 took 3m 41s, and the same tests with Postgres took 4m 11s. Running a single test locally in PyCharm also executes in less than 1 second. You notice some bootstrap happening, but once the container image is downloaded locally, it's really quite fast.
yeah, w/o py-pglite attempt this should be the only approach, the pglite ideally could make it more flexibly/lightweight in unittest cases, but as you mentioned it's never felt slow, it should be fine to working on it.
And actually, more e2e cases I think it's way better to not use the lite backend.
the non-container solutions would do more like the lifecycle mgmt/isolated env prep/tear-down with elegantly designed abstractions. While I think similar abstractions could be done on top of containers.
Maybe we ideally could have unified abstractions on both container-based, wasm evantually to boost dx yet with different expectation of speed vs compatibility.
Comments
There’s also https://testcontainers.com/ not sure about the speed difference but testcontainers has never felt slow to me when using it from node js unittests.
Ding ding ding. Testcontainers is a fanatstic way to write the most important tests (arguably) for your app. Don't test E2E with a mock, just use a real database.
If that feels hard to you (to set up your app pointing to another DB, run a single E2E-testable part of your app with it's own DB connection, etc), fix that.
Yep. I do create fresh db's from a fixture db using postgres's ability to create a database from a template. Very quick, always correct.
If you feel you need E2E tests, that means that you have implementation details leaking across different areas of your application. That's what you really need to fix. Bandaids may be found useful, but even better is not cut yourself in the first place.
If you just know when you aren't cutting yourself then why do you need tests?
Normally you would write tests at the user boundaries (like a set of APIs) for the purposes of documenting the behavioural intent for the user as to not leave unspecified behaviour. While you could theoretically do that in Word instead, testing, as we call it, offers a couple of advantages:
1. It is written in code, which is what coders want to read. There is nothing worse than having to read documentation in English (or insert your natural language of choice).
2. It is able to automatically prove that what is documented is true. Programmers tend to have a penchant for lying if you let them, so this is a surprisingly big advantage.
You might not need them per se, but unless you are trying to be an asshole, it is socially expected that you will be kind to your users by filling them in on important details; to not leave them guessing.
Now, if your application only has one user boundary (a.k.a. spaghetti code) then any tests you have will technically end up being E2E tests by there being only one end (although calling that E2E would be rather silly), that is true, but if that's what your code looks like then you're going to have trouble connecting to a test database anyway, and per the earlier comment you need to fix that for the sake of E2E tests. But if you reach that point, you may as well go all the way and fix it properly.
For Python specifically I use pytest-docker: https://pypi.org/project/pytest-docker/
It's remarkably easy to integrate. We have a tests/db module with a single conftest.py file that does everything. Every test suite run spins up a fresh container (no state), runs all migrations against it to prepare it for duty, and then the test suite continues. You could do this on a per-test basis for ultra isolation, but this works for us. That change would simply be to modify the `scope` value.
Also a convenient `conn` fixture at the end which allows any test in the module to get a new db connection.
linky: https://gist.github.com/whalesalad/6ecd284460ac3836a6c2b9ca8...
That's exactly what we are using for our tests in a new pull request to add support to Postgres, https://github.com/BSC-ES/autosubmit/pull/2187
The last GH Actions jobs with SQLite and Python 3.9 took 3m 41s, and the same tests with Postgres took 4m 11s. Running a single test locally in PyCharm also executes in less than 1 second. You notice some bootstrap happening, but once the container image is downloaded locally, it's really quite fast.
yeah, w/o py-pglite attempt this should be the only approach, the pglite ideally could make it more flexibly/lightweight in unittest cases, but as you mentioned it's never felt slow, it should be fine to working on it.
And actually, more e2e cases I think it's way better to not use the lite backend.
the non-container solutions would do more like the lifecycle mgmt/isolated env prep/tear-down with elegantly designed abstractions. While I think similar abstractions could be done on top of containers.
Maybe we ideally could have unified abstractions on both container-based, wasm evantually to boost dx yet with different expectation of speed vs compatibility.