if you're not running the test-suite on pre-push your tests are slow
Of course tests are slow. If they're fast they're skipping something. You probably want to be skipping something most of the time, because most of the time your change shouldn't cause any side-effects in the system, but you definitely want a thorough test suite to run occasionally. Maybe before merge, maybe before release, maybe on some other schedule.
Each individual change might be fine in isolation but cause excessive memory pressure when accumulated with other changes, for example. Unit tests won't catch everything, integration & functional (hardware in-the-loop) tests are needed. I even sometimes have to run tests in a thermal chamber repeatedly to cover the whole -40-105°C temperature range, since the firmware I work on runs on hardware that allows that range & performance varies with temperature.
What is a good time to run slower tests? My full test suite takes around 4 minutes to run, and it’s trending upwards. I am a solo developer, so I run it pre-push.
And what about tests that only need to run intermittently, such as broken link checkers?
Is there a reason for that? I figure that catching bugs before pushing is faster than getting an email from GitHub, and my hardware already has all the containers running. Spinning up machines in the cloud for this feels wasteful.
if you want to reuse the local setup for ci or pushing broken code is somehow undesirable, then run all the tests pre-push
i tend to run only the fast set of tests, or a subset of the environments matrix, on pre-push — CI can take care of the rest while, i move onto do something else
when working with other the CI status is what matters
Certainly not the formatting you have done manually, because the auto-formatter will likely throw it away. You likely do that, because you care about the consistency of the formatting, not for the formatting already present in the file.
Personally, I don't like auto-formatters. It's like having someone else cleaning up your room. I have put every byte of whitespace where it belongs, I certainly don't want someone else to mess that up.
Comments
Pre commit hooks shine with fast formatters. Keep the hook under half a second or so and it's great.
it's the tools. if they're slow people disable them or shift them right. you want your defect-detection and fixing to shift left
if you're not running auto-format on file-save, your auto-formatter is slow
if you're not running a code checker with auto-fix on pre-commit, your code checker is slow
if you're not running the test-suite on pre-push your tests are slow
if your tooling is slow you need to pick better tooling or make them fast
you want to keep that loop tight and active
Of course tests are slow. If they're fast they're skipping something. You probably want to be skipping something most of the time, because most of the time your change shouldn't cause any side-effects in the system, but you definitely want a thorough test suite to run occasionally. Maybe before merge, maybe before release, maybe on some other schedule.
Each individual change might be fine in isolation but cause excessive memory pressure when accumulated with other changes, for example. Unit tests won't catch everything, integration & functional (hardware in-the-loop) tests are needed. I even sometimes have to run tests in a thermal chamber repeatedly to cover the whole -40-105°C temperature range, since the firmware I work on runs on hardware that allows that range & performance varies with temperature.
that's what ci is for
anyway, if you're doing embedded development and can distinguish between different kinds of testing then my previous post is not meant for you
What is a good time to run slower tests? My full test suite takes around 4 minutes to run, and it’s trending upwards. I am a solo developer, so I run it pre-push.
And what about tests that only need to run intermittently, such as broken link checkers?
your ci setup should take care of the slow tests — for most folks this is github actions
pre-push can run the faster unittest set, ci will run the integration set
aim to keep pre-push duration under 10s
Is there a reason for that? I figure that catching bugs before pushing is faster than getting an email from GitHub, and my hardware already has all the containers running. Spinning up machines in the cloud for this feels wasteful.
if you want to reuse the local setup for ci or pushing broken code is somehow undesirable, then run all the tests pre-push
i tend to run only the fast set of tests, or a subset of the environments matrix, on pre-push — CI can take care of the rest while, i move onto do something else
when working with other the CI status is what matters
Or you care about your formatting and don't want to throw them away.
i don't know what you think i care about, when i run the auto-formatter on file-save, if not for the formatting of the code
Certainly not the formatting you have done manually, because the auto-formatter will likely throw it away. You likely do that, because you care about the consistency of the formatting, not for the formatting already present in the file.
Personally, I don't like auto-formatters. It's like having someone else cleaning up your room. I have put every byte of whitespace where it belongs, I certainly don't want someone else to mess that up.