I think the question there is how do you make test actually test? Allowing them to break their test env so they can actually fix something they broke, imo. It's enabling and it's educational. Too often I see it so abstracted away that devs don't even know what their environment is made out of, so how can you expect them to make performant decisions that align with the infra?
Maybe the cost you pay is in the complexity of security. And then where does that land? Is it ops or security when some ephemeral rotating key or token that's controlled by some test service account doesn't work?
IMO having a test environment is a bit of a failure: you should be able to spin up and stop a full test environment easily, ideally (and feasibly for most systems) on the developer's local machine.
Depends on the type of testing. If you're doing integration testing between systems, then it's not just a matter of spinning it up on a developer's local machine.
You need a testing environment, especially if there are other physical elements that are being integrated (embedded and other equipment).
It's bizarre to me that we live in a world where whole machines and their networking stack can be spun up and virtualized and yet integration testing STILL actually has to go out and touch a real resource running on a different machine.
Yeah, obviously this isn't gonna work for embedded development, but not that many people have that specific problem. Is it really so hard to spin up ephemeral testing clusters on a single machine? Everywhere that I've seen it done, it's a horrible kludge of scripts and hacks that's almost as difficult to understand as the code under test.
When you are integrating to other people's stacks, then you need a test facility where people can bring systems together, sometimes there are also physical elements that are required.
I work in Automatic Fare Collection (think railway passenger gates, ticket vending, contactless readers, bus driver consoles and readers etc as well as the backends that deal with the fare calculation and payments).
There are usually multiple vendors involved, and even with the best API and interface documentation and individual system testing against the specifications, you still need to bring the entire environment, including the hardware, together.
These ITFs (Integration Test Facilities) tend to be in existence for the life of the systems, including after hardware refreshes etc.
Yes you can test performance with jmeter etc, but sometimes you need 20 people walking in a circle between an entrance gate and an exit gate.
Fair enough, but IMO even a system like this one should have good mock implementations of the system hardware, down to and including a simple simulator for the physical layout of people near the machine.
In past years I wouldn't say that, but these days it's so easy to get an LLM to write something like this plus a quick and dirty visualization.
I'm not saying this can be a replacement for a physical test setup, but it's so much easier to make changes when the tests can just give a clear cut "yes, this acts like you expect" or "no, this is broken."
Docker has made this much much easier than it used to be, and the ease of publishing images has led to many SaaS companies providing "dockerized" versions of their services.
I write a lot of Terraform to manage internal infrastructure with dependencies on other internal and external infrastructure that's outside of my control. I define a module parameter for each dependency (usually with a default value that points to the real service URL) and then create a second module that uses Docker to spin up and configure local instances of each dependency and overrides the corresponding parameter.
This makes local testing very straightforward: `terraform -chdir=tests apply`
Sure, for integration with some specific hardware, there may be some limits (though I would ask if that could be simulated for most testing, or if multiple could be made available). But it's a goal that should be aimed for and is achievable for most software, if some effort is spent on it. I see a lot of places seem to cargo-cult a dev/staging/prod setup (each of which inevitably winds up slightly different and hard to reproduce for incidental reasons) and it seems like an approach with a lot of headaches that you probably don't want to use unless you're forced to.
When you have a lot of interdepended systems it is not always possible. In our case our test environment cannot be replicated locally on developer machines because it integrates to various 3rd parties to which we only have one set of test credentials.
In our case, "testing" as an environment has the goal to be a fully integrated setup of the entire service landscape. Each team is responsible for the uptime of their service, just like in production.
The teams are running automated end-to-end tests against this environment. On top, pre-sales uses it to build demo and sample environments, sales shows demos on it, solution engineers use it to test/try out the important parts of projects.
That generates a good production-like experience for the team: You have customers using the system, and possibly screaming if you break it. And it has been pretty successful at catching problems missed or integration failures.
Comments
I think the question there is how do you make test actually test? Allowing them to break their test env so they can actually fix something they broke, imo. It's enabling and it's educational. Too often I see it so abstracted away that devs don't even know what their environment is made out of, so how can you expect them to make performant decisions that align with the infra?
Maybe the cost you pay is in the complexity of security. And then where does that land? Is it ops or security when some ephemeral rotating key or token that's controlled by some test service account doesn't work?
IMO having a test environment is a bit of a failure: you should be able to spin up and stop a full test environment easily, ideally (and feasibly for most systems) on the developer's local machine.
Depends on the type of testing. If you're doing integration testing between systems, then it's not just a matter of spinning it up on a developer's local machine.
You need a testing environment, especially if there are other physical elements that are being integrated (embedded and other equipment).
It's bizarre to me that we live in a world where whole machines and their networking stack can be spun up and virtualized and yet integration testing STILL actually has to go out and touch a real resource running on a different machine.
Yeah, obviously this isn't gonna work for embedded development, but not that many people have that specific problem. Is it really so hard to spin up ephemeral testing clusters on a single machine? Everywhere that I've seen it done, it's a horrible kludge of scripts and hacks that's almost as difficult to understand as the code under test.
When you are integrating to other people's stacks, then you need a test facility where people can bring systems together, sometimes there are also physical elements that are required.
I work in Automatic Fare Collection (think railway passenger gates, ticket vending, contactless readers, bus driver consoles and readers etc as well as the backends that deal with the fare calculation and payments).
There are usually multiple vendors involved, and even with the best API and interface documentation and individual system testing against the specifications, you still need to bring the entire environment, including the hardware, together.
These ITFs (Integration Test Facilities) tend to be in existence for the life of the systems, including after hardware refreshes etc.
Yes you can test performance with jmeter etc, but sometimes you need 20 people walking in a circle between an entrance gate and an exit gate.
Fair enough, but IMO even a system like this one should have good mock implementations of the system hardware, down to and including a simple simulator for the physical layout of people near the machine.
In past years I wouldn't say that, but these days it's so easy to get an LLM to write something like this plus a quick and dirty visualization.
I'm not saying this can be a replacement for a physical test setup, but it's so much easier to make changes when the tests can just give a clear cut "yes, this acts like you expect" or "no, this is broken."
At some point, it becomes hard to assert “we are NOT going to test the product the way a customer would actually use it”
Docker has made this much much easier than it used to be, and the ease of publishing images has led to many SaaS companies providing "dockerized" versions of their services.
I write a lot of Terraform to manage internal infrastructure with dependencies on other internal and external infrastructure that's outside of my control. I define a module parameter for each dependency (usually with a default value that points to the real service URL) and then create a second module that uses Docker to spin up and configure local instances of each dependency and overrides the corresponding parameter.
This makes local testing very straightforward: `terraform -chdir=tests apply`
Sure, for integration with some specific hardware, there may be some limits (though I would ask if that could be simulated for most testing, or if multiple could be made available). But it's a goal that should be aimed for and is achievable for most software, if some effort is spent on it. I see a lot of places seem to cargo-cult a dev/staging/prod setup (each of which inevitably winds up slightly different and hard to reproduce for incidental reasons) and it seems like an approach with a lot of headaches that you probably don't want to use unless you're forced to.
We can do that on our developers' machines no problem, but there still needs to be a place for QA to do their testing.
Can QA not do the same thing?
When you have a lot of interdepended systems it is not always possible. In our case our test environment cannot be replicated locally on developer machines because it integrates to various 3rd parties to which we only have one set of test credentials.
In our case, "testing" as an environment has the goal to be a fully integrated setup of the entire service landscape. Each team is responsible for the uptime of their service, just like in production.
The teams are running automated end-to-end tests against this environment. On top, pre-sales uses it to build demo and sample environments, sales shows demos on it, solution engineers use it to test/try out the important parts of projects.
That generates a good production-like experience for the team: You have customers using the system, and possibly screaming if you break it. And it has been pretty successful at catching problems missed or integration failures.