I've been trying to gradually introduce Nix more into my development.[0]
One roadblock I haven't been able to overcome is integrating Nix in CI. In my case, I use CircleCI. I know there are Nix-centric CI platforms, but I've invested a lot into CircleCI and would like to stay.
The problem I hit is that if I switch from something like a sqlfluff Docker container to a Nix Docker container, my build job goes from 15s to 2m+ because now I have to download several gigs within the CI job and build my Nix environment from scratch.[1]
I tried Cachix, but the best I could get was bringing the job down to 55s (a roughly 3x performance penalty from an off-the-shelf Docker container).[2]
How does FlakeHub Cache's performance in CI compare to the default Nix cache or to Cachix?
What worked great for me for speeding up CI was using Cachix + namespace.so. This works so well I moved all my personal and our company CI to GitHub Actions.
With "it works so well", I mean:
* CI is cheap because namespace.so runners are faster & cheaper than GitHub Actions
* the UX is exactly the same as using vanilla GitHub Actions, namespace.so's runner is a drop-in replacement
* the CI starts up very fast because `/nix/store` is stored on namespace.so's "Cached Volumes" which are like virtual disks attached to the CI runner, so there is nothing to download to prime the cache, Nix can just start running immediately
* https://github.com/cachix/cachix-action uploads whatever Nix builds to Cachix in the background so from this point on if any of our devs pulls the latest code, they don't have to build any derivations locally, and lose time waiting, everything is pulled automatically from Cachix to their machines, they can get right to work. Or if I have to purge the Cache Volume for some reason, nothing needs to be rebuilt, just pulled from Cachix.
It's taken a few years to get this dialed in, but now that it works, it's sooo good!
IIUC Magic Nix Cache uses Github Actions Cache, which means there is waiting for the cache contents to be loaded onto the runner instance. This can in no way be as fast as what namespace.so does, which is mounting a disk image with /nix/store.
I'm going to have to look into this mounting option. I wanted to try making volumes very similar to this for reusing existing /target directories for internal company Rust builds. Sounds like Namespace could make this very easy to try out.
Just need to figure out what the cache size limitations are..
To be clear, I would not expect it to be as fast! But I do wonder what the percentage drop-off would be and, if small enough, if that would make staying on Actions more compelling than a bespoke solution.
I'd love to find out how it compares! We're not quite ready to support CircleCI, but when we are -- want to do a test? You could hop on to our discord: https://determinate.systems/discord
What I've been doing for now is to pre-build the container to include the Nix environment in the nix store. I still run nix in the CI so I only have to rebuild the container from time to time, when the env has changed a lot. It's certainly not an ideal solution but it works without special support.
You indeed need some kind of nix-aware runner (e.g. with a persistent /nix) if you want both the flexibility and performance...
I am this close to building my own CI pipeline. Building a new CI pipeline is a really bad idea, I would be the first to tell you, and it's what I've been telling myself for five plus years now, but it's becoming harder to ignore the disparity between off the shelf tools that solve parts of the CI/CD problem and the ones that are actually used in garbage like Bamboo.
I think the fact that there is no bidding process for agents is a mistake that we need to correct. It's a little used concept in distributed computing, but a powerful one when applicable, and I think the preconditions for a fast build, with all of the docker images and other bootstrapping pipelines is making that cost harder to ignore. We are de facto using heterogeneous networks of worker processes now, and it's starting to reach 8 Fallacies level.
You build a distributed queue for response time or throughput, and we are using the throughput model, more or less, while CI/CD is mostly about response time.
I don’t know about CircleCI but if you can persist the /nix/store then that will give you the performance improvements you’re looking for.
I use Gitlab CI and with self hosted runners I can make sure the /nix/store is persistent across CI jobs and then my builds take ~5 seconds if no code has changed, 5 seconds of which is just nix build evaluating all my nix files and realising no work needs to be done.
The problem people often run in to when persisting the nix store is that it can be very large, and updates often. This adds a significant tax on setup/teardown time and sometimes storage costs. The completely free and unsubsidized Magic Nix Cache's design alleviates that on GitHub Actions by persisting each path individually in the actions cache. FlakeHub Cache brings a for-cost component to that, which exposes the cache to your workstation and production infrastructure.
To clarify I meant having /nix/store on some kind of persistent volume rather than copying it at the start and end of every job, that indeed would be painfully slow and extremely inefficient. I don't know if this is possible with CircleCI.
Have you considered bundling that env into your CI image? I have a CI/CD process that will build and update an image with the environment pre-loaded. It doesn't save you the data transfer, but it likely provides a better warm cache.
I've thought about it, but I'm not sure how that would work. Because for me, the value proposition of having a Nix flake for my project is that I can bind the exact dependencies to the state of the code in my repo. So if I want to update from Go 1.19 to Go 1.20, I just update a line or two in my flake.nix. If I have to go update a separate repo, build a Docker image from that repo, then update the tag in my real repo, it kills the convenience.
Is there an easier way to do it?
Also, wouldn't it be pretty similar performance-wise to caching the /nix path on a generic Nix image? I've tried that, but it has problems because the Nix path ends up being a few GB, so that ends up adding a long time to the build as well.
It might help to describe how I employ this. I own the build and infra for a fairly large monorepo, and I moved all of the deps to Nix. I have a few ways to source/build things:
- A bare image with just Nix can pull the deps from a cache.
- A pre-built image contains the deps from the latest push to default branch and nothing else.
I do not maintain a separate repo for this. Part of my CI is building and caching the paths. And then CD is building and pushing the image, which the CI has verified as working. This requires using a mutable tag, but you could also have the CD process auto-update the tag if this is not an option.
You're correct that this doesn't save you from having to get the same amount of data to the same place, but the image repo can usually be cached more advantageously for whatever is pulling images, so it should be considerably faster overall.
My ideal setup is to have my dev env be the CI env, to stop the chance that every so often things break (my CI breaks my delicate android incantions with some regularity).
Comments
I've been trying to gradually introduce Nix more into my development.[0]
One roadblock I haven't been able to overcome is integrating Nix in CI. In my case, I use CircleCI. I know there are Nix-centric CI platforms, but I've invested a lot into CircleCI and would like to stay.
The problem I hit is that if I switch from something like a sqlfluff Docker container to a Nix Docker container, my build job goes from 15s to 2m+ because now I have to download several gigs within the CI job and build my Nix environment from scratch.[1]
I tried Cachix, but the best I could get was bringing the job down to 55s (a roughly 3x performance penalty from an off-the-shelf Docker container).[2]
How does FlakeHub Cache's performance in CI compare to the default Nix cache or to Cachix?
[0] https://mtlynch.io/notes/nix-dev-environment/
[1] https://github.com/cachix/cachix/issues/579
[2] https://github.com/cachix/cachix/issues/579#issuecomment-175...
What worked great for me for speeding up CI was using Cachix + namespace.so. This works so well I moved all my personal and our company CI to GitHub Actions.
With "it works so well", I mean:
* CI is cheap because namespace.so runners are faster & cheaper than GitHub Actions
* the UX is exactly the same as using vanilla GitHub Actions, namespace.so's runner is a drop-in replacement
* the CI starts up very fast because `/nix/store` is stored on namespace.so's "Cached Volumes" which are like virtual disks attached to the CI runner, so there is nothing to download to prime the cache, Nix can just start running immediately
* https://github.com/cachix/cachix-action uploads whatever Nix builds to Cachix in the background so from this point on if any of our devs pulls the latest code, they don't have to build any derivations locally, and lose time waiting, everything is pulled automatically from Cachix to their machines, they can get right to work. Or if I have to purge the Cache Volume for some reason, nothing needs to be rebuilt, just pulled from Cachix.
It's taken a few years to get this dialed in, but now that it works, it's sooo good!
I'd be quite curious to see how using the Magic Nix Cache Action compares to your setup in terms of performance: https://github.com/DeterminateSystems/magic-nix-cache-action
(full disclosure: Determinate Systems employee)
IIUC Magic Nix Cache uses Github Actions Cache, which means there is waiting for the cache contents to be loaded onto the runner instance. This can in no way be as fast as what namespace.so does, which is mounting a disk image with /nix/store.
I'm going to have to look into this mounting option. I wanted to try making volumes very similar to this for reusing existing /target directories for internal company Rust builds. Sounds like Namespace could make this very easy to try out.
Just need to figure out what the cache size limitations are..
To be clear, I would not expect it to be as fast! But I do wonder what the percentage drop-off would be and, if small enough, if that would make staying on Actions more compelling than a bespoke solution.
I'd love to find out how it compares! We're not quite ready to support CircleCI, but when we are -- want to do a test? You could hop on to our discord: https://determinate.systems/discord
What I've been doing for now is to pre-build the container to include the Nix environment in the nix store. I still run nix in the CI so I only have to rebuild the container from time to time, when the env has changed a lot. It's certainly not an ideal solution but it works without special support.
You indeed need some kind of nix-aware runner (e.g. with a persistent /nix) if you want both the flexibility and performance...
I am this close to building my own CI pipeline. Building a new CI pipeline is a really bad idea, I would be the first to tell you, and it's what I've been telling myself for five plus years now, but it's becoming harder to ignore the disparity between off the shelf tools that solve parts of the CI/CD problem and the ones that are actually used in garbage like Bamboo.
I think the fact that there is no bidding process for agents is a mistake that we need to correct. It's a little used concept in distributed computing, but a powerful one when applicable, and I think the preconditions for a fast build, with all of the docker images and other bootstrapping pipelines is making that cost harder to ignore. We are de facto using heterogeneous networks of worker processes now, and it's starting to reach 8 Fallacies level.
You build a distributed queue for response time or throughput, and we are using the throughput model, more or less, while CI/CD is mostly about response time.
I don’t know about CircleCI but if you can persist the /nix/store then that will give you the performance improvements you’re looking for.
I use Gitlab CI and with self hosted runners I can make sure the /nix/store is persistent across CI jobs and then my builds take ~5 seconds if no code has changed, 5 seconds of which is just nix build evaluating all my nix files and realising no work needs to be done.
The problem people often run in to when persisting the nix store is that it can be very large, and updates often. This adds a significant tax on setup/teardown time and sometimes storage costs. The completely free and unsubsidized Magic Nix Cache's design alleviates that on GitHub Actions by persisting each path individually in the actions cache. FlakeHub Cache brings a for-cost component to that, which exposes the cache to your workstation and production infrastructure.
To clarify I meant having /nix/store on some kind of persistent volume rather than copying it at the start and end of every job, that indeed would be painfully slow and extremely inefficient. I don't know if this is possible with CircleCI.
Yes, this is exactly the problem I hit when I try to cache the Nix store on CircleCI.
This is part of why we built https://garnix.io/
Have you considered bundling that env into your CI image? I have a CI/CD process that will build and update an image with the environment pre-loaded. It doesn't save you the data transfer, but it likely provides a better warm cache.
I've thought about it, but I'm not sure how that would work. Because for me, the value proposition of having a Nix flake for my project is that I can bind the exact dependencies to the state of the code in my repo. So if I want to update from Go 1.19 to Go 1.20, I just update a line or two in my flake.nix. If I have to go update a separate repo, build a Docker image from that repo, then update the tag in my real repo, it kills the convenience.
Is there an easier way to do it?
Also, wouldn't it be pretty similar performance-wise to caching the /nix path on a generic Nix image? I've tried that, but it has problems because the Nix path ends up being a few GB, so that ends up adding a long time to the build as well.
It might help to describe how I employ this. I own the build and infra for a fairly large monorepo, and I moved all of the deps to Nix. I have a few ways to source/build things:
- A bare image with just Nix can pull the deps from a cache.
- A pre-built image contains the deps from the latest push to default branch and nothing else.
I do not maintain a separate repo for this. Part of my CI is building and caching the paths. And then CD is building and pushing the image, which the CI has verified as working. This requires using a mutable tag, but you could also have the CD process auto-update the tag if this is not an option.
You're correct that this doesn't save you from having to get the same amount of data to the same place, but the image repo can usually be cached more advantageously for whatever is pulling images, so it should be considerably faster overall.
Also in the same boat.
I have the complication of android + Rust (com[compiling rust for android + SQLite require gimnastics: https://www.reddit.com/r/rust/comments/1872ftd/exist_a_solid...).
My ideal setup is to have my dev env be the CI env, to stop the chance that every so often things break (my CI breaks my delicate android incantions with some regularity).