Suppose you have an Artifactory server that mirrors/caches a lot of public stuff so one is (hopefully) a good citizen and don’t spam public mirrors with constant requests for the same thing.
But every tool has its own config to set to use the Artifactory. One setting for the OS package manager (which is different for different Linux distributions), another for PyPI, another for NPM (or Yarn or whatever), another for Maven/Gradle, something else for Go, then I need to download this Postgres extension and build it from source - the list goes on. So almost inevitably something gets missed and one ends up not being as good a citizen as one ought, and then one day some random Jenkins job is failing because some external dependency could not be downloaded.
I wish there was an easier way. Like some standard mechanism for saying “for this URL use this proxy”.
I guess one could just use a proxy server (http_proxy environment variable) but with most things on HTTPS it needs to MITM the TLS which then means you need that certificate installed in the build process - which is another one of those “everything can do it but everything does it differently” problems. And in any event, MITM is a bad smell.
It's always seemed ridiculous to me that `apt` by default isn't just implemented as a global hash lookup. Once I have my package indexes and signatures, where a package comes from really doesn't matter - I should be able to fire a request into the ether and get routed to whoever has it, not depend on one specific mirror not breaking mid-update.
But every tool has its own config to set to use the Artifactory. One setting for the OS package manager (which is different for different Linux distributions), another for PyPI, another for NPM (or Yarn or whatever), another for Maven/Gradle, something else for Go, then I need to download this Postgres extension and build it from source - the list goes on.
Tools like Artifactory are a hack built upon a hack built upon a hack; redirecting certain HTTP requests to proxies would just be piling even more crap on top.
Content-addressing is a much cleaner option: identify files by their hash, rather than as the result of a HTTP request to some particular URL (many tools will already use these hashes to verify the result anyway; that's what "lock files" are for!). Content-addressed data is agnostic about how its retrieved, which makes caching trivial. There's no need to care about the data format, whether it's RPM, Deb, a source tarball, a patch, a Python "egg", or whatever.
For example, I lead a transition to Nix at a previous employer. We had a bunch of projects with various build processes (Maven, SBT, Gradle, PyPI, NPM). After wrapping these in Nix, the whole lot could be cached on S3 by simply copying files around (see https://nix.dev/manual/nix/2.22/store/types/s3-binary-cache-... )
I have been twirling around in my mind lately what it will take to deploy a Pulp Project instance for my business. Even in a small company it’s a bunch of work. I have a Kubernetes cluster, Ansible AWX running a bunch of playbooks using custom Execution Environments, a bunch of Ubuntu servers managed by AWX, and I’m evaluating the idea of migrating from Github to Gitea which would include Github Actions. A few critical apps are written in Laravel or Python and so that’s in the package/artifact caching mix too. keep punting because these workloads keep feeling like a time consuming chicken and an egg problem.
Pulp is perfect for this but the demands on my time make it hard to see around the corner.
Proxy auto config (PAC) supports specifying different proxies for different URLs. Unfortunately, a PAC file is just a file that contains a JavaScript function to pick the proxy, so they're crazily over-powered for the task, and support for them isn't very broad. Browsers support them, but I guess most command line tools wouldn't.
Another solution: a HTTP proxy server listening on localhost to which you sent HTTPS requests using GET https:// instead of CONNECT. Then the proxy server could have all the logic about which requests to handle via the cache versus which to fetch directly. It could also handle authentication to a cache server if that is required.
The problem is most clients don’t do GET https://, because in your old-school corporate web proxy use case, the proxy server is remote, and sending HTTPS requests to it over HTTP eliminates the security of HTTPS.
If only there was some standard environment variable like artifact_proxy, which had to be a localhost http URI, and which tools would understand as meaning “send HTTP GET to this proxy, even for https://, delegating all the TLS stuff to it, but only if you are trying to download a build artifact, not for any runtime use”
The hard part wouldn’t be implementing this idea (the local proxy server and the environment variable), the hard part would be getting all the different tool developers to agree to support it
Comments
Suppose you have an Artifactory server that mirrors/caches a lot of public stuff so one is (hopefully) a good citizen and don’t spam public mirrors with constant requests for the same thing.
But every tool has its own config to set to use the Artifactory. One setting for the OS package manager (which is different for different Linux distributions), another for PyPI, another for NPM (or Yarn or whatever), another for Maven/Gradle, something else for Go, then I need to download this Postgres extension and build it from source - the list goes on. So almost inevitably something gets missed and one ends up not being as good a citizen as one ought, and then one day some random Jenkins job is failing because some external dependency could not be downloaded.
I wish there was an easier way. Like some standard mechanism for saying “for this URL use this proxy”.
I guess one could just use a proxy server (http_proxy environment variable) but with most things on HTTPS it needs to MITM the TLS which then means you need that certificate installed in the build process - which is another one of those “everything can do it but everything does it differently” problems. And in any event, MITM is a bad smell.
It's always seemed ridiculous to me that `apt` by default isn't just implemented as a global hash lookup. Once I have my package indexes and signatures, where a package comes from really doesn't matter - I should be able to fire a request into the ether and get routed to whoever has it, not depend on one specific mirror not breaking mid-update.
Tools like Artifactory are a hack built upon a hack built upon a hack; redirecting certain HTTP requests to proxies would just be piling even more crap on top.
Content-addressing is a much cleaner option: identify files by their hash, rather than as the result of a HTTP request to some particular URL (many tools will already use these hashes to verify the result anyway; that's what "lock files" are for!). Content-addressed data is agnostic about how its retrieved, which makes caching trivial. There's no need to care about the data format, whether it's RPM, Deb, a source tarball, a patch, a Python "egg", or whatever.
For example, I lead a transition to Nix at a previous employer. We had a bunch of projects with various build processes (Maven, SBT, Gradle, PyPI, NPM). After wrapping these in Nix, the whole lot could be cached on S3 by simply copying files around (see https://nix.dev/manual/nix/2.22/store/types/s3-binary-cache-... )
I have been twirling around in my mind lately what it will take to deploy a Pulp Project instance for my business. Even in a small company it’s a bunch of work. I have a Kubernetes cluster, Ansible AWX running a bunch of playbooks using custom Execution Environments, a bunch of Ubuntu servers managed by AWX, and I’m evaluating the idea of migrating from Github to Gitea which would include Github Actions. A few critical apps are written in Laravel or Python and so that’s in the package/artifact caching mix too. keep punting because these workloads keep feeling like a time consuming chicken and an egg problem.
Pulp is perfect for this but the demands on my time make it hard to see around the corner.
Proxy auto config (PAC) supports specifying different proxies for different URLs. Unfortunately, a PAC file is just a file that contains a JavaScript function to pick the proxy, so they're crazily over-powered for the task, and support for them isn't very broad. Browsers support them, but I guess most command line tools wouldn't.
https://en.m.wikipedia.org/wiki/Proxy_auto-config
Another solution: a HTTP proxy server listening on localhost to which you sent HTTPS requests using GET https:// instead of CONNECT. Then the proxy server could have all the logic about which requests to handle via the cache versus which to fetch directly. It could also handle authentication to a cache server if that is required.
The problem is most clients don’t do GET https://, because in your old-school corporate web proxy use case, the proxy server is remote, and sending HTTPS requests to it over HTTP eliminates the security of HTTPS.
If only there was some standard environment variable like artifact_proxy, which had to be a localhost http URI, and which tools would understand as meaning “send HTTP GET to this proxy, even for https://, delegating all the TLS stuff to it, but only if you are trying to download a build artifact, not for any runtime use”
The hard part wouldn’t be implementing this idea (the local proxy server and the environment variable), the hard part would be getting all the different tool developers to agree to support it