The argument here I suppose is live on the bleeding edge? Or swap compilers/allocators as necessary? Or use O3? I'm not entirely sure.
It wraps up with:
>> COMPILE YOUR SOFTWARES. It's going to help you understand it, make your app faster, more secure (by removing the mail gateway of NGiNX for example), and it will show you which softwares are easily maintanable and reliable.
It is PITA is keeping up with compiler changes and library changes of third party software. In this case, throwing in different malloc implementations in there too, not to mention different libc implementations. With all those variables for each component you deploy you're probably less likely to understand what's going in your software.
Maybe for a small app with 2-3 extra pieces or something that needs to really be optimized this is good advice, but it sounds like a lot of work for significant footprints.
It would be nice if the official docker images had some tags that were better optimized, within reason.
The argument here I suppose is live on the bleeding edge? Or swap compilers/allocators as necessary? Or use O3? I'm not entirely sure.
My point is that when you are using a software, especially open source, you should evaluate it, and understand it. BTW, we are running arch in production, with our own repos.
It is PITA is keeping up with compiler changes and library changes of third party software. In this case, throwing in different malloc implementations in there too, not to mention different libc implementations. With all those variables for each component you deploy you're probably less likely to understand what's going in your software.
in my example, the libc implementation is always the same! it's just that it's outdated on all main docker images. Furthermore, Redis already use a different malloc implementation (jemalloc), but in the makefile, they support also the standard malloc and tcmalloc, so throwing in another one is very easy.
Maybe for a small app with 2-3 extra pieces or something that needs to really be optimized this is good advice, but it sounds like a lot of work for significant footprints.
Or when you need security/performance. of course, if you have 2 servers, it's useless, but we have more than 4000 servers running in production on arch, so it's worth it for us.
It would be nice if the official docker images had some tags that were better optimized, within reason.
or just be up to date
redis:latest uses glibc though? looks like glibc 2.28.
Yeah - if you're targeting a single platform for a single app it's probably much easier, and DIY is probably even easier with Arch or Gentoo. I've been in the "business" of maintaining multiple systems/libraries (mysql, boost, numpy, scipy, matplotlib, even the JDK) over multiple platforms (centos+devtoolset, macos, clang) and it's a nightmare. I've also, of course, done lots of custom builds of nginx/openresty for specific things which isn't so bad.
We've since moved on to conda-forge for getting most our third parties, which moved faster than the defaults channel even though the libc version compatibility is very old (but improving). Compilers update more frequently than glibc there, but compiler upgrades there are still deliberate.
BTW, we are running arch in production, with our own repos.
I believe the ability to deploy from one's own fork is a minimum requirement for a distro of choice. I was wondering, do you have any resources to recommend for how to accomplish this with Arch?
So much advice in software development presupposes that everyone has the same single goal and at least one inexhaustible resource. In this case, time:
- To read and understand all relevant parts of the build pipelines for each performance-critical part of the infrastructure. In a relatively simple web application that could mean at least a web server, a caching framework, a database and a message bus.
- To debug any build failures, which could be plentiful and hard to parse until you're very familiar with the build infrastructure for that particular piece of software.
- To benchmark the build outputs in comparable ways with realistic configuration and inputs.
- To repeat the above whenever any part of the stack changes significantly.
Is it required for open source software to support every combination of compiler, kernel, libc etc? We wouldn't say it has to support every version of any other dependency (not provided by system)
It's definitely a nice to have, but I think it's also valid for some project to say we support the version we used in CI and for releases. On the other hand you could say being stuck on a specific version is skin to relying on implementation details, which is a code smell atleast.
Is it required for open source software to support every combination of compiler, kernel, libc etc?
If the versions of the components are within the range of what is defined at supported, then yes, absolutely. Most open source software supports a certain range of all the dependencies it needs to compile, like libfoo >= 0.2 - <= 0.5, libbar >= 1.0.0 - <= 1.8.0 and so on. This is encoded in the build scripts that check for this and set up the final Makefiles or whatever to actually compile the thing.
In practice it's more difficult because distributions often "fix" build scripts to allow versions outside this range, or the libraries they ship are heavily patched, but in that case it's still a bug, just some else's.
I think writing something against only a single specific version of anything is a shoddy practice and leads to brittle and ultimately hard to maintain software. It means you can't easily do updates and will depend on bugs in whatever version you use.
If you seriously suggested that way to handle things to an open source project most would laugh at you.
Its only a bug if the software was stated to support that combination. If it was designed and documented to suppot three combinations, and you tried it on a fourth, found it not to work, that’s an enhancement request not a bug report.
I buy that for a bottleneck, and also when you encounter bugs (it happens). This post didn’t go into day 2 though, and having dealt with a very large code base. It was very hard to maintain it over the years. We started sourcing most things from conda-forge because it was easier (data science pipelines) but a few things we still built.
Comments
The argument here I suppose is live on the bleeding edge? Or swap compilers/allocators as necessary? Or use O3? I'm not entirely sure.
It wraps up with:
It is PITA is keeping up with compiler changes and library changes of third party software. In this case, throwing in different malloc implementations in there too, not to mention different libc implementations. With all those variables for each component you deploy you're probably less likely to understand what's going in your software.
Maybe for a small app with 2-3 extra pieces or something that needs to really be optimized this is good advice, but it sounds like a lot of work for significant footprints.
It would be nice if the official docker images had some tags that were better optimized, within reason.
Author here.
My point is that when you are using a software, especially open source, you should evaluate it, and understand it. BTW, we are running arch in production, with our own repos.
in my example, the libc implementation is always the same! it's just that it's outdated on all main docker images. Furthermore, Redis already use a different malloc implementation (jemalloc), but in the makefile, they support also the standard malloc and tcmalloc, so throwing in another one is very easy.
redis:latest uses glibc though? looks like glibc 2.28.
Yeah - if you're targeting a single platform for a single app it's probably much easier, and DIY is probably even easier with Arch or Gentoo. I've been in the "business" of maintaining multiple systems/libraries (mysql, boost, numpy, scipy, matplotlib, even the JDK) over multiple platforms (centos+devtoolset, macos, clang) and it's a nightmare. I've also, of course, done lots of custom builds of nginx/openresty for specific things which isn't so bad.
We've since moved on to conda-forge for getting most our third parties, which moved faster than the defaults channel even though the libc version compatibility is very old (but improving). Compilers update more frequently than glibc there, but compiler upgrades there are still deliberate.
I believe the ability to deploy from one's own fork is a minimum requirement for a distro of choice. I was wondering, do you have any resources to recommend for how to accomplish this with Arch?
Not really, but it could be an article soon!
So much advice in software development presupposes that everyone has the same single goal and at least one inexhaustible resource. In this case, time:
- To read and understand all relevant parts of the build pipelines for each performance-critical part of the infrastructure. In a relatively simple web application that could mean at least a web server, a caching framework, a database and a message bus.
- To debug any build failures, which could be plentiful and hard to parse until you're very familiar with the build infrastructure for that particular piece of software.
- To benchmark the build outputs in comparable ways with realistic configuration and inputs.
- To repeat the above whenever any part of the stack changes significantly.
It also could introduce bugs given your company might be the only one earth with that combination of compiler, kernel and libraries.
You mean, expose bugs. If the software doesn't support a certain combination, this is a bug in the software.
Is it required for open source software to support every combination of compiler, kernel, libc etc? We wouldn't say it has to support every version of any other dependency (not provided by system)
It's definitely a nice to have, but I think it's also valid for some project to say we support the version we used in CI and for releases. On the other hand you could say being stuck on a specific version is skin to relying on implementation details, which is a code smell atleast.
If the versions of the components are within the range of what is defined at supported, then yes, absolutely. Most open source software supports a certain range of all the dependencies it needs to compile, like libfoo >= 0.2 - <= 0.5, libbar >= 1.0.0 - <= 1.8.0 and so on. This is encoded in the build scripts that check for this and set up the final Makefiles or whatever to actually compile the thing.
In practice it's more difficult because distributions often "fix" build scripts to allow versions outside this range, or the libraries they ship are heavily patched, but in that case it's still a bug, just some else's.
I think writing something against only a single specific version of anything is a shoddy practice and leads to brittle and ultimately hard to maintain software. It means you can't easily do updates and will depend on bugs in whatever version you use.
If you seriously suggested that way to handle things to an open source project most would laugh at you.
That's what people are doing with Alpine and Musl!
Its only a bug if the software was stated to support that combination. If it was designed and documented to suppot three combinations, and you tried it on a fourth, found it not to work, that’s an enhancement request not a bug report.
I'm sure that'll be reassuring when production goes down at 3am... and you're seeing an error message that nobody else in the world has encountered.
Only if you do it for everything. But it might be worth it for the most performance critical components.
I buy that for a bottleneck, and also when you encounter bugs (it happens). This post didn’t go into day 2 though, and having dealt with a very large code base. It was very hard to maintain it over the years. We started sourcing most things from conda-forge because it was easier (data science pipelines) but a few things we still built.