Why not let the upstream application developer decide, rather than choosing for them?
This is one of the downsides of using C/C++ rather than a modern programming language like Rust or Zig. In the former case, the system maintainers reach across the table and change the settings, despite what the actual application developer has chosen. In the latter, the upstream developers' choices are respected more, mainly because the tooling is less standardized.
Because the upstream developer is not the only one who needs to profile the program.
For one thing, you might be the upstream developer of a library the program links against, and your stack frames might be hidden under frames from the program that didn’t save the frame pointer. Or if it’s a library that isn’t saving the frame pointer, vice versa.
But the more interesting use case is if you’re not the upstream developer of anything, but a skilled user who wants to get a view of your entire system and diagnose problems yourself. Personally I do that all the time, both in my spare time and at work. With respect to performance in particular I have more experience with the macOS tooling than the Linux tooling, but it’s analogous – with tools like Instruments and dtrace I can get a profile of any process I want or of the entire system, and I find that incredibly valuable. And that’s made possible in part by the stock macOS toolchain enabling frame pointers by default.
The NixOS case you linked sounds rather annoying, but turning on optimizations and -Werror and PIC by default is very different from just enabling frame pointers by default.
Not clearly written in the original article is that "many" (I'm not sure what actual percentage, but vaguely "most") packages already have frame pointers enabled.
The problem packages that don't are exactly all of those upstream projects that intentionally compile with -fomit-frame-pointer because of those small performance gains. And those are also usually the exact same projects you end up wanting to profile or otherwise analyse :)
I work in the Support organisation at Canonical and the two most frequent projects I run into this with are Ceph and Openvswitch - they both compile with -fomit-frame-pointer by default upstream (and currently in the Ubuntu packages) which makes using perf (which I often need to do with both of those) a pain.
While you can do it, you have to record ~8kB of stack extra for every sample (times 1000 per second, times the number of CPUs...) and then unwind it later with the DWARF debug symbols. The resulting perf exports are multiple gigabytes for 0-2 minutes. Compared to maybe 25-200MB for frame-pointer enabled cases where I can usually then easily captured.
The problem is the product or upstream project wants to claim the absolute best performance, even 1% better, but the end user rarely needs that last 1% and both them and their support team would like to be able to easily use profiling in production to fix the random much more significant 10-100% performance bugs that inevitably crop up when actually using it and not benchmarking it :)
Comments
Why not let the upstream application developer decide, rather than choosing for them?
This is one of the downsides of using C/C++ rather than a modern programming language like Rust or Zig. In the former case, the system maintainers reach across the table and change the settings, despite what the actual application developer has chosen. In the latter, the upstream developers' choices are respected more, mainly because the tooling is less standardized.
Shoutouts to this NixOS bug which is still ongoing after causing much pain for many years: https://github.com/NixOS/nixpkgs/issues/18995
Because the upstream developer is not the only one who needs to profile the program.
For one thing, you might be the upstream developer of a library the program links against, and your stack frames might be hidden under frames from the program that didn’t save the frame pointer. Or if it’s a library that isn’t saving the frame pointer, vice versa.
But the more interesting use case is if you’re not the upstream developer of anything, but a skilled user who wants to get a view of your entire system and diagnose problems yourself. Personally I do that all the time, both in my spare time and at work. With respect to performance in particular I have more experience with the macOS tooling than the Linux tooling, but it’s analogous – with tools like Instruments and dtrace I can get a profile of any process I want or of the entire system, and I find that incredibly valuable. And that’s made possible in part by the stock macOS toolchain enabling frame pointers by default.
The NixOS case you linked sounds rather annoying, but turning on optimizations and -Werror and PIC by default is very different from just enabling frame pointers by default.
Yeah, I agree with this.
Not clearly written in the original article is that "many" (I'm not sure what actual percentage, but vaguely "most") packages already have frame pointers enabled.
The problem packages that don't are exactly all of those upstream projects that intentionally compile with -fomit-frame-pointer because of those small performance gains. And those are also usually the exact same projects you end up wanting to profile or otherwise analyse :)
I work in the Support organisation at Canonical and the two most frequent projects I run into this with are Ceph and Openvswitch - they both compile with -fomit-frame-pointer by default upstream (and currently in the Ubuntu packages) which makes using perf (which I often need to do with both of those) a pain.
While you can do it, you have to record ~8kB of stack extra for every sample (times 1000 per second, times the number of CPUs...) and then unwind it later with the DWARF debug symbols. The resulting perf exports are multiple gigabytes for 0-2 minutes. Compared to maybe 25-200MB for frame-pointer enabled cases where I can usually then easily captured.
The problem is the product or upstream project wants to claim the absolute best performance, even 1% better, but the end user rarely needs that last 1% and both them and their support team would like to be able to easily use profiling in production to fix the random much more significant 10-100% performance bugs that inevitably crop up when actually using it and not benchmarking it :)