There are two phases I've seen in becoming a security engineer. The first phase is moving beyond the "I am an engineer" mindset, where the goal is to build systems that fit into a specific set of design constraints. You have to realize that systems can operate outside of those design constraints. This is shockingly hard for good engineers to learn -- I've been in rooms full of them when they have their "aha" moment.
The second phase is when engineers realize that just whacking specific vulnerabilities is not going to end bugs -- that you need to take systematic actions to close entire vulnerability classes. That's where formal verification, sandboxing, MTE etc. come from. But in practice so far, this doesn't end vulnerabilities, it just leads to a bunch of new and more exciting ones.
I want to believe that with enough of a push we can get AIs to finish all of this and we'll be security-bug free. But if we can't, at least we can get to the point where new vulnerabilities are costly again.
When everyone suddenly started calling themselves engineers instead of programmers I cringed (yes, that's a lot of cringe over the past decade+), precisely because no formal guarantees were ever given (and still largely aren't).
It's like eyeballing the dimensions of a building and saying it should hold. That said, I see the discipline excused. The stakes weren't that high and it was all very new... And e.g. mechanical engineering went through this as well, except a lot of people actually died.
Yeah every other engineering discipline is licensed, bonded, insured, and (typically) unionized.
Emphasis on "licensed" as in "you need to complete an ABET-accredited program and pass a licensure exam and complete continuing education courses to practice, because there are standards and regulations to prevent accidents".
The phrase "regulations written in blood" also comes to mind.
But in practice so far, this doesn't end vulnerabilities, it just leads to a bunch of new and more exciting ones.
I kind of wish you wrote "potentially worse" rather than "exciting", because that happens too, and it's deceptively subtle and underappreciated.
To make this very concrete with a programming example, C and C++ are (somewhat counterintuitively) examples here, because if you guaranteed the absence of an entire classes of vulnerabilities - say, guaranteeing that uninitialized memory is zero, to prevent secret leaks - then you simultaneously make it much harder to detect logic bugs that this would've surfaced, since you no longer have that degree of freedom to detect logic bugs (say, via sanitizers). Say, an initialized UID that would've appeared as 0xDEADBEEF might now be well-defined as UID 0, giving you root access instead of tripping an alarm...
In other words, it's like natural selection and antibiotics: being too good at solving one class of problems selects for other classes that are more resilient and harder to find, whereas in some of those cases, whack-a-mole would've actually uncovered the root cause. Like with antibiotics, that's sometimes worth it, but definitely not always! Some infections just aren't worth preventing at all costs.
I'm obviously not saying we should write unsafe code or that we shouldn't try to eliminate entire classes of bugs, but that HOW we do it matters. We don't want to end up in a situation where problems still lurk but we push their detection beyond our ability because of the way we "solved" other problems.
(C++ was just for illustration here; this extends far beyond programming.)
class User {
explicit User(const char* name) {
if (!look_up_uid(&uid, name)) {
abort();
}
}
bool is_root() const {
return uid == 0;
}
int uid;
};
Let's say look_up_uid() forgot to fill in uid for certain special kinds of users. Like maybe you have a dummy 'nobody' user that was introduced specially after the fact and which is not in the database like the rest.
As C++ is right now, uid would contain garbage. Which means that, at run time, you would often get invalid UIDs if you attempted to log in with such a user, triggering some logging or reporting you to Santa or whatever. And which means that sanitizers would immediately tell you that you forgot to initalialize the field if you ever try to use it (say, in is_root()). Both of these would flag the bug the moment that that kind of user attempts to log in, and make you dig into look_up_uid()'s body to figure out why it's not returning the UID when it's supposed to.
However, if C++ were to zero-initialize everything by default, then neither of those would be true - you would silently get a root user, which is capable of doing everything that nobody can do. And someone who reads the code wouldn't immediately know that you have such a bug; it would sit there idly until someone exploits it.
Thank you, I understand the example better now. Squashing the class of problems of using non-initialized variables by using zero-initialization causes potentially worse bugs since 0 could inadvertently be a correct value. That makes sense!
Going back to GP:
being too good at solving one class of problems selects for other classes that are more resilient and harder to find
Rather than this being too good at solving this class of problem, it seems to me that zero-initialization is the wrong approach; if the default value were present in the program, it'd be eas(y|ier) to spot. Initializer checks can do that without introducing this issue. You're also using the fact that non-initialized values are "random" by default- we could also use fuzzer checkers for that.
I think the general lesson from the example is that the way you solve a class of problems could introduce more pernicious ones, rather than the fact that it's solved.
Rather than this being too good at solving this class of problem, it seems to me that zero-initialization is the wrong approach
That's exactly why I wrote this here:
> I'm obviously not saying we should write unsafe code or that we shouldn't try to eliminate entire classes of bugs, but that HOW we do it matters.
After you get past the hurdle of noticing this problem (which, as you saw, is very much not obvious), the harder question becomes: what is the right approach?
In this particular case it's not too hard to think of a better approach once you concede the obvious solution isn't so great, but in other cases it is, and often the better alternatives put some kind of selection pressure too... just less frequently. And even in this case, it's not at all obvious that this approach is bad - plenty of people think it's better to force a default value you can rely on, and they want to remove undefined behavior from C++ by forcing initialization on everything. For longstanding examples elsewhere, just look at how Java and C# initialize fields, for example.
Outside of programming it's even harder to notice and find a better alternative, but selection pressure has these kinds of effects in other areas too.
The problem is that 0 is a valid UID in POSIX systems like Linux.
If you automatically initialize variables with a garbage value, use-before-initialization (the program’s initialization) them the use of that variable will likely fail due to error. By using what turns out to be a legit UID on every system you have the opportunity for this case not to be detected, perhaps causing a problem immediately or else allowing some nefarious actor to write what they want into that variable instead.
Comments
There are two phases I've seen in becoming a security engineer. The first phase is moving beyond the "I am an engineer" mindset, where the goal is to build systems that fit into a specific set of design constraints. You have to realize that systems can operate outside of those design constraints. This is shockingly hard for good engineers to learn -- I've been in rooms full of them when they have their "aha" moment.
The second phase is when engineers realize that just whacking specific vulnerabilities is not going to end bugs -- that you need to take systematic actions to close entire vulnerability classes. That's where formal verification, sandboxing, MTE etc. come from. But in practice so far, this doesn't end vulnerabilities, it just leads to a bunch of new and more exciting ones.
I want to believe that with enough of a push we can get AIs to finish all of this and we'll be security-bug free. But if we can't, at least we can get to the point where new vulnerabilities are costly again.
When everyone suddenly started calling themselves engineers instead of programmers I cringed (yes, that's a lot of cringe over the past decade+), precisely because no formal guarantees were ever given (and still largely aren't).
It's like eyeballing the dimensions of a building and saying it should hold. That said, I see the discipline excused. The stakes weren't that high and it was all very new... And e.g. mechanical engineering went through this as well, except a lot of people actually died.
Time to shine, theoretical CS, time to shine.
Yeah every other engineering discipline is licensed, bonded, insured, and (typically) unionized.
Emphasis on "licensed" as in "you need to complete an ABET-accredited program and pass a licensure exam and complete continuing education courses to practice, because there are standards and regulations to prevent accidents".
The phrase "regulations written in blood" also comes to mind.
I kind of wish you wrote "potentially worse" rather than "exciting", because that happens too, and it's deceptively subtle and underappreciated.
To make this very concrete with a programming example, C and C++ are (somewhat counterintuitively) examples here, because if you guaranteed the absence of an entire classes of vulnerabilities - say, guaranteeing that uninitialized memory is zero, to prevent secret leaks - then you simultaneously make it much harder to detect logic bugs that this would've surfaced, since you no longer have that degree of freedom to detect logic bugs (say, via sanitizers). Say, an initialized UID that would've appeared as 0xDEADBEEF might now be well-defined as UID 0, giving you root access instead of tripping an alarm...
In other words, it's like natural selection and antibiotics: being too good at solving one class of problems selects for other classes that are more resilient and harder to find, whereas in some of those cases, whack-a-mole would've actually uncovered the root cause. Like with antibiotics, that's sometimes worth it, but definitely not always! Some infections just aren't worth preventing at all costs.
I'm obviously not saying we should write unsafe code or that we shouldn't try to eliminate entire classes of bugs, but that HOW we do it matters. We don't want to end up in a situation where problems still lurk but we push their detection beyond our ability because of the way we "solved" other problems.
(C++ was just for illustration here; this extends far beyond programming.)
I'm not following your example very well. Why is it that this makes the new vulnerabilities potentially worse?
You can initialise a UID to 0 whether or not you're using a compiler or checks for initialisation. Do you have another example?
Here's a trivial example to illustrate:
Let's say look_up_uid() forgot to fill in uid for certain special kinds of users. Like maybe you have a dummy 'nobody' user that was introduced specially after the fact and which is not in the database like the rest.As C++ is right now, uid would contain garbage. Which means that, at run time, you would often get invalid UIDs if you attempted to log in with such a user, triggering some logging or reporting you to Santa or whatever. And which means that sanitizers would immediately tell you that you forgot to initalialize the field if you ever try to use it (say, in is_root()). Both of these would flag the bug the moment that that kind of user attempts to log in, and make you dig into look_up_uid()'s body to figure out why it's not returning the UID when it's supposed to.
However, if C++ were to zero-initialize everything by default, then neither of those would be true - you would silently get a root user, which is capable of doing everything that nobody can do. And someone who reads the code wouldn't immediately know that you have such a bug; it would sit there idly until someone exploits it.
Thank you, I understand the example better now. Squashing the class of problems of using non-initialized variables by using zero-initialization causes potentially worse bugs since 0 could inadvertently be a correct value. That makes sense!
Going back to GP:
Rather than this being too good at solving this class of problem, it seems to me that zero-initialization is the wrong approach; if the default value were present in the program, it'd be eas(y|ier) to spot. Initializer checks can do that without introducing this issue. You're also using the fact that non-initialized values are "random" by default- we could also use fuzzer checkers for that.
I think the general lesson from the example is that the way you solve a class of problems could introduce more pernicious ones, rather than the fact that it's solved.
That's exactly why I wrote this here:
After you get past the hurdle of noticing this problem (which, as you saw, is very much not obvious), the harder question becomes: what is the right approach?
In this particular case it's not too hard to think of a better approach once you concede the obvious solution isn't so great, but in other cases it is, and often the better alternatives put some kind of selection pressure too... just less frequently. And even in this case, it's not at all obvious that this approach is bad - plenty of people think it's better to force a default value you can rely on, and they want to remove undefined behavior from C++ by forcing initialization on everything. For longstanding examples elsewhere, just look at how Java and C# initialize fields, for example.
Outside of programming it's even harder to notice and find a better alternative, but selection pressure has these kinds of effects in other areas too.
The problem is that 0 is a valid UID in POSIX systems like Linux.
If you automatically initialize variables with a garbage value, use-before-initialization (the program’s initialization) them the use of that variable will likely fail due to error. By using what turns out to be a legit UID on every system you have the opportunity for this case not to be detected, perhaps causing a problem immediately or else allowing some nefarious actor to write what they want into that variable instead.