In an attempt to go fast and beat benchmarks and other computers, CPUs attempt to speculatively execute code, and then later undo the results of the speculation if it turns out it was wrong. This causes all sorts of security issues (spectre, meltdown, and friends et al.) even when it's done relatively competently.
When it's done incompetently as on this ARM implementation, then you can't even run perfectly good and correct code, because the CPU will attempt speculative execution on a location that you never asked it to execute code at, and then bork itself when it realizes that can't possibly work.
Naturally, this is the sort of problem that requires tedious dissection of what exactly happened, and copious amounts of alcohol.
So in this specific case, if I understood correctly, here’s what should happen:
Interrupt(?) fires to trigger hypervisor, hypervisor figures out what it needs to do, jumps to that code, does its job, returns.
The “figured out what it needs to do” is the issue right? So what was actually happening was:
Same start… CPU predicts what hypervisor will do, speculatively loads instructions from mispredicted branch target, that wrong instruction reads memory(?) against the “no data prefetch” settings for that part of memory, CPU blows up/halts/whatever.
The fix is to mark the area the branch was mispredicted to in such a way that the CPU won’t prefetch instructions. Thus that won’t be run and prefetch data, thus no violation. CPU execution continues taking the correct branch and everything is fine.
No, that instruction does not read memory. That instruction is itself IN the inaccessible memory.
It does not really matter how execution ended up in the HV in the first place. The misprediction happens due to having a branch-to-register instruction.
Well, that was the whole issue. After marking the memory as not accessible via a data load, the CPU was still executing there and borking itself.
He had to also mark the memory as inaccessible for code loads.
Which, honestly is kinda stupid. Because nobody asked the processor to start executing there. It just took it upon itself to try to start executing there, later decided that was a bad thing, and then borked itself.
People talk about how terrible x86 is, but honestly, one of the reasons that x86 won for decades was because of making things that programmers did that might be suboptimal still work, even if a bit slowly (misaligned data accesses, for example). ARM does that now for that specific case, but didn't before 2002.
So there's an implementation tradeoff for whether you decide to spend transistors to reduce the number of sharp edges on the tool. Obviously ARM just doesn't give a shit about this particular sharp edge.
Speculative instruction fetches fetch code from "a location that you never asked it to execute code at" by design. If it already knew you asked it to execute code there, it wouldn't be speculative.
The Armv8/9-A architecture reference manual is clear that speculative instruction fetches are permitted in Device memory unless that memory is also marked NX. So if your hardware has side-effects from a certain address, but it maps it as Device non-NX memory, then your code is not "perfectly good and correct". Assigning correct memory attributes is one of the many things needed for correct code.
Speculative instruction fetches fetch code from "a location that you never asked it to execute code at" by design. If it already knew you asked it to execute code there, it wouldn't be speculative.
Sure, but (a) the speculation is supposed to be transparent; and (b) ARM's response to a failure in speculative execution is arguably broken.
The Armv8/9-A architecture reference manual
Sure, but before then marking memory as not readable would make it non-executable, and also mostly before then there was no or minimal speculation.
then your code is not "perfectly good and correct".
Certainly it won't work on that machine. It might have worked perfectly well on prior machines, and we could argue all day about how reasonable ARM's implementation decisions were, but I will merely say that a documented flaw is, in fact, still a flaw.
Comments
In an attempt to go fast and beat benchmarks and other computers, CPUs attempt to speculatively execute code, and then later undo the results of the speculation if it turns out it was wrong. This causes all sorts of security issues (spectre, meltdown, and friends et al.) even when it's done relatively competently.
When it's done incompetently as on this ARM implementation, then you can't even run perfectly good and correct code, because the CPU will attempt speculative execution on a location that you never asked it to execute code at, and then bork itself when it realizes that can't possibly work.
Naturally, this is the sort of problem that requires tedious dissection of what exactly happened, and copious amounts of alcohol.
So in this specific case, if I understood correctly, here’s what should happen:
Interrupt(?) fires to trigger hypervisor, hypervisor figures out what it needs to do, jumps to that code, does its job, returns.
The “figured out what it needs to do” is the issue right? So what was actually happening was:
Same start… CPU predicts what hypervisor will do, speculatively loads instructions from mispredicted branch target, that wrong instruction reads memory(?) against the “no data prefetch” settings for that part of memory, CPU blows up/halts/whatever.
The fix is to mark the area the branch was mispredicted to in such a way that the CPU won’t prefetch instructions. Thus that won’t be run and prefetch data, thus no violation. CPU execution continues taking the correct branch and everything is fine.
No, that instruction does not read memory. That instruction is itself IN the inaccessible memory.
It does not really matter how execution ended up in the HV in the first place. The misprediction happens due to having a branch-to-register instruction.
Ah, thanks. So instruction loads also count as data loads?
Well, that was the whole issue. After marking the memory as not accessible via a data load, the CPU was still executing there and borking itself.
He had to also mark the memory as inaccessible for code loads.
Which, honestly is kinda stupid. Because nobody asked the processor to start executing there. It just took it upon itself to try to start executing there, later decided that was a bad thing, and then borked itself.
People talk about how terrible x86 is, but honestly, one of the reasons that x86 won for decades was because of making things that programmers did that might be suboptimal still work, even if a bit slowly (misaligned data accesses, for example). ARM does that now for that specific case, but didn't before 2002.
So there's an implementation tradeoff for whether you decide to spend transistors to reduce the number of sharp edges on the tool. Obviously ARM just doesn't give a shit about this particular sharp edge.
Speculative instruction fetches fetch code from "a location that you never asked it to execute code at" by design. If it already knew you asked it to execute code there, it wouldn't be speculative.
The Armv8/9-A architecture reference manual is clear that speculative instruction fetches are permitted in Device memory unless that memory is also marked NX. So if your hardware has side-effects from a certain address, but it maps it as Device non-NX memory, then your code is not "perfectly good and correct". Assigning correct memory attributes is one of the many things needed for correct code.
Sure, but (a) the speculation is supposed to be transparent; and (b) ARM's response to a failure in speculative execution is arguably broken.
Sure, but before then marking memory as not readable would make it non-executable, and also mostly before then there was no or minimal speculation.
Certainly it won't work on that machine. It might have worked perfectly well on prior machines, and we could argue all day about how reasonable ARM's implementation decisions were, but I will merely say that a documented flaw is, in fact, still a flaw.