FWIW, just having a mask in the interrupt controller is normally enough to give you the same thing at the cost of a dozen or so cycles in the critical path. Basically you just keep a mask per priority that can be built up cheaply at init time (or even compile time if you're cute about it), you apply the appropriate mask in the interrupt prologues and epilogues, and pretty much as soon as you apply the new mask in the prologue you go ahead and acknowledge the interrupt.
You can do this on x86 as well at a cost a merely tens to hundreds (possibly lots of hundreds) of thousands of cycles. This is part of why x86 is so popular in the embedded space.
(I’m being sarcastic, obviously. x86 interrupts and interrupt returns are hilariously slow. FRED may improve this by quite a bit.)
Mostly tons of speculative state that needs to be unwound, combined with spectre mitigations, plus tons of committed state that the interrupt prologue needs to save, plus a huge song and dance to do that correctly (that FRED should help with).
All combined with the fact that there's a good chance the memory the interrupt handler is going to touch isn't in the cached working set anymore, both in the actual L* caches and in subtler places like the branch predictors and TLBs.
You’re missing the big ones: both the interrupt delivery and the IRET (interrupt return) mechanisms use incredibly complicated data structures to determine what the new state should be. They need to dig around in the IDT, the GDT, the TSS and possibly the LDT to find all the register values they need to set, and they need to handle all kinds of backwards compatibility. And they “serialize”, which is an extra heavyweight fence, although that only likely accounts for a few hundred cycles in each direction.
Check out the pseudocode in the SDM — there are pages of it, and the pseudocode isn’t even complete.
FRED simplifies the state transitions such that the new state is mostly a foregone conclusion based on MSR contents.
A 1GHz superscalar Cortex-M7 can still hit a 12 cycle interrupt latency if the system stack is kept in the 2x32 bit wide zero cycle dTCM, and the VTOR and handler are in the iTCM thanks to lots of bandwidth and a reasonably short pipeline. The NVIC is truly the underappreciated superpower of the Cortex-M family. You can implement a full RTOS or complex bare metal application without ever having to disable interrupts. On other MCU architectures (most RISC-V, PIC32, even older crap refusing to die) you'll easily end up with closer to 100 then 10 cycles of jitter from critical sections having to run with disabled interrupts e.g. context switching code.
That’s fair. I’m not saying x86 is a pig for interrupt latency. But M still isn’t a fair comparison. X86 runs general purpose OSes which like to do things like load balance interrupts (not that I’m convinced this is actually a good design) which NVIC doesn’t have a full solution to especially for multicore (A series also uses a GIC).
The other problem is that A series and x86 runs out of DRAM typically whereas M is generally set up to run its ISRs out of SRAM so it can actually realistically hit its low latencies. But even though A also has significantly better nominal interrupt latency (competitive with M actually), in practice it’s similar to x86 because DRAM dominates anyway. Also of course best case latencies are when you don’t use the FPU which is more common on M series than it would be on A or x86 (and x86 also has generally more SIMD stuff to handle)
The gap is a low narrower than those exaggerated numbers. Cortex-M goes to 1 GHz and beyond with MIMXRT117x, while loads of x86 chips down-clocked to the couple GHz or are capped lower, like laptop CPUs
I am aware. That "dozen or so" is a problem: when everything is an interrupt, there are no interrupts: it's just scheduling, and things that must be scheduled frequently can't suffer "a dozen or so" overhead. For the SRP model to really hum, you need the silicon that solves this.
I've found that it doesn't matter except for something that you want at the absolute highest priority anyway, which then by definition doesn't need to jump through the same hoops because nothing can preempt it anyway.
It's more about frequency than priority. When something has to be serviced tens of thousands of times a second, "a dozen or so" becomes a problem. If you have the silicon that solves this, you can retain the model. If you don't then you have to resort to workarounds.
Fortunately we typically have more efficient means to deal with such hardware, but not always, and it would be a shame to break the intended model when this is the case.
Comments
FWIW, just having a mask in the interrupt controller is normally enough to give you the same thing at the cost of a dozen or so cycles in the critical path. Basically you just keep a mask per priority that can be built up cheaply at init time (or even compile time if you're cute about it), you apply the appropriate mask in the interrupt prologues and epilogues, and pretty much as soon as you apply the new mask in the prologue you go ahead and acknowledge the interrupt.
You can do this on x86 as well at a cost a merely tens to hundreds (possibly lots of hundreds) of thousands of cycles. This is part of why x86 is so popular in the embedded space.
(I’m being sarcastic, obviously. x86 interrupts and interrupt returns are hilariously slow. FRED may improve this by quite a bit.)
What's the technical reason for them being slow? Book keeping with caches or something?
Mostly tons of speculative state that needs to be unwound, combined with spectre mitigations, plus tons of committed state that the interrupt prologue needs to save, plus a huge song and dance to do that correctly (that FRED should help with).
All combined with the fact that there's a good chance the memory the interrupt handler is going to touch isn't in the cached working set anymore, both in the actual L* caches and in subtler places like the branch predictors and TLBs.
You’re missing the big ones: both the interrupt delivery and the IRET (interrupt return) mechanisms use incredibly complicated data structures to determine what the new state should be. They need to dig around in the IDT, the GDT, the TSS and possibly the LDT to find all the register values they need to set, and they need to handle all kinds of backwards compatibility. And they “serialize”, which is an extra heavyweight fence, although that only likely accounts for a few hundred cycles in each direction.
Check out the pseudocode in the SDM — there are pages of it, and the pseudocode isn’t even complete.
FRED simplifies the state transitions such that the new state is mostly a foregone conclusion based on MSR contents.
Thanks. Surely this was a performance hit even before Spectre?
Also, any good technical resources that concisely describe FRED?
I found this, which isn't bad but it's a bit more dumbed down than I'd like: https://www.tomshardware.com/pc-components/cpus/amd-adopts-f...
But you’re also comparing dozens of cycles at 10mhz to 100ks at 5ghz. That’s probably comparable in terms of wall clock, no?
A 1GHz superscalar Cortex-M7 can still hit a 12 cycle interrupt latency if the system stack is kept in the 2x32 bit wide zero cycle dTCM, and the VTOR and handler are in the iTCM thanks to lots of bandwidth and a reasonably short pipeline. The NVIC is truly the underappreciated superpower of the Cortex-M family. You can implement a full RTOS or complex bare metal application without ever having to disable interrupts. On other MCU architectures (most RISC-V, PIC32, even older crap refusing to die) you'll easily end up with closer to 100 then 10 cycles of jitter from critical sections having to run with disabled interrupts e.g. context switching code.
That’s fair. I’m not saying x86 is a pig for interrupt latency. But M still isn’t a fair comparison. X86 runs general purpose OSes which like to do things like load balance interrupts (not that I’m convinced this is actually a good design) which NVIC doesn’t have a full solution to especially for multicore (A series also uses a GIC).
The other problem is that A series and x86 runs out of DRAM typically whereas M is generally set up to run its ISRs out of SRAM so it can actually realistically hit its low latencies. But even though A also has significantly better nominal interrupt latency (competitive with M actually), in practice it’s similar to x86 because DRAM dominates anyway. Also of course best case latencies are when you don’t use the FPU which is more common on M series than it would be on A or x86 (and x86 also has generally more SIMD stuff to handle)
The gap is a low narrower than those exaggerated numbers. Cortex-M goes to 1 GHz and beyond with MIMXRT117x, while loads of x86 chips down-clocked to the couple GHz or are capped lower, like laptop CPUs
I am aware. That "dozen or so" is a problem: when everything is an interrupt, there are no interrupts: it's just scheduling, and things that must be scheduled frequently can't suffer "a dozen or so" overhead. For the SRP model to really hum, you need the silicon that solves this.
I've found that it doesn't matter except for something that you want at the absolute highest priority anyway, which then by definition doesn't need to jump through the same hoops because nothing can preempt it anyway.
It's more about frequency than priority. When something has to be serviced tens of thousands of times a second, "a dozen or so" becomes a problem. If you have the silicon that solves this, you can retain the model. If you don't then you have to resort to workarounds.
Fortunately we typically have more efficient means to deal with such hardware, but not always, and it would be a shame to break the intended model when this is the case.