It's not really obvious unless you go in depth of the details on modern GPU architecture. GPUs aren't really SIMD, they're SIMT (single instruction multiple thread). The silicon looks a lot like SIMD, but the programming model is different.
If you go look at AMD's ISA docs (they're public) you'll see you don't have the equivalent of a __mm256 register like on x86. Each 'thread' just deals with single scalar values like int32 of float32. The hardware, however, groups 32 or 64 threads together which all run the same program and runs them together. Each 'thread' loosely maps to a SIMD lane. The SIMD is implicit, not explicit.
The main difference is that the 'SIMD' execution is somewhat opaque to the program. You just write plain scalar code and the hardware model dispatches it efficiently to SIMD execution units. It's not really an abstraction because to extract maximum performance you have to understand how it works. You can use this kind of programming model on a CPU too, Intel did it with [0] ISPC. It's a C-like language that has execution semantics similar to GPU shader languages but compiles to regular CPU code, and maps threads to your CPUs SIMD lanes like a GPU.
Yes, of course writing naive code assuming each lane in a thread group is a real thread is going to cause problems, but I didn't feel like I needed to go into that level of detail replying to someone just learning about GPU internals. I tried to cover this loosely by mentioning how you need to know how it works for maximum performance.
If you want to get more pedantic you also need to look at your target hardware and their specific micro-architectural quirks and features to get the best performance. AMD specifically benefits a lot from exploiting the scalar unit over the vector unit, you save loads of register file space if you can keep data in SGPRs over VGPRs. There's lots of traps you can fall into where you can load data from buffers into SGPRs but they get promoted to VGPRs because the scalar unit lacks an opcode for like one math operation you did to the value somewhere.
While each lane isn't truly a thread because it doesn't have its own PC the programming model definitely tries to make it seem that way. The threads can terminate at different points too. And again, the ISA isn't a vector ISA. Your register values are scalar.
While each lane isn't truly a thread because it doesn't have its own PC the programming model definitely tries to make it seem that way. The threads can terminate at different points too. And again, the ISA isn't a vector ISA. Your register values are scalar.
This is not correct. If you check AMD's documentation there are explicit mentions of vector registers (VGPR), vector ALUs, and vector instructions. The introduction to Chapter 2 describes it as a vector ISA.
RDNA4 shader programs (kernels) are programs executed by the shader processor. Conceptually, the shader program is executed independently on every work-item, but in reality the processor groups up to 32 or 64 work-items into a wave, that executes the shader program on all 32 or 64 work-items in one pass ("wave32" or
"wave64").
A VGPR is not the same thing as a vector register like in SSE4 or AVX. Each addressed register contains a single 32-bit value. A VGPR differs from an SGPR in that each thread in a thread group can have a different value in that register. An SGPR will have a uniform value shared with all threads in a group.
An add instruction on an AMD GPU adds two scalar values. If they're in a VGPR then each thread will add two values unique to that thread. A SIMD ISA as is common on a CPU is different because an add instruction explicitly adds a vector of values. xmm1 stores 128-bits of data. VGPR[1] stores 32-bits of data vectored over 32-64 threads in a thread group.
Without special instructions a thread can't access the VGPR values stored in other threads.
False. If they were threads they'd have their own PC. They do not - only the warp has a PC.
They are using the term SIMT as it is normally used[1]. The "single instruction" part means that there is only one PC shared across multiple 'threads'.
Actually not so false anymore. (But still they don't expect you to use this knowledge while coding, and you should treat all threads in a warp as moving in lockstep)
In GPUs of compute capability 7.0 and later, independent thread scheduling allows full concurrency between threads, regardless of warp. With independent thread scheduling, the GPU maintains execution state per thread, including a program counter and call stack... [1]
This might be more confusing than it needs to be. SIMD and SIMT are not mutually exclusive.
People commonly think of things like vector registers when they talk about SIMD, and each "thread" in a GPU warp definitely deals with local vector registers. Granted, they may be slices of superwide registers shared by the whole warp, or whatever else, but from the programmer's perspective, that's a valid way to think about it.
Put another way, it would be a mistake to think that each lane of a vec4 in a shader gets processed by a separate unit.
Comments
It's not really obvious unless you go in depth of the details on modern GPU architecture. GPUs aren't really SIMD, they're SIMT (single instruction multiple thread). The silicon looks a lot like SIMD, but the programming model is different.
If you go look at AMD's ISA docs (they're public) you'll see you don't have the equivalent of a __mm256 register like on x86. Each 'thread' just deals with single scalar values like int32 of float32. The hardware, however, groups 32 or 64 threads together which all run the same program and runs them together. Each 'thread' loosely maps to a SIMD lane. The SIMD is implicit, not explicit.
The main difference is that the 'SIMD' execution is somewhat opaque to the program. You just write plain scalar code and the hardware model dispatches it efficiently to SIMD execution units. It's not really an abstraction because to extract maximum performance you have to understand how it works. You can use this kind of programming model on a CPU too, Intel did it with [0] ISPC. It's a C-like language that has execution semantics similar to GPU shader languages but compiles to regular CPU code, and maps threads to your CPUs SIMD lanes like a GPU.
[0] https://ispc.github.io/
False. If they were threads they'd have their own PC. They do not - only the warp has a PC.
Absolutely not. If you don't write coalesced loads, bank-conflict free, predication-free, cooperative code you will get worse than CPU performance.
Yes, of course writing naive code assuming each lane in a thread group is a real thread is going to cause problems, but I didn't feel like I needed to go into that level of detail replying to someone just learning about GPU internals. I tried to cover this loosely by mentioning how you need to know how it works for maximum performance.
If you want to get more pedantic you also need to look at your target hardware and their specific micro-architectural quirks and features to get the best performance. AMD specifically benefits a lot from exploiting the scalar unit over the vector unit, you save loads of register file space if you can keep data in SGPRs over VGPRs. There's lots of traps you can fall into where you can load data from buffers into SGPRs but they get promoted to VGPRs because the scalar unit lacks an opcode for like one math operation you did to the value somewhere.
While each lane isn't truly a thread because it doesn't have its own PC the programming model definitely tries to make it seem that way. The threads can terminate at different points too. And again, the ISA isn't a vector ISA. Your register values are scalar.
This is not correct. If you check AMD's documentation there are explicit mentions of vector registers (VGPR), vector ALUs, and vector instructions. The introduction to Chapter 2 describes it as a vector ISA.
Sources:
https://gpuopen.com/amd-gpu-architecture-programming-documen...
https://docs.amd.com/v/u/en-US/rdna4-instruction-set-archite...
A VGPR is not the same thing as a vector register like in SSE4 or AVX. Each addressed register contains a single 32-bit value. A VGPR differs from an SGPR in that each thread in a thread group can have a different value in that register. An SGPR will have a uniform value shared with all threads in a group.
An add instruction on an AMD GPU adds two scalar values. If they're in a VGPR then each thread will add two values unique to that thread. A SIMD ISA as is common on a CPU is different because an add instruction explicitly adds a vector of values. xmm1 stores 128-bits of data. VGPR[1] stores 32-bits of data vectored over 32-64 threads in a thread group.
Without special instructions a thread can't access the VGPR values stored in other threads.
But those instructions exist, see Section 7.9 "Cross-Lane and Data Parallel Processing (DPP)"
AMD documentation describes RDNA as a vector ISA, so I don't understand why you say it is not.
They are using the term SIMT as it is normally used[1]. The "single instruction" part means that there is only one PC shared across multiple 'threads'.
[1] https://en.wikipedia.org/wiki/Single_instruction,_multiple_t...
Actually not so false anymore. (But still they don't expect you to use this knowledge while coding, and you should treat all threads in a warp as moving in lockstep)
1: https://docs.nvidia.com/cuda/cuda-programming-guide/03-advan...
This might be more confusing than it needs to be. SIMD and SIMT are not mutually exclusive.
People commonly think of things like vector registers when they talk about SIMD, and each "thread" in a GPU warp definitely deals with local vector registers. Granted, they may be slices of superwide registers shared by the whole warp, or whatever else, but from the programmer's perspective, that's a valid way to think about it.
Put another way, it would be a mistake to think that each lane of a vec4 in a shader gets processed by a separate unit.