Given that PCIe allows data to be piped directly from one device to another without going through the host CPU[1][2], I guess it might make sense to just have the GPU read blocks straight from the NVMe (or even NVMe-of[3]) rather than having the CPU do a lot of work.
edit: blind as a bat, says so right in the paper of course:
PMem is mapped directly to the GPU, and NVMe memory is accessed via Peer to Peer-DMA (P2PDMA)
I'm not sure they're actually doing NVMe yet; using Optane PMem is a bit of a cheat so that accessing storage is just plain memory reads and writes over PCIe. Implementing an NVMe device driver to set up and interact with command queues would be an extra layer of complexity that I think they left for future work.
But this work used the Optane DC Persistent Memory DIMMs that only work with certain Intel server CPUs. I'm not sure what the typical price people actually paid for those was, but it probably was not actually more expensive than DRAM.
For GPUs where Nvidia has turned off P2P, can RAM or NVMe drives be used for emulating P2P? Let’s assume you have a RAID AIC with 4 or 8 high speed SSDs. Could you make 3 3090s work as well as 3 A5000 RTX for training a model?
Comments
Given that PCIe allows data to be piped directly from one device to another without going through the host CPU[1][2], I guess it might make sense to just have the GPU read blocks straight from the NVMe (or even NVMe-of[3]) rather than having the CPU do a lot of work.
edit: blind as a bat, says so right in the paper of course:
PMem is mapped directly to the GPU, and NVMe memory is accessed via Peer to Peer-DMA (P2PDMA)
[1]: https://nvmexpress.org/wp-content/uploads/Enabling-the-NVMe-...
[2]: https://lwn.net/Articles/767281/
[3]: https://www.nvmexpress.org/wp-content/uploads/NVMe_Over_Fabr...
I'm not sure they're actually doing NVMe yet; using Optane PMem is a bit of a cheat so that accessing storage is just plain memory reads and writes over PCIe. Implementing an NVMe device driver to set up and interact with command queues would be an extra layer of complexity that I think they left for future work.
Sure, but my point was that it should be quite possible to get regular NVMes working.
Once you got that then the CPU is just the orchesterator, and wouldn't necessarily need to be so beefy.
That's just called DirectStorage and was added as part of Windows 10 (erm.... some update in Windows 10).
The PS5 and Xbox both have GPU-access of NVMe Flash.
------
So you are right. But what you are talking about happened like 5 years ago.
EDIT: https://devblogs.microsoft.com/directx/directstorage-develop...
Looks like 3 years ago for Win10. But I feel like I heard it sooner than that as NVidia or AMD specific API calls.
Didn't they stop making Optane? :(
Also, Optane was like $4 per GB, so a moderately-sized drive, like 256GB, is already above $1000.
The Optane NVMe drives were more like $1-2 per GB when they were new, and are a fair bit cheaper now that they're basically on clearance: https://www.newegg.com/intel-optane-ssd-905p-series-960gb/p/...
But this work used the Optane DC Persistent Memory DIMMs that only work with certain Intel server CPUs. I'm not sure what the typical price people actually paid for those was, but it probably was not actually more expensive than DRAM.
Yes, Optane isn't produced anymore
For GPUs where Nvidia has turned off P2P, can RAM or NVMe drives be used for emulating P2P? Let’s assume you have a RAID AIC with 4 or 8 high speed SSDs. Could you make 3 3090s work as well as 3 A5000 RTX for training a model?