That's right, and that means most transforms in big data. The fact that the dataset can be distributed at all typically implies that the task is parallel.
The fact that the dataset can be distributed at all typically implies that the task is parallel.
It depends, big data tasks aren’t necessarily CPU bound but IO bound so you won’t see any speed up throwing GPUs at the problem but you will see a speed up throwing more worker nodes that come with their own network bandwidth and memory. CPU bound problems aren’t all appropriate for GPUs either. I suspect the article author is thinking of ML pipelines where at large scales GPUs are definitely necessary but at lower scales you can get away with ordinary CPUs.
If they aren't network/disk bound, then GPUs have significantly higher memory bandwidth (per $). If you're so network bound that your nodes must be <8 cores, then sure. Otherwise, adding a cheap GPU to your nodes would likely be cheaper.
If you're so network bound that your nodes must be <8 cores, then sure. Otherwise, adding a cheap GPU to your nodes would likely be cheaper.
Suppose you're so network bound that your nodes stop being compute bound at 64 cores; more than 8. Then any GPU is useless because you're still network bound and the CPU and GPU both have more memory bandwidth than the network. A cheap GPU is even worse because the CPU may have >100 PCIe lanes to use for network I/O but a PCIe GPU would only have 16 -- and it would take 16 PCIe lanes the system could otherwise have used for more network I/O.
Let's assume a cloud datacenter context, using GCP as an example for real, verifiable numbers.
An L4 GPU node (with 4 CPU cores) is approximately the cost of a 16-core VM. So for the price of 64 cores, you could have ~4 L4 nodes (or perhaps one VM with 4 L4s). Each GPU has 300 GB/s of memory bandwidth, which is approximately 60 cores worth (at ~5GB/s per core). In fact, the 64 "cores" are typically 32 physical cores only, so you would expect lower than 300 GB/s for a 64-core VM.
Let's say network I/O is between 10 Gbits/s and 800 Gbits/s.
PCIe 4.0x16 is 64 GB/s, i.e. 512 Gbits/s. PCIe 5.0x16 is double that, which would easily saturate the network link.
Also, L4 GPUs have RDMA enabled, bypassing the CPU when ingesting data.
I also highly doubt that a 64-(v)core VM can access ">100 PCIe lanes" unless you have some kind of sole tenancy setup.
Then, what if: instead of 4 L4 GPUs (which would have a similar cost to the 64-core node), you used 2 L4 GPUs and found that setup to process just as fast as your 64-core node? You'd have just halved your cost.
Of course, this depends on whether or not the GPU nodes actually process your data with the same SLA, but generally GPU network ingest > CPU network ingest (per $ of processing).
That would be a bad fit for this kind of workload because they're not optimized for I/O.
Each GPU has 300 GB/s of memory bandwidth, which is approximately 60 cores worth (at ~5GB/s per core).
CPU cores don't have a specific amount of bandwidth, sockets do. For example, Epyc Genoa with 12 channels of DDR5-4800 has ~460GB/s per socket. This is the same regardless of whether the physical CPU in the socket has 32 cores or 128. But also, the memory bandwidth is still irrelevant, because you're I/O bound. If the network is 800Gbits/s, i.e. 100GB/s, it doesn't matter if 64 CPU cores have 200GB/s or 900GB/s or if the GPU has more or less because none of these are the bottleneck when they're all faster than the network.
I also highly doubt that a 64-(v)core VM can access ">100 PCIe lanes" unless you have some kind of sole tenancy setup.
Epyc systems have 128 PCIe lanes, dual socket Epyc systems can have slightly more, common Xeon systems slightly less. A handful of these are used for system devices, but there would be ~100 remaining. That is assuming that you would have the entire machine, but that's what you'd want here. Otherwise you'd need more partial machines to get more I/O.
For network-bound workloads you would then fill the machine's I/O with network interfaces. For storage-bound workloads you would do the same thing with SSDs. 100 PCIe 5.0 lanes thereby get you up to 6400Gbits/sec, i.e. 800GB/s. Even a quarter of that would not allow a PCIe 5.0x16 GPU to saturate the network/storage interface, you would need four GPUs to get to half and more than four GPUs would have you out of PCIe lanes.
That kind of configuration would cost more per instance than getting some Google Cloud VMs, but those VMs max out at 200Gbits/s, and then how many more of them do you need?
Probably the interesting use case for cloud VMs when you're I/O bound is the one you excluded initially, when you're thoroughly I/O bound and hardly need any compute at all, and then you can get cheap VMs with low core counts and a medium amount of I/O and just use a ton of them distributed across separate physical machines to get high aggregate I/O bandwidth.
The situation you keep trying to find is the one where you're not actually I/O bound. If you're compute or memory bound on the types of computation GPUs are suited to then obviously GPUs will be cheaper.
Sure, everything I said likely goes out the window if we're talking on-prem. No idea how I/O works in that setting, and in fact I am completely unfamiliar with workloads which saturate 800 GB/s network, especially since that approaches the limit of main memory bandwidth. Really appreciate your response! And you're definitely right in noting that the situation I'm talking about is when we're not thoroughly I/O bound.
Comments
That's right, and that means most transforms in big data. The fact that the dataset can be distributed at all typically implies that the task is parallel.
It depends, big data tasks aren’t necessarily CPU bound but IO bound so you won’t see any speed up throwing GPUs at the problem but you will see a speed up throwing more worker nodes that come with their own network bandwidth and memory. CPU bound problems aren’t all appropriate for GPUs either. I suspect the article author is thinking of ML pipelines where at large scales GPUs are definitely necessary but at lower scales you can get away with ordinary CPUs.
If they aren't network/disk bound, then GPUs have significantly higher memory bandwidth (per $). If you're so network bound that your nodes must be <8 cores, then sure. Otherwise, adding a cheap GPU to your nodes would likely be cheaper.
Suppose you're so network bound that your nodes stop being compute bound at 64 cores; more than 8. Then any GPU is useless because you're still network bound and the CPU and GPU both have more memory bandwidth than the network. A cheap GPU is even worse because the CPU may have >100 PCIe lanes to use for network I/O but a PCIe GPU would only have 16 -- and it would take 16 PCIe lanes the system could otherwise have used for more network I/O.
Let's assume a cloud datacenter context, using GCP as an example for real, verifiable numbers.
An L4 GPU node (with 4 CPU cores) is approximately the cost of a 16-core VM. So for the price of 64 cores, you could have ~4 L4 nodes (or perhaps one VM with 4 L4s). Each GPU has 300 GB/s of memory bandwidth, which is approximately 60 cores worth (at ~5GB/s per core). In fact, the 64 "cores" are typically 32 physical cores only, so you would expect lower than 300 GB/s for a 64-core VM.
Let's say network I/O is between 10 Gbits/s and 800 Gbits/s. PCIe 4.0x16 is 64 GB/s, i.e. 512 Gbits/s. PCIe 5.0x16 is double that, which would easily saturate the network link. Also, L4 GPUs have RDMA enabled, bypassing the CPU when ingesting data. I also highly doubt that a 64-(v)core VM can access ">100 PCIe lanes" unless you have some kind of sole tenancy setup.
Then, what if: instead of 4 L4 GPUs (which would have a similar cost to the 64-core node), you used 2 L4 GPUs and found that setup to process just as fast as your 64-core node? You'd have just halved your cost.
Of course, this depends on whether or not the GPU nodes actually process your data with the same SLA, but generally GPU network ingest > CPU network ingest (per $ of processing).
Here's some benchmark results from Voltron Data (I'm not affiliated with them): https://voltrondata.com/benchmarks/theseus
Here's more data from PayPal: https://medium.com/paypal-tech/leveraging-spark-3-and-nvidia...
That would be a bad fit for this kind of workload because they're not optimized for I/O.
CPU cores don't have a specific amount of bandwidth, sockets do. For example, Epyc Genoa with 12 channels of DDR5-4800 has ~460GB/s per socket. This is the same regardless of whether the physical CPU in the socket has 32 cores or 128. But also, the memory bandwidth is still irrelevant, because you're I/O bound. If the network is 800Gbits/s, i.e. 100GB/s, it doesn't matter if 64 CPU cores have 200GB/s or 900GB/s or if the GPU has more or less because none of these are the bottleneck when they're all faster than the network.
Epyc systems have 128 PCIe lanes, dual socket Epyc systems can have slightly more, common Xeon systems slightly less. A handful of these are used for system devices, but there would be ~100 remaining. That is assuming that you would have the entire machine, but that's what you'd want here. Otherwise you'd need more partial machines to get more I/O.
For network-bound workloads you would then fill the machine's I/O with network interfaces. For storage-bound workloads you would do the same thing with SSDs. 100 PCIe 5.0 lanes thereby get you up to 6400Gbits/sec, i.e. 800GB/s. Even a quarter of that would not allow a PCIe 5.0x16 GPU to saturate the network/storage interface, you would need four GPUs to get to half and more than four GPUs would have you out of PCIe lanes.
That kind of configuration would cost more per instance than getting some Google Cloud VMs, but those VMs max out at 200Gbits/s, and then how many more of them do you need?
Probably the interesting use case for cloud VMs when you're I/O bound is the one you excluded initially, when you're thoroughly I/O bound and hardly need any compute at all, and then you can get cheap VMs with low core counts and a medium amount of I/O and just use a ton of them distributed across separate physical machines to get high aggregate I/O bandwidth.
The situation you keep trying to find is the one where you're not actually I/O bound. If you're compute or memory bound on the types of computation GPUs are suited to then obviously GPUs will be cheaper.
Sure, everything I said likely goes out the window if we're talking on-prem. No idea how I/O works in that setting, and in fact I am completely unfamiliar with workloads which saturate 800 GB/s network, especially since that approaches the limit of main memory bandwidth. Really appreciate your response! And you're definitely right in noting that the situation I'm talking about is when we're not thoroughly I/O bound.