Using the CPU with quantized weights on GPT models makes sense, an example is llama.cpp, that’s because these models are constrained by memory bandwidth and not compute (low arithmetic density)
LLAMA is a LLM, very little to do with a model trying to learn MNIST. CNNs in particular gain from using a GPU (or ten), as they're optimizing weights for convolutional/spatial kernels.
Comments
No, the bottleneck would be not utilizing the idling GPU.
Using the CPU with quantized weights on GPT models makes sense, an example is llama.cpp, that’s because these models are constrained by memory bandwidth and not compute (low arithmetic density)
https://github.com/ggerganov/llama.cpp
LLAMA is a LLM, very little to do with a model trying to learn MNIST. CNNs in particular gain from using a GPU (or ten), as they're optimizing weights for convolutional/spatial kernels.
Pls see the post from version_five nearby: https://news.ycombinator.com/item?id=35699260
You can go 100x faster using SIMD on the CPU, instead of doing the linear algebra by hand, then another order of magnitude or two on the GPU.