what is considered affordable hardware for running large language models locally today?
I’d say “under $20k” is considered affordable. In comparison, a single H100 server is $250k. At least if you want to run decent models (>70B) at bearable speeds (>1t/s).
Your optimal choice today is Mac Studio with 192GB of unified memory (~$7k). But it will be too slow to run something like llama 400B.
What criteria are you using to define “optimal”? And what is your use case (e.g. how large of a model would you like to run)? If you want to maximize the amount of high-bandwidth memory available, Macs are decent. But they are not the best value IMHO.
If you just want to play with 13B parameter models or smaller, an RTX 4060 Ti 16GB is a great option at $450 or less.
If you want the ability to use larger models, RTX 3090s are a pretty good value. They can be had on the secondary market for $700ish, and are quite fast and have 24GB each. For 70B models, you’ll want to use 4-5 bit quantization and have two 3090s. You could probably run larger models on 4 or 6 of them.
Both of these options require a PC to install them into, but are nowhere near the cost of a Mac mini with 192GB of RAM. Yes, the Mac will give you more memory total, but won’t be as fast at inference and costs several multiples of a dual 3090 setup.
I want to play with the best model I can get my hands on. In the nearest future, this will probably be llama 400B. Even that model will probably be dumber than GPT4, and GPT4 feels pretty dumb sometimes.
Comments
what is considered affordable hardware for running large language models locally today?
I’d say “under $20k” is considered affordable. In comparison, a single H100 server is $250k. At least if you want to run decent models (>70B) at bearable speeds (>1t/s).
Your optimal choice today is Mac Studio with 192GB of unified memory (~$7k). But it will be too slow to run something like llama 400B.
What criteria are you using to define “optimal”? And what is your use case (e.g. how large of a model would you like to run)? If you want to maximize the amount of high-bandwidth memory available, Macs are decent. But they are not the best value IMHO.
If you just want to play with 13B parameter models or smaller, an RTX 4060 Ti 16GB is a great option at $450 or less.
If you want the ability to use larger models, RTX 3090s are a pretty good value. They can be had on the secondary market for $700ish, and are quite fast and have 24GB each. For 70B models, you’ll want to use 4-5 bit quantization and have two 3090s. You could probably run larger models on 4 or 6 of them.
Both of these options require a PC to install them into, but are nowhere near the cost of a Mac mini with 192GB of RAM. Yes, the Mac will give you more memory total, but won’t be as fast at inference and costs several multiples of a dual 3090 setup.
I want to play with the best model I can get my hands on. In the nearest future, this will probably be llama 400B. Even that model will probably be dumber than GPT4, and GPT4 feels pretty dumb sometimes.