Comment on Ask HN: Affordable hardware for running local large language models?Comments−wokwokwok2yYou can use a raspberry pi with 8 GB of ram to run a quantised 7B model (eg. https://huggingface.co/TheBloke/Llama-2-7B-Chat-GGUF), or any cheap stick pc.For larger models, or GPU accelerated inference, there is no “cheap” solution.Why do you think everyone is so in love with the 7B models?It’s not because they’re good. They’re just ok, and it’s expensive to run larger models.
Comments
You can use a raspberry pi with 8 GB of ram to run a quantised 7B model (eg. https://huggingface.co/TheBloke/Llama-2-7B-Chat-GGUF), or any cheap stick pc.
For larger models, or GPU accelerated inference, there is no “cheap” solution.
Why do you think everyone is so in love with the 7B models?
It’s not because they’re good. They’re just ok, and it’s expensive to run larger models.