That's already the case. I run a quantized 70 billion parameter llama 3.1 model on my framework 13 inch laptop. Only cost ~$300 to get the 96GB of ram (which I purchased for unrelated non-AI reasons before the AI boom). It certainly isn't fast, but it is fast enough. I run it via vulkan compute using llama.cpp with an anythingllm web interface in front of it.
Comments
That's already the case. I run a quantized 70 billion parameter llama 3.1 model on my framework 13 inch laptop. Only cost ~$300 to get the 96GB of ram (which I purchased for unrelated non-AI reasons before the AI boom). It certainly isn't fast, but it is fast enough. I run it via vulkan compute using llama.cpp with an anythingllm web interface in front of it.