Comment on Show HN: Running LLMs in one line of Python without DockerparentComments−jiayq84OP2yIn theory one can have 640G = 8 * 80G A100s memory and launch it. 180B Falcon with fp16 will be 360G, so there would be enough memory. It's definitely going to be very expensive indeed.−sbierwagen2yLlama.cpp can run quantized Falcon on a top end Mac Studio, which is only five grand: https://twitter.com/ggerganov/status/1699791226780975439If I'm paying a third party a hundred bucks a month, I'd at least want them to be able to match the capacities of consumer hardware.
Comments
In theory one can have 640G = 8 * 80G A100s memory and launch it. 180B Falcon with fp16 will be 360G, so there would be enough memory. It's definitely going to be very expensive indeed.
Llama.cpp can run quantized Falcon on a top end Mac Studio, which is only five grand: https://twitter.com/ggerganov/status/1699791226780975439
If I'm paying a third party a hundred bucks a month, I'd at least want them to be able to match the capacities of consumer hardware.