Comment on Show HN: Running LLMs in one line of Python without DockerComments−sbierwagen2yLlama 70B but no Falcon 180B?−bddppq2yAre there some examples(prompts) that Falcon 180B is performing better than Llama 70B?−moffkalast2yHardly anyone can even run a 70B model, let alone 180B. Any anecdata will be extremely rare.−jiayq84OP2yIn theory one can have 640G = 8 * 80G A100s memory and launch it. 180B Falcon with fp16 will be 360G, so there would be enough memory. It's definitely going to be very expensive indeed.−sbierwagen2yLlama.cpp can run quantized Falcon on a top end Mac Studio, which is only five grand: https://twitter.com/ggerganov/status/1699791226780975439If I'm paying a third party a hundred bucks a month, I'd at least want them to be able to match the capacities of consumer hardware.
Comments
Llama 70B but no Falcon 180B?
Are there some examples(prompts) that Falcon 180B is performing better than Llama 70B?
Hardly anyone can even run a 70B model, let alone 180B. Any anecdata will be extremely rare.
In theory one can have 640G = 8 * 80G A100s memory and launch it. 180B Falcon with fp16 will be 360G, so there would be enough memory. It's definitely going to be very expensive indeed.
Llama.cpp can run quantized Falcon on a top end Mac Studio, which is only five grand: https://twitter.com/ggerganov/status/1699791226780975439
If I'm paying a third party a hundred bucks a month, I'd at least want them to be able to match the capacities of consumer hardware.