I'm pretty sure $20,000 per LPU isn't actually the cost of these LPUs. I saw someone else on HN asking if $20,000 could get them something and an employee said to reach out. Which makes me think $20,000 is enough to get some sort of model running at least, even if it's not necessarily an LLM.
Comments
They need 568 LPUs to load both Mixtral 8x7B and LLaMA 70B, because they need both those models available for the demo.
I imagine Mixtral by itself would only take something like 200-300 LPUs
Only $5M then.
I'm pretty sure $20,000 per LPU isn't actually the cost of these LPUs. I saw someone else on HN asking if $20,000 could get them something and an employee said to reach out. Which makes me think $20,000 is enough to get some sort of model running at least, even if it's not necessarily an LLM.
$5M once, upfront. But given the significantly increased throughput, how fast does that pay for itself?
You need computers for all of them and megawatts of power, power supplies, cooling, and power distribution.
Naturally, but you need that for GPUs as well, no? What is the actual difference when running, when measured per token generated?
Depends on power usage. I’m curious how power hungry those are compared to server/workstation cards.