Comment on Mesh LLM: distributed AI computing on irohparentComments−wlesieutre2moEquivalent M3 machines no longer for sale from Apple (only up to 96 GB) but can be had on eBay for around $14,000 each−drawnwren2moIt's notable that they're so valuable because they feature 800Gbps of memory bandwidth. About twice what's available on the top end of M5, and exactly what makes llm inference fast.−embedding-shape2mobecause they feature 800Gbps of memory bandwidth. About twice what's available on the top end of M5Ouch, about half of the memory bandwidth of a dedicated GPU though :/ Running LLMs on Apple hardware still doesn't make any sense to me.−khalic2moM5 max has 614GB/s, you mean the m4?−drawnwren2moOh, I was looking at every M5 except for the 40-core M5 Max. They have 460.
Comments
Equivalent M3 machines no longer for sale from Apple (only up to 96 GB) but can be had on eBay for around $14,000 each
It's notable that they're so valuable because they feature 800Gbps of memory bandwidth. About twice what's available on the top end of M5, and exactly what makes llm inference fast.
Ouch, about half of the memory bandwidth of a dedicated GPU though :/ Running LLMs on Apple hardware still doesn't make any sense to me.
M5 max has 614GB/s, you mean the m4?
Oh, I was looking at every M5 except for the 40-core M5 Max. They have 460.