Ah yea after watching one of the creators youtube videos I realize these benchmarks are combining prefill and decode which isn't super helpful - it seems this struggles with the exact same bottlenecks as all strix halo setups, memory bandwidth. It seems this is still significantly slower than equivalent memory sizing on Mac hardware.
The apple silicon chips basically beat everything in bandwidth. Highest amount of memory controllers (i.e. channels) for a given capacity. That's the main party trick.
Comments
Ah yea after watching one of the creators youtube videos I realize these benchmarks are combining prefill and decode which isn't super helpful - it seems this struggles with the exact same bottlenecks as all strix halo setups, memory bandwidth. It seems this is still significantly slower than equivalent memory sizing on Mac hardware.
How are the memory bandwidths specs of Macbooks vs this?
The apple silicon chips basically beat everything in bandwidth. Highest amount of memory controllers (i.e. channels) for a given capacity. That's the main party trick.
I looked it up: 512 GB/s for the two node AMD cluster, Macbook Pro with M5 CPU has 153 GB/s. But you can get faster Macs with M5 Pro or M5 Max.
Strix halo I believe is 256GB/s max memory bandwidth and M5 Max is 614GB/s - M3 Ultra is up to 800gb/s