I tried this in my mac mini m2 16GB, unfortunately I have to use an usb disk for the model weights, and I’m getting 0.5 tok/s. Still, being able to run (heh maybe crawl is more accurate) a 100B model on this computer AT ALL is pretty cool.
I see disk maxing out at 400 MB/s, this disk should be able to hit 1GB/s (it hits that eg when verifying the check sum of the weights), so there might be some optimization to be done there (I’m guessing it’s because the weights access is not pure sequential reads but involves some randomness depending on which expert)
Comments
I tried this in my mac mini m2 16GB, unfortunately I have to use an usb disk for the model weights, and I’m getting 0.5 tok/s. Still, being able to run (heh maybe crawl is more accurate) a 100B model on this computer AT ALL is pretty cool.
I see disk maxing out at 400 MB/s, this disk should be able to hit 1GB/s (it hits that eg when verifying the check sum of the weights), so there might be some optimization to be done there (I’m guessing it’s because the weights access is not pure sequential reads but involves some randomness depending on which expert)