Comment on Releasing 3B and 7B RedPajamaparentComments−Joeri3yThe critical factor seems to be the ability to fit the whole model in RAM (--mlock option in oobabooga). With Apple's RAM prices most M1/M2 owners probably don't have the 32 GB RAM required to fit a 4bit 30B model.−Semaphor3yI have 64 GB RAM, but only a Ryzen 5 3600, and the larger models are very slow ;)
Comments
The critical factor seems to be the ability to fit the whole model in RAM (--mlock option in oobabooga). With Apple's RAM prices most M1/M2 owners probably don't have the 32 GB RAM required to fit a 4bit 30B model.
I have 64 GB RAM, but only a Ryzen 5 3600, and the larger models are very slow ;)