Comment on Apple/OpenELM: Efficient Open-Source Family Language ModelsparentComments−SushiHippie2y* ~3GB with 8bit quantization. Without quantization it is ~6GB [0].8 bits = 1 byte3 billion * 1 byte = 3 gigabyte+ Some memory for the context of the LLM[0]3b-instruct has a total file size of 4.94GB + 1.13GB which is 6.07GB which can be seen here:https://huggingface.co/apple/OpenELM-3B-Instruct/tree/mainA bit of overhead will always be there, as you probably want to store some metadata next to the raw weights.
Comments
* ~3GB with 8bit quantization. Without quantization it is ~6GB [0].
8 bits = 1 byte
3 billion * 1 byte = 3 gigabyte
+ Some memory for the context of the LLM
[0]
3b-instruct has a total file size of 4.94GB + 1.13GB which is 6.07GB which can be seen here:
https://huggingface.co/apple/OpenELM-3B-Instruct/tree/main
A bit of overhead will always be there, as you probably want to store some metadata next to the raw weights.