Comment on Apple/OpenELM: Efficient Open-Source Family Language ModelsComments−sunflowerfly2yAny idea how much ram this requires?−vineyardmike2yThe model sizes are:270M, 450M, 1.1B and 3B parametersWhich roughly translates to 3GB for the highest end one, depending on context length used.−SushiHippie2y* ~3GB with 8bit quantization. Without quantization it is ~6GB [0].8 bits = 1 byte3 billion * 1 byte = 3 gigabyte+ Some memory for the context of the LLM[0]3b-instruct has a total file size of 4.94GB + 1.13GB which is 6.07GB which can be seen here:https://huggingface.co/apple/OpenELM-3B-Instruct/tree/mainA bit of overhead will always be there, as you probably want to store some metadata next to the raw weights.
Comments
Any idea how much ram this requires?
The model sizes are:
Which roughly translates to 3GB for the highest end one, depending on context length used.
* ~3GB with 8bit quantization. Without quantization it is ~6GB [0].
8 bits = 1 byte
3 billion * 1 byte = 3 gigabyte
+ Some memory for the context of the LLM
[0]
3b-instruct has a total file size of 4.94GB + 1.13GB which is 6.07GB which can be seen here:
https://huggingface.co/apple/OpenELM-3B-Instruct/tree/main
A bit of overhead will always be there, as you probably want to store some metadata next to the raw weights.