Please note that it is advised to avoid using the Hugging Face fast tokenizer for now, as we’ve observed that the auto-converted fast tokenizer sometimes gives incorrect tokenization
This is great. Based on the throughout of 2200 tokens/sec and the 1,000,000,000,000 tokens used to train this was at least $183k worth of compute (that's based on the three year committed use rate). And now we can have it for free!
Comments
be sure to read the warning in their repo: https://github.com/openlm-research/open_llama#loading-the-we...
This is great. Based on the throughout of 2200 tokens/sec and the 1,000,000,000,000 tokens used to train this was at least $183k worth of compute (that's based on the three year committed use rate). And now we can have it for free!
The price for training their 7B, as stated by MosaicML[0] and Falcon 7B, is roughly the same.
[0] https://twitter.com/MosaicML/status/1660738892306485248