Comment on Show HN: LLaMA 3 tokenizer runs in the browserparentComments−belladoreaiOP2yIf you need to work with multiple LLMs, you probably want to use transformers.js−mrbishalsaha2yIsn't it to much for just calculating the number of token?−belladoreaiOP2yIt's the best option you have if you need to work with multiple LLMs in the browser.
Comments
If you need to work with multiple LLMs, you probably want to use transformers.js
Isn't it to much for just calculating the number of token?
It's the best option you have if you need to work with multiple LLMs in the browser.