Comment on Show HN: LLaMA 3 tokenizer runs in the browserComments−mrbishalsaha2yReally good. I am actually using js-tiktokken and wish there was a package to handle all the other LLMs also but still something I can work with.−belladoreaiOP2yIf you need to work with multiple LLMs, you probably want to use transformers.js−mrbishalsaha2yIsn't it to much for just calculating the number of token?−belladoreaiOP2yIt's the best option you have if you need to work with multiple LLMs in the browser.
Comments
Really good. I am actually using js-tiktokken and wish there was a package to handle all the other LLMs also but still something I can work with.
If you need to work with multiple LLMs, you probably want to use transformers.js
Isn't it to much for just calculating the number of token?
It's the best option you have if you need to work with multiple LLMs in the browser.