Comment on Count tokens used by GPT-4 and Llama for large texts (> 50k characters)Comments−xenovaOP3yThis web-app fixes the two main problems of OpenAI's tokenizer playground: (1) being capped at 50k characters, and (2) not supporting GPT-4/GPT-3.5 tokenizers.Everything runs in-browser thanks to Transformers.js.
Comments
This web-app fixes the two main problems of OpenAI's tokenizer playground: (1) being capped at 50k characters, and (2) not supporting GPT-4/GPT-3.5 tokenizers.
Everything runs in-browser thanks to Transformers.js.