Hey, I'm Angel, one of the people behind HolaClaw. We recently had to optimize different prompts for local inference and understanding how much tokens the system and user prompts consume were quite important.
I found several token counters around, but most of them were using estimations. In this tool, all the calculations are real. The website loads the public tokenizer.json files from those models and encode them to count. Everything happens in your browser.
For models / families that don't publish their tokenizer, we had to do an estimation. If you are curious, it worked by compiling the tokenizers library from HuggingFace to Wasm (they already have a demo).
Hope you like it! We plan to add some extra visualizations and stats. Also, feel free to suggest any missing model.
Comments
Hey, I'm Angel, one of the people behind HolaClaw. We recently had to optimize different prompts for local inference and understanding how much tokens the system and user prompts consume were quite important.
I found several token counters around, but most of them were using estimations. In this tool, all the calculations are real. The website loads the public tokenizer.json files from those models and encode them to count. Everything happens in your browser.
For models / families that don't publish their tokenizer, we had to do an estimation. If you are curious, it worked by compiling the tokenizers library from HuggingFace to Wasm (they already have a demo).
Hope you like it! We plan to add some extra visualizations and stats. Also, feel free to suggest any missing model.