Comment on Show HN: LLaMA tokenizer that runs in browserparentComments−sp3323yhttps://platform.openai.com/tokenizer or the official python library tiktoken https://github.com/openai/tiktoken or this JS port of tiktoken https://github.com/dqbd/tiktoken−oofsa3yhttps://platform.openai.com/tokenizer is not for GPT-4 but GPT-3. https://tiktokenizer.vercel.app/ supports GPT-4.−adroitboss3yThanks for the recommendations. I found the one I needed because of this comment thread.
Comments
https://platform.openai.com/tokenizer or the official python library tiktoken https://github.com/openai/tiktoken or this JS port of tiktoken https://github.com/dqbd/tiktoken
https://platform.openai.com/tokenizer is not for GPT-4 but GPT-3. https://tiktokenizer.vercel.app/ supports GPT-4.
Thanks for the recommendations. I found the one I needed because of this comment thread.