Tokenization is a probable issue here, the longer the context the longer the initial processing with llama I think. Possible tokenizer is not optimized.
That’s very unlikely, tokenization is really simple and usually quite fast (scales with input size). Unexpected slowness with a larger context window might point to a non-existent or unoptimized KV cache.
Comments
It appears to generate 30 tokens/s on iPhone 14 pro: https://i.imgur.com/AWTXtGA.png
But for some reason it dramatically slows down after a few messages
Edit:
Oh no, this one also gives lectures instead of answering questions.
https://i.imgur.com/eiuGzK4.jpg
I'm afraid, in near future the only organic content on the internet would be only the type of content that LLMs refuse to generate.
Tokenization is a probable issue here, the longer the context the longer the initial processing with llama I think. Possible tokenizer is not optimized.
That’s very unlikely, tokenization is really simple and usually quite fast (scales with input size). Unexpected slowness with a larger context window might point to a non-existent or unoptimized KV cache.