I've found that for the most part the articles that I want summarized are those which only fit the largest context models such as Claude. Because otherwise I can skim-read the article possibly in reader mode for legibility.
Is llama 2 a good fit considering its small context window?
Personally I use llama3.1:8b or mistral-nemo:latest which have a decent contex window (even if it is less than the commercial ones usually). I am working on a token calculator / division of the content method too but is very early
I applied (for now) a pre-filled table with a 4096 default limit. Users can also specify an upper or lower limit from the UI directly now. Added chunk and recursive summarization too.
Comments
I've found that for the most part the articles that I want summarized are those which only fit the largest context models such as Claude. Because otherwise I can skim-read the article possibly in reader mode for legibility.
Is llama 2 a good fit considering its small context window?
Personally I use llama3.1:8b or mistral-nemo:latest which have a decent contex window (even if it is less than the commercial ones usually). I am working on a token calculator / division of the content method too but is very early
why not llama3.2:3B? it has fairly large context window too
I assume because the 8B model is smarter than the 3B model; it outperforms it on almost every benchmark: https://huggingface.co/meta-llama/Llama-3.2-3B
If you have the compute, might as well use the better model :)
The 3.2 series wasn't the kind of leap that 3.0 -> 3.1 was in terms of intelligence; it was just:
1. Meta releasing multimodal vision models for the first time (11B and 90B), and
2. Meta releasing much smaller models than the 3.1 series (1B and 3B).
I don't think this is intended for Llama 2? The Llama 3.1 and 3.2 series have very long context windows (128k tokens).
What about using a Modelfile for ollama that tweaks the context window size? I seem to remember parameters for that in the ollama GitHub docs.
I applied (for now) a pre-filled table with a 4096 default limit. Users can also specify an upper or lower limit from the UI directly now. Added chunk and recursive summarization too.
do multi stage summarization?
Hi! This was a good suggestion! I implemented it in v 1.1 which is already out :)