Comment on Understanding, using, and finetuning GemmaparentComments−brucethemoose22yexl2 is Nvidia/AMD only.But GGUF Mixtral should fit in 32GB... just not with the full 32K context. Long context is very memory intense in llama.cpp, at least until they fully implement flash attention and a quantized cache.
Comments
exl2 is Nvidia/AMD only.
But GGUF Mixtral should fit in 32GB... just not with the full 32K context. Long context is very memory intense in llama.cpp, at least until they fully implement flash attention and a quantized cache.