DeepSeek-R1-Distill models are fine-tuned based on open-source models, using samples generated by DeepSeek-R1.
The amount of confusion on the internet because of this seems surprisingly high. DeepSeek R1 has 670B parameters, and it's not easy to run it on local hardware.
There are some ways to run it locally, like https://unsloth.ai/blog/deepseekr1-dynamic which should let you fit the dynamic quant into 160GBs of VRAM, but the quality will suffer.
Agreed - these are Qwen/Llama that have been finetuned against data FROM Deepseek. It kind of annoys me that the names of these models hosted on Ollama start with "DeepSeek-R1-XXX-XXX" since I think it's confusing a lot of people.
The 2 DeepSeek R1 distilled models available through Ollama are actually very low quality Qwen and Llama models frankensteinged with DeepSeek R1. They cannot be used to judge capabilities of the original DeepSeek R1 in any way, shape, or form.
Ollama and LM Studio cause so much confusion because people simply use their pre-packaged models and not exploring and comparing to what else is available on the market (HuggingFace).
Ollama makes it kind of cumbersome to download straight from Huggingface, unless something changed lately. It doesn't help that they felt the need to store files differently on disk either (inspired by docker, seemingly), making it even harder to share stuff between applications.
LM Studio though allows you to browse and download straight from Huggingface (assuming GGUF), so people could spend more time looking for models, but I don't think many have the interest to do what many of us do, download 10s of models and compare them against each other to find the best one for our use case.
For people who don't want to click into X or don't have an account to see the thread: The rig in question uses dual-socket EPYCs and no GPUs, relying on 768 GB of fast DDR5 RAM. It gets about 6 - 8 tokens/sec for the full DeepSeek R1 model.
I haven't played with the distillations extensively, but I think the internal monologue helps the LLM produce a higher quality output, even if its just Qwen or Llama
According to this, you can fit it on a CPU-only setup (no GPUs) with 2 x AMD EPYC CPUs and 24 x 32GB DDR5-RDIMM RAM. About $6000 MSRP for the rig. Doubt you are going to get very many tokens/sec out of it though (6-8, according to the author).
The minimum deployment unit of the decoding stage consists of 40 nodes with 320 GPUs.
but realistically, >=671GB of VRAM to run at full precision on GPU, or >=131G VRAM to run the most heavily quantized version[1], or >=671GB RAM, and a dose of patience to run on CPU.
Comments
These mini models are NOT the DeepSeek R1.
The amount of confusion on the internet because of this seems surprisingly high. DeepSeek R1 has 670B parameters, and it's not easy to run it on local hardware.
There are some ways to run it locally, like https://unsloth.ai/blog/deepseekr1-dynamic which should let you fit the dynamic quant into 160GBs of VRAM, but the quality will suffer.
Also MLX attempt on a cluster of Mac Ultras: https://x.com/awnihannun/status/1881412271236346233
Agreed - these are Qwen/Llama that have been finetuned against data FROM Deepseek. It kind of annoys me that the names of these models hosted on Ollama start with "DeepSeek-R1-XXX-XXX" since I think it's confusing a lot of people.
The 2 DeepSeek R1 distilled models available through Ollama are actually very low quality Qwen and Llama models frankensteinged with DeepSeek R1. They cannot be used to judge capabilities of the original DeepSeek R1 in any way, shape, or form.
Ollama and LM Studio cause so much confusion because people simply use their pre-packaged models and not exploring and comparing to what else is available on the market (HuggingFace).
Ollama makes it kind of cumbersome to download straight from Huggingface, unless something changed lately. It doesn't help that they felt the need to store files differently on disk either (inspired by docker, seemingly), making it even harder to share stuff between applications.
LM Studio though allows you to browse and download straight from Huggingface (assuming GGUF), so people could spend more time looking for models, but I don't think many have the interest to do what many of us do, download 10s of models and compare them against each other to find the best one for our use case.
You are correct, except it's not that hard to run locally. Here, somebody made a 6k rig to do it: https://x.com/carrigmat/status/1884244369907278106
For people who don't want to click into X or don't have an account to see the thread: The rig in question uses dual-socket EPYCs and no GPUs, relying on 768 GB of fast DDR5 RAM. It gets about 6 - 8 tokens/sec for the full DeepSeek R1 model.
That Link is being discussed as well:
https://news.ycombinator.com/item?id=42865575
I haven't played with the distillations extensively, but I think the internal monologue helps the LLM produce a higher quality output, even if its just Qwen or Llama
ollama is making this worse by not denoting what you’re getting in the model name. You have to look at individual model cards to see it’s distilled.
Both ollama and groq are spreading this misinfo for some reason.
So what hardware do I need to run DeepSeek R1 with 670B parameters?
https://www.reddit.com/r/LocalLLaMA/comments/1ic8cjf/6000_co...
According to this, you can fit it on a CPU-only setup (no GPUs) with 2 x AMD EPYC CPUs and 24 x 32GB DDR5-RDIMM RAM. About $6000 MSRP for the rig. Doubt you are going to get very many tokens/sec out of it though (6-8, according to the author).
Pretty impressive TPS numbers for CPU-only
Per the technical report:
but realistically, >=671GB of VRAM to run at full precision on GPU, or >=131G VRAM to run the most heavily quantized version[1], or >=671GB RAM, and a dose of patience to run on CPU.
[1]: https://news.ycombinator.com/item?id=42850222
Dang, that's a bad oversight. It's truly misinformation indeed.