"used to" in your sentence already answers your curiosity.
ROCm runs most things just fine without any ceremony or difficulty. I have a couple of the same cards as covered in the article, bought before they got expensive (I'd recommend current Radeon AI Pro 9700 cards over the V620 now, though, as they have increased in price since I bought mine), and they're at the long end of the supported chart for ROCm...nearly EOL. But, they currently work great with llama.cpp and current ROCm 7.14. They're pretty fast and stable. I also have a Strix Halo 128GB, and while the Strix Halo can run bigger models, the dedicated GPUs are quite a bit faster. And, since the best models you can run at home are probably Qwen 3.6 27B and Gemma 4 31B, and those both fit comfortably on dual 32GB GPUs, I find I use the desktop more often than the Strix Halo.
Anyway, there's very little reason to spend 3x or more for the Nvidia ecosystem these days. The only exception is the Strix Halo vs the Nvidia GB10 platform. Strix Halo was a no-brainer when it was half the price, but it's risen in price to be almost the same price as an Asus GX10. In that one instance, I think the Nvidia based Asus is a better choice. The Strix Halo is almost too slow to make use of 128GB. Some MoE models are comfortable, like Laguna S2.1, so it's probable that there will someday be an MoE model that is better then Qwen 3.6 27B or Gemma 4 31B that won't run on 64GB but will run on 128GB. I don't know of one, yet, though.
AMD also used to focus almost entirely on their data center line and not pay any attention to their lower end stuff for more advanced AI features, but that's been changing, and ROCm support has broadened to include almost everything AMD ships now, including the smaller embedded stuff. I think they've realized that as long as the only way to develop AI for AMD was to have a quarter million dollars worth of hardware, they would always fall behind a platform that can be developed for and tested on consumer hardware.
Ah, the price has gone up on the AI Pro 9700, again, I see. So, yeah, it's a worse deal now. But, the V620 is not really a good deal, either, at $450 on eBay. It's roughly half the performance of the newer card. Usable but not blazing. And, you probably don't want to deal with cooling them; if you already have a 3D printer, you can print the shrouds for a few bucks and buy the fans for a few more bucks. But, if you don't have a 3D printer, the shrouds you can buy on eBay are for the tiny fans that have to run at ludicrous speed and have to be so fucking loud it should be illegal. I went through three different shroud and fan combos before finding a set that worked in my case with this cards (about $100 worth of experimentation, probably).
I should say, though, that I actually don't recommend buying anything right now. I wrote up my setup and made recommendations (and the main recommendation was "don't"). https://swelljoe.com/post/how-i-run-local-llms/
The V620 at $450 or the Radeon AI Pro 9700 at ~$1400 are a good deal compared to everything else right now, but buying tokens from DeepSeek is a better deal. DeepSeek V4 Flash 0731 is better than anything you can host locally, they'll serve it to you at blistering fast speeds for pennies a day, and with 1 million token context. You could host a 2-bit quantization of it on a Strix Halo or four of these V620s, and it would run at a crawl on either one. The Strix Halo gets 9-13 t/s. Four V620s would, I guess, get two-three times that. Which is still too slow for comfortable interactive agentic use, and much slower than getting it from DeepSeek, and at two bits there is measurable loss. You're paying a lot more for self-hosted and you're getting worse models.
Also, if you want four cards, you need a server-class motherboard and CPU and RAM. More money.
It's all just a bad investment. Self-hosting is a bad idea if you don't already have the hardware, unless and until memory and GPU prices come down. Even at the prices I paid (before RAMpocalypse really kicked off; $2k for the Strix Halo, ~$350 for each V620), I wouldn't recommend it if you don't have a strong urge to tinker with hardware and it'll probably never pay for itself vs. buying inference from DeepSeek directly.
Edit: The same shrouds for all the old Instinct cards also work for the V620, as they have the same dimensions and screw and cable layout. I tried several and ended up with this one: https://www.thingiverse.com/thing:7296707
I totally agree. I have a very similar setup as OP. I have 4x AMD Instinct Mi50. They are 32GB each with 1TB/s memory bandwidth. I got very lucky and bought them at 220 euro a piece back when prices were sane. Now they cost almost tripple of that. For the server I bought a used X99 xeon that has enough PCIE lanes for all four GPUs, with ASUS X99 WS motherboard. The CPU was only ~22 usd from Aliexpress. The build made some small sense when prices were good. I just did it for fun. Now it makes no sense at all.
I agree with your overall and also use DS4. Economically it makes no sense to run locally (electricity alone would be more expensive than what I pay our brothers in communist faith). But as a matter of fact I do have a tinkering urge and a 3d printer, so I better stay away from this stuff.
For shits and giggles I had Claude build out a full voice cloning pipeline that runs completely locally on my Steam Deck. I’ve got Gemma e4b, Qwen and Chatterbox all running on the AMD.
Uhh.... I have a box with two AMD Radeon AI PRO R9700 32GB.
It was the most trouble-free AI setup that I have done. The driver worked right out of the box with the stock Fedora kernel, ROCM can be installed from the regular repo, and most AI software has ROCM builds by now.
Recently: running Laguna-S with reasonable speed. Before that, I was able to run multiple Qwens or a Qwen and several copies of a smaller model for subagents.
The cards that I have are not loud at all. The cards are capped at 300W each, so that's really not that much heat to move.
Comments
Quite courageous to go with AMD. I am very curious to see Chapter 2 and how he solves the software stability that used to plague AMD AI applications.
"used to" in your sentence already answers your curiosity.
ROCm runs most things just fine without any ceremony or difficulty. I have a couple of the same cards as covered in the article, bought before they got expensive (I'd recommend current Radeon AI Pro 9700 cards over the V620 now, though, as they have increased in price since I bought mine), and they're at the long end of the supported chart for ROCm...nearly EOL. But, they currently work great with llama.cpp and current ROCm 7.14. They're pretty fast and stable. I also have a Strix Halo 128GB, and while the Strix Halo can run bigger models, the dedicated GPUs are quite a bit faster. And, since the best models you can run at home are probably Qwen 3.6 27B and Gemma 4 31B, and those both fit comfortably on dual 32GB GPUs, I find I use the desktop more often than the Strix Halo.
Anyway, there's very little reason to spend 3x or more for the Nvidia ecosystem these days. The only exception is the Strix Halo vs the Nvidia GB10 platform. Strix Halo was a no-brainer when it was half the price, but it's risen in price to be almost the same price as an Asus GX10. In that one instance, I think the Nvidia based Asus is a better choice. The Strix Halo is almost too slow to make use of 128GB. Some MoE models are comfortable, like Laguna S2.1, so it's probable that there will someday be an MoE model that is better then Qwen 3.6 27B or Gemma 4 31B that won't run on 64GB but will run on 128GB. I don't know of one, yet, though.
AMD also used to focus almost entirely on their data center line and not pay any attention to their lower end stuff for more advanced AI features, but that's been changing, and ROCm support has broadened to include almost everything AMD ships now, including the smaller embedded stuff. I think they've realized that as long as the only way to develop AI for AMD was to have a quarter million dollars worth of hardware, they would always fall behind a platform that can be developed for and tested on consumer hardware.
That's about 4x the price of v620, which is 450 eurobucks. Can't imagine buying 4 of those at 4x the price.
Ah, the price has gone up on the AI Pro 9700, again, I see. So, yeah, it's a worse deal now. But, the V620 is not really a good deal, either, at $450 on eBay. It's roughly half the performance of the newer card. Usable but not blazing. And, you probably don't want to deal with cooling them; if you already have a 3D printer, you can print the shrouds for a few bucks and buy the fans for a few more bucks. But, if you don't have a 3D printer, the shrouds you can buy on eBay are for the tiny fans that have to run at ludicrous speed and have to be so fucking loud it should be illegal. I went through three different shroud and fan combos before finding a set that worked in my case with this cards (about $100 worth of experimentation, probably).
I should say, though, that I actually don't recommend buying anything right now. I wrote up my setup and made recommendations (and the main recommendation was "don't"). https://swelljoe.com/post/how-i-run-local-llms/
The V620 at $450 or the Radeon AI Pro 9700 at ~$1400 are a good deal compared to everything else right now, but buying tokens from DeepSeek is a better deal. DeepSeek V4 Flash 0731 is better than anything you can host locally, they'll serve it to you at blistering fast speeds for pennies a day, and with 1 million token context. You could host a 2-bit quantization of it on a Strix Halo or four of these V620s, and it would run at a crawl on either one. The Strix Halo gets 9-13 t/s. Four V620s would, I guess, get two-three times that. Which is still too slow for comfortable interactive agentic use, and much slower than getting it from DeepSeek, and at two bits there is measurable loss. You're paying a lot more for self-hosted and you're getting worse models.
Also, if you want four cards, you need a server-class motherboard and CPU and RAM. More money.
It's all just a bad investment. Self-hosting is a bad idea if you don't already have the hardware, unless and until memory and GPU prices come down. Even at the prices I paid (before RAMpocalypse really kicked off; $2k for the Strix Halo, ~$350 for each V620), I wouldn't recommend it if you don't have a strong urge to tinker with hardware and it'll probably never pay for itself vs. buying inference from DeepSeek directly.
Edit: The same shrouds for all the old Instinct cards also work for the V620, as they have the same dimensions and screw and cable layout. I tried several and ended up with this one: https://www.thingiverse.com/thing:7296707
I totally agree. I have a very similar setup as OP. I have 4x AMD Instinct Mi50. They are 32GB each with 1TB/s memory bandwidth. I got very lucky and bought them at 220 euro a piece back when prices were sane. Now they cost almost tripple of that. For the server I bought a used X99 xeon that has enough PCIE lanes for all four GPUs, with ASUS X99 WS motherboard. The CPU was only ~22 usd from Aliexpress. The build made some small sense when prices were good. I just did it for fun. Now it makes no sense at all.
I agree with your overall and also use DS4. Economically it makes no sense to run locally (electricity alone would be more expensive than what I pay our brothers in communist faith). But as a matter of fact I do have a tinkering urge and a 3d printer, so I better stay away from this stuff.
For shits and giggles I had Claude build out a full voice cloning pipeline that runs completely locally on my Steam Deck. I’ve got Gemma e4b, Qwen and Chatterbox all running on the AMD.
lemon-server is amazing and gets you the right models for your hardware, along with a model router
Add openwebui and/or pi and away you go.
https://lemonade-server.ai/
Uhh.... I have a box with two AMD Radeon AI PRO R9700 32GB.
It was the most trouble-free AI setup that I have done. The driver worked right out of the box with the stock Fedora kernel, ROCM can be installed from the regular repo, and most AI software has ROCM builds by now.
What can you do with "only" 64G of VRAM that a 32G can't? Also, the R9700 are so loud!
Recently: running Laguna-S with reasonable speed. Before that, I was able to run multiple Qwens or a Qwen and several copies of a smaller model for subagents.
The cards that I have are not loud at all. The cards are capped at 300W each, so that's really not that much heat to move.