I love these efforts to get proper models running on lower cost hardware and I think this is where the next real breakthrough will come from. The more efficient this sort of thing can be done the bigger the chance to democratize this tech, 'good enough' is what you need and as long 'top of the line' gives a competitive edge even if it is at a cost there is a substantial risk of the door closing on general computing at some point in the near future. Keep in mind that there is no guarantee that the pendulum has to swing back, it can swing one way and get stuck, and then you're going to have to beg for crumbs from the haves.
I think there's a very good chance that history will rhyme a bit.
DOS/Windows and PC clones were by no means the best available, but they were cheap, ubiquitous, and versatile compared to alternatives that were either much better at one task but more expensive or better at everything but wildly expensive. They were "good enough" and represented a solid improvement over what many existing computer users had as well as a good entry point for new users. As such they spread like wildfire and became the standard while the expensive alternatives either became hardcore niche or vanished.
I am trying to make it easier to use LLMs on older, cheaper, smaller GPUs. I'm taking a similar approach (move MoE expert weights to disk, avoid wasting VRAM on these). My goal is also to run models that do not fit. My work also suffers from AI documentation issues. Where my approach differs is that instead of running an LLM that doesn't fit slowly, run many agents in parallel sharing the streams of MoE experts weights, to increase throughput. I envision a team of AI agents sharing a pretty-good-at-coding LLM that does not fit to collaborate on a set of related features, being developed in parallel.
I am doing this work initially on a 6-Xeon-cores Linux workstation with an RTX5060-16G to run MoE models larger than that. Then I will be moving this to a server with a lot more cores (Dual 32-cores) and a mix of SAS HD and SSD drives, using older GPUs.
Ultimately, I hope to build some FPGA/MCU "accelerators" that process the expert weights on systems with not enough CPU cores to offload the experts. If I can enable large capable models to run on older hardware, keeping the limited GPU VRAM for context and things that must be in VRAM, I can get useful work out of my old refurbished systems without paying today's RAM and VRAM/GPU prices.
Comments
I love these efforts to get proper models running on lower cost hardware and I think this is where the next real breakthrough will come from. The more efficient this sort of thing can be done the bigger the chance to democratize this tech, 'good enough' is what you need and as long 'top of the line' gives a competitive edge even if it is at a cost there is a substantial risk of the door closing on general computing at some point in the near future. Keep in mind that there is no guarantee that the pendulum has to swing back, it can swing one way and get stuck, and then you're going to have to beg for crumbs from the haves.
I think there's a very good chance that history will rhyme a bit.
DOS/Windows and PC clones were by no means the best available, but they were cheap, ubiquitous, and versatile compared to alternatives that were either much better at one task but more expensive or better at everything but wildly expensive. They were "good enough" and represented a solid improvement over what many existing computer users had as well as a good entry point for new users. As such they spread like wildfire and became the standard while the expensive alternatives either became hardcore niche or vanished.
SUN Apollo SGI
Though to be fair it was Linux more than Windows that killed them. Dos and Windows were competition for DEC and - ironically - IBM.
I am trying to make it easier to use LLMs on older, cheaper, smaller GPUs. I'm taking a similar approach (move MoE expert weights to disk, avoid wasting VRAM on these). My goal is also to run models that do not fit. My work also suffers from AI documentation issues. Where my approach differs is that instead of running an LLM that doesn't fit slowly, run many agents in parallel sharing the streams of MoE experts weights, to increase throughput. I envision a team of AI agents sharing a pretty-good-at-coding LLM that does not fit to collaborate on a set of related features, being developed in parallel.
My work is showing promising results (if you can get past the way the AI tries to describe what I am doing). https://sw-ml-study.github.io/emufpga/index.html
I am doing this work initially on a 6-Xeon-cores Linux workstation with an RTX5060-16G to run MoE models larger than that. Then I will be moving this to a server with a lot more cores (Dual 32-cores) and a mix of SAS HD and SSD drives, using older GPUs.
Ultimately, I hope to build some FPGA/MCU "accelerators" that process the expert weights on systems with not enough CPU cores to offload the experts. If I can enable large capable models to run on older hardware, keeping the limited GPU VRAM for context and things that must be in VRAM, I can get useful work out of my old refurbished systems without paying today's RAM and VRAM/GPU prices.