That's incredible. Sure, not practical for most applications, but if you really want a local top tier model, you can run it on anything as long as you are patient.
As someone with a healthy amount of RAM, but just a 16GB GPU, I am wondering what kind of work I could queue up for overnight runs. I thought the best models were fully out of reach, but the 128GB CPU only test had a 1.8 tokens/second. While not speedy, you could probably do something with that given extensive coffee breaks. This speed simulator[0] demos what it looks like.
Comments
That's incredible. Sure, not practical for most applications, but if you really want a local top tier model, you can run it on anything as long as you are patient.
As someone with a healthy amount of RAM, but just a 16GB GPU, I am wondering what kind of work I could queue up for overnight runs. I thought the best models were fully out of reach, but the 128GB CPU only test had a 1.8 tokens/second. While not speedy, you could probably do something with that given extensive coffee breaks. This speed simulator[0] demos what it looks like.
[0] https://shir-man.com/tokens-per-second/?speed=1.8
I used to run Povray as a kid by setting up a scene in the evening and then letting it render all night and admiring the output in the morning.
I could see where you could set up some coding task and let it churn all night.