Plug it in, get the cuda drivers, run ollama or equivalent, and go from there. I use an 4GB Quadro card from the same generation and depending on the model can get faster than I can read output. It’s not as good as commercial models, but smaller Qwens and Gemmas work well. The Quadro came with a Dell refurb that cost about as much as the card would have by itself at the time.
(I’m doing this with passthrough on a FreeBSD host with bhyve to a Linux guest. Works great.)
I’d like a beefier card, but until prices come down, even older cards like this can do some interesting things if you give them enough time.
Comments
My 1080ti is my local inference machine now. With the release of the Bonsai 27B models I can run a genuine dense model with usable context. GOAT card.
What exact runtime are you running Bonsai 27B with right now and what variants of the weights are you using?
I have a 1080ti currently not plugged in. More info please.
Plug it in, get the cuda drivers, run ollama or equivalent, and go from there. I use an 4GB Quadro card from the same generation and depending on the model can get faster than I can read output. It’s not as good as commercial models, but smaller Qwens and Gemmas work well. The Quadro came with a Dell refurb that cost about as much as the card would have by itself at the time.
(I’m doing this with passthrough on a FreeBSD host with bhyve to a Linux guest. Works great.)
I’d like a beefier card, but until prices come down, even older cards like this can do some interesting things if you give them enough time.