I use a Proxmox server with RTX 3060 to generate paintings (I have a couple of old jailbroken Amazon Kindle's attached to walls for that purpose), and to run ollama, which is connected to Home Assistant & their voice preview device, allowing me to talk with LLM without transmitting anything to cloud services.
Admittedly with that amount of VRAM the models I can run are fairly useless for stuff like controlling lights via Home Assistant, occasionally does what I tell it to do but usually not. It is pretty okay for telling me information, like temperature or value of some sensors I have connected to HA. For generating AI paintings it's enough. My server also hosts tons of virtual machines, docker containers and is used for remote gameplay, so the AI thing is just an extra.
That's good to know. The model I was using supported function calling, but seemed to get the calls often wrong. Perhaps I should try a more fine-tuned model for the purpose.
Comments
I use a Proxmox server with RTX 3060 to generate paintings (I have a couple of old jailbroken Amazon Kindle's attached to walls for that purpose), and to run ollama, which is connected to Home Assistant & their voice preview device, allowing me to talk with LLM without transmitting anything to cloud services.
Admittedly with that amount of VRAM the models I can run are fairly useless for stuff like controlling lights via Home Assistant, occasionally does what I tell it to do but usually not. It is pretty okay for telling me information, like temperature or value of some sensors I have connected to HA. For generating AI paintings it's enough. My server also hosts tons of virtual machines, docker containers and is used for remote gameplay, so the AI thing is just an extra.
Why do you say that? You can easily finetune 8B parameter model for function calling.
That's good to know. The model I was using supported function calling, but seemed to get the calls often wrong. Perhaps I should try a more fine-tuned model for the purpose.