Comment on Building an AI server on a budgetComments−teleforce1yDECISION: Nvidia RTX 4070I'm curiuos why OP didn't go for the more recent Nvidia RTX 4060 Ti with 16 GB VRAM that cost cheaper (~USD500) brand new and lesser power consumption at 165W [1].[1] RTX 5060 Ti 16GB sucks for gaming, but seems like a diamond in the rough for AI:https://news.ycombinator.com/item?id=44196991−qingcharles1yAnd if you're gonna be fine with 12GB, why not a 2080ti instead?−Fluorescence1yOnly 11GB... but I guess it will allow you to not do anything useful just as well as 12GB will :)You can however solder on double-capacity memory chips to get 22GB:https://forums.overclockers.com.au/threads/double-your-gpu-m...I hoped the article would be more along these lines than calling an unremarkable second-hand last-gen gaming pc an "AI Server".−iamnotagenius1y4060ti has abysmal bandwidth 288 Gb/sec which is a no go for llms.
Comments
I'm curiuos why OP didn't go for the more recent Nvidia RTX 4060 Ti with 16 GB VRAM that cost cheaper (~USD500) brand new and lesser power consumption at 165W [1].
[1] RTX 5060 Ti 16GB sucks for gaming, but seems like a diamond in the rough for AI:
https://news.ycombinator.com/item?id=44196991
And if you're gonna be fine with 12GB, why not a 2080ti instead?
Only 11GB... but I guess it will allow you to not do anything useful just as well as 12GB will :)
You can however solder on double-capacity memory chips to get 22GB:
https://forums.overclockers.com.au/threads/double-your-gpu-m...
I hoped the article would be more along these lines than calling an unremarkable second-hand last-gen gaming pc an "AI Server".
4060ti has abysmal bandwidth 288 Gb/sec which is a no go for llms.