Still rocking a 3090 so can't speak from experience but general vibe around simple at home inference seems like it has improved (esp since both vulkan and rocm are now viable paths on newer cards).
development using pytorch
Would probably still play it nvidia safe for more adventurous stuff than token generation even if it has improved
Comments
Still rocking a 3090 so can't speak from experience but general vibe around simple at home inference seems like it has improved (esp since both vulkan and rocm are now viable paths on newer cards).
Would probably still play it nvidia safe for more adventurous stuff than token generation even if it has improved