I find, since 30B models are actually quite usable locally if you have good hardware, that I really want something like a Vicuna 30B. That would be amazing. I can only run 65B locally at a speed of 1 token per second, which is too slow to be usable unfortunately.
Comments
I find, since 30B models are actually quite usable locally if you have good hardware, that I really want something like a Vicuna 30B. That would be amazing. I can only run 65B locally at a speed of 1 token per second, which is too slow to be usable unfortunately.