Comment on State-of-the-Art Chatbot, Vicuna-7B, now runs on MacBook with GPU accelerationparentComments−matwood3yThe shared ram and neural engine make for an interesting/powerful platform if people are willing to port to it.−steve_adams_863yAre the neural engines able to be leveraged by 3rd parties yet? I thought there was no API available yet.−rnosov3yThey are leveraging Apple’s Metal Performance Shaders[1] not the neural engine. From the chart, it looks like you might get ~20x max boost on inference over plain CPU. Obviously, it's not like having RTX 4090 but better than nothing.[1] https://pytorch.org/blog/introducing-accelerated-pytorch-tra...−wmf3yCoreML is the API.
Comments
The shared ram and neural engine make for an interesting/powerful platform if people are willing to port to it.
Are the neural engines able to be leveraged by 3rd parties yet? I thought there was no API available yet.
They are leveraging Apple’s Metal Performance Shaders[1] not the neural engine. From the chart, it looks like you might get ~20x max boost on inference over plain CPU. Obviously, it's not like having RTX 4090 but better than nothing.
[1] https://pytorch.org/blog/introducing-accelerated-pytorch-tra...
CoreML is the API.