NPUs, at least for apple, are built for relatively small scale efficient background processing. Theyre taking care of extracting structured metadata from freetext, recognizing people & text in photos, indexing file contents, etc. they do this really well with crazy low power draw.
Theyre generally not even a target for current inference frameworks ala ollama, pytorch, etc.
I did see one prototype someone built using private ios APIs a few months back. They were able to run a reasonable SLM about an order of magnitude slower than GPU but with a tiny power draw.
Comments
NPUs, at least for apple, are built for relatively small scale efficient background processing. Theyre taking care of extracting structured metadata from freetext, recognizing people & text in photos, indexing file contents, etc. they do this really well with crazy low power draw. Theyre generally not even a target for current inference frameworks ala ollama, pytorch, etc.
I did see one prototype someone built using private ios APIs a few months back. They were able to run a reasonable SLM about an order of magnitude slower than GPU but with a tiny power draw.