There are uses for them but as a sibling comment mentions they're currently extra work to target per vendor. If you squint a bit they're all basically stripped down GPU compute elements with everything except the matrix multiplication hardware removed, a few MB of cache, no RAM of their own, and a nonstandard API.
OpenCL and Vulkan are supposed to gain support eventually but given their extremely constrained instruction sets and the wide variance in capabilities between different hardware models I don't imagine they're the sort of thing you'd want to target manually.
NPUs, at least for apple, are built for relatively small scale efficient background processing. Theyre taking care of extracting structured metadata from freetext, recognizing people & text in photos, indexing file contents, etc. they do this really well with crazy low power draw.
Theyre generally not even a target for current inference frameworks ala ollama, pytorch, etc.
I did see one prototype someone built using private ios APIs a few months back. They were able to run a reasonable SLM about an order of magnitude slower than GPU but with a tiny power draw.
I dont think llama does. There has been some work. They were originally designed for image processing (remove background from calls) and many exist because Microsoft mandated them for "AI laptops". Often they don't have much memory bandwidth, and are limited in compute and they are all incompatible with each other.
Comments
Tangent question: what about NPU performance? I always read about CPU vs GPU, but nobody talks about NPUs; I don't even know if llama supports them.
Why do phones and laptops get equipped with NPUs if there's no use for them?
There are uses for them but as a sibling comment mentions they're currently extra work to target per vendor. If you squint a bit they're all basically stripped down GPU compute elements with everything except the matrix multiplication hardware removed, a few MB of cache, no RAM of their own, and a nonstandard API.
OpenCL and Vulkan are supposed to gain support eventually but given their extremely constrained instruction sets and the wide variance in capabilities between different hardware models I don't imagine they're the sort of thing you'd want to target manually.
NPUs, at least for apple, are built for relatively small scale efficient background processing. Theyre taking care of extracting structured metadata from freetext, recognizing people & text in photos, indexing file contents, etc. they do this really well with crazy low power draw. Theyre generally not even a target for current inference frameworks ala ollama, pytorch, etc.
I did see one prototype someone built using private ios APIs a few months back. They were able to run a reasonable SLM about an order of magnitude slower than GPU but with a tiny power draw.
I dont think llama does. There has been some work. They were originally designed for image processing (remove background from calls) and many exist because Microsoft mandated them for "AI laptops". Often they don't have much memory bandwidth, and are limited in compute and they are all incompatible with each other.