I'm actually confused by the post - indeed why choosing the 27B dense and the 35B MoE models? Given the structure of the post I honestly have the impression that the Spark was purchased out of curiosity more than for a specific purpose (LLMs).
Ah yeah, your reaction makes sense. In the circles I run in, there is a lot of hype around Sparks for inference, so my gut reaction is to respond with this type of warning.
I did not intend to imply that the post author was advocating that they're great for inference, as they're obviously not.
Comments
I'm actually confused by the post - indeed why choosing the 27B dense and the 35B MoE models? Given the structure of the post I honestly have the impression that the Spark was purchased out of curiosity more than for a specific purpose (LLMs).
The 27b dense is _remarkably_ better at coding tasks.
Especially at larger quanitizations (Q4 is pretty crap).
Ah yeah, your reaction makes sense. In the circles I run in, there is a lot of hype around Sparks for inference, so my gut reaction is to respond with this type of warning.
I did not intend to imply that the post author was advocating that they're great for inference, as they're obviously not.