I love using encoder models, and they are generally a better technology for this kind of application. But the price of GPU instances is too damn high.

I won’t lie that I’ve been unreasonably annoyed that I have to use a lot more compute than I need, for no other reason than an LLM API exists and it’s good enough in a relatively small throughput application.