Hey! Author of the blog post here. Yes you should be able to use any model. Your mileage may vary with the smaller models but asking them to “return x in json” tends to help with accuracy (anecdotally).
Do you happen to know if got-4o would be negatively affected by the addition of “return x in json”? I’m debating whether I could use the same prompt across all models, hosted and ollama.
Comments
So I can use this with any supported models? The reason I'm asking is because I can only run 1b-3b models reliably on my hardware.
Hey! Author of the blog post here. Yes you should be able to use any model. Your mileage may vary with the smaller models but asking them to “return x in json” tends to help with accuracy (anecdotally).
Do you happen to know if got-4o would be negatively affected by the addition of “return x in json”? I’m debating whether I could use the same prompt across all models, hosted and ollama.
Why do smaller models fail to follow? Isn't the objective of constraints that it always provides the right output type?
The constraints will always be met. It’s the data inside that might be inaccurate. YMMV with smaller models in that sense.