Have you found the output for arbitrary grammars to be satisfactory? My naive assumption has been that these models will produce better JSON than other formats simply by virtue of having seen so much of it.
If you want to get a good result, the grammar should be following the expect output from the prompt, especially if you use a small model. Normally I would manually fine-tune the prompt to output the grammar format first, and then apply the grammar in production.
Grammars don't have to just be JSON, which means you could have it format responses as anything with a formal grammar. XML, HTTP responses, SQL, algebraic notation of math, etc.
Comments
If anyone needs a more powerful constrain outputs, llama.cpp support gbnf:
https://github.com/ggerganov/llama.cpp/blob/master/grammars/...
Thats is exactly what they are using
Have you found the output for arbitrary grammars to be satisfactory? My naive assumption has been that these models will produce better JSON than other formats simply by virtue of having seen so much of it.
If you want to get a good result, the grammar should be following the expect output from the prompt, especially if you use a small model. Normally I would manually fine-tune the prompt to output the grammar format first, and then apply the grammar in production.
Who would downvote this perfectly reasonable question?
edit: Nm
This is amazing, thank you for the link
How is it more powerful?
Grammars don't have to just be JSON, which means you could have it format responses as anything with a formal grammar. XML, HTTP responses, SQL, algebraic notation of math, etc.