Comment on Structured Outputs with OllamaparentComments−mmoskal1yIf you want avoid startup cost, llguidance [0] has no compilation phase and by far the fullest JSON support [1] of any library. I did a PoC llama.cpp integration [2] though our focus is mostly server-side [3].[0] https://github.com/guidance-ai/llguidance [1] https://github.com/guidance-ai/llguidance/blob/main/parser/s... [2] https://github.com/ggerganov/llama.cpp/pull/10224 [3] https://github.com/guidance-ai/llgtrt−HanClinto1yI have been thinking about your PR regularly, and pondering about how we should go about getting this merged in.I really want to see support for additional grammar engines merged into llama.cpp, and I'm a big fan of the work you did on this.−parthsareen1yThis looks really useful. Thank you!
Comments
If you want avoid startup cost, llguidance [0] has no compilation phase and by far the fullest JSON support [1] of any library. I did a PoC llama.cpp integration [2] though our focus is mostly server-side [3].
[0] https://github.com/guidance-ai/llguidance [1] https://github.com/guidance-ai/llguidance/blob/main/parser/s... [2] https://github.com/ggerganov/llama.cpp/pull/10224 [3] https://github.com/guidance-ai/llgtrt
I have been thinking about your PR regularly, and pondering about how we should go about getting this merged in.
I really want to see support for additional grammar engines merged into llama.cpp, and I'm a big fan of the work you did on this.
This looks really useful. Thank you!