I really don't get what people are doing wrong here.
I have a 7000 token prompt that generates JSON chugging away in production and at scale I'm seeing ~1 in 4000 generations require a re-generation, and even that could probably be killed with some basic "healing" code.
OSS are prone to outputting garbage in my experience, but OP mentions ChatGPT:
How are you running into issues if you simply prefill the response with ```json and set ``` as your stop token?
Also, are people also just not trying to parse the opening and closing bracket and treating it as broken if there's a preamble? The prefill gets rid of the preamble, but if you're not willing/able to prefill, how hard is getting JSON out of a string?
Comments
I really don't get what people are doing wrong here.
I have a 7000 token prompt that generates JSON chugging away in production and at scale I'm seeing ~1 in 4000 generations require a re-generation, and even that could probably be killed with some basic "healing" code.
OSS are prone to outputting garbage in my experience, but OP mentions ChatGPT:
How are you running into issues if you simply prefill the response with ```json and set ``` as your stop token?
Also, are people also just not trying to parse the opening and closing bracket and treating it as broken if there's a preamble? The prefill gets rid of the preamble, but if you're not willing/able to prefill, how hard is getting JSON out of a string?