Comment on Who is building LLM Chatbots, and what issues are you running into?parentComments−petervandijckOP2yImprove probably, but speed also matter, I worry a lot about chaining lots of LLM calls together, each of them takes a bunch of seconds, and then the experience becomes just really slow.−BoorishBears2yYou don't make multiple calls, you ask for a structured output and use some of the keys as the "chain links" in the chain of thought−anoy88882yIs there an example on how to do this ?−BoorishBears2yContrived example, but say I'm summarizing an arbitrary document, I might ask for JSON with the following keys and instructions:- purpose: one sentence on why would someone want to summarize this document- effect: one sentence on how this affects the strategy we should use to summarize it vs a naive approach- summary: one paragraph with the summary of the document based on purpose and effectThe LLM will change its summary based on the keys that come before summary in a meaningful way−quickthrower22yYeah you better be 99.9% perfect if you are going to take 60 seconds of plain spinner time to come back to me.−BoorishBears2yWhich model are you using in production that's so slow you'd be waiting for 10 seconds, let alone 60?
Comments
Improve probably, but speed also matter, I worry a lot about chaining lots of LLM calls together, each of them takes a bunch of seconds, and then the experience becomes just really slow.
You don't make multiple calls, you ask for a structured output and use some of the keys as the "chain links" in the chain of thought
Is there an example on how to do this ?
Contrived example, but say I'm summarizing an arbitrary document, I might ask for JSON with the following keys and instructions:
- purpose: one sentence on why would someone want to summarize this document
- effect: one sentence on how this affects the strategy we should use to summarize it vs a naive approach
- summary: one paragraph with the summary of the document based on purpose and effect
The LLM will change its summary based on the keys that come before summary in a meaningful way
Yeah you better be 99.9% perfect if you are going to take 60 seconds of plain spinner time to come back to me.
Which model are you using in production that's so slow you'd be waiting for 10 seconds, let alone 60?