Most of the Pipecat examples we've been working on are focused on speech-to-speech. The examples guide you through how to do that (or you can give the hosted storytelling example a try: https://storytelling-chatbot.fly.dev/)
We should probably update the example in the README to better represent that, thank you!
Your project is amazing and I'm not trying to take away from what you have accomplished.
But..I looked at the code but didn't see any audio-to-audio service or model. Can you link to an example of that?
I don't mean speech to text to LLM to text to speech. I mean speech-to-speech directly, as in the ML model takes audio as input and outputs audio. As they have now in OpenAI.
I am very familiar with the typical multi-model workflow and have implemented it several times.
An audio-to-audio model is definitely a step forward. And I do think that's where things are going to go, generally speaking.
For context relating to real-time voice AI: once you're down below ~800ms things are fast enough to feel naturally responsive for most people and use cases.
The GPT-4o announcement page says they average ~320ms time to first token from an audio prompt. Which is definitely next level and is really, really exciting. You can't get to 800ms with any pipeline that includes GPT-4 Turbo today, so this is a big deal.
It's possible to do ~500ms time to first token by pipelining today's fastest transcription, inference, and tts models. (For example, Deepgram transcription, Groq Llama-3, Deepgram Aura voices.)
Every opening phrase is a platitude like ‘sure let’s do it’. So the OpenAI latency is probably higher, they are just using clever orchestration to generate some filler tokens to make latency lower. Unlikely the initial response at OpenAI is coming from the main model.
I'm familiar with Deepgram, groq, and Eleven Labs. I have recently built something on those and it's really not too bad as far as latency. But OpenAI has shown that audio-to-audio can't be beat.
Comments
This is great but we really need an audio-to-audio model like they demoed in the open source world. Does anyone know of anything like that?
Edit: someone found one: https://news.ycombinator.com/item?id=40346992
Most of the Pipecat examples we've been working on are focused on speech-to-speech. The examples guide you through how to do that (or you can give the hosted storytelling example a try: https://storytelling-chatbot.fly.dev/)
We should probably update the example in the README to better represent that, thank you!
Your project is amazing and I'm not trying to take away from what you have accomplished.
But..I looked at the code but didn't see any audio-to-audio service or model. Can you link to an example of that?
I don't mean speech to text to LLM to text to speech. I mean speech-to-speech directly, as in the ML model takes audio as input and outputs audio. As they have now in OpenAI.
I am very familiar with the typical multi-model workflow and have implemented it several times.
That's absolutely amazing, both visually and technically! Do you share any insights of the development process, perhaps some code?
I just realized this is exactly the example provided in the repo which I haven't run yet! Thanks for adding this!
An audio-to-audio model is definitely a step forward. And I do think that's where things are going to go, generally speaking.
For context relating to real-time voice AI: once you're down below ~800ms things are fast enough to feel naturally responsive for most people and use cases.
The GPT-4o announcement page says they average ~320ms time to first token from an audio prompt. Which is definitely next level and is really, really exciting. You can't get to 800ms with any pipeline that includes GPT-4 Turbo today, so this is a big deal.
It's possible to do ~500ms time to first token by pipelining today's fastest transcription, inference, and tts models. (For example, Deepgram transcription, Groq Llama-3, Deepgram Aura voices.)
Every opening phrase is a platitude like ‘sure let’s do it’. So the OpenAI latency is probably higher, they are just using clever orchestration to generate some filler tokens to make latency lower. Unlikely the initial response at OpenAI is coming from the main model.
I'm familiar with Deepgram, groq, and Eleven Labs. I have recently built something on those and it's really not too bad as far as latency. But OpenAI has shown that audio-to-audio can't be beat.
Gazelle is (or will be) significantly faster than that.