ChatGPT voice takes the cascaded approach - Whisper to transcribe speech to text, then to GPT, then to TTS. We skip the transcription step.
Latency: OpenAI's implementation is quite slow - 5+ seconds to get a reply - but even optimized solutions are far slower than this approach. Fundamentally, most transcription models do audio -> audio encoder -> text decoder. By skipping the text decoder, we are always going to win on speed, we just do less work.
Emotion: transcription cannot transcribe emotion, only the text, but emotion - eg if the user is sarcastic, angry, happy, etc - plays a big role.
Transcription errors: in a cascaded approach, any mistake in an earlier step propagates to the latter steps. Mistranscription means that the LLM will make a mistake. But the LLM's are really really smart, certainly smarter than the transcription models - they'll get it right more often.
Comments
ChatGPT voice takes the cascaded approach - Whisper to transcribe speech to text, then to GPT, then to TTS. We skip the transcription step.
Latency: OpenAI's implementation is quite slow - 5+ seconds to get a reply - but even optimized solutions are far slower than this approach. Fundamentally, most transcription models do audio -> audio encoder -> text decoder. By skipping the text decoder, we are always going to win on speed, we just do less work.
Emotion: transcription cannot transcribe emotion, only the text, but emotion - eg if the user is sarcastic, angry, happy, etc - plays a big role.
Transcription errors: in a cascaded approach, any mistake in an earlier step propagates to the latter steps. Mistranscription means that the LLM will make a mistake. But the LLM's are really really smart, certainly smarter than the transcription models - they'll get it right more often.