Interesting. Just a couple of hours ago I came across MetaVoice-1B [0] (Demo [1]) and was amazed by the quality of their TTS in English (sadly no other languages available).
If this year becomes the year when high quality Open Source TTS and ASR models appear that can run in real-time on an Nvidia RTX 40x0 or 30x0, then that would be great. On CPU even better.
Also note the Ethical Statement on BASE TTS:
An application of this model can be to create synthetic voices of people who have lost the ability to speak due to accidents or illnesses, subject to informed consent and rigorous data privacy reviews. However, due to the potential misuse of this capability, we have decided against open-sourcing this model as a precautionary measure.
Metavoice is one of a dozen GPT-based TTS systems around starting from Tortoise. And not that great honestly. You can clearly hear "glass scratches" in their sound, it is because they trained on MP3-compressed data.
There are much more clear sounding systems around. You can listen for StyleTTS2 to compare.
Is the crispness of compressed audio really the benchmark of TTS improvements? I feel like that's an aside. A valid point, but not much of a detractor..
I had forgotten about StyleTTS2, and it was discussed here on HN a couple of months ago. Maybe that's what made me feel that there's something going on.
I am running the smaller models in near real-time on a 3rd gen i7, with good results even using my terrible built-in laptop mic from a distance. The medium and large models are impressively accurate for technical language.
I'm using Whisper to transcribe notes I record with a lavalier mic during my bike rides (wind is no problem), but am using OpenAI's service. When it was released I tested it on a Ryzen 5950x and it was too slow and memory hungry for my taste. Using large was necessary for that use case (also, I'm recording in German).
xtts2 with deepspeed and whisper +
Ctranslate2 with or without distil-whisper weights already run at many multiples of realtime on GPU.
For the top-top end Whisper Large with distil-whisper and TensorRT-LLM hits at least 50x realtime on an RTX 4090.
Note that my application only uses very short speech segments. Longer speech segments increase the realtime multiple SIGNIFICANTLY (as in hitting 150x realtime) due to batching, etc.
There’s also Nvidia Canary which is smaller, faster, and more accurate. It’s pretty new and the ecosystem around it is more or less nonexistent but it’s increasingly well supported in Nvidia world at least.
Comments
Interesting. Just a couple of hours ago I came across MetaVoice-1B [0] (Demo [1]) and was amazed by the quality of their TTS in English (sadly no other languages available).
If this year becomes the year when high quality Open Source TTS and ASR models appear that can run in real-time on an Nvidia RTX 40x0 or 30x0, then that would be great. On CPU even better.
Also note the Ethical Statement on BASE TTS:
[0] https://github.com/metavoiceio/metavoice-src
[1] https://ttsdemo.themetavoice.xyz/
Metavoice is one of a dozen GPT-based TTS systems around starting from Tortoise. And not that great honestly. You can clearly hear "glass scratches" in their sound, it is because they trained on MP3-compressed data.
There are much more clear sounding systems around. You can listen for StyleTTS2 to compare.
Is the crispness of compressed audio really the benchmark of TTS improvements? I feel like that's an aside. A valid point, but not much of a detractor..
Yes, it is one of the important aspects. In particular if you use TTS to create an audiobook or in a video production.
Especially as any finished product may end up being compressed again. Lossy to lossy audio transcodes ALWAYS cause additional audio data to be lost.
I had forgotten about StyleTTS2, and it was discussed here on HN a couple of months ago. Maybe that's what made me feel that there's something going on.
I've tested both. StyleTTS2 is impressive, especially its speed, but the prosody is lacking, compared to Metavoice.
Is it possible to run Metavoice and other pytorch systems on Apple silicon EG the M1? I keep getting issues.
Check out `whisper` and `whisper-cpp` for ASR.
I am running the smaller models in near real-time on a 3rd gen i7, with good results even using my terrible built-in laptop mic from a distance. The medium and large models are impressively accurate for technical language.
I'm using Whisper to transcribe notes I record with a lavalier mic during my bike rides (wind is no problem), but am using OpenAI's service. When it was released I tested it on a Ryzen 5950x and it was too slow and memory hungry for my taste. Using large was necessary for that use case (also, I'm recording in German).
The original release was full precision model weights running in an old version of PyTorch with no optimizations.
Fast forward to now and you have faster-whisper (using Ctranslate2) and distil-whisper optimized weights.
Between the two of them Whisper Large uses something like 1/8th the memory and is likely at least an order of magnitude faster on your hardware.
German has no effect on these metrics and for accuracy it actually has a lower word error rate than English.
With Whisper, you can find many smaller models that are fine-tuned for a particular language, so even smaller models can perform adequately.
Whisper is for STT though right?
The term STT is not used, it's called ASR, Automatic Speech Recognition. I mean, I was referring to both TTS and ASR in my comment.
Not used by who? It’s a better term. Let’s use it.
I also use STT but the parent poster wrote ASR so for clarity I responded in kind.
xtts2 with deepspeed and whisper + Ctranslate2 with or without distil-whisper weights already run at many multiples of realtime on GPU.
For the top-top end Whisper Large with distil-whisper and TensorRT-LLM hits at least 50x realtime on an RTX 4090.
Note that my application only uses very short speech segments. Longer speech segments increase the realtime multiple SIGNIFICANTLY (as in hitting 150x realtime) due to batching, etc.
There’s also Nvidia Canary which is smaller, faster, and more accurate. It’s pretty new and the ecosystem around it is more or less nonexistent but it’s increasingly well supported in Nvidia world at least.