For one thing, to get a very good quality, lot of resources are needed. Studio quality recordings lasting many hours, voice directors and voice experts who can sift through wav files and ensure phoneme boundaries are aligned etc. And even with this, the quality may not be predictable - but they have gotten reasonably good. It is a hard task to do it at scale (The HMM HTS synthesis used in my app is scalable - but quality is not that great and is robotic).
That means that you can't simply reverse engineer voice from, say, a sample text read by an voice actor? I mean, down to the tiny bits of audio waveform? I mean, how hard could it be? :)
If you can have a speaker read through a specific list of items, a useful singing model can be constructed. That's how Vocaloid works.
What hasn't been done well yet is extracting a model from existing uncontrolled voice samples. That's what this is trying to do. Once this works well, software clones of dead singers will be popular. The RIAA is going to hate this.
That's exactly how unit selection systems like Festival work. The trouble starts when you hit a previously unencountered phoneme sequence and you have to interpolate. Sometimes good, sometimes bad.
Edit: Text to pronunciation is a whole other problem.
Comments
I am wondering, what exacly is holding back the technology? Why isn't it there yet?
For one thing, to get a very good quality, lot of resources are needed. Studio quality recordings lasting many hours, voice directors and voice experts who can sift through wav files and ensure phoneme boundaries are aligned etc. And even with this, the quality may not be predictable - but they have gotten reasonably good. It is a hard task to do it at scale (The HMM HTS synthesis used in my app is scalable - but quality is not that great and is robotic).
That means that you can't simply reverse engineer voice from, say, a sample text read by an voice actor? I mean, down to the tiny bits of audio waveform? I mean, how hard could it be? :)
If you can have a speaker read through a specific list of items, a useful singing model can be constructed. That's how Vocaloid works.
What hasn't been done well yet is extracting a model from existing uncontrolled voice samples. That's what this is trying to do. Once this works well, software clones of dead singers will be popular. The RIAA is going to hate this.
That's exactly how unit selection systems like Festival work. The trouble starts when you hit a previously unencountered phoneme sequence and you have to interpolate. Sometimes good, sometimes bad.
Edit: Text to pronunciation is a whole other problem.
Given a text and waveform, how do you know how to match them up exactly?