That's exactly how unit selection systems like Festival work. The trouble starts when you hit a previously unencountered phoneme sequence and you have to interpolate. Sometimes good, sometimes bad.

Edit: Text to pronunciation is a whole other problem.