I recall one series where R. C. Bray had been doing the narration for several books, then for undisclosed reasons they replaced him with another narrator. The drop in quality was so bad I eventually gave up trying to finish the series (though admittedly the author(s) didn't seem to be helping much with the content).

Some narrators, like Wil Wheaton, are so entertaining to me I actively search by what they have voiced.

In general, I have to agree the narrator can make or break a series.