One thing I have found with prezto (as well as oh-my-zsh) is that over time, starting up a new terminal window can take a while. On the other hand, when I tried fish, I found certain aspects of it to be sluggish (autocomplete, for example).