I wasn't aware that yelp provided a dataset, that's very interesting!

Since I had this very problem as I was working on using the output from sentiment analysis to modify sentences so to invert the sentiment polarity (positive to negative, negative to positive), the datasets I found were never general enough (movie reviews, many domain specific terms, hard in the text generation step), or had a lot of noise (twitter dataset).

Though evaluating the system was very hard, due to the reasons you stated, inter-annotator agreement was beyond terrible.

I'll have to look into if other review services expose their data as well, seems appropriate.