Probably it happens when the real text encoding is different from the declared one (at least that's the problem I encountered when aggregating content from unfiltered wild web).
If it really bothers, you can use chardet [1] to try to detect the real encoding (BeautifulSoup should use it if it's installed). But even this is not 100% foolproof.
Comments
Probably it happens when the real text encoding is different from the declared one (at least that's the problem I encountered when aggregating content from unfiltered wild web).
If it really bothers, you can use chardet [1] to try to detect the real encoding (BeautifulSoup should use it if it's installed). But even this is not 100% foolproof.
[1] http://chardet.feedparser.org/
Yep, ’ means that the pages contains the UTF-8 quote character, but the browser renders the bytestream as if it's a single byte character stream.
The crux is that the basic alphabet is encoded the same. So you only notice it with special characters such as the curly quote and the em-dash.