Does IE7 matter any more, or IE9? Do people with those browsers almost expect sites to look wrong?

It depends what sort of sites they use the most, and just the sheer number of sites on the Internet, most of which have been around for a long time, means that only the newest ones tend to be incompatible (and as a data point, HN is usable with IE6.)