I actually loved this part of the page being discussed.
The whole idea of versioning content based on who accesses it is broken and fundamentally at odds with the idea of the open web. Same goes for user-agent string madness, by the way. Yes, we should be able to tell robots from humans, but otherwise, it's supposed to be the Web.
Incidentally, this hits close to a pain point: I find it extremely annoying when publishers (like Elsevier) hide content behind a paywall, but still expose it to Googlebot for indexing. The result is that you are able to find a scientific article, which is not accessible (but Googlebot has cached snippets). This goes against Google's own guidelines (they used to tell people that Googlebot must not see different content from browsers). And it goes against the whole idea of the Web: if you want to hide stuff behind a paywall, do so — but then it is no longer accessible.
Going back to Applebot, I love the fact that they will now follow Googlebot instructions. Hopefully people will stop distinguishing who accesses content.
I remember a big issue with user-agent filtering when Google was still trying to make Google TV work. At the time, loads of TV networks had free, full streaming episodes of shows on their websites as a way to capture some ad revenue that would be lost if people were torrenting or whatever in order to see shows they missed. Google TV was attempting to list those episodes alongside whatever was currently live on TV via your cable/sat/antenna and stuff from other sites like Youtube.
The idea was that instead of having to go to all sorts of places to find content, it would show you what was available at a given time based on what you were looking for.
...and then all of the network sites and the free Hulu stuff got put behind a user agent filter and essentially wiped out a huge part of GTV's reason for existence. The goal was to bring all of the free content into one place but the networks didn't want you watching a free stream in lieu of a cable broadcast. They wanted you to watch cable on your living room TV and only use the free streaming episodes from the computer in your office as a backup.
Same goes for services where web viewing/listening is free but if you try to access it from a mobile web browser, you have to either fool the site or subscribe to some mobile version.
Comments
I actually loved this part of the page being discussed.
The whole idea of versioning content based on who accesses it is broken and fundamentally at odds with the idea of the open web. Same goes for user-agent string madness, by the way. Yes, we should be able to tell robots from humans, but otherwise, it's supposed to be the Web.
Incidentally, this hits close to a pain point: I find it extremely annoying when publishers (like Elsevier) hide content behind a paywall, but still expose it to Googlebot for indexing. The result is that you are able to find a scientific article, which is not accessible (but Googlebot has cached snippets). This goes against Google's own guidelines (they used to tell people that Googlebot must not see different content from browsers). And it goes against the whole idea of the Web: if you want to hide stuff behind a paywall, do so — but then it is no longer accessible.
Going back to Applebot, I love the fact that they will now follow Googlebot instructions. Hopefully people will stop distinguishing who accesses content.
I remember a big issue with user-agent filtering when Google was still trying to make Google TV work. At the time, loads of TV networks had free, full streaming episodes of shows on their websites as a way to capture some ad revenue that would be lost if people were torrenting or whatever in order to see shows they missed. Google TV was attempting to list those episodes alongside whatever was currently live on TV via your cable/sat/antenna and stuff from other sites like Youtube.
The idea was that instead of having to go to all sorts of places to find content, it would show you what was available at a given time based on what you were looking for.
...and then all of the network sites and the free Hulu stuff got put behind a user agent filter and essentially wiped out a huge part of GTV's reason for existence. The goal was to bring all of the free content into one place but the networks didn't want you watching a free stream in lieu of a cable broadcast. They wanted you to watch cable on your living room TV and only use the free streaming episodes from the computer in your office as a backup.
Same goes for services where web viewing/listening is free but if you try to access it from a mobile web browser, you have to either fool the site or subscribe to some mobile version.