Definitely accurate. I've spent some time working on a web crawler with a friend, and eventually you just realize that no matter what you do, requesting pages is going to break, hyperlinks lie, content is borked, ect ect ect. After countless hours debugging you just say "FUCK IT, WE'LL JUST GO THE NEXT PAGE". And really, that's what you would have done anyway if you had filtered the page out...
Comments
Definitely accurate. I've spent some time working on a web crawler with a friend, and eventually you just realize that no matter what you do, requesting pages is going to break, hyperlinks lie, content is borked, ect ect ect. After countless hours debugging you just say "FUCK IT, WE'LL JUST GO THE NEXT PAGE". And really, that's what you would have done anyway if you had filtered the page out...