This implies that providers are either severely limiting their caches, or expiring in a shorter than posted TTL. Even though pinkbike looks like it has thousands of users, one would expect the front pages to be largely identical for most user sessions, so the ISP dns caches should already have most of those username.domain.com records cached. Either that or the ISP's DNS servers are more numerous and distributed, with fewer customers each, or something like that.
Anybody at an ISP that can fill us in on DNS TTL mangling or cache limiting?
When I worked on Akamai's DNS server, I was told that most ISPs cached much more aggressively than we told them to. We would set TTLs at...I want to say 5 seconds...because we were directing traffic based on real-time network conditions; but most ISPs would cache for at least an hour.
Exact numbers may be off, of course; it was a few years ago.
username subdomains are a very popular feature with many websites, especially older ones. users like it too. there is nothing abusive about it. He cant scrap several years worth of links and throw everyone at www.example.com/username
Comments
This implies that providers are either severely limiting their caches, or expiring in a shorter than posted TTL. Even though pinkbike looks like it has thousands of users, one would expect the front pages to be largely identical for most user sessions, so the ISP dns caches should already have most of those username.domain.com records cached. Either that or the ISP's DNS servers are more numerous and distributed, with fewer customers each, or something like that.
Anybody at an ISP that can fill us in on DNS TTL mangling or cache limiting?
When I worked on Akamai's DNS server, I was told that most ISPs cached much more aggressively than we told them to. We would set TTLs at...I want to say 5 seconds...because we were directing traffic based on real-time network conditions; but most ISPs would cache for at least an hour.
Exact numbers may be off, of course; it was a few years ago.
See my reply here: http://news.ycombinator.com/item?id=2306788
He is creating thousands of unique subdomains, so nobody can cache anything.
username subdomains are a very popular feature with many websites, especially older ones. users like it too. there is nothing abusive about it. He cant scrap several years worth of links and throw everyone at www.example.com/username
And? Someone asked why the DNS wasn't caching and I explained.
Why can't this be handled with DNS wildcards? Surely they would allow caching on the nearest server.