Thanks! We've got a fairly mature spidering architecture, so we're just using that to scrape the posts at scale. It's possible Google will block us, but we're pretty careful to rate limit and load balance across IPs. Plus, you know Google's business is built on scraping :) They don't seem to mind other apps that do it, so I figure we're moderately safe. Once there's an API, we'll use that.
We check every 5 mins if you post frequently, progressively slowing out at around once every 20 mins currently. If traffic builds up we'll slow it down to max at around 1 hour.
Comments
Thanks! We've got a fairly mature spidering architecture, so we're just using that to scrape the posts at scale. It's possible Google will block us, but we're pretty careful to rate limit and load balance across IPs. Plus, you know Google's business is built on scraping :) They don't seem to mind other apps that do it, so I figure we're moderately safe. Once there's an API, we'll use that.
We check every 5 mins if you post frequently, progressively slowing out at around once every 20 mins currently. If traffic builds up we'll slow it down to max at around 1 hour.
Nice job. Looking forward to the API-enabled version.