Infrastructure wise scaling will continue to be possible with CS innovations. Algo wise, I am not sure if we can handle all the additional adversarial content and noise. A lot of index pruning happens just to reduce adversarial content and noise. However, ultimately it all comes down to cost in long run. Cost of crawling and serving extra Y% needs to be equal or lower than the potential drop in revenue in long run. At current stage, it is likely that vast majority of crawlable internet is not actually in index. By some measure, just 50B pages were sufficient to keep most users fairly happy. Going to 150B pages has marginal gain that small players cannot afford. The reachable size of internet is well over 1T pages.
Comments
Infrastructure wise scaling will continue to be possible with CS innovations. Algo wise, I am not sure if we can handle all the additional adversarial content and noise. A lot of index pruning happens just to reduce adversarial content and noise. However, ultimately it all comes down to cost in long run. Cost of crawling and serving extra Y% needs to be equal or lower than the potential drop in revenue in long run. At current stage, it is likely that vast majority of crawlable internet is not actually in index. By some measure, just 50B pages were sufficient to keep most users fairly happy. Going to 150B pages has marginal gain that small players cannot afford. The reachable size of internet is well over 1T pages.