Maybe the big tech providers should play fair and host the downloadable database for those bots as well as crawlable mirrors.