Because ~75% of clicks on search results are the top 5 results, we were less interested in the long tail of results, and focused on the top results from the most popular sites.
We developed this experiment to see if the top 5 results in context of the top sites would give a better result set than the top 15-20 results from any single search engine.
Well unfortunately, they just don't. I too have helped built a limited domain search engine. The site, www.trailbehind.com, crawls about 20 of the best hiking related sites, including wikipedia, everytrail, localhikes, etc.
Unfortunately, the long tail is where all the action is, and my site and your site will both be marginal compared to Google.
One way to think about it is they can always do what you do, but they can also use the context of the link graph that you don't have, so you really cannot ever hope to compare with what they do.
Everything you/I can do with 20 websites, Google could do if they wanted. But it's just not worth it.
Comments
You know you are heading down the wrong track with this when you say "searches Wikipedia, Twitter, and more..."
Please tell me one search engine that doesn't search these. It's almost like you are calling out a weakness in your product.
Google doesn't search Wikipedia and Twitter and more. They search everything, and if you are going to be in the search game, you better follow suit.
Because ~75% of clicks on search results are the top 5 results, we were less interested in the long tail of results, and focused on the top results from the most popular sites.
We developed this experiment to see if the top 5 results in context of the top sites would give a better result set than the top 15-20 results from any single search engine.
Well unfortunately, they just don't. I too have helped built a limited domain search engine. The site, www.trailbehind.com, crawls about 20 of the best hiking related sites, including wikipedia, everytrail, localhikes, etc.
Unfortunately, the long tail is where all the action is, and my site and your site will both be marginal compared to Google.
One way to think about it is they can always do what you do, but they can also use the context of the link graph that you don't have, so you really cannot ever hope to compare with what they do.
Everything you/I can do with 20 websites, Google could do if they wanted. But it's just not worth it.