It'd be nice to see a more systematic review comparing the search engines head to head on the same set of queries or tasks. So far the reviews here seem pretty unscientific.
Also, does anyone know if BOSS is now serving Bing rather than Yahoo results after the switchover? It would seem kind of bizarre for Yahoo to continue serving its own results just for this one property (unless they're serving stale results). In any case, gaming at aimed Yahoo or Bing is going to be an issue for anything built on top of them (although DDG of course is free to do its own extra filtering).
A systematic review would be really hard because a fair sample size is probably hundreds if not thousands, and it may not always be easy to define which one did "best".
I think he's stopped using BOSS altogether. Just Bing API, a few other API's that I've never heard of, and his own crawling.
Also, he does a lot of reordering of search results, so things meant to game Bing wouldn't be that effective.
Even if it was just 3 or 4 queries, I think it would still have value over what we've seen so far, and some people might have an idea on how to evaluate a few more paired comparisons: http://blog.crowdflower.com/2009/06/bing-an-improvement-over... . It would also be interesting to see an estimate of how much the ordering for his results differs from Bing's on the query stream hitting his site. (I wonder if Microsoft does this in order to look for places where they might improve.)
Comments
It'd be nice to see a more systematic review comparing the search engines head to head on the same set of queries or tasks. So far the reviews here seem pretty unscientific.
Also, does anyone know if BOSS is now serving Bing rather than Yahoo results after the switchover? It would seem kind of bizarre for Yahoo to continue serving its own results just for this one property (unless they're serving stale results). In any case, gaming at aimed Yahoo or Bing is going to be an issue for anything built on top of them (although DDG of course is free to do its own extra filtering).
A systematic review would be really hard because a fair sample size is probably hundreds if not thousands, and it may not always be easy to define which one did "best".
I think he's stopped using BOSS altogether. Just Bing API, a few other API's that I've never heard of, and his own crawling.
Also, he does a lot of reordering of search results, so things meant to game Bing wouldn't be that effective.
Even if it was just 3 or 4 queries, I think it would still have value over what we've seen so far, and some people might have an idea on how to evaluate a few more paired comparisons: http://blog.crowdflower.com/2009/06/bing-an-improvement-over... . It would also be interesting to see an estimate of how much the ordering for his results differs from Bing's on the query stream hitting his site. (I wonder if Microsoft does this in order to look for places where they might improve.)