OK, thanks I think I get what thriftDB is a little better. It's hosted (only) -- so instead of booting a solr on your own boxes, you just hit api.thriftdb.com to create indexes, add data and do queries. So thus my "API" confusion -- I was assuming this was software I run on my own box and I don't know what I would get from the API that I wouldn't get hitting solr direct. But a turnkey search service is a nice idea.
But do you guys have a lot of experience with solr / lucene scaling (I mean well beyond the 15m docs octopart has)? What happens when an API customer starts ingesting >100m docs? And then 10 do it at once? Are the indexes on different boxes? Are you on AWS? What's the disk backing it? Does each index fit in RAM? Does the user have any control over caches, tokenizers, stemming, triefields? Like... when I add a date, you're not indexing it with millisecond accuracy, are you?
After EN gets bought on accident by the norwegian fish cannery I plan to retire on giving speeches of 5 years of Solr scaling woes. Unless Otis G is running it there's no way I would trust a hosted solr solution that I didn't have full control over.
Exactly! Would love to get more feedback if you have a chance to try out ThriftDB (http://www.thriftdb.com).
We have a lot of experience scaling faceted search with solr. 15M docs might not sound like a lot if you're doing full-text search but faceted-search adds another level of complexity:
As far as scaling goes, most of the demand right now is for smaller indexes so that's not a problem. We know how to scale out though so we're pretty excited to get customers with >100M docs.
We're hosted on AWS but we're still iterating on the architecture. The indexes fit in RAM. Currently you can't control caches, tokenizers, stemming, etc. but that's in the pipeline.
Good luck with your new Norwegian cannery owners! I hear you on hosted search. We're trying our best to create a headache-free hosted solution.
Comments
OK, thanks I think I get what thriftDB is a little better. It's hosted (only) -- so instead of booting a solr on your own boxes, you just hit api.thriftdb.com to create indexes, add data and do queries. So thus my "API" confusion -- I was assuming this was software I run on my own box and I don't know what I would get from the API that I wouldn't get hitting solr direct. But a turnkey search service is a nice idea.
But do you guys have a lot of experience with solr / lucene scaling (I mean well beyond the 15m docs octopart has)? What happens when an API customer starts ingesting >100m docs? And then 10 do it at once? Are the indexes on different boxes? Are you on AWS? What's the disk backing it? Does each index fit in RAM? Does the user have any control over caches, tokenizers, stemming, triefields? Like... when I add a date, you're not indexing it with millisecond accuracy, are you?
After EN gets bought on accident by the norwegian fish cannery I plan to retire on giving speeches of 5 years of Solr scaling woes. Unless Otis G is running it there's no way I would trust a hosted solr solution that I didn't have full control over.
Exactly! Would love to get more feedback if you have a chance to try out ThriftDB (http://www.thriftdb.com).
We have a lot of experience scaling faceted search with solr. 15M docs might not sound like a lot if you're doing full-text search but faceted-search adds another level of complexity:
http://octopart.com/partsearch/#search/requestData&q=cap...
As far as scaling goes, most of the demand right now is for smaller indexes so that's not a problem. We know how to scale out though so we're pretty excited to get customers with >100M docs.
We're hosted on AWS but we're still iterating on the architecture. The indexes fit in RAM. Currently you can't control caches, tokenizers, stemming, etc. but that's in the pipeline.
Good luck with your new Norwegian cannery owners! I hear you on hosted search. We're trying our best to create a headache-free hosted solution.