It's my understanding that the amount of code is so small that they just have it all in ram (and maybe less accessed data goes to disk) and do a distributed grep.
Right, that's more or less what all the approaches I've seen use. It's not indexing in the sense of full-text indexing for fast retrieval, all of the data is more or less greped over. It's basically just a full-text search, not an index.
Comments
> Even more interesting than a simple full-text search would be a regular-expression full-text search of the internet
It would seem to me that indexing data in such a way as to support arbitrary regex would be difficult. Anybody know of an existing system?
Yes, Google code search: http://codesearch.google.com
It's my understanding that the amount of code is so small that they just have it all in ram (and maybe less accessed data goes to disk) and do a distributed grep.
Right, that's more or less what all the approaches I've seen use. It's not indexing in the sense of full-text indexing for fast retrieval, all of the data is more or less greped over. It's basically just a full-text search, not an index.