Some quick questions:

1) How frequently would you crawl?

2) What are you defining as a 'massive scale'? It'll probably grow i'm sure but what is a reasonable range you're working with?

3) Which languages will you be supporting?