We did initially try dedicating ruby processes to particular locales -- when we saw a big improvement we knew we were on the right track. Doing so permanently would be more difficult. To "follow the sun" we would need to shift capacity to Europe and Asia during some hours and back to the US in others.
Parallelizing memcache queries is difficult because we don't know ahead of time what translations will be required to render a page.
We /are/ only working with the most recently used strings. Strings not accessed in the last 4 days are not loaded.
I'm not sure what "denormalize locales" means exactly.
Sparkey is fast because the files end up in the filesystem cache and most of the work is done in C. Going through the dalli gem to grab the translations out of memcache causes a lot of temporary ruby objects to be created.
1. Have a database of rows which are say: translation_id, locale_id, translation_text
2. That is really a 2-D array, translation[translation_id][locale_id] = translation_text
3. Reshuffle to translation[locale_id][translation_id] = translation_text (note the swapping of the indexes)
4. Generate a map of page_url => [arary of tranlation IDs]. You can do this because the number of translation_ids of a given page doesn't change any faster than the translation_ids themselves.
5. Create an in-memory object of all the specific translation_ids for a specific page in a specific locale
For an implementation of 5 I think it'd be preferable to have one giant string that contains all the translation text in it and then a list/array of indices that you use to decide where in that one big string you pull your substring from. That might well be more memory efficient than many individual substrings (this is what they did to make HBase about 5x faster) and it'll definitely give you some cache locality goodness.
Comments
Good points!
We did initially try dedicating ruby processes to particular locales -- when we saw a big improvement we knew we were on the right track. Doing so permanently would be more difficult. To "follow the sun" we would need to shift capacity to Europe and Asia during some hours and back to the US in others.
Parallelizing memcache queries is difficult because we don't know ahead of time what translations will be required to render a page.
We /are/ only working with the most recently used strings. Strings not accessed in the last 4 days are not loaded.
I'm not sure what "denormalize locales" means exactly.
Sparkey is fast because the files end up in the filesystem cache and most of the work is done in C. Going through the dalli gem to grab the translations out of memcache causes a lot of temporary ruby objects to be created.
How I read "denormalize locales" as:
1. Have a database of rows which are say: translation_id, locale_id, translation_text
2. That is really a 2-D array, translation[translation_id][locale_id] = translation_text
3. Reshuffle to translation[locale_id][translation_id] = translation_text (note the swapping of the indexes)
4. Generate a map of page_url => [arary of tranlation IDs]. You can do this because the number of translation_ids of a given page doesn't change any faster than the translation_ids themselves.
5. Create an in-memory object of all the specific translation_ids for a specific page in a specific locale
For an implementation of 5 I think it'd be preferable to have one giant string that contains all the translation text in it and then a list/array of indices that you use to decide where in that one big string you pull your substring from. That might well be more memory efficient than many individual substrings (this is what they did to make HBase about 5x faster) and it'll definitely give you some cache locality goodness.