It appears from slides 14, 21, and 24, that copying all of the static files that needed be served onto a single machine could take over 12 hours. Another way to say that is that the totality of content that is served from that single host is too large to put on a single machine. The "groupcache" abstraction lets the http-serving machines dynamically serve the hot subsets of the larger file set (represented by "storage svc" on slide 55).
But nginx can work as cache, acting as reverse proxy on misses. Minor amount of Lua scripting or C (or, better, whatever fancies one's taste if that could be called from C) hacking can teach it any reasonably complex semantics (fancy ACLs, not-before dates etc.). Many modules (dynamic archive generation, GeoIP, various cache backends) are already there, well tested in production environments.
Surely, one would probably need a separate external tool that would pre-fill (nginx wouldn't know that some file's pending before it's requested) and clean up caches (provided that rules are complex than trivial LRU removal on some threshold), as one probably wouldn't like webserver doing this unsuitable job, but that should be another story.
Comments
It appears from slides 14, 21, and 24, that copying all of the static files that needed be served onto a single machine could take over 12 hours. Another way to say that is that the totality of content that is served from that single host is too large to put on a single machine. The "groupcache" abstraction lets the http-serving machines dynamically serve the hot subsets of the larger file set (represented by "storage svc" on slide 55).
But nginx can work as cache, acting as reverse proxy on misses. Minor amount of Lua scripting or C (or, better, whatever fancies one's taste if that could be called from C) hacking can teach it any reasonably complex semantics (fancy ACLs, not-before dates etc.). Many modules (dynamic archive generation, GeoIP, various cache backends) are already there, well tested in production environments.
Surely, one would probably need a separate external tool that would pre-fill (nginx wouldn't know that some file's pending before it's requested) and clean up caches (provided that rules are complex than trivial LRU removal on some threshold), as one probably wouldn't like webserver doing this unsuitable job, but that should be another story.