I wondered the same thing. Why not handling the connexion and serving the files through a regular event-based server like nginx, and handle the specific business logic in java or any other popular language inside Google.
I guess that some details on the slides would answer those questions, but if anybody here know the answer (some slides were pretty obscure if you're not already familiar with file serving and/or google architecture).
You have two options here, assuming you ignore option 0:
0) copy all the files from the DFS to the local storage.
1) Attempt to make a distributed filesystem available as a mount point (for example, Hadoop FUSE) on the nginx server so it can serve the data.
2) Make the nginx server know the distributed filesystem API and talk to it via user space directly.
The former leads to insanity (writing a really good, fast FUSE implementation is devlishly hard). The latter can work; I've seen a number of open source codes adapted to distributed filesystems.
This is made easiest if the underlying includes an IO abstraction layer. From a quick skim, it seems nginx has a OS abstraction layer (win32 and unix implementations):, which is more than sufficient.: http://trac.nginx.org/nginx/browser/nginx/src/os/{win32,unix.... From that, it wouldn't be hard to write a "distributed FS variant" (although IO and OS are not orthogonal concepts).
The next problem you're going to have is performance. nginx and other systems really are written with the expectation that IO is local, high throughput, and low-latency. Opening a remote file often takes some time (hundreds of milliseconds), and serial roundtrips cause additional latency, especially if DFS and server are not within 5ms RTT.
The next logical step is to write a readahead layer (if you're serving files larger than a single read block to the DFS) to get better throughput.
A combination of the above techniques, with a bit of tuning and elbow grease, is a good foundation for scalable (in terms of file size, total # of files, etC) serving.
Comments
I wondered the same thing. Why not handling the connexion and serving the files through a regular event-based server like nginx, and handle the specific business logic in java or any other popular language inside Google.
I guess that some details on the slides would answer those questions, but if anybody here know the answer (some slides were pretty obscure if you're not already familiar with file serving and/or google architecture).
nginx cannot read from Google's fancy-pants distributed filesystems, for one thing among many.
You have two options here, assuming you ignore option 0:
0) copy all the files from the DFS to the local storage.
1) Attempt to make a distributed filesystem available as a mount point (for example, Hadoop FUSE) on the nginx server so it can serve the data.
2) Make the nginx server know the distributed filesystem API and talk to it via user space directly.
The former leads to insanity (writing a really good, fast FUSE implementation is devlishly hard). The latter can work; I've seen a number of open source codes adapted to distributed filesystems.
This is made easiest if the underlying includes an IO abstraction layer. From a quick skim, it seems nginx has a OS abstraction layer (win32 and unix implementations):, which is more than sufficient.: http://trac.nginx.org/nginx/browser/nginx/src/os/{win32,unix.... From that, it wouldn't be hard to write a "distributed FS variant" (although IO and OS are not orthogonal concepts).
The next problem you're going to have is performance. nginx and other systems really are written with the expectation that IO is local, high throughput, and low-latency. Opening a remote file often takes some time (hundreds of milliseconds), and serial roundtrips cause additional latency, especially if DFS and server are not within 5ms RTT.
The next logical step is to write a readahead layer (if you're serving files larger than a single read block to the DFS) to get better throughput.
A combination of the above techniques, with a bit of tuning and elbow grease, is a good foundation for scalable (in terms of file size, total # of files, etC) serving.