From the looks of it, they wanted support for putting files in place before a release date and easy per-file header/caching/access control. Add in a few other miscellaneous features and make it available to everyone and you're at a point where an HTTP server probably won't cut it.
Files also apparently come from some central repository and need to be synced.
I think the biggest reason would be that almost any off-the-shelf server software would struggle in a Google server environment. They have solved scale at the machine level, and he alludes to this on one slide:
... why aren't you using the cluster file systems like everybody else?
... cluster file systems own disk time on your machine, not you.
Slide 62 mentions the proprietary bits: ACL policies and RPC storage access. Does an off-the-shelf httpd support ACLs? How easy would it be to make them support google storage instead of a file system?
(I wrote a rough equivalent of "payload_server" in Go at my current employer to solve authentication, access control, and some other business logic issues.)
Does an off-the-shelf httpd support ACLs?
Not really. You inevitably end up writing custom code to conform to your particular requirements and/or existing systems. If you want high-performance, you end up writing it in C as a module for Apache/nginx/whatever.
How easy would it be to make them support google storage instead of a file system?
Unless said storage system is presented to userspace through ordinary file interfaces, same as above. There's no general turn-key solution built into webservers. The problem space is too wide.
Using Go in this way gets you good performance, simple architecture, maintainability, and easy deployment with total flexibility to do whatever you need in order to solve your version of the problem. There are no straightjackets, you don't have to conform to (or find ways around) anyone else's conception of the problem space.
Comments
From the looks of it, they wanted support for putting files in place before a release date and easy per-file header/caching/access control. Add in a few other miscellaneous features and make it available to everyone and you're at a point where an HTTP server probably won't cut it.
Files also apparently come from some central repository and need to be synced.
I think the biggest reason would be that almost any off-the-shelf server software would struggle in a Google server environment. They have solved scale at the machine level, and he alludes to this on one slide:
... why aren't you using the cluster file systems like everybody else?
... cluster file systems own disk time on your machine, not you.
Slide 62 mentions the proprietary bits: ACL policies and RPC storage access. Does an off-the-shelf httpd support ACLs? How easy would it be to make them support google storage instead of a file system?
(I wrote a rough equivalent of "payload_server" in Go at my current employer to solve authentication, access control, and some other business logic issues.)
Not really. You inevitably end up writing custom code to conform to your particular requirements and/or existing systems. If you want high-performance, you end up writing it in C as a module for Apache/nginx/whatever.
Unless said storage system is presented to userspace through ordinary file interfaces, same as above. There's no general turn-key solution built into webservers. The problem space is too wide.
Using Go in this way gets you good performance, simple architecture, maintainability, and easy deployment with total flexibility to do whatever you need in order to solve your version of the problem. There are no straightjackets, you don't have to conform to (or find ways around) anyone else's conception of the problem space.