I really don't understand why dl.google.com isn't just running Nginx or some web server. It's just serving files. Why does it need software written in house?
Nginx was built with 'single server' architecture in mind while Google generally operates on a 'distributed everything' architecture. This involves stuff like distributed file systems/data stores. My guess is it's not compatible with the single file system interfaces of Nginx. Also note that dl.google.com serves stuff on a massively bigger scale than the majority of the rest of the internet. On that scale I'd guess they face some rather uncommon challenges and bottlenecks that can only be addressed by custom software.
True, but a static-file server should be trivially scalable. That is, scalable by placing it behind a load balancer(or using any number of alternative methods that don't acutally touch the server itself).
Even if the server implements Access Control, it should still be trivially scalable.
I mean, if by "rather uncommon challenges" you're referring to the "what are threads?" design philosophy of the original... then yeah, it does require "custom software".
dl.google.com isn't just a place to dump files to the public web, it's also got complicated ACL schemes that define restrictions on data access. dl.google.com is used internally.
Google is the only company on the planet with anything approaching its scale requirements. It's entirely possible that even nginx breaks down at their level of speed/concurrency/distribution.
True, but it isn't the only company serving apt packages, and it isn't even the biggest one. Considering that their apt server was as slow as it was(http://talks.golang.org/2013/oscon-dl.slide#9) - they obviously did something wrong that everyone else was doing right.
Perhaps the problem was that dl.google.com was being used for too many things at once, but this doesn't excuse building custom software that offers worse performance than free off-the-shelf products that literally everyone else has been using for years.
Comments
I really don't understand why dl.google.com isn't just running Nginx or some web server. It's just serving files. Why does it need software written in house?
Nginx was built with 'single server' architecture in mind while Google generally operates on a 'distributed everything' architecture. This involves stuff like distributed file systems/data stores. My guess is it's not compatible with the single file system interfaces of Nginx. Also note that dl.google.com serves stuff on a massively bigger scale than the majority of the rest of the internet. On that scale I'd guess they face some rather uncommon challenges and bottlenecks that can only be addressed by custom software.
True, but a static-file server should be trivially scalable. That is, scalable by placing it behind a load balancer(or using any number of alternative methods that don't acutally touch the server itself). Even if the server implements Access Control, it should still be trivially scalable.
Also, nginx doesn't have such...interesting ideas as the custom server: http://talks.golang.org/2013/oscon-dl.slide#19
I mean, if by "rather uncommon challenges" you're referring to the "what are threads?" design philosophy of the original... then yeah, it does require "custom software".
dl.google.com isn't just a place to dump files to the public web, it's also got complicated ACL schemes that define restrictions on data access. dl.google.com is used internally.
Google is the only company on the planet with anything approaching its scale requirements. It's entirely possible that even nginx breaks down at their level of speed/concurrency/distribution.
True, but it isn't the only company serving apt packages, and it isn't even the biggest one. Considering that their apt server was as slow as it was(http://talks.golang.org/2013/oscon-dl.slide#9) - they obviously did something wrong that everyone else was doing right.
Perhaps the problem was that dl.google.com was being used for too many things at once, but this doesn't excuse building custom software that offers worse performance than free off-the-shelf products that literally everyone else has been using for years.