I'll be that guy; I'm a little confused as to why this would require $200,000 (the requested total funding requested) to solve. From the site itself :
614,680,691 requests per month come down to ~230 request per second. Allowing for some spikiness that boils down to perhaps 1k request/second at peak. Requests in these cases are mostly relatively simple queries on version-ed, highly cacheable data. I say highly cacheable because it is relatively static data for which most (if not all) of the data fields relevant for these requests can fit in memory of perhaps even a single node (NPM currently includes 48,799 packages. That leaves a very healthy chunk of data per package on 16Gb-128Gb RAM server boxes).
The downloads are a bit of a puzzle to me as well. On my machine the average NPM package is about 200Kb (YMMV). 114,626,717 downloads are mentioned on the site. 200k times 114 million downloads lands us on roughly 23 TB. Even on a relatively expensive CDN such as Amazon CloudFront the total monthly cost for that bandwidth and content request load for CloudFront and the required S3 costs land on about $3k/month and that's ignoring all bulk discounts, reserved capacity and so on (which are very significant at these volumes).
I'm more than likely oversimplifying a few things here and there (or failed horribly at math) but I'd still be very interested to hear why this requires such a large investment. Also, wouldn't the more obvious solution be to open source the npmjs software and allow the community to contribute knowledge and time instead?
EDIT: Quickly wanted to point out that I use npmjs.org often , is a great service and that donations are very well deserved. After re-reading my post it turned out more negative sounding than intended.
Every fuzzy-versioned dependency means one request to npm that you can not (really) cache - at least not without doing and relying on active cache invalidation. And if I look at my average npm install log, that's about half of the requests. For storage estimates you should also include that it's not only storing the packages but also all versions of the package. ~A year ago the registry was 25GB in size IIRC. And that was a freshly downloaded & compacted one on my machine. Especially considering that the growth will not suddenly stop, things get complicated.
Also: both the website and the registry is already completely open source. But hosting it with "perfect" uptime is a real problem and requires not only network/hardware but also people. Including testing and migrating to a new, more scalable solution - say a handful of people (2-4) will work on that for a month, which does not yet include future maintenance. That can easily mean $50000 just in salaries - or in "people that would normally bring value to paying customers" if you assume that those people will be fine with working that month without pay.
I see what you mean but allow me to theorycraft a bit more : Changes and removal of packages are considerably more rare than additions and either happens relatively infrequently. This makes cluster wide cache invalidation relatively trivial (it's easy but has scalability issues, and scalability won't be an issue here). Also, when I said "cached" I probably should have said keeping various indexes in memory to facilitate queries. I actually work with systems in a roughly similar technical domain (way different space though, I work on large scale TV systems).
Your other point is definitely the big challenge but that is exactly the main motivation to use hosted CDNs and other hosted services that have solved this challenge for you o a large extent.
Instead of resolving those requests on the server, why not have the client download an index file with all the available versions for every package, or maybe just an index file with the top 20% of packages responsible for 80% of the load.
Comments
I'll be that guy; I'm a little confused as to why this would require $200,000 (the requested total funding requested) to solve. From the site itself :
614,680,691 requests per month come down to ~230 request per second. Allowing for some spikiness that boils down to perhaps 1k request/second at peak. Requests in these cases are mostly relatively simple queries on version-ed, highly cacheable data. I say highly cacheable because it is relatively static data for which most (if not all) of the data fields relevant for these requests can fit in memory of perhaps even a single node (NPM currently includes 48,799 packages. That leaves a very healthy chunk of data per package on 16Gb-128Gb RAM server boxes).
The downloads are a bit of a puzzle to me as well. On my machine the average NPM package is about 200Kb (YMMV). 114,626,717 downloads are mentioned on the site. 200k times 114 million downloads lands us on roughly 23 TB. Even on a relatively expensive CDN such as Amazon CloudFront the total monthly cost for that bandwidth and content request load for CloudFront and the required S3 costs land on about $3k/month and that's ignoring all bulk discounts, reserved capacity and so on (which are very significant at these volumes).
I'm more than likely oversimplifying a few things here and there (or failed horribly at math) but I'd still be very interested to hear why this requires such a large investment. Also, wouldn't the more obvious solution be to open source the npmjs software and allow the community to contribute knowledge and time instead?
EDIT: Quickly wanted to point out that I use npmjs.org often , is a great service and that donations are very well deserved. After re-reading my post it turned out more negative sounding than intended.
Every fuzzy-versioned dependency means one request to npm that you can not (really) cache - at least not without doing and relying on active cache invalidation. And if I look at my average npm install log, that's about half of the requests. For storage estimates you should also include that it's not only storing the packages but also all versions of the package. ~A year ago the registry was 25GB in size IIRC. And that was a freshly downloaded & compacted one on my machine. Especially considering that the growth will not suddenly stop, things get complicated.
Also: both the website and the registry is already completely open source. But hosting it with "perfect" uptime is a real problem and requires not only network/hardware but also people. Including testing and migrating to a new, more scalable solution - say a handful of people (2-4) will work on that for a month, which does not yet include future maintenance. That can easily mean $50000 just in salaries - or in "people that would normally bring value to paying customers" if you assume that those people will be fine with working that month without pay.
I see what you mean but allow me to theorycraft a bit more : Changes and removal of packages are considerably more rare than additions and either happens relatively infrequently. This makes cluster wide cache invalidation relatively trivial (it's easy but has scalability issues, and scalability won't be an issue here). Also, when I said "cached" I probably should have said keeping various indexes in memory to facilitate queries. I actually work with systems in a roughly similar technical domain (way different space though, I work on large scale TV systems).
Your other point is definitely the big challenge but that is exactly the main motivation to use hosted CDNs and other hosted services that have solved this challenge for you o a large extent.
I'm almost tempted to have a go at this.
Instead of resolving those requests on the server, why not have the client download an index file with all the available versions for every package, or maybe just an index file with the top 20% of packages responsible for 80% of the load.
It sounded like they might hire against it in their description. That's not much compared to a potential salary over a few years.