Has npm finally figured out how to de-dupe dependencies? In one project, I have something like 47 copies of the same version of the same library, distributed at all levels of the node_modules hierarchy.
I try not to think about that JS tooling too hard, lest I start pulling my hair out and devolve into a screaming crazy person.
Totally. We just ran into this issue. Running npm install inside a linux vm resulted in an endless loop, because the nfs mapping created path names that exceeded the length limit.
NPM 3 tries, not entirely without success, to flatten the tree under node_modules/; I've had good results using it to resolve the kind of path length issues (in this case, with NTFS directories mounted in a VM) that you describe. Might be worth a look in your case as well.
Both a and b depend on c version 1.0.0, but since there's a version 2.0.0 in the root node_modules folder c can't be placed there, and has to be duplicated in a and b's own node_modules folder, otherwise Node couldn't find it for each of them.
I've built a fairly hacky solution to this before (for a different package manager) - it can be pretty simple:
node_modules/
versions/
a@1.0.0
node_modules/
c -> ../../c@1.0.0
b@1.0.0
node_modules/
c -> ../../c@1.0.0
c@1.0.0/
c@2.0.0/
a -> versions/a@1.0.0
b -> versions/b@1.0.0
c -> versions/c@2.0.0
I was just about to write a comment to ask why it can't be done like this (using CAS / symlinks). So I guess it can. Are there any disadvantages to using ied over npm?
To me it looks like a silly architectural mistake made by NPM/Node developers, considering that there's already pretty good dependency management solution on the market that does the things right.
Maven repositories have following structure, that allows to avoid duplication and take versions into account:
/<vendor namespace>/<library>/<version>/<library artifact.ext>
Vendor namespace itself is hierarchical and usually related to domain name, e.g. "com/mycompany/myapp".
No idea, why this approach is not yet used in JS world (except the webjars), but it's high time to fix it this way.
Comments
Has npm finally figured out how to de-dupe dependencies? In one project, I have something like 47 copies of the same version of the same library, distributed at all levels of the node_modules hierarchy.
I try not to think about that JS tooling too hard, lest I start pulling my hair out and devolve into a screaming crazy person.
This sucks so hard. It's also quite easy to hit path length limits for certain operations (on windows) when you have this mess there.
Totally. We just ran into this issue. Running npm install inside a linux vm resulted in an endless loop, because the nfs mapping created path names that exceeded the length limit.
it's like an entire eco-system is fundamentally broken
Maybe, but it's not like there is no way of solving that issue.
It's just that this problem can't be solved entirely by npm. Node has to make changes to it's module loader as well.
NPM 3 tries, not entirely without success, to flatten the tree under node_modules/; I've had good results using it to resolve the kind of path length issues (in this case, with NTFS directories mounted in a VM) that you describe. Might be worth a look in your case as well.
This used to be the case. In npm3 and beyond it installs dependencies in a flat way, which seems to have solved the Windows file length limit problem.
It's actually not possible due to the way Node looks up dependencies. You could have a situation like:
Both a and b depend on c version 1.0.0, but since there's a version 2.0.0 in the root node_modules folder c can't be placed there, and has to be duplicated in a and b's own node_modules folder, otherwise Node couldn't find it for each of them.It's entirely possible with symbolic links. https://github.com/rstacruz/pnpm
You could use links, version in names, ... You'd just have problem to take out one submodule I guess, but I doubt that's a use case anywhere.
I've built a fairly hacky solution to this before (for a different package manager) - it can be pretty simple:
Some other people have linked to npm alternatives ied and pnpm which do essentially this.
npm v3 tries to dedupe packages during installation to reduce this problem. However, it still does not install packages in a deterministic way...
[0]: https://docs.npmjs.com/how-npm-works/npm3-dupe
[1]: https://docs.npmjs.com/how-npm-works/npm3-nondet
https://github.com/alexanderGugel/ied
I was just about to write a comment to ask why it can't be done like this (using CAS / symlinks). So I guess it can. Are there any disadvantages to using ied over npm?
It's not fully stable yet. I tried a while ago, and after running into 3 showstoping bugs (and filing bugreports) i had to give up.
Great idea though, and it probably has improved a bunch since then.
This is one of the reasons why I love the JS ecosystem. You can even choose among a few tools you use to install packages from a package manager.
It does not support external packages from git remote url (github).
Update:
Looks like it is supported now:
https://github.com/alexanderGugel/ied/issues/7#issuecomment-...
Private modules seem to be an issue at the moment, but a PR for this is pending.
I used it successfully in a project and it even worked with native modules.
To me it looks like a silly architectural mistake made by NPM/Node developers, considering that there's already pretty good dependency management solution on the market that does the things right.
Maven repositories have following structure, that allows to avoid duplication and take versions into account: /<vendor namespace>/<library>/<version>/<library artifact.ext>
Vendor namespace itself is hierarchical and usually related to domain name, e.g. "com/mycompany/myapp".
No idea, why this approach is not yet used in JS world (except the webjars), but it's high time to fix it this way.
Yes, that's already solved with npm 3.