NPM packages doesn’t take up that much disk space for me with standard pnpm deduplication. The much bigger offender for me is Rust target dir: when people talk about vibe coding in Rust for performance, what they don’t mention (at least I’ve hardly ever seen it mentioned) is every trivial little tool pushes 1GB on disk and anything slightly nontrivial easily racks up multi-GB. Which hurts when you have lots of vibed small tools. I wonder if anyone bothered to look into this problem.
Rust has the deadly combo of static linking, dependency unification and a feature flag system. That makes it hard to share cached dependencies between projects, and when your compiler is as complex as rustc the individual files will end up big.
It's definitely possible to optimize the rlib format, like representing generic types as a tree of short IDs instead of a string or separating them into an optional debug file. Or do it like Zig with a new IR designed to be compact and easy to assemble into final executables. Even a simple global cache for the times you get lucky and end up with the same hash could help.
The problem is no one on the core team has time to do it because they are always overworked from the amount of bugs that need fixing. They also have a culture of making sure all changes are perfect before they go stable as a overreaction to C++ shipping half baked proposals. I think these are the reasons everything is moving so slow.
I think the parent was referring to hosting the entire NPM registry, not having a project that uses NPM. In that case there's substantial duplication from version to version.
I was informed many eons ago that one of the hot features of Rational was that when one user made a change that required recompiling the headers, those headers were uploaded to the server so that other users didn't have to recompile them.
It had the nice effect that whoever caused the headers to need to be recompiled was the only one who had to pay the recompilation tax instead of everyone.
You may be cleaving the problem along the wrong plane if you're worried about sharing between multiple projects on the same machine versus across all of the members of a single team.
Artifactory is older than dirt. And its main feature isn't even saving bandwidth, it's ensuring that versioned libraries can never be overwritten in place with a different version potentially containing a trojan. And it can be set up not to download new versions until they've survived long enough for someone to hopefully notice a supply chain attack.
Generally agree with you. But for this specific project the goal was to host the _entire_ npm registry as a mirror on consumer grade hardware, not just the slice of it I personally used for local development.
Comments
NPM packages doesn’t take up that much disk space for me with standard pnpm deduplication. The much bigger offender for me is Rust target dir: when people talk about vibe coding in Rust for performance, what they don’t mention (at least I’ve hardly ever seen it mentioned) is every trivial little tool pushes 1GB on disk and anything slightly nontrivial easily racks up multi-GB. Which hurts when you have lots of vibed small tools. I wonder if anyone bothered to look into this problem.
Rust has the deadly combo of static linking, dependency unification and a feature flag system. That makes it hard to share cached dependencies between projects, and when your compiler is as complex as rustc the individual files will end up big.
It's definitely possible to optimize the rlib format, like representing generic types as a tree of short IDs instead of a string or separating them into an optional debug file. Or do it like Zig with a new IR designed to be compact and easy to assemble into final executables. Even a simple global cache for the times you get lucky and end up with the same hash could help.
The problem is no one on the core team has time to do it because they are always overworked from the amount of bugs that need fixing. They also have a culture of making sure all changes are perfect before they go stable as a overreaction to C++ shipping half baked proposals. I think these are the reasons everything is moving so slow.
Don't they have nightly for imperfect experiments like this?
It was the second most reported problem on the 2025 Rust survey, after compilation times.
I think the parent was referring to hosting the entire NPM registry, not having a project that uses NPM. In that case there's substantial duplication from version to version.
I was informed many eons ago that one of the hot features of Rational was that when one user made a change that required recompiling the headers, those headers were uploaded to the server so that other users didn't have to recompile them.
It had the nice effect that whoever caused the headers to need to be recompiled was the only one who had to pay the recompilation tax instead of everyone.
You may be cleaving the problem along the wrong plane if you're worried about sharing between multiple projects on the same machine versus across all of the members of a single team.
Artifactory is older than dirt. And its main feature isn't even saving bandwidth, it's ensuring that versioned libraries can never be overwritten in place with a different version potentially containing a trojan. And it can be set up not to download new versions until they've survived long enough for someone to hopefully notice a supply chain attack.
This sounds like a security nightmare - there is a reason we have build servers (well there are many reasons, but build safety is one of them).
But I have to admit the idea is clever.
I think the idea was that the IDE trusted itself, the same way we trust the CI server. Ken Thompson's Trusting Trust holds pretty equally for both.
Generally agree with you. But for this specific project the goal was to host the _entire_ npm registry as a mirror on consumer grade hardware, not just the slice of it I personally used for local development.
It was part of the (now defunct) frea project: https://freajs.github.io/frea-website/
Use a cargo workspace. Helps with both compilation speed and disk usage.