Honest question: have you actually tried working with modules and dependencies in Node? It's better now, but it's still not pleasant and often installing a module starts with debugging.
And what's the core missing, would you say? Personally, I thought the cluster stuff was already getting heavy.
> Honest question: have you actually tried working with modules and dependencies in Node? It's better now, but it's still not pleasant and often installing a module starts with debugging.
Yes. And you're right, but then it's not often that much better than other languages. Personally I'd rather they get the core language finished first (though their concept of "finished" and mine seem to differ).
> And what's the core missing, would you say? Personally, I thought the cluster stuff was already getting heavy.
There's no way to lock a file (making file writes completely fragile). There's no way to seek() in a file. There's no way to open a file with O_EXCL set (meaning lockless file writing is impossible). There's no temp file functionality in core, and without O_EXCL the solutions available for this on npm are utterly broken. There's still major bugs in the Crypto routines (they take Strings instead of Buffers in various places)... Honestly I could go on.
You can open a file with O_EXCL if you pass in the open flags as a number. (You can find them on require("constants"), and they need to be binary-OR'ed together.) This isn't documented. It should be. It should probably also be exposed in a cleaner way.
Most of the rest of what you describe is APIs that need to be polished and refined a bit. The boundaries are well defined at this point, though. We probably won't add another builtin module at this point, or dramatically expand what any of them can do. (I don't consider seek() dramatic, it's just tricky to get right given JavaScript's annoying Number problems.)
> You can open a file with O_EXCL if you pass in the open flags as a number. (You can find them on require("constants"), and they need to be binary-OR'ed together.) This isn't documented. It should be. It should probably also be exposed in a cleaner way.
That's great to know. Obviously I'm just following the docs.
> Most of the rest of what you describe is APIs that need to be polished and refined a bit
My concern is merely that there have been a number of statements put out saying "we won't be adding anything more to the API", and that we are basically almost at 1.0, at which point there won't ever be anything added to the core API. Lack of flock() is huge (you can't write to an existing file safely without it - and Node developers are doing that all the time, including your own NPM). Lack of an ability to create temporary files safely seems a fundamental weakness - especially when so many Node apps are dealing with file uploads - that's a disaster waiting to happen. We are going to be dealing with Node.js security bugs because of these issues for a VERY long time.
flock() is not trivial to do in a portable way. For a unix-only flock(), check out the fs-ext addon. Same for mktemp.
I wouldn't be opposed to either being in core if it could be done in a clean way, but this is just adding another knob that can be done with an addon easily enough. If you care more about having flock() than about writing portable programs, then that's what the fs-ext addon is for.
> We are going to be dealing with Node.js security bugs because of these issues for a VERY long time.
Of course we'll be "dealing with Node.js security bugs for a VERY long time", because we'll be using Node.js for a very long time. Software is buggy, and many bugs are security hazards. We'll be dealing with "Unix security bugs" and "C security bugs" and "Java security bugs" forever as well.
Please do not make vague suggestions about security issues. Either you've found an issue, and should be submitting it, or you haven't, and are just spreading fud.
> For a unix-only flock(), check out the fs-ext addon
Which I wrote.
> Of course we'll be "dealing with Node.js security bugs for a VERY long time", because we'll be using Node.js for a very long time.
That's not quite what I meant - I mean that people right now are writing temp files in LOTS of Node.js applications in an insecure way. It's good that O_EXCL is available, I'll try and submit a patch to node-temp, but really temp file creation should be in core (amongst other things).
This isn't a vague suggestion. There are COUNTLESS security bugs created every day by insecure temp file creation. Let's see, from npm these packages have security bugs because they rely on the insecure node-temp: ShipItJS assetgraph-builder confy filerepl gracie joose js-loader muffin nerve redisfs.
I disagree that it is "huge" that flock() is missing; if you are relying on advisory (emphasis on advisory) file and record locking to implement coherence, then your architecture is likely an accident waiting to happen -- it's a highly problematic interface for many reasons. Of course, if you really need it, use fs-ext as Isaac mentioned -- NS more generally, if you need some system functionality that isn't present in core, develop an add-on and knock yourself out. It's not entirely pleasant, of course -- but don't pretend that an interface not being in core means that some bit of system functionality is somehow off limits...
Huge might be the wrong word, but I've definitely been frustrated that it's missing. Java has a standard library class that works well enough for whole-file advisory (or mandatory if you're on Windows, I suppose) locks on a local filesystem. It's eminently helpful for excluding a second process from starting with a lock that the OS will clean up for you if/when your process terminates for whatever reason.
smf(5) can get you part of the way there, but not if you're (say) writing some system bits that you need to deploy on platforms in addition to SunOS. Plus, if you can avoid depending on any not-just-pure-JS modules then you can deploy one tree onto all of your platforms without additional build steps, C++ compile/ABI issues, etc.
THANK YOU. This addresses a question I've had on stack overflow for a couple of days. It also addresses at least two issues in Node.js's github tracker under the 'fs' flag.
I don't know that anyone is suggesting that it's "finished". In fact, it's not even to a 1.x release yet. I consider anything below that a work-in-progress at at minimum. Generally anything under a 2.x release has lots of room for improvement.
The fact that they're shifting focus isn't a big concern. I'd at least wait until Isaac comes out and says what the future roadmap is before criticizing it.
You have to remember, developers close to a platform always live 6-12 months in the future, at least. (Talk to Brendan Eich about JavaScript, he's like 5 years in the future.)
What "finished" means in this context is that we've established which problems we're going to solve, and the solutions to those are well-understood. There aren't new problems that node-core is setting out to address by making changes to the binary itself.
The new hard things to tackle are above the core layer - CI, userland module documentation and discoverability, binary deployments, etc. There's quite a bit of polish and refinement to be done at the lower level, but the paths are clear at this point, as far as that goes.
Comments
Honest question: have you actually tried working with modules and dependencies in Node? It's better now, but it's still not pleasant and often installing a module starts with debugging.
And what's the core missing, would you say? Personally, I thought the cluster stuff was already getting heavy.
> Honest question: have you actually tried working with modules and dependencies in Node? It's better now, but it's still not pleasant and often installing a module starts with debugging.
Yes. And you're right, but then it's not often that much better than other languages. Personally I'd rather they get the core language finished first (though their concept of "finished" and mine seem to differ).
> And what's the core missing, would you say? Personally, I thought the cluster stuff was already getting heavy.
There's no way to lock a file (making file writes completely fragile). There's no way to seek() in a file. There's no way to open a file with O_EXCL set (meaning lockless file writing is impossible). There's no temp file functionality in core, and without O_EXCL the solutions available for this on npm are utterly broken. There's still major bugs in the Crypto routines (they take Strings instead of Buffers in various places)... Honestly I could go on.
You can open a file with O_EXCL if you pass in the open flags as a number. (You can find them on require("constants"), and they need to be binary-OR'ed together.) This isn't documented. It should be. It should probably also be exposed in a cleaner way.
Most of the rest of what you describe is APIs that need to be polished and refined a bit. The boundaries are well defined at this point, though. We probably won't add another builtin module at this point, or dramatically expand what any of them can do. (I don't consider seek() dramatic, it's just tricky to get right given JavaScript's annoying Number problems.)
> You can open a file with O_EXCL if you pass in the open flags as a number. (You can find them on require("constants"), and they need to be binary-OR'ed together.) This isn't documented. It should be. It should probably also be exposed in a cleaner way.
That's great to know. Obviously I'm just following the docs.
> Most of the rest of what you describe is APIs that need to be polished and refined a bit
My concern is merely that there have been a number of statements put out saying "we won't be adding anything more to the API", and that we are basically almost at 1.0, at which point there won't ever be anything added to the core API. Lack of flock() is huge (you can't write to an existing file safely without it - and Node developers are doing that all the time, including your own NPM). Lack of an ability to create temporary files safely seems a fundamental weakness - especially when so many Node apps are dealing with file uploads - that's a disaster waiting to happen. We are going to be dealing with Node.js security bugs because of these issues for a VERY long time.
O_EXCL is in for 0.8: https://github.com/bnoordhuis/node/compare/O_EXCL
flock() is not trivial to do in a portable way. For a unix-only flock(), check out the fs-ext addon. Same for mktemp.
I wouldn't be opposed to either being in core if it could be done in a clean way, but this is just adding another knob that can be done with an addon easily enough. If you care more about having flock() than about writing portable programs, then that's what the fs-ext addon is for.
> We are going to be dealing with Node.js security bugs because of these issues for a VERY long time.
Of course we'll be "dealing with Node.js security bugs for a VERY long time", because we'll be using Node.js for a very long time. Software is buggy, and many bugs are security hazards. We'll be dealing with "Unix security bugs" and "C security bugs" and "Java security bugs" forever as well.
Please do not make vague suggestions about security issues. Either you've found an issue, and should be submitting it, or you haven't, and are just spreading fud.
> flock() is not trivial to do in a portable way
Perl, Python and Ruby manage it.
> For a unix-only flock(), check out the fs-ext addon
Which I wrote.
> Of course we'll be "dealing with Node.js security bugs for a VERY long time", because we'll be using Node.js for a very long time.
That's not quite what I meant - I mean that people right now are writing temp files in LOTS of Node.js applications in an insecure way. It's good that O_EXCL is available, I'll try and submit a patch to node-temp, but really temp file creation should be in core (amongst other things).
This isn't a vague suggestion. There are COUNTLESS security bugs created every day by insecure temp file creation. Let's see, from npm these packages have security bugs because they rely on the insecure node-temp: ShipItJS assetgraph-builder confy filerepl gracie joose js-loader muffin nerve redisfs.
I disagree that it is "huge" that flock() is missing; if you are relying on advisory (emphasis on advisory) file and record locking to implement coherence, then your architecture is likely an accident waiting to happen -- it's a highly problematic interface for many reasons. Of course, if you really need it, use fs-ext as Isaac mentioned -- NS more generally, if you need some system functionality that isn't present in core, develop an add-on and knock yourself out. It's not entirely pleasant, of course -- but don't pretend that an interface not being in core means that some bit of system functionality is somehow off limits...
Huge might be the wrong word, but I've definitely been frustrated that it's missing. Java has a standard library class that works well enough for whole-file advisory (or mandatory if you're on Windows, I suppose) locks on a local filesystem. It's eminently helpful for excluding a second process from starting with a lock that the OS will clean up for you if/when your process terminates for whatever reason.
smf(5) can get you part of the way there, but not if you're (say) writing some system bits that you need to deploy on platforms in addition to SunOS. Plus, if you can avoid depending on any not-just-pure-JS modules then you can deploy one tree onto all of your platforms without additional build steps, C++ compile/ABI issues, etc.
THANK YOU. This addresses a question I've had on stack overflow for a couple of days. It also addresses at least two issues in Node.js's github tracker under the 'fs' flag.
I don't know that anyone is suggesting that it's "finished". In fact, it's not even to a 1.x release yet. I consider anything below that a work-in-progress at at minimum. Generally anything under a 2.x release has lots of room for improvement.
The fact that they're shifting focus isn't a big concern. I'd at least wait until Isaac comes out and says what the future roadmap is before criticizing it.
The point is that people (core developers) are suggesting it is finished. Follow them on twitter, and subscribe to the mailing list and you'll see.
You have to remember, developers close to a platform always live 6-12 months in the future, at least. (Talk to Brendan Eich about JavaScript, he's like 5 years in the future.)
What "finished" means in this context is that we've established which problems we're going to solve, and the solutions to those are well-understood. There aren't new problems that node-core is setting out to address by making changes to the binary itself.
The new hard things to tackle are above the core layer - CI, userland module documentation and discoverability, binary deployments, etc. There's quite a bit of polish and refinement to be done at the lower level, but the paths are clear at this point, as far as that goes.