moe on Asynchronous Iteration Patterns (in Node.js)

Comment on Asynchronous Iteration Patterns (in Node.js)

This is a great demonstration of the fundamental issue with node. His code goes from:

   function insertCollection(collection) {
     for(var i = 0; i < collection.length; i++) {
       db.insert(collection[i]);
     }
   }

To this:

  function insertCollection(collection, callback) {
    var coll = collection.slice(0); // clone collection
    (function insertOne() {
      var record = coll.splice(0, 1)[0];
      try {
        db.insert(record, function(err) {
          if (err) { callback(err); return }
          if (coll.length == 0) {
            callback();
          } else {
            insertOne();
          }
        }
      } catch (exception) {
        callback(exception);
      }
    })();
  }

Hopefully this article will work as an eye-opener for anyone who was not yet convinced that node badly needs a concurrency abstraction.

substack 15y

First, why wrap your second example in try {} catch {} but not the first? The second example (final example from the article) seems like it was intentionally made more complicated to prove its point, but it could just be unfamiliarity. Node idioms are just different from other environments and it takes time to get a good feel for them.

(Edit: noticed that the article does it this way.)

How about just:

    function insertCollection(collection, cb) {
        if (collection.length === 0) cb(null);
        else db.insert(collection[0], function (err) {
            if (err) cb(err)
            else insertCollection(collection.slice(1), cb)
        }
    }

That's hardly much more complex, and as a bonus won't lock up your whole program while db.insert() is working. You could use something like my own library Seq() too:

    var Seq = require('seq');
    function insertCollection (collection, cb) {
        Seq.ap(collection).seqEach(function () {
            db.insert(c, this);
        }).seq(cb).catch(cb)
    }

That is, if you actually want the inserts to go sequentially. Usually you want them to go in parallel with say a maximum of 10 pending requests:

    var Seq = require('seq');
    function insertCollection (collection, cb) {
        Seq.ap(collection).parEach(10, function () {
            db.insert(c, this);
        }).seq(cb).catch(cb)
    }

I don't see any "fundamental issue" with node here. It's just a very new ecosystem and the idioms and libraries are rapidly evolving.

moe 15y

why wrap your second example

That's hardly much more complex

Well, sorry to be snarky, but you couldn't have supported my point better.

Both you and the article author (where the code snippets are from) missed this critical detail during the first iteration. And not on some complex, esoteric problem but on one of the most basic language constructs.

I blame neither of you, I only ask that we please not discount the extra complexity.

substack 15y

Ah, your point makes much more sense now. Yes, the author had some overly complex examples, but I'm optimistic about using user-space libraries to handle flow control without having to add new language primitives.

jimmyjazz14 15y

I agree and I keep hearing people say node should use co-routines but I did not think that co-routines where available until javascript 1.7. Would it be possible to implement them without support in V8?

pshapiroOP15y

There are a number of options. Have a look at the comments on the article page where people list a few, like Step and FutureJS.

moe 15y

Well, Step is apparently just the Sequence-subset of FutureJS and FutureJS as a whole is mostly syntactic sugar and baseline infrastructure (Deferreds, Promises) without tackling the fundamental problem.

Quite frankly, I've been down that rabbit hole with Twisted which also has most of these primitives. More magic does not help!

I do not want to deal with 'Future-ish object for the purpose of synchronizing other Futures'. Just let me write my code in sequential order where appropriate - which is about 90% of the business logic.

I'll quote jerf from this thread: You shouldn't even have to think about this, let alone argue which way is the best way to do it.

And Larry Wall: The computer should be doing the hard work. That's what it's paid to do, after all.

Comments