STRML on Use Node.js to Extract Data from the Web

Comment on Use Node.js to Extract Data from the Web

Don't forget streams, the more `node.js` way to parse HTML:

    var http = require('http');
    var tr = require('trumpet')();
    var request = require('request');
    request.get('http://www.echojs.com")
      .pipe(tr.createReadStream("article > span"))
      .pipe(process.stdout);

That's it! See https://github.com/substack/node-trumpet and their tests for more.

substack 13y

You probably meant:

    var tr = require('trumpet')();
    tr.createReadStream('article > span')
      .pipe(process.stdout);
    
    var request = require('request');
    request.get('http://www.echojs.com').pipe(tr);

Bonus: I just noticed a simple bug in the selector engine from running your intended code that I just fixed in trumpet@1.5.6.

kanzure 13y

And then there's hyperquest because maybe you want to do more than five simultaneous requests:

https://github.com/substack/hyperquest

ssafejava 13y

True - you can also disable the globalAgent or change the number of pooled connections. Connection pooling was generally a bad idea (tm) in Node and afaik will be removed in the near future.

Comments