Looks like you took the good ideas from Scrapy's crawling engine and combined it with a great scraping API, which is all I ever wanted in a bot framework!
I'm especially excited about the unified API for browser and HTML scraping, which is something I've had to hack on top of Scrapy in the past and it really wasn't a good experience. That, along with puppeteer-heap-snapshot, will make the common case of "we need this to run NOW, you can rewrite it later" so much easier to handle.
While I'm not particularly happy to see JavaScript begin taking over another field as it truly is an awful language, more choice is always better and this project looks valuable enough to make dealing with JS a worthwhile tradeoff.
While I'm not particularly happy to see JavaScript begin taking over another field as it truly is an awful language, more choice is always better and this project looks valuable enough to make dealing with JS a worthwhile tradeoff.
Love that comment :D
Yeah, the ability to switch between headless and http is very important to us in production. We often hack something up quickly with headless and then later optimize it to use HTTP when we find the time.
people trash JavaScript while using python? They should look at the mirror!
Anyone can trash js except for all these C like languages with boringly similar designs, doubly so for python. Itself took over the world due to the fact that amateur (scientists and other professions as opposed to programmers) can easily play with it.
node.js dependency hell that often makes java's transitive dependencies on bigger projects look like hello world? Exception handling? Performance? No real multithreading afaik?
I have no skin in the game (anymore), but boy the feeling repeated gazillion times left and right in past decade about javascript crawling to be used in places it shouldn't be strongly resonated with me back then (not particularly for this project, rant in general).
You have one file which lists your dependencies and it comes as a standard with the framework compared to Java & co where you have lots of flavours.
The amount of dependencies your dependencies have is up to your choice.
Exception handling?
Synchronous code uses the standard try / catch method.
Asynchronous code has been using async / await combined with try / catch.
A strategy that Nose.js invented and that other languages like Java, Rust and Python copied.
Performance?
V8 and Node.ja are pretty much the fastest dynamic language platform.
Years ago, when companies started switching to Node.js, a lot of companies actually reduced the amount of servers they used when switching from Java to Node.js
No real multithreading afaik?
I/O is multithreaded through libuv and C-Ares.
The rest can simply run as a multi-process app.
Worker pools have also recently been introduced.
In any case that wouldn't be an issue for crawling which doesn't require multithreading.
JavaScript is the language used in the front-end so it seems like the most fitting language to use for crawling and scraping, especially since the introduction of headless browsers.
Nothing against dynamically typed languages, Python is one as well. Maybe "awful" was a bit of a strong word, but to quickly summarise my issues with JS:
- all the syntax and duck typing footguns (see all those "js is weird" memes)
- the complexity of the build chain, especially if you want typing support
- the unmanageable dependency mess that seems to be the norm in the ecosystem
Comments
Looks like you took the good ideas from Scrapy's crawling engine and combined it with a great scraping API, which is all I ever wanted in a bot framework!
I'm especially excited about the unified API for browser and HTML scraping, which is something I've had to hack on top of Scrapy in the past and it really wasn't a good experience. That, along with puppeteer-heap-snapshot, will make the common case of "we need this to run NOW, you can rewrite it later" so much easier to handle.
While I'm not particularly happy to see JavaScript begin taking over another field as it truly is an awful language, more choice is always better and this project looks valuable enough to make dealing with JS a worthwhile tradeoff.
Love that comment :D
Yeah, the ability to switch between headless and http is very important to us in production. We often hack something up quickly with headless and then later optimize it to use HTTP when we find the time.
people trash JavaScript while using python? They should look at the mirror!
Anyone can trash js except for all these C like languages with boringly similar designs, doubly so for python. Itself took over the world due to the fact that amateur (scientists and other professions as opposed to programmers) can easily play with it.
Python is anything but a C-like language, I really don't know what you're talking about.
Because of indentation? Gimme a break, most of the semantics are the same. Different would be like lisp or SML
Sorry but this irked me. What exactly are you hangups with JS ? It's just a JiT dynamic typed language. By design.
It has its qwirks for sure, but again, its just a language. Truly awful it isn't.
node.js dependency hell that often makes java's transitive dependencies on bigger projects look like hello world? Exception handling? Performance? No real multithreading afaik?
I have no skin in the game (anymore), but boy the feeling repeated gazillion times left and right in past decade about javascript crawling to be used in places it shouldn't be strongly resonated with me back then (not particularly for this project, rant in general).
You have one file which lists your dependencies and it comes as a standard with the framework compared to Java & co where you have lots of flavours. The amount of dependencies your dependencies have is up to your choice.
Synchronous code uses the standard try / catch method. Asynchronous code has been using async / await combined with try / catch. A strategy that Nose.js invented and that other languages like Java, Rust and Python copied.
V8 and Node.ja are pretty much the fastest dynamic language platform. Years ago, when companies started switching to Node.js, a lot of companies actually reduced the amount of servers they used when switching from Java to Node.js
I/O is multithreaded through libuv and C-Ares. The rest can simply run as a multi-process app. Worker pools have also recently been introduced. In any case that wouldn't be an issue for crawling which doesn't require multithreading.
JavaScript is the language used in the front-end so it seems like the most fitting language to use for crawling and scraping, especially since the introduction of headless browsers.
Nothing against dynamically typed languages, Python is one as well. Maybe "awful" was a bit of a strong word, but to quickly summarise my issues with JS: - all the syntax and duck typing footguns (see all those "js is weird" memes) - the complexity of the build chain, especially if you want typing support - the unmanageable dependency mess that seems to be the norm in the ecosystem