Comment on Ask HN: Fastest Crawl of HN ArticlesparentComments−krapp4yThe fastest way to get that would probably still be through HN's API, you just have to take the URL field for stories and ignore everything else.−tedunangst4yAnd how do you get the content once you have the URL?−arinlen4yAnd how do you get the content once you have the URL?I don't understand your question. If you have the URL, you just GET it, like any regular URL? Is there something that I'm missing?−agenciesOP4yMany domains have expired or content is no longer available.−krapp4yUse IA more responsibly, perhaps. Instead of scraping it, convert the list of links from HN to point to IA? You still have to work with whatever limits the site puts up in any case.−agenciesOP4yIf a HN story is a link to Wikipedia, the HN api serves the content of the Wikipedia page??
Comments
The fastest way to get that would probably still be through HN's API, you just have to take the URL field for stories and ignore everything else.
And how do you get the content once you have the URL?
I don't understand your question. If you have the URL, you just GET it, like any regular URL? Is there something that I'm missing?
Many domains have expired or content is no longer available.
Use IA more responsibly, perhaps. Instead of scraping it, convert the list of links from HN to point to IA? You still have to work with whatever limits the site puts up in any case.
If a HN story is a link to Wikipedia, the HN api serves the content of the Wikipedia page??