Skip to content

Ask HN: Is there a library or a package that converts a web page to JSON?

2 pointsaliirz1 comment
On HN

I am living in a country where open data is a very alien concept for the public and government. Except for a handful of civic hackers who find it very difficult to create apps or services due to lack of open APIs or datasets. Data is available on government websites but its not technically open (as in a RESTful API). I am thinking about writing a library that can crawl all over a page and convert the HTML into JSON that can be consumed over REST. Is there something like that out there already? What are some of the things I need to consider before I start writing this?

Comments

Back in the day Yahoo Pipes would have been something I'd have recommended as being a tool in this space. Here's a blog post that lists some alternatives to Yahoo Pipes and some of them look interesting for your use case, though I don't know that any of them is exactly what you're looking for.

http://www.makeuseof.com/tag/12-best-yahoo-pipes-alternative...

Also, CloudScrape:

http://cloudscrape.com/

AboutSource Built by g1lg1l

Hackerly is an independent reader for Hacker News, built on the public HN API. Not affiliated with Y Combinator.