Hey, I'm working on an open source project that I think could application for this. Its something I've termed a "web gatherer" basically it provides the framework for crawling web pages and then has workflows where custom code is written to determine certain things about each page, if it meets criteria that is programmed for that workflow then the page is added to the results queue, the others are filtered out. I'm planning to implement an NLP component at some point using one of the open source NLP libs availabe. Overall, I think of this project as sort of a web scraper / search engine that sits above the base layer (such as Google) which can be used to refine results. Anyways, you may be interested, if so feel free to contact me: https://github.com/devs1010/WebGatherer---Scraper-and-Analyz...
Comments
Hey, I'm working on an open source project that I think could application for this. Its something I've termed a "web gatherer" basically it provides the framework for crawling web pages and then has workflows where custom code is written to determine certain things about each page, if it meets criteria that is programmed for that workflow then the page is added to the results queue, the others are filtered out. I'm planning to implement an NLP component at some point using one of the open source NLP libs availabe. Overall, I think of this project as sort of a web scraper / search engine that sits above the base layer (such as Google) which can be used to refine results. Anyways, you may be interested, if so feel free to contact me: https://github.com/devs1010/WebGatherer---Scraper-and-Analyz...