NLP is hard, particularly for highly general problems conducted on small samples of text.
Here's a problem case that you will find to be very common: a 20-second post with the right answer to a difficult problem by someone who's busy and typing on their phone. Riddled with typos, weird corrections, transposition errors, etc. - but still something you'd want to be a high-ranking result.
You can use a large set of sample articles that are well written, the machine learning is done on these, then the search engine would compare against these for each web page if finds when searching. You could easily have a search go a few hundred pages, just have it take 20+ pages of google results. I worked on a project that used NLP (for a startup), I wasn't the one doing most NLP related stuff but it was definitely an interesting experience and the guy who was in charge of the NLP was able to get some pretty interesting results, even from fairly small data-sets.
Comments
NLP is hard, particularly for highly general problems conducted on small samples of text.
Here's a problem case that you will find to be very common: a 20-second post with the right answer to a difficult problem by someone who's busy and typing on their phone. Riddled with typos, weird corrections, transposition errors, etc. - but still something you'd want to be a high-ranking result.
You can use a large set of sample articles that are well written, the machine learning is done on these, then the search engine would compare against these for each web page if finds when searching. You could easily have a search go a few hundred pages, just have it take 20+ pages of google results. I worked on a project that used NLP (for a startup), I wasn't the one doing most NLP related stuff but it was definitely an interesting experience and the guy who was in charge of the NLP was able to get some pretty interesting results, even from fairly small data-sets.