"It filters out the posts that are least interesting to you and leaves only the stuff that is relevant."
How well does this service work in practice? I imagine telling spam and non-spam emails apart is far easier then "predicting" which blog post I will like.
Hi Maro, I invite you to try it out yourself. I've released 500 "hackernews" invites so everyone can give it a try.
You're right, it's a lot easier to filter out spam emails with the word "V1AGRA" than it is to filter out news posts. You'll want to take some time to train the filter properly before you unsubscribe from your junk feed.
Is there anything fancier than a naive Bayes classifier going on?
Do you do any calculus on the relative values of false positives and false negatives? It seems that a person with a large number of feeds would put a greater importance on limiting the number of false positives (non-interesting messages classified as interesting by your program), whereas a person with a small number of feeds would be more worried about false negatives (interesting stories classified as uninteresting).
Also, see this comment in this thread: http://news.ycombinator.com/item?id=435837 . For something like this, I doubt a machine learning algorithm will work as well as social algorithms. Web search is a very similar problem (find the most interesting link given input from the user) and social algorithms won out. Of course, the kind and quality of input that you are getting from the user is different. However, your problem is much less well-defined (finding interesting things in general instead of interesting things related to a particular topic).
Interesting points! We are using a naive Bayes classifier right now, and we're working with some social aspects, especially as they relate to Latent Semantic Analysis. We've considered doing a mini Netflix Prize for further improving our algorithm :-).
We've also done a lot of tweaking to get the current filter as smart as possible (thanks PG for your thoughts in A Plan for Spam!) Tweaked things like the number of interesting words to count, weighting of title vs body, etc.
We'd love to talk with some machine learning experts, drop me a note if you are one!
I'm not an expert, just interested in the subject.
I thought of an idea, and this is completely fresh so take it with a grain of salt, but you could use social data from all users subscribed to the same feed to modify the prior probability that any given post was interesting, and then use each individual's data to modify the conditional posterior probability terms.
I'm not sure how effective it would be, but it would create a hybrid sort of classifier that might be interesting.
I'm studying natural language processing this quarter, so I'll let you know in three months if I come up with something more clever.
Comments
"It filters out the posts that are least interesting to you and leaves only the stuff that is relevant."
How well does this service work in practice? I imagine telling spam and non-spam emails apart is far easier then "predicting" which blog post I will like.
Hi Maro, I invite you to try it out yourself. I've released 500 "hackernews" invites so everyone can give it a try.
You're right, it's a lot easier to filter out spam emails with the word "V1AGRA" than it is to filter out news posts. You'll want to take some time to train the filter properly before you unsubscribe from your junk feed.
Is there anything fancier than a naive Bayes classifier going on?
Do you do any calculus on the relative values of false positives and false negatives? It seems that a person with a large number of feeds would put a greater importance on limiting the number of false positives (non-interesting messages classified as interesting by your program), whereas a person with a small number of feeds would be more worried about false negatives (interesting stories classified as uninteresting).
Also, see this comment in this thread: http://news.ycombinator.com/item?id=435837 . For something like this, I doubt a machine learning algorithm will work as well as social algorithms. Web search is a very similar problem (find the most interesting link given input from the user) and social algorithms won out. Of course, the kind and quality of input that you are getting from the user is different. However, your problem is much less well-defined (finding interesting things in general instead of interesting things related to a particular topic).
Interesting points! We are using a naive Bayes classifier right now, and we're working with some social aspects, especially as they relate to Latent Semantic Analysis. We've considered doing a mini Netflix Prize for further improving our algorithm :-).
We've also done a lot of tweaking to get the current filter as smart as possible (thanks PG for your thoughts in A Plan for Spam!) Tweaked things like the number of interesting words to count, weighting of title vs body, etc.
We'd love to talk with some machine learning experts, drop me a note if you are one!
I'm not an expert, just interested in the subject.
I thought of an idea, and this is completely fresh so take it with a grain of salt, but you could use social data from all users subscribed to the same feed to modify the prior probability that any given post was interesting, and then use each individual's data to modify the conditional posterior probability terms.
I'm not sure how effective it would be, but it would create a hybrid sort of classifier that might be interesting.
I'm studying natural language processing this quarter, so I'll let you know in three months if I come up with something more clever.