"Kaggle is doing ML wrong. It has the opportunity to teach some people about these perils":
We have made several (reasonably successful) attempts to encourage participants to use cross validation (including a competition called "Don't Overfit: http://www.kaggle.com/c/overfitting). The incidence of over fitting has decreased dramatically since the site first launched - but it hasn't been eradicated. If people have ideas about how we can do better at educating our community, we're all ears. (Unsurprisingly, once someone makes the over fitting mistake once, they never make it again.)
As an aside, one of the key ways we try and promote education is the "How I did it" posts on the Kaggle blog (blog.kaggle.com).
"Black box machine learning":
Actually our number one learning on ML best practice is that for a competition with rich data (like the essay scoring competition mentioned elsewhere in the comments), feature engineering (sometimes known as variable creation and variable selection) is WAY more important than whether or not one uses SVM or random forest.
(Sigh. The point of my joking remark is not to dump on Anthony Goldbloom, it's that he's slightly more involved than "working there". He founded Kaggle).
Comments
(Disclaimer: I work at Kaggle.)
"Kaggle is doing ML wrong. It has the opportunity to teach some people about these perils": We have made several (reasonably successful) attempts to encourage participants to use cross validation (including a competition called "Don't Overfit: http://www.kaggle.com/c/overfitting). The incidence of over fitting has decreased dramatically since the site first launched - but it hasn't been eradicated. If people have ideas about how we can do better at educating our community, we're all ears. (Unsurprisingly, once someone makes the over fitting mistake once, they never make it again.)
As an aside, one of the key ways we try and promote education is the "How I did it" posts on the Kaggle blog (blog.kaggle.com).
"Black box machine learning": Actually our number one learning on ML best practice is that for a competition with rich data (like the essay scoring competition mentioned elsewhere in the comments), feature engineering (sometimes known as variable creation and variable selection) is WAY more important than whether or not one uses SVM or random forest.
> (Disclaimer: I work at Kaggle.)
So modest.
(Sigh. The point of my joking remark is not to dump on Anthony Goldbloom, it's that he's slightly more involved than "working there". He founded Kaggle).