Could you explain what you mean by its 'designed not to allow for this'?
That seems a very sweeping criticism.
I took part in the last essay scoring competition on Kaggle.
In that competition, the data set was the original text of the essays, scored by human raters. A lot of the effort in our submission was in developing features. We didn't spend much time on our entry, but we did use domain specific features (e.g. writing code to calculate http://en.wikipedia.org/wiki/Gunning_fog_index), implementing spell checking, etc.
That entire competition took place within the Kaggle framework.
Furthermore, in addition to the training-set (which we did most of our development with, using cross validation) and the test-set, on which the final scores were calculated, there was also a validation-set, from which the public scoreboard was calculated. We could only submit our predictions for the validation set twice a day, and were told only our overall accuracy on the set -- not given the scores of each essay.
The role of the validation set in Kaggles framework was surely to prevent errors such as overfitting. In fact, we had a situation where our cross-validation accuracy increased, and validation-set accuracy decreased, because of a bug where our feature selection didn't properly implement cross-validation (similar to the linked article).
The difference between the cross-validation and validation-set scores drew our attention to this problem.
That all happened within the Kaggle framework.
So I don't really understand criticisms as general as 'Kaggle is doing ML wrong'?
I think that the Kaggle competition you describe has essentially the same structure as the Kaggle competition outlined in the blog post, and is susceptible to the same potential hazard of overfitting to the validation score.
Here by validation score I mean the combination of both your own cross-validation procedure on the training data, and the validation score that kaggle calculates and displays on the leaderboard while the competition is running.
I imagine if the essay scoring competition had been extended to run for another 10 years before the final scores on held-back test data were reported we would see a similar trend of overfitting in most if not all of the entries.
edit: I might be underestimating the sophistication of your approach or the approaches of others.
At least in the Kaggle competitions I have personally entered, I have used my own cross-validation scores and the Kaggle leaderboard scores to tune my approaches, without properly taking into account that this breaks the validity of the subsequent validation procedures, since I have no way of validating my tuning.
Yes - even if you are only allowed look at the validation set twice a day, you can still overfit it, with sufficient days.
But thats hardly a criticism of Kaggle.
Its the responsibility of the person building the ML model to take steps to avoid overfitting. Even if competitors fail to take these steps, the fact that Kaggle has a built in validation-set helps avoid overfitting.
Yes, with enough time it would still be possible to overfit, but I don't see how that is Kaggle's fault.
If anything, they've a framework to discourage overfitting, not encourage it.
Being a machine learning (and Kaggle competitions) novice, can you explain what it means that your feature selection didn't properly implement cross-validation?
What it means is that we did something stupid. I'm not sure how novice you are, so I'll take it from the top.
So, lets say you have a large training set - a large number of randomly chosen example essays, and a score for each essay.
You want to develop a model that will predict/guess scores of other essays, in future.
Your model is going to be evaluated against a set of essays - the test set - which you do not see until after the competition is over (at which point you may no longer make changes to your model).
Now, in an ideal world, we'd be able to take just one look at the training set, and instantly learn the best way there is to predict similar essay scores in future. But, in practice, that's not going to be the case. We, the designers of the ML approach, initially don't know how to build essay grading systems. So we are going to need to do a development process, where we think, then try a particular approach, then see how well it works. And we have to iterate on that process.
So, the key bit there, is that we need to see whether a particular approach is working. We can't check the test-set - we don't get access to that until the end of the competition. So, instead, we take the training set, and partition it up into 2 parts, lets call them A and B. Essays from both A and B are in the training set, as we've just said, hence we know the scores for the essays of both A and B.
We now train our model with essays from A, without it being allowed look at essays from B. Then, afterwards, we have it make predictions/guesses of the scores of essays in B, and see how well it does.
This allows us to evaluate how well our modelling approach is doing.
In practice, we repeat this procedure for many different partitions of the training set, into many different A and Bs.
So, thats cross validation.
The key thing to realize is that if we use any information from B, when training A, its not really a fair evaluation. The model we make from A will have been contaminated with information from B.
Which is simple enough.
The thing is, in practice, before doing any of this, people generally go through a conceptually separate process, of deciding what features their model is going to use - 'Feature Selection'. Some part of this process is intuitive, but there are also algorithms people use, to help them do feature selection.
The wrong way to do Feature Selection, is to run your Feature Selection algorithms on the entire training set, and then afterwards train your model on one partition A of the training set, and test it on separate essays from B.
This is wrong, because the essays from B were seen in the feature selection process. Sometimes that doesn't matter - if you are dealing with large data sizes, etc. But on the Kaggle essay competition, in particular because we had rather a large amount of features (e.g. bag-of-words n-grams, parts-of-speech n-grams), relative to the number of training examples, it certainly did matter.
As a result, we saw much higher scores on the training set, than we would have gotten on the test set.
The right way to do things is to do any automatic feature selection only on the training partition of the training set, A. One way to do this is to build any algorithmic feature selection into your process after the point you partition into A and B, during cross validation. But its very easy, when hacking on a competition solution, over a weekend, to accidentally put things in the wrong order, in the codebase.
For this reason, if you have limited training data, its generally a good idea, at the very start of the process, to take some training data, and put it somewhere completely separate, until you think your algorithm is ready to go, and then use it as a final test. People often call this the validation-set.
Comments
Could you explain what you mean by its 'designed not to allow for this'? That seems a very sweeping criticism.
I took part in the last essay scoring competition on Kaggle.
In that competition, the data set was the original text of the essays, scored by human raters. A lot of the effort in our submission was in developing features. We didn't spend much time on our entry, but we did use domain specific features (e.g. writing code to calculate http://en.wikipedia.org/wiki/Gunning_fog_index), implementing spell checking, etc.
That entire competition took place within the Kaggle framework.
Furthermore, in addition to the training-set (which we did most of our development with, using cross validation) and the test-set, on which the final scores were calculated, there was also a validation-set, from which the public scoreboard was calculated. We could only submit our predictions for the validation set twice a day, and were told only our overall accuracy on the set -- not given the scores of each essay.
The role of the validation set in Kaggles framework was surely to prevent errors such as overfitting. In fact, we had a situation where our cross-validation accuracy increased, and validation-set accuracy decreased, because of a bug where our feature selection didn't properly implement cross-validation (similar to the linked article).
The difference between the cross-validation and validation-set scores drew our attention to this problem.
That all happened within the Kaggle framework.
So I don't really understand criticisms as general as 'Kaggle is doing ML wrong'?
I think that the Kaggle competition you describe has essentially the same structure as the Kaggle competition outlined in the blog post, and is susceptible to the same potential hazard of overfitting to the validation score.
Here by validation score I mean the combination of both your own cross-validation procedure on the training data, and the validation score that kaggle calculates and displays on the leaderboard while the competition is running.
I imagine if the essay scoring competition had been extended to run for another 10 years before the final scores on held-back test data were reported we would see a similar trend of overfitting in most if not all of the entries.
edit: I might be underestimating the sophistication of your approach or the approaches of others.
At least in the Kaggle competitions I have personally entered, I have used my own cross-validation scores and the Kaggle leaderboard scores to tune my approaches, without properly taking into account that this breaks the validity of the subsequent validation procedures, since I have no way of validating my tuning.
Yes - even if you are only allowed look at the validation set twice a day, you can still overfit it, with sufficient days.
But thats hardly a criticism of Kaggle.
Its the responsibility of the person building the ML model to take steps to avoid overfitting. Even if competitors fail to take these steps, the fact that Kaggle has a built in validation-set helps avoid overfitting.
Yes, with enough time it would still be possible to overfit, but I don't see how that is Kaggle's fault.
If anything, they've a framework to discourage overfitting, not encourage it.
Being a machine learning (and Kaggle competitions) novice, can you explain what it means that your feature selection didn't properly implement cross-validation?
What it means is that we did something stupid. I'm not sure how novice you are, so I'll take it from the top.
So, lets say you have a large training set - a large number of randomly chosen example essays, and a score for each essay. You want to develop a model that will predict/guess scores of other essays, in future.
Your model is going to be evaluated against a set of essays - the test set - which you do not see until after the competition is over (at which point you may no longer make changes to your model).
Now, in an ideal world, we'd be able to take just one look at the training set, and instantly learn the best way there is to predict similar essay scores in future. But, in practice, that's not going to be the case. We, the designers of the ML approach, initially don't know how to build essay grading systems. So we are going to need to do a development process, where we think, then try a particular approach, then see how well it works. And we have to iterate on that process.
So, the key bit there, is that we need to see whether a particular approach is working. We can't check the test-set - we don't get access to that until the end of the competition. So, instead, we take the training set, and partition it up into 2 parts, lets call them A and B. Essays from both A and B are in the training set, as we've just said, hence we know the scores for the essays of both A and B.
We now train our model with essays from A, without it being allowed look at essays from B. Then, afterwards, we have it make predictions/guesses of the scores of essays in B, and see how well it does.
This allows us to evaluate how well our modelling approach is doing.
In practice, we repeat this procedure for many different partitions of the training set, into many different A and Bs.
So, thats cross validation.
The key thing to realize is that if we use any information from B, when training A, its not really a fair evaluation. The model we make from A will have been contaminated with information from B.
Which is simple enough.
The thing is, in practice, before doing any of this, people generally go through a conceptually separate process, of deciding what features their model is going to use - 'Feature Selection'. Some part of this process is intuitive, but there are also algorithms people use, to help them do feature selection.
The wrong way to do Feature Selection, is to run your Feature Selection algorithms on the entire training set, and then afterwards train your model on one partition A of the training set, and test it on separate essays from B.
This is wrong, because the essays from B were seen in the feature selection process. Sometimes that doesn't matter - if you are dealing with large data sizes, etc. But on the Kaggle essay competition, in particular because we had rather a large amount of features (e.g. bag-of-words n-grams, parts-of-speech n-grams), relative to the number of training examples, it certainly did matter.
As a result, we saw much higher scores on the training set, than we would have gotten on the test set.
The right way to do things is to do any automatic feature selection only on the training partition of the training set, A. One way to do this is to build any algorithmic feature selection into your process after the point you partition into A and B, during cross validation. But its very easy, when hacking on a competition solution, over a weekend, to accidentally put things in the wrong order, in the codebase.
For this reason, if you have limited training data, its generally a good idea, at the very start of the process, to take some training data, and put it somewhere completely separate, until you think your algorithm is ready to go, and then use it as a final test. People often call this the validation-set.
Not that novice, but your explanation was exquisite, thanks!