I think that the Kaggle competition you describe has essentially the same structure as the Kaggle competition outlined in the blog post, and is susceptible to the same potential hazard of overfitting to the validation score.
Here by validation score I mean the combination of both your own cross-validation procedure on the training data, and the validation score that kaggle calculates and displays on the leaderboard while the competition is running.
I imagine if the essay scoring competition had been extended to run for another 10 years before the final scores on held-back test data were reported we would see a similar trend of overfitting in most if not all of the entries.
edit: I might be underestimating the sophistication of your approach or the approaches of others.
At least in the Kaggle competitions I have personally entered, I have used my own cross-validation scores and the Kaggle leaderboard scores to tune my approaches, without properly taking into account that this breaks the validity of the subsequent validation procedures, since I have no way of validating my tuning.
Yes - even if you are only allowed look at the validation set twice a day, you can still overfit it, with sufficient days.
But thats hardly a criticism of Kaggle.
Its the responsibility of the person building the ML model to take steps to avoid overfitting. Even if competitors fail to take these steps, the fact that Kaggle has a built in validation-set helps avoid overfitting.
Yes, with enough time it would still be possible to overfit, but I don't see how that is Kaggle's fault.
If anything, they've a framework to discourage overfitting, not encourage it.
Comments
I think that the Kaggle competition you describe has essentially the same structure as the Kaggle competition outlined in the blog post, and is susceptible to the same potential hazard of overfitting to the validation score.
Here by validation score I mean the combination of both your own cross-validation procedure on the training data, and the validation score that kaggle calculates and displays on the leaderboard while the competition is running.
I imagine if the essay scoring competition had been extended to run for another 10 years before the final scores on held-back test data were reported we would see a similar trend of overfitting in most if not all of the entries.
edit: I might be underestimating the sophistication of your approach or the approaches of others.
At least in the Kaggle competitions I have personally entered, I have used my own cross-validation scores and the Kaggle leaderboard scores to tune my approaches, without properly taking into account that this breaks the validity of the subsequent validation procedures, since I have no way of validating my tuning.
Yes - even if you are only allowed look at the validation set twice a day, you can still overfit it, with sufficient days.
But thats hardly a criticism of Kaggle.
Its the responsibility of the person building the ML model to take steps to avoid overfitting. Even if competitors fail to take these steps, the fact that Kaggle has a built in validation-set helps avoid overfitting.
Yes, with enough time it would still be possible to overfit, but I don't see how that is Kaggle's fault.
If anything, they've a framework to discourage overfitting, not encourage it.