The user's algorithms are going to have try to control for overfitting on their own. Try to make it hard for the users to evaluate the model on the same data they trained it on. Also educate them that in-sample results are meaningless.
I guess I'd want an easy way to get data in chunks for testing. You can either do it as a rolling window or regular cross validation. (Rolling would be train on N years of data, test on the next year. Then drop the last year and add another year and repeat. CV would be just divide in 10 chunks, train on all combination of 9 and test on the 10th.) It would be good to support both.
Comments
The user's algorithms are going to have try to control for overfitting on their own. Try to make it hard for the users to evaluate the model on the same data they trained it on. Also educate them that in-sample results are meaningless.
I guess I'd want an easy way to get data in chunks for testing. You can either do it as a rolling window or regular cross validation. (Rolling would be train on N years of data, test on the next year. Then drop the last year and add another year and repeat. CV would be just divide in 10 chunks, train on all combination of 9 and test on the 10th.) It would be good to support both.