Cfrgtkky

I hope this is not a stupid question. Let us say I have a data generation process that is quite stationary and I do not care about arriving at generalizable knowledge but more about accurate predictions. Would it be acceptable in this scenario to overfit a powerful model (e.g. random forest => fully saturated-ish model) by refreshing it daily using all retrospective data and using it to predict next day’s dependent variable?

It will eventually be a balance that you need to test (e.g cross validation).

• If you are too conservative then you won't capture the model and the predictions will be bad.


• If you are too liberal then you will capture too much of the noise (aside from the model) and the predictions will be bad.

It can be that a slightly more conservative model than the 'real' model (e.g the true model is a polynomial of order 5 and the optimal model to fit it is of order 4) works better, but this depends entirely on the specific circumstances and needs to be tested on a case-by-case basis. However, in general it is better to add some little bias (it will reduce the variability, if done correctly ).

In case your question is about adding new data to the data that you used to train your model, then I would guess that this is rarely gonna be a problem. In most cases adding more data should make the model better unless the modelfit has the behaviour that it is not gonna improve with more data (e.g. when the model is not constant in time, but then the predictions are not going be good anyway).

We say that model overfitts when it has good performance on training data, but not on unseen data. It is not a statement about data generating process, but about the sample that you use for training, versus any other sample that can be drawn. So if model has good predictive performance on unseen data, it does not overfit.

Overfitting would not be a problem if you didn't want to make predictions on unseen data and didn't want to make any conclusions about it given the model. You are right that if you can be perfectly sure that the future data would be identical to your training sample, then it wouldn't matter, but I can't imagine any scenario where you could be sure about it. Notice that even if you had perfectly representative sample, or population data, it still can happen that the phenomenon of interest would change over time and the past data wouldn't be relevant any more.

See also the Which model is better: One that overfits or one that underfits? thread.

Overfitting is bad, because it means the model you learned from your training data may not work well for new data points. You can imagine a perfectly overfit model that simply memorizes each training point and returns the appropriate output. When confronted with data that it wasn't trained on, it outputs a random number. You could train a model like this on a ton of retrospective data, but unless you get identical data tomorrow, you'll do no better than random. I suppose an approach like this could work with a limited and discrete input space, but you don't really need machine learning models for that anyway.