What is Overfitting?
Fitting noise instead of signal, so a model looks brilliant on old data and fails live.
An overfitted model has memorised the training period, including its accidents. It scores well in-sample and poorly out-of-sample, which is why the walk-forward test exists.
Defences are boring: fewer features, regularisation, held-out slices and a gate that refuses to publish a model with no measured edge.
Related terms
- Walk-forward backtestOur model
Testing the model only on data after each training window, the way it would have been used.
- Model gateOur model
The rule that hides a forecast when the model could not beat a coin flip on it.
- Brier scoreOur model
Mean squared error of probability forecasts; 0.25 is the score of a coin flip.