The distinction between in-sample and out-of-sample performance is the most fundamental dividing line in quantitative strategy validation.
In-sample (IS) performance is measured on the historical data used to develop, optimize, or fit the strategy. Because the strategy was designed using this data, it tends to fit it well — but this fit reflects the specific noise patterns of the training period, not necessarily genuine predictive structure that will persist.
Out-of-sample (OOS) performance is measured on data that was held out and not used in any way during development. OOS performance is a more honest estimate of how the strategy will perform in live trading.
IS/OOS divergence
A large gap between IS and OOS performance (high IS Sharpe, low OOS Sharpe) is the signature of backtest overfitting. The ratio of OOS to IS Sharpe provides a rough measure of how much the IS result can be trusted as a forward-looking estimate.
Techniques that produce more reliable performance estimates include walk-forward analysis and purged cross-validation.