Leakage is when information that will not be available at prediction time finds its way into training or evaluation. The model is not cheating on purpose; the pipeline is handing it the answer. It is the most common reason a model that tested brilliantly disappoints in use, and every example below comes from my own projects.
- Leakage inflates test results; it never shows up as an error.
- Ask of every feature: would I know this at the moment I need the prediction?
- Ask of every preprocessing step: did it see the test data?
1. A feature that arrives after the event
In a bank marketing model, call duration was by far the strongest predictor of whether a contact subscribed to a term deposit. Gradient Boosting reached a test ROC AUC of 0.9485. But call duration is only known once the call has ended, so it cannot help decide whom to call. The score describes what successful calls looked like; it does not score leads in advance.
The check for every feature, write down when its value becomes known. Anything known only after the outcome belongs in an explanation, not a predictor.
From the project: Decision Trees and SHAP: Bank Marketing
2. A feature that already contains the outcome
A student non-completion model was built in three stages as more data arrived. With academic results added, the final stage reached a ROC AUC of 0.998. On a hard human outcome, a near-perfect score usually means a feature is a proxy for the answer itself.
The check treat any near-perfect score as a suspected leak. Remove the newest feature group and see how far the score falls before trusting it.
From the project: Student Non-Completion Prediction
3. Preprocessing that saw the test period
In a book sales forecast, one title’s weekly series was smoothed before the data was split into training and test periods. A smoothed test period is easier to forecast than the raw sales it stands for, so that title’s forecast error looks better than it should.
The check split first, then transform. Smoothing, scaling, imputation and encoding should be fitted on training data only and applied to the test data afterwards.
From the project: Book Sales Demand Forecasting
4. The target inside an exploratory step
In a car price project, PCA was fitted on all 190 vehicles with price among the inputs. That is fine for a picture of the data, but the components carry information about the target, so they cannot be used as model inputs. The project treats the PCA and t-SNE views as exploratory only for exactly that reason.
The check keep the target out of anything that will feed a model, including dimensionality reduction and feature selection.
From the project: Automobile Price Prediction
5. Choosing the model on the test set
The quietest form of leakage is in model selection. In a spam classifier, grid-search settings were picked by test accuracy, which makes the reported best result optimistic. A bank churn model had the same issue with its hyperparameters. Every time the test set is used to make a choice, it stops being an honest estimate of performance on new data.
The check hold out a validation split, or use cross-validation, for every choice. Touch the test set once, at the end.
From the project: Neural Network Design and Hyperparameter Search
A leakage checklist
- Is every feature known at the moment the prediction is needed?
- Does any feature encode the outcome, directly or by proxy?
- Was every transformation fitted on training data only?
- Is the target excluded from exploratory steps that feed the model?
- Was the test set used for any decision before the final evaluation?
- For time series: is the split by time, with no future values in the training window?
RJ Data Voyage builds and tests data science and AI systems for teams that need to trust the results. If you want a model checked for leakage before it goes live, get in touch.