Six ways a model can pass its tests and still be wrong

A model that scores well has passed a test. Whether it is right depends on whether the test was. Every example below comes from my own projects, where a result looked fine, or looked excellent, and turned out to need a second look.

1. The evaluation itself was wrong

In a student non-completion model, a threshold-based evaluation step had inverted the predicted labels for the class that mattered: the students who would not complete. The numbers describing the model were wrong, and they had been published.

The check review the evaluation code as carefully as the model. A later review found this one: the code was corrected and every figure produced by the inverted step was withdrawn.

From the project: Student Non-Completion Prediction

2. The result was too good

The same project predicted non-completion in three stages as more data arrived. The final stage, with academic results added, reached a ROC AUC of 0.998. Near-perfect scores on a hard human outcome usually mean the features already contain the answer.

The check treat a near-perfect score as a warning. The stage is flagged for a leakage check before it can be used as an early-warning signal, and the stage reported as the useful one is the earlier, more modest one.

3. Preprocessing touched the test set

In a book sales forecast, one title’s weekly series was smoothed before the data was split into training and test periods. A smoothed test period is easier to forecast than the raw sales it stands for, so the forecast error for that title looks better than it should.

The check confirm that no preprocessing step has touched the test period. The issue is stated alongside the result.

From the project: Book Sales Demand Forecasting

4. A model failed silently

In the same project, the weekly XGBoost forecasts stayed flat at about 50 units while actual sales ran between 1,000 and 3,500. Nothing crashed and no warning appeared. The cause was a bug in how features were updated from one forecast step to the next.

The check plot forecasts against actuals, not only an error number. XGBoost was excluded from the comparison rather than reported with a broken score.

5. The test agreed by construction

A PDF compliance harness matched the known answer for all 300 letters in its test corpus. That sounds conclusive. But the letters and the rules were generated from the same specification, with only five defect types seeded, so full agreement was expected before a single test ran.

The check ask what the test data cannot represent. The result is published as evidence that the harness works as designed, not as a measure of how it would perform on real-world drift.

From the project: Automated PDF Compliance Test Harness

6. The brief was wrong

A crime forecasting brief asked for a 70-week seasonal period to be confirmed. The autocorrelation function showed the cycle is 52 weeks. Building to the brief would have produced a model that fitted the instructions and not the data.

The check test the brief’s assumptions before building on them. The 52-week cycle was used, and the change is stated in the write-up.

From the project: Crime Incident Forecasting with SARIMA

And one that is not a bug: complexity that did not help

Two fixes that sound sensible did not improve anything. SMOTE oversampling lowered F1 in two of the three student stages. In the book sales forecast, neither hybrid model beat the single models, and the optimised blend weight came out at zero. Reporting that is as useful as reporting a win: it tells the next person where not to spend time.

A short checklist