What to test in a model you can't fully specify

Software testing starts from a specification: given this input, expect that output. A model breaks the pattern, because nobody can write down the right answer for every input; if they could, they would not need the model. That does not make models untestable. It changes what you test, and much of it is familiar to anyone who has tested software.

1. Test against a baseline, not a target

Without an expected answer, the first question is whether the model beats something trivial. In a five-class movie review task, a bidirectional LSTM reached 38.5% test accuracy. That sounds poor until it is set against a majority-class baseline of about 23%. The same comparison exposed two models that had failed outright: an LSTM and a GRU both scored 23.1%, because they had learned to predict a single class.

The check always report a trivial baseline next to the model. A score at baseline level is a failed model, whatever the number looks like.

From the project: Movie Review Sentiment Analysis

2. Build a corpus where you do know the answer

If real data has no known answers, make some. A PDF compliance harness was tested on 300 synthetic letters: 210 valid and 90 with one seeded defect each, so every verdict could be scored. The same project is honest about the limit: the corpus and the rules came from the same specification, so full agreement shows the harness works as designed, not how it handles real-world drift.

The check seed known defects and measure what is caught. Then state what the seeded set cannot represent.

From the project: Automated PDF Compliance Test Harness

3. Separate the parts that can be specified

Most systems built around a model are not all model. A private document-AI prototype answers most data questions with deterministic logic, a first tier of 16 patterns and a schema engine, and only sends questions that need reasoning to the language model. Those deterministic tiers were validated against 96 test queries like any other software.

The check draw the line between deterministic and probabilistic components, and test the deterministic side to a full specification.

From the project: Private Document Analysis AI

4. Test that the result is not set by a setting

In an engine anomaly project, One-Class SVM and Isolation Forest each flagged about 5% of readings. That figure was not a finding: their nu and contamination settings decide roughly what share is flagged. Only the IQR rule’s 2.16% was not fixed in advance.

The check change the setting and watch the result. If the output moves in step with a parameter you chose, the parameter is the result.

From the project: Ship Engine Anomaly Detection

5. Test reproducibility

A result that only appears once is an anecdote. The PDF harness verdicts were reproduced in three runs. At the other end, a spam classifier’s tuned model was evaluated over 10 runs, which showed it scoring below the simple baseline rather than above it.

The check re-run with different seeds and splits. Report the spread, not the best run.

6. Test the evaluation code

In a student non-completion model, a threshold-based evaluation step inverted the predicted labels for the class that mattered. The figures it produced were published, and they were wrong. A later review caught it, and the affected figures were withdrawn.

The check hand-check the evaluation on a few rows: compute the confusion matrix yourself and confirm it matches.

From the project: Student Non-Completion Prediction

The tester’s view