No change to a model, prompt, or pipeline is judged by anecdote. A fixed, versioned eval set is the regression suite for behavior.
First recorded July 17, 2026 · Last revised July 17, 2026
Rationale
Single impressive outputs are the weakest possible evidence about a system. Model and prompt changes routinely improve the examples you looked at while regressing the ones you did not.
The doctrine: no change to a model, prompt, or pipeline is judged by anecdote. Keep a fixed, versioned evaluation set that reflects real usage; score every candidate change against it; compare distributions, not favorites. When a new failure appears in production, it becomes a new eval case before it becomes a fix.
This is the AI analogue of “no merge without tests” — the eval set is the regression suite for behavior. Vibes still matter for discovering what to measure; they are simply not admissible as the verdict.