The problem: the model grades itself high
Ask a model to grade its own answer from 0 to 10 and it almost always returns a high number. The score looks like measurement, but it came from the same process that produced the answer. If the answer was wrong, the score tends to be wrong in the same way.
The four levels
- Deterministic test. Code or a rule that always gives the same answer. The calculation matches or it does not.
- Environment feedback. The real world answers: the code runs, the email is delivered, the client replies.
- Isolated model judge. Another model grades the result against written criteria, without seeing how it was produced.
- Self-assessment. The same model judges its own work. As proof, it is worth almost nothing.
How to pick the level for each step
| Type of step | Example | Level that fits |
|---|---|---|
| Format or fixed rule | Valid JSON, a sum that adds up, a link that exists | Deterministic test |
| Action in the world | Email sent, deploy live, payment approved | Environment feedback |
| Subjective quality | Tone of voice, clarity, brand fit | Isolated model judge, with criteria |
Self-assessment is left out of the table on purpose. It can improve a text, but it cannot prove the text is right.
Real examples
- In cognitive-claude, each session's cost is calculated by a script that reads the logs. Run it again and the number is the same.
- In Branding OS, a validator checks the brand rules before a piece reaches a person.
- In experiment #003, two auditors ran in parallel without seeing each other's notes and found 16 issues a single one would have missed.
- This site runs eight tests on its own code on every visit, on the This site, inside page.
Frequently asked questions
Does asking the AI to review its own answer help?
It can improve the form. It does not prove the answer is right.
Is a model judge reliable?
More than self-assessment, as long as it has written criteria and does not see how the answer was produced. Less than a test.
Isn't checking everything expensive?
Each level up takes more work to build. Start with the steps where an error costs the business the most.
Want to apply this to your operation?
The first call is a free one-hour diagnosis.