Building the
evaluation flywheel
How human judgment becomes an evaluator you can run on every change.
- 01Who has built an eval?
- 02Who runs an eval on every change, in CI or
on production traffic? - 03Who has checked an LLM judge against human labels?
- 04Who would let agents merge PRs autonomously?
Two correct answers.
Only one you want
“Where is the station?”
52.3791° N, 4.9003° E.
Two streets to your left, about five minutes on foot.
We came for optimization.
We got stuck on the signal.
Self-improvement runs on the judge's critiques. A wrong judge, wrong improvements.
Evaluation moved from correctness to alignment
You cannot use evals to improve an agent
before the eval is aligned.
What is orq.ai?
Generative AI collaboration platform. One control plane to build, ship, and optimize AI products.
A B2B wholesaler of physical home appliances. The board wants to understand the quality of growth.
Does the answer support the decision?
Pass or fail
The critique carries the nuance
The grey zone
Every evaluation has one.
All plausible. All different.
Every failure has two suspects
Aligning the judge to humans is how you tell the two apart.
The quality-control process
has not changed
In both cases, agreement with the expert decides whether to trust the delegate.
But we are lazy
We don't want to annotate all 50 cases.
Let the LLM judges find the ambiguous cases, then spend human time only there.
The most valuable thing ishuman attention.
Two ways of disagreement
Three judges · three repetitions
Judges disagree with each other, and one judge disagrees with itself.
Both signals exist for every case in the pool.
The judge
sorts the queue
Flagged cases go first. A random sample comes along to catch the cases the judges agreed on and got wrong.
Disagreement gives us
a question
The flagged cases produced no labels. They produced the question the criterion never answered.
One answer exposed another ambiguity
The human answered one boundary question: claims visible in the evidence must be valid. Then the rule went into the evaluator.
4cases the panel split on
8cases the panel split on
We aligned the principle, but not what counts as unsupported.
The grey-zone loop
Disagreement shows where the evaluator still needs a human decision.
Run the jury
Surface instability
Collaborator reads the reasons
Ask one boundary question
Update the evaluator
Clarify the metric first
Valid evidence supports the scoped answer.
Best month net
Visible evidence contradicts the stated definition.
Earlier context still counts
The response established the context earlier in the conversation.
Human labels reveal the judge limits
Jury signals helped us find the unresolved questions. Once the human decisions became labels, we could see which judges reproduced them.
The answer is only the endpoint
Agents require evaluating behavior, rather than final answers.
Fifty Sphere.com cases. Each bar is one run, each block one message, sized by how much context it added.
The Eval Lifecycle
Build with humans
Hard cases and edge cases. A low pass rate is the eval working; one that never fails is too easy.
Guard every commit
Known behavior and core paths. Everything passes, or something broke and the commit stops.
When and where to run evaluations
The same criterion runs in all three, and it holds only while production stays inside the slice humans validated.
Most software will ship
without a human reading it.
Evals in the software factory
Same throughput.
Different factory.
Most pull requests merge untouched.
Most pull requests merge untouched.
The factory reports how much moved. Only an eval tells you which of these you are running.
The evaluation flywheel
Judge disagreement directs attention.
Human judgement sets the boundary.
The resulting eval guards every change.
Run this on your own agent
Q&A
