method · design criteria
Six decisions to clarify before giving a task to an AI system.
In my projects, I separate three questions: which work to give the model, how to measure the result and where a human decision is needed. τ²-bench and the quote review interface show two applications of these criteria.
Before choosing a model, define the input, expected result and exceptions. Some steps can be deterministic; others need language interpretation or human confirmation.
A useful comparison starts with real examples and a verifiable success criterion. On τ²-bench, for instance, the final score alone did not show whether the agent had communicated poorly or changed a booking incorrectly.
A rule or service call may be enough for a predictable step. A model helps when free text needs interpretation or cases are hard to describe with rules.
When the sequence is known, code can control it and call the model only where needed. Agentic search is more suitable when the next step depends on newly found information.
for each step: code, model or person; decide when the model belongs inside a fixed workflow
In a chain of steps, even rare errors accumulate. The chart assumes independent steps that each succeed 98% of the time: after ten steps, complete success is about 82%. An effective intermediate check can catch some failures.
theoretical example: the value of checks depends on how many errors they catch
Tools should also be chosen by their effects: reading a source differs from changing data or sending a message. The quote prototype shows a point where a person can check output before it is sent.
Langfuse traces link calls, results and judgments. I used them to analyze errors in τ²-bench and to bring automated and human evaluations together in the review prototype.
A useful test defines the outcomes that matter in advance: task completion, errors, time and cost. Behavior on unanswerable cases also needs observation.
Repeating the same cases reveals variation. On τ²-bench, four runs of the same version scored from 74% to 86%; one run would have hidden that range.
An automated judge can check rules, data or text quality; a person can step in where context must be interpreted or release must be authorized. The choice depends on the output and the cost of an error.
In the quote prototype, an automated judge evaluates the quote and message. A person sees the same case and decides whether to approve or correct it before sending.
Both judgments are attached to the same Langfuse trace. A disagreement marks a case for review without confusing an agent error with an evaluation error.
Where an objective check exists, inspect the final state: for example, whether a record has the expected values or a code test passes. The check should be separate from the action being evaluated.
If the criterion depends on context or personal responsibility, the model can prepare a proposal but the decision stays with the user. Corrections show what deserves attention in later tests.
For low-impact tasks, sample review may suffice when errors are easy to find and fix. Review frequency needs to be measured for the actual use case.
Langfuse traces help explain what the system did. To review an output, a person needs the context and result without navigating every technical call.
In the human-review prototype, I show one case at a time: the original message, corrected proposal and space for a judgment. Any correction becomes useful data when analyzing disagreement with the automated judge.
one case can be read by the automated judge, reviewer and trace analyst
In the prototype, human judgments are linked to automated evaluations in Langfuse. Disagreements help select new test cases; any change to the judge then needs a separate check.