AI Agent Evaluation: The 7-Point Checklist Before You Ship

AI agent evaluation has an awkward secret: one green score tells you almost nothing about how the agent will behave on its hundredth run. A framework from deepsense.ai, shared on Hacker News on 20 October 2026, says you should ship only when an agent passes the same test every time, takes an acceptable path, and does it at a cost you can live with.

Key takeaways

• An agent’s output changes from run to run, so a single passing run is one sample, not proof.
• Gate releases on pass^k (all k runs succeed), not pass@k (at least one succeeds).
• Check how the agent reached the answer: tool choice, order, failure handling and cost.
• LLM judges need their own tests against human labels before you trust them.
• The framework ends in a seven-point release checklist you can adopt as-is.

What is the deepsense.ai framework for AI agent evaluation?

On 6 October 2026, Kamil Pluciński, a senior machine learning engineer at deepsense.ai, published “Your Agent Passed the Eval. Should You Ship It?”. His opening line sums up the problem: “An agent can pass an evaluation and still fail the business process it was built to run.”

Agents don’t just write text. They issue refunds, update records and call internal systems. So the bar shifts from “did it answer well?” to “will it do the right thing repeatedly, within limits, at an acceptable price?” The framework splits that question into four layers: the dataset, the execution setup, behavior checks and the metric.

Why does one passing run prove so little?

Language models are nondeterministic: the same input can produce different outputs. Pluciński notes this holds even at temperature 0, the setting meant to make a model’s output predictable. In his words, “One run is one sample from the agent.”

The fix is to run every test case k times and decide how to combine the results. Two metrics matter, and they pull in opposite directions.

Metric	Passes when	As k grows	Use it for
Single run	One attempt succeeds	n/a	Quick smoke tests
pass@k	At least one of k runs succeeds	Rises	Measuring what the agent can do at its best
pass^k	All k runs succeed	Falls	Deciding whether to ship

The arithmetic is unforgiving. Take an agent that succeeds 90% of the time on a task, with runs independent of each other. Its chance of passing five runs in a row is 0.9 to the fifth power, about 59%. You wouldn’t fly with a pilot who lands safely nine times out of ten.

The idea isn’t new. The τ-bench paper (Yao, Shinn, Razavi and Narasimhan, June 2024) introduced pass^k and found that GPT-4o succeeded on fewer than half of its tasks, with pass^8 below 25% in the retail domain. Anthropic’s January 2026 guide, “Demystifying evals for AI agents”, makes the same point and says consistency matters most for customer-facing agents.

How do you build an eval set before you have users?

Start small and write the cases by hand, covering the main use cases and happy paths. Then grow the set in two ways:

• Synthetic generation: a model drafts new cases. A human must review them, because a flawed test set can share the agent’s blind spots.
• Simulated users: a model plays the customer across multi-turn chats. Pluciński calls this too noisy to gate releases.

Once real traffic arrives, fold it in. Early pass rates will look bad, and that’s the point. He also warns against scaling the dataset before the system works reliably for roughly 75% of real users. Anthropic’s guide agrees on size: “20-50 simple tasks drawn from real failures is a great start.”

Should you grade the outcome or the path?

This is where the experts differ. Anthropic advises grading what the agent produced rather than the exact steps, since rigid tool-call orders make brittle tests and agents often find valid routes nobody planned for.

Pluciński agrees that many paths can be valid, but argues “a 100% completion rate says nothing on its own about how the agent got there.” His middle ground: describe acceptable behavior and check four things.

1. Tool choice: the right tool, with arguments taken from the conversation.
2. Order: steps that must happen in sequence, such as checking identity before a refund.
3. Failure handling: how it copes with timeouts or malformed data.
4. Cost and latency: turns, tokens, time and dollars per run, reported next to the score.

Our view: if the agent can move money or change records, check the steps that must happen and leave the rest free. That keeps tests firm where it matters without breaking every time the agent finds a new route.

Can you trust an LLM judge to score your agent?

Only after you test the judge. Pluciński wants metrics that are bounded (a fixed range) and monotone (a better run always scores higher). Judges using 1-10 scales often fail that second test, so he prefers lots of yes/no checks, counted up.

Validate each judge on a held-out set labeled by humans, and measure agreement with Cohen’s kappa, a statistic that adjusts for agreement by chance. His “Business Utility” score also penalizes variance: an agent that scores a steady 70% beats one that swings between 0% and 100%.

What does this look like on a real project?

The worked example is a technical-support agent for a supply-chain system. Every test case came from production. When the agent failed, the team kept the conversation up to the bad step, rewrote that step correctly and moved the case to a regression suite once it passed. That suite grew to about 1,000 cases.

The payoff came during a major restructuring. The suite replaced weeks of manual checks, the refactored system matched its earlier performance, and it shipped after code review. If you’re picking an orchestration stack for this kind of work, our guide to LangChain vs LangGraph covers the tradeoffs.

The 7-point release checklist

1. Coverage: every claimed capability is tested, and real production cases outnumber generated ones.
2. Same harness: the tools and permissions you tested match what ships.
3. Clean environment: each case runs in a seeded, isolated sandbox, with state compared afterwards.
4. Repeats: each case runs several times and is gated on pass^k.
5. Path, not just outcome: checks cover how the agent got there.
6. Validated judges: each LLM judge was tested against human labels.
7. Readable metrics: ship thresholds are set in advance, along with what to fix if you miss them.

Passing the list doesn’t make an agent safe. It makes the decision informed: you know what was tested, what wasn’t, and which risks the business is accepting.

What happens next

There is still no single standard. Academic work such as the “Beyond Task Completion” framework, presented at the AGENT workshop at ICSE 2026 in April 2026, also argues that pass/fail task metrics hide behavioral variability. A 2026 study, “On the Reliability of Computer Use Agents”, goes further and says pass^k is itself an aggregate that can hide per-task changes, proposing paired statistical tests over repeated runs.

The direction is clear, though. Repeated runs, process checks and tested judges are becoming the minimum for teams that put agents in front of customers. For a wider view of where evals fit, see our breakdown of the 7-level maturity model for production-grade AI agents.

FAQ

What is the difference between pass@k and pass^k?

pass@k counts a task as solved if at least one of k attempts works, so it rises as k grows. pass^k needs every attempt to work, so it falls as k grows, which makes it the better test of reliability.

How many test cases do I need to start evaluating an agent?

Fewer than you think. Anthropic suggests 20 to 50 simple tasks drawn from real failures. deepsense.ai advises growing the set with production traffic once the agent works for about 75% of real users.

Are LLM-as-judge scores reliable?

Only once you’ve checked them. Use many narrow yes/no judges instead of a 1-10 scale, and measure their agreement with human labels on a held-out set using Cohen’s kappa.

Your next step: take your 20 most important agent tasks, run each five times and look at pass^5. That number is what your customers will actually get.