Most AI agents fail on the way from the demo to customers, and the fix is usually engineering, not a bigger model. That is the argument behind a new field guide to production-grade AI agents on the Stack Overflow Blog, and it comes with a seven-level maturity model you can check your own system against.
Key takeaways
- The guide was published on 19 October 2026. It ranks agent systems from Level 0 (a notebook demo) to Level 6 (a disciplined build process).
- Its main rule: you get to production by containing the model more, not by trusting it more.
- A 10-point self-assessment is included. If fewer than half the points are true, your system is less mature than its demo makes it look.
- Survey data points the same way. Quality is the top barrier to production, and evals trail far behind observability.
What is the production-grade AI agents field guide?
Engineer Varun Jindal wrote the piece for the Stack Overflow Blog. It works as the map for a series on running large language model (LLM) systems in production. The deep-dives on each level went up from 7 October 2026.
The starting point is simple. LLMs are probabilistic, so they can give different answers to the same input, and you can’t make them fully deterministic. What you can do is shrink the model’s job to the smallest decision that needs judgment. Then you wrap that decision in ordinary code that is testable, auditable and observable.
In Jindal’s words, there is “a canyon between an agent that demos well and an agent you’d put in front of customers or money.”
The 7 levels, and the warning sign at each one
Each level assumes the ones below it are in place. The warning sign column is the guide’s “tell”, the symptom that shows a level is missing.
| Level | What you have | Warning sign |
|---|---|---|
| 0: Notebook | A prompt, a model, a happy-path demo | “It works on my examples” |
| 1: Determinism | Agents propose, never act; a fixed graph per task; bounded loops | The agent writes business data directly |
| 2: Evaluation | Evals gate CI; baselines catch drift | Prompt changes ship on hope |
| 3: Confidence | Calibrated confidence; low scores go to a human | The model’s self-reported confidence triggers actions |
| 4: Safety and governance | Layered guardrails; PII handled at the boundary; append-only audit log | One moderation filter; raw user data in logs |
| 5: Operability | Decision dashboards, cost control, model fallback, kill switch | Problems surface on the invoice or from customers |
| 6: Production-grade build | A decision log that humans and coding agents both read | Same decisions re-debated every few weeks |
The order matters. Determinism comes before evals, because you can’t measure behaviour that changes with whatever path the agent took. Evals come before you trust confidence scores, and safety comes before you scale.
What does “agents propose, they don’t act” mean in practice?
This is the core idea of Part 1, “Make your AI agents boring”. The agent returns a structured proposal, such as “refund this order”. A separate component applies the change, and only after approval.
Think of a junior bank clerk who fills in the form but never touches the vault. Their judgment is useful, and the system still has the final say.
The code examples back this up. Output must match a schema and fails closed if it doesn’t. A routing function sends decisions below a 0.85 confidence threshold to human review. Every decision becomes one row in a ledger that can never be edited, and corrections are added as new rows. Open-ended loops are allowed only with a hard step cap and an allow-list of tools.
Why does this matter now?
Agents are already in production, but their safety nets lag behind. LangChain’s State of Agent Engineering report surveyed 1,340 people between 18 November and 2 December 2025. In that survey, 57.3% said they had agents in production. Quality was the top barrier, cited by 32%.
The gap is in the numbers: 89% had some form of observability, but only 52.4% ran offline evaluations and 37.3% ran online ones. In other words, many teams can see what their agent did without being able to say whether it was right. That is roughly Level 5 tooling sitting on a Level 1 foundation.
The business stakes are real too. In June 2025, Gartner predicted that more than 40% of agentic AI projects will be cancelled by the end of 2027. It blamed rising costs, unclear value and weak risk controls. Cost control and risk controls are exactly what Levels 4 and 5 of the guide address.
How to score your own system in 10 questions
Here is a condensed version of the guide’s self-assessment. Count how many are true for you.
- Agents only propose; a separate component applies changes.
- Each capability is a fixed sequence, with the model as one step.
- A prompt or model change that lowers quality fails CI.
- Every release is compared against a baseline.
- Low, calibrated confidence routes to a human.
- Guardrails are layered and fail closed.
- Sensitive data is redacted or hashed at the boundary.
- A dashboard shows decisions, blocks, token cost and override rate.
- Automated decisions can be stopped within seconds, without a deploy.
- Key decisions live in a log that people and agents both read.
If you scored under five, start at your lowest missing level, not the most exciting one. Our opinion: item 9, the kill switch, is the cheapest one on the list to build and the one teams most regret skipping.
What happens next
The series is still growing. The field guide lists about 25 planned topics, from LLM-as-judge (one model grading another’s output) and multi-tenant memory to agent identity and fallback routing. Only one post per level was linked when the guide went up on 19 October 2026.
Level 6 is the one to watch. It treats the coding agents that build your system as needing the same discipline as the agents you ship. As more code is written by agents working in parallel, a shared decision log may matter as much as any test suite. If you are weighing how AI fits a smaller team first, our explainer on how AI is changing small businesses covers the basics.
FAQ
What makes an AI agent production-grade?
The field guide’s answer is containment. The model makes one narrow decision, and deterministic code handles validation, approval, audit logging, monitoring and the ability to shut it off.
Should I add evals or observability first?
The guide puts evaluation at Level 2 and observability at Level 5. Make behaviour repeatable first, then gate releases on evals, then build out operations dashboards.
Can I trust an LLM’s own confidence score?
Not on its own. The guide recommends building confidence from several independent signals, calibrating it, and sending low scores to a human instead of acting on them.
Your next step: run the 10-point check on your agent this sprint, write down your lowest missing level, and fix that before you add a single new feature.
2 comments