How to Make AI Agents Reliable

The model gets the credit when the agent works and the blame when it doesn't. That's almost always backwards. Most of the time you're looking at a context problem wearing a model costume.

Here's the answer up front: if your AI agent is unreliable in production, the model is probably fine. The data feeding it isn't. Stabilize the data layer and most of the "the AI is hallucinating" problems quietly go away.

Most AI projects that look great in a demo never make it to durable production. The numbers people throw around land somewhere between 70 and 85 percent that stall out after the pilot — Gartner has cited 85, RAND found more than 80, and the failure usually isn't the model. The model isn't dumber in production than it was in the lab. The data around it is messier, and nobody built the part that keeps it from being messy.

There's a keynote on exactly this failure mode — "How to make AI agents reliable: data context done right" by Tom Kaltofen — and it names the unsexy fix plainly: build a deterministic context layer. Stable interface. Reproducible data. Clear lineage. Almost none of that sounds like "the AI part," which is exactly why it's the part nobody funds, and exactly why it's the part that decides whether the agent survives contact with real users.

Why this matters

If you're building any agent that touches real data — a RAG system, a support agent, an analytics assistant, anything where the answer depends on a database underneath — the model is at most half the system. The other half is the contract between the model and the data. And that contract is where almost every production failure I've watched has actually lived.

Quick plain-language note, because I don't want to lose anyone: RAG just means the agent looks things up in your data before it answers, instead of answering from memory. The lookup is the part that breaks. The model is doing fine. It's being handed bad context and answering accordingly. To the user that reads as "the AI is making stuff up." On the engineering side it's the agent eating stale or wrong data and doing exactly what you'd expect.

"Data quality" usually isn't a data quality problem

This is the part of the keynote that landed hardest for me, because I've lived it. Most of what teams call a data quality problem isn't dirty data. It's a data understanding problem. The data is fine. People just disagree on what it means.

The example that makes it click: "revenue." Three teams in the same company, three definitions. Finance counts it after refunds. Sales counts it gross, before deductions. Product only counts paid plans and ignores trials. So when your agent asks for "revenue" and pulls a number from one of those three sources, the number is correct — it's just not the right one. And the agent has no way to know that, because nobody wrote down which one it was supposed to use.

You can't clean your way out of that. There's nothing dirty to clean. You fix it by writing down what "revenue" means in your system, in one place, and making every consumer — human or agent — read that definition before they trust the number.

Takeaway: most production failures aren't bad data. They're inconsistent interpretation of fine data. Define your terms in one place and most of it goes away.

What a deterministic context layer actually is

Don't let the name scare you. A deterministic context layer is just the stuff that sits between your agent and your raw data and gives you three guarantees:

1. Stability. Same input, same data shape — today, tomorrow, six months from now. Someone renames a column and the contract doesn't silently change underneath you.
2. Lineage. For any answer the agent gave, you can trace which records produced it. Not the query plan — the actual data the model saw.
3. Reproducibility. If you need to audit, debug, or replay a decision, you can rebuild the exact context the agent had at the moment it decided.

What provides those guarantees depends on what you're building:

• Feature stores — versioned data features with explicit schemas and freshness rules.
• Semantic layers — your business definitions in one place, behind an API. Revenue means revenue.
• Knowledge graphs — when the relationships between things matter more than a flat table.
• Direct database access — fine for a simple, low-stakes coding agent that can just re-query and restart to refresh.

The right pick changes. The principle doesn't: the agent reads from a layer you control, not a raw database it has to guess at.

Takeaway: pick the layer that fits the job, but pick one. "Agent queries the prod database with whatever schema it assumes" isn't a strategy. It's the absence of one.

Lineage is the part everybody underbuilds

Lineage is the least glamorous piece and the one teams skip first — the record of what data the agent actually used, not what it was supposed to use.

In a regulated environment — healthcare, finance, anything touching personal data — somebody will eventually point at one decision and ask you to explain it. "Why was this person denied?" The good answer is: here are the exact values, from these exact tables, at this exact timestamp. The bad answer is: the model said no. Both might be true. Only one survives an audit.

And it's not just regulated work. Lineage is what lets you debug an agent that's slowly drifting. Without it, when something goes weird you're guessing — did the model change, the data change, the pipeline change, or did somebody ship a fix that broke a downstream consumer? Logs tell you what the agent did. Lineage tells you what it did it with.

Takeaway: build lineage before you need it. The day you need it and don't have it, rebuilding the context from scratch costs ten times what logging it alongside the agent would have.

From my own bench

I'm not preaching from the outside on this one. I build agents that sit on private knowledge — RAG over my own vault of notes, conversational agents over personal data, running on my own hardware with safety gates in front of them. They fail in exactly the ways the keynote describes. Stale snapshots. Schema drift the second I rename a field. Two code paths reading the same field two different ways because I never wrote down what the field actually means.

The piece I've been slowest to build is lineage. My agents log the prompt and the response. They mostly don't log "here are the exact records I retrieved, with their timestamps and versions." So when I go back to debug a weird answer, I'm reconstructing the context by hand — and a couple of times I just couldn't. That's a bad place to be, and I put myself there by skipping the boring part.

So I'm rewiring my retrieval layers to capture lineage at the record level by default. Not because I need it today. Because the next time something breaks, I want to answer why in an hour instead of a day. That's the whole trade.

Try it today

Step	What you do	Why it pays off
1. Write down what your three most-used data fields mean	One markdown file, one paragraph per field — the definition you'd hand a new analyst on day one.	This is the cheapest semantic layer there is. It catches the "three teams, three definitions" failure before it ever reaches the agent.
2. Log the data the agent actually retrieved, with every response	Not just the query — the records, with timestamps and source IDs. A JSON blob in your trace log is enough to start.	The day you need lineage, you'll have it. The day you don't, it cost you almost nothing.
3. Pin a schema version to every interface the agent reads	Version the API your agent calls. Don't ship a breaking change without bumping it.	Schema drift is the silent killer. Versioning makes drift loud, intentional, and reviewable.

Where people get burned

• Treating "data quality" as a cleaning job. It usually isn't — it's a meaning problem. Fix: document what fields mean, not just whether they're filled in.
• Letting the agent query production directly. Convenient right up until it isn't. Fix: put an interface between the agent and the database that you control, even a thin one.
• Skipping lineage because the system is "small." Small systems become big systems with the same blind spot baked in. Fix: log what data was used from day one, in whatever cheap form you can manage.
• Versioning the model but not the data interface. You'll catch model drift and miss data drift, and data drift is worse. Fix: version the model, the prompts, and the schemas the agent reads.

A tool and a question

• A tool to look at: dbt has become the common answer for the semantic-layer piece — define your metrics once, expose them through a stable interface, let agents and BI tools consume the same contract. If dbt feels like overkill for where you are, a markdown definitions file in your repo is a real start. Start where you are.
• A question to actually sit with: if someone asked me to explain one specific decision my agent made six months ago, could I? If not — what do I need to start logging today so that next time, I can?

The bottom line

The model gets the credit and the blame. The reliability actually lives in the data context underneath it. Every agent I've seen survive in production converges on the same boring shape: a stable interface to the data, with definitions written down, with lineage, with versioning.

It's not glamorous. It doesn't go on a demo slide. It's the part that decides whether the demo ever becomes a product.

— Dru Edwards