Making AI Agents Suitable for Regulated Industries

Agents are already in production in regulated industries. They draft appeals, prepare submissions, triage cases, and assemble audit logs. What almost none of them do is decide anything on their own.

That gap is not about capability. It is about what happens when somebody asks you to justify the outcome.

Key takeaways

1. The hard part is responsibility, not technology. The question a regulator asks is not whether the system worked, but whether you can explain why it did what it did.
2. Production agents today are assistive on purpose. The top use cases in regulated industries are document drafting and case preparation, because drafting a document that a human signs off is a risk an organization can carry.
3. An agent is an architecture, not a language model. The term predates LLMs by decades, and nothing requires every part of an agent to be one.
4. Use GenAI to write the decision model, not the decision. The model is an artifact a human can read and approve. The engine executes it the same way every time.
5. Accountability is more than determinism. Generated code is deterministic too. What regulated work needs is a versioned artifact a person reviewed before it shipped.
6. The agent asks, the model decides. Given a published model as a skill, the agent refuses to invent inputs and routes the decision to the model.
7. This is also the cheaper architecture. Offloading policy to a decision engine keeps a full qualification under ten thousand tokens.

---

This post is a summary of the Aletyx webinar with Paul Vincent, an industry analyst formerly of Gartner whose work in AI goes back to the 1980s, and Alex Porcelli, CEO of Aletyx and builder with two decades on the Drools team.

The hardest decisions are not technical

A line from the AI ethics team at Oxford’s Schwarzman Centre sets the frame for everything that follows.

“The hardest decisions about AI are not purely technical. They are decisions about power, values, and responsibility. And that responsibility thing is exactly what we’re talking about today.”

This is not only a compliance concern. Any organization that has to answer for an outcome has the same problem, whether or not a regulator is asking.

Why an LLM alone cannot carry a regulated decision

The limitations are well known individually. What matters is that they compound in exactly the settings where the stakes are highest.

“You have hallucinations, which are not things you want to have in active healthcare systems, certainly not in financial services. You don’t want wrong answers, and I need to be able to verify decisions and behaviour, and that’s very difficult to do with the closed LLM world.”

Four problems, and the fourth is the one people underestimate. Responses vary between runs, so the same case can come back differently on Tuesday than it did on Monday. And token cost is climbing: one AWS fraud researcher saw execution costs come in far above forecast, which pushed the team to look at other decisioning technologies.

This is why the market looks the way it does. Ask for the top five production grade agentic use cases in regulated industries and the list comes back as healthcare claims denial, regulatory document preparation, anti-money laundering triage, compliance audit logging, and claims adjudication. Four of the five are document drafting.

An agent is an architecture, not a language model

The structural point is easy to miss because the vocabulary has shifted so fast. The word “agent” predates the current wave by decades. It came out of call centre software, then robotic process automation, and the academic literature on cooperating agents goes back to James Odell’s work twenty years ago.

That history matters for one practical reason: nothing requires every component of an agent to be a language model.

“What I really want to do is switch from black box decisioning, which is what the LLM basically gives me, to a more verifiable or accountable decision capability… There’s no requirement for me to use purely only LLMs inside my agents.”

So you add a component whose entire job is to decide, and you choose a technology for it that can be read, tested, and defended. This is not hypothetical. Run the same exercise for agentic use cases built on Drools and you find credit and lending underwriting, insurance fraud triage, healthcare prior authorization, AML reporting, and trade reconciliation already in production.

Use GenAI to write the decision model, not the decision

“Instead of using AI in the agent, we’re using GenAI to create the decision model, and the decision model then creates the rules to execute within the agent using a decision engine… All I’m doing is exploiting GenAI to create that decision model, which I can then verify.”

This is model driven engineering. GenAI is fast at producing a first draft and unreliable at being the system of record, so you point it at the artifact rather than the outcome. The decision model is readable by compliance teams and business experts, it can be tested, and the engine executes it identically every time.

The AI accelerates the work. The decision itself stays explicit, versioned, and owned by the business. If that sounds familiar, it is the same argument as what separates a good decision model from a merely correct one: the value is in being reviewable, not just in being right.

What accountability actually requires

“Accountability is not only reproducibility, the deterministic aspect. Deterministic is a given. Even without decision models you can get deterministic behaviour by generating code. Here you get explainability, but you also have a version that can be reviewed by a human before getting into production.”

This is worth sitting with, because “deterministic” is often where the conversation stops. Generated code is deterministic. A hash function is deterministic. Determinism only tells you the system will do the same thing again; it says nothing about whether anyone understood what it would do the first time.

What regulated work needs on top of that is an artifact a person actually read and approved, carrying a version you can point at when the decision is challenged.

The agent asks, the model decides

In practice, that looks like an agent built in Claude Code with a single system prompt, pointed at a published loan pre-qualification model exposed as a skill, and asked to qualify a borrower.

“It’s the model, it’s not thinking by itself. It’s being guardrailed to some extent to execute following the model that we just published. So it is asking for more information, it’s not coming up with their own values.”

The behaviour to watch is the refusal. Given an incomplete case, the agent does not guess the missing values to produce a plausible answer. It asks for them, then hands the decision to the model.

The result comes back as not qualified, with a link to the execution. Opening it shows the inputs, the rules that fired, and the path through the model. Change the numbers and the same trace shows why the answer changed.

This is also the cheaper architecture

“When you’re writing code and you ask for a search on your code base, the agents are not opening file by file or line by line. They’re using the best tools, and one really well known tool is grep… Here it is exactly the same thing to execute a decision. Instead of thinking, they are narrowed to use the decision model.”

Nobody would ask a language model to read a repository line by line when grep exists. A decision model is that tool for policy. Reasoning your way to an eligibility answer token by token is the expensive, unreliable path to a result the engine can return exactly.

So the governance argument and the cost argument point the same way, which is unusual and worth noticing. Accountability here is not a tax you pay for compliance. It is the cheaper design.

Where this goes next

There is a speculative phase two worth naming: take an existing LLM agent’s inputs and outputs and reverse engineer them through GenAI into a decision model plus a description, using the agent as a prototype and then graduating to accountable execution. Aletyx is not there yet, and getting there means consolidating execution data and feeding it back into the models.

The closing read of what has shifted:

“The whole shift of decision automation development has taken an entire step sideways to exploiting GenAI. This is a really interesting approach, a great use of LLMs, and it avoids all the accountability issues of LLMs actually being used for decisioning at runtime.”

A step sideways is the right description. The decision technology did not need replacing. What changed is who writes the model and how quickly, and that turns out to be enough to move agents from drafting documents to making decisions somebody can defend.

---

FAQ

Can an AI agent make a regulated decision today?

Yes, if the decision itself does not come from the language model. The pattern in production is an agent that gathers context and calls a decision engine executing an approved model. The agent handles interpretation and orchestration; the model determines the outcome, reproducibly and with a trace.

Why are most production agents in regulated industries only assistive?

Because drafting is a risk organizations can carry. Four of the top five production use cases are document centric: claims appeals, regulatory submissions, AML case files, audit logs. A human signs off, so the human remains accountable. Automating the decision itself requires being able to justify it later, which a language model on its own cannot do.

Is determinism enough for accountability?

No. Generated code is deterministic and so is a decision engine, but determinism only guarantees the same output for the same input. Accountability also needs explainability and a versioned artifact that a human reviewed before it reached production, so the decision can be defended when it is challenged months later.

Does using a decision model make an agent more expensive?

The opposite. Offloading policy to a decision engine keeps a full qualification process without spending excessive tokens. The agent calls the model the way it would call grep instead of reading a repository line by line, which is both cheaper and more reliable.

Do I need to replace my existing rules to do this?

Not necessarily. Decision technologies are already present in agentic use cases in production across underwriting, fraud triage, prior authorization, AML reporting, and trade reconciliation. The shift is less about new execution technology than about using AI to author and maintain the models, and exposing them so agents can call them.

---