A retrieval-augmented generation (RAG) pipeline looks like one call to a large language model from the outside. Inside, the system makes many small model calls for every new document and every question. It classifies documents, flags sensitive content, picks a source, scores chunks, checks whether the context is enough and checks every claim of the answer. Only one of these calls generates new content, which is the answer itself. Decision models belong in a RAG pipeline because most of its model calls choose from a fixed set of options. A model built to return a probability per option does that faster and cheaper than a model built to write prose.
![]()
Jayanth Penumarthi of Thoughtworks made this observation in October 2026 in How Jev can help improve the efficiency of RAG pipelines. His article maps the steps of a RAG pipeline to Jev, the first decision model from TypeSafe AI, and reports no measurements. This post takes the idea to the enterprise architecture level and covers:
- Which calls in a RAG pipeline are decisions, and why the generative LLM is the wrong default for them
- What a decision model returns, and where rerankers, guard models and LLM judges still fit
- How the ingestion, query and escalation paths differ in latency and in who runs them
- Confidence bands per decision and the decision log as an audit trail
- When a decision model is the wrong tool, and how to start without rebuilding the pipeline
A RAG Pipeline Makes Many Decisions and Writes One Answer
A RAG pipeline retrieves documents that fit a question and passes them to a language model, which writes the answer from them. A document is any piece of enterprise content that the pipeline indexes. Examples are a PDF, a wiki page, a contract, a support ticket, an email or a record from a business application. The usual view puts the LLM in the center and treats everything else as plumbing. A more useful view follows the output type of each step. The picture below shows a typical pipeline with its ingestion path for documents and its query path for questions, and marks what each step returns.

The Steps of a RAG Pipeline and Their Answer Types
Each step returns one of a few answer types:
- Classification at ingestion. A new document gets a type, a business domain and a retention class. The answer is a choice from a known list.
- Sensitivity at ingestion. Does the document contain personal data or trade secrets? The answer is a yes or no for each category. The search index then enforces access rights as a filter, so the model only sets the label.
- Routing and rewrite. Does the question need retrieval at all, which source fits, and does the question need a rewrite first? Each is a choice or a yes or no. If a rewrite is needed, a small LLM writes the new query, which adds a second generation step in some pipelines.
- Reranking. Each retrieved chunk gets a relevance score.
- Context sufficiency. Is the retrieved context enough to answer? The answer is a yes or no for the whole set.
- Generation. The step that creates new content. In most pipelines this is text for a person. It can also be structured output for an application or an agent, such as a summary in JSON. The format makes no difference. This step writes content that did not exist before, and every other step picks from options that are known in advance.
- Grounding per claim. A claim is grounded when the retrieved context supports it. A claim without this support comes from the training data of the model or is invented. This is the usual meaning of a hallucination. The check asks for each claim whether the context supports it. The answer is a yes or no per claim.
Why the Generative LLM Is the Wrong Default for Decisions
One question triggers a routing decision, a score for every candidate chunk, a sufficiency check and a grounding check for every claim. All of them surround a single generation call.
Research on AI agents points in the same direction. An agent is software that calls an LLM many times to complete one task, for example to write code or to operate an application. NVIDIA researchers looked at three popular open source agents and estimated that small specialized models could handle about 40 to 70 percent of the LLM calls these agents make. The study did not look at RAG pipelines. Its reasoning applies to them as well, because most calls repeat a small number of narrow tasks and need no general-purpose model.
Most pipelines started with an LLM for all of these calls, because it was the only model that took an instruction in natural language without a training set. The cost shows up in three places.
- Latency. The LLM generates its answer token by token, and every decision in the query path adds to the user’s waiting time.
- Price. Each classification pays for a general-purpose model, and the bill grows with every added check.
- Confidence. The pipeline needs a number to branch on. Structured outputs from OpenAI or Anthropic guarantee a valid format. They say nothing about how reliable the answer is. The token probabilities of chat models are a weak basis for a threshold. A benchmark from 2026 tested up to 20 LLMs on scientific question answering. It found that instruction tuning pushes token probabilities towards the extremes and undermines them as a confidence signal.
Every call that chooses from known options deserves a model that returns a probability per option and answers in milliseconds. The generative LLM keeps the answer and the hard cases.
What a Decision Model Returns in a RAG Pipeline
A decision model reads a piece of state, answers typed questions about it in one pass and returns a probability for every allowed option. It writes no text. The questions come in three forms: a choice from a list, a score on an ordered scale, or a yes or no. In a RAG pipeline, the state is a document, a question, a chunk or a claim. The pipeline compares the probability with a threshold and branches without parsing any text.
TypeSafe AI introduced this class as System One models with Jev in September 2026. Within weeks, the market called the category decision models, and other vendors copied the request schema. Open-weight decision models run inside your own infrastructure. Platform variants run inside data platforms next to the data. OpenAI’s commercial Decisions API puts one of its existing LLMs behind the same kind of typed interface. A typical decision takes tens to a few hundred milliseconds and costs a fraction of a cent. My earlier post on the category covers the vendors and how to evaluate them.
The probability needs a check before it drives an automatic action. Amplitude tested Jev on a small sample from a banking intent dataset and measured 87.4 percent mean confidence against 75.3 percent accuracy, about twelve points of overconfidence. Hence, every team has to check the calibration on its own labelled data before it sets a threshold.
Decision Models Next to Rerankers, Guard Models and LLM Judges
RAG pipelines use specialized models for some decisions already, and a decision model complements most of them. The picture below compares five model types by the questions they answer and by what they return.

Each of the existing model types keeps its place:
- Cross-encoder rerankers read the question and a chunk together and return a relevance score. They are fast, cheap and trained for exactly this question, so they stay.
- Detectors for personal data, such as Microsoft Presidio or Amazon Comprehend, return a score per detected entity. They stay for the patterns they know.
- Grounding checks in cloud services return a score as well. The commercial contextual grounding check in Amazon Bedrock Guardrails, for example, blocks answers below a threshold you configure. Cloud providers already treat grounding as a decision with a threshold.
- LLM judges make the same kind of decision with a generative model. Google built its findings on sufficient context into an LLM re-ranker in the Vertex AI RAG Engine. A large model can be the best judge for hard cases, at a higher price and latency.
What remains is the long tail: routing by your own business taxonomy, sufficiency, sensitivity in your company’s terms, and every new check a team invents next quarter. No specialized model exists for most of these, and training one per question does not pay off. A decision model answers them through one interface, and the team writes each question in natural language. For each call, pick the cheapest model that makes the decision well enough, and keep the generative LLM for the answer and for the cases where the cheaper models are unsure.
Decision Models in the RAG Ingestion, Query and Escalation Paths
The model calls of a RAG pipeline fall into three paths with very different timing, and each path needs a different runtime. The picture below shows the three paths, their latency budgets and the decision log that all of them write to.

The Ingestion Path Runs Event-Driven or in Batch
Documents arrive through uploads, change data capture from content systems, crawlers and nightly exports. Each document is classified, checked for sensitivity, chunked, embedded and indexed. Nobody waits for the result in real time, and the volume comes in bursts. Seconds per document are fine, but the path needs retries, backfills and a record of what happened.
This is a job for workflow orchestration. A flow starts on an event from the content system or a Kafka topic, or on a schedule for the nightly export. It runs the decisions as tasks, writes labels and probabilities into the index metadata, and sends documents below their confidence band to a review queue. When a new model version arrives, the same flow scores a sample again and compares the results before the new version goes live. I described the event-driven side of RAG ingestion with Kafka and Flink in an earlier post on real-time GenAI.
Which orchestrator runs the flow depends on where the pipeline lives. Apache Airflow is the common choice when ingestion is part of the data platform. Camunda fits when the review of uncertain documents is a business process with human tasks. Kestra spans data, infrastructure, applications and business processes in one engine. A team can also start with it in this single pipeline through declarative YAML and event triggers.
The Query Path Needs Decisions in Milliseconds
In the query path, a person or an agent is waiting. Routing, reranking, the sufficiency check and the grounding check all sit between the question and the answer. The whole path has a budget of a few seconds, and generation takes most of it. The decisions have to fit into the rest.
A workflow orchestrator has no place here. It persists state and manages retries over seconds to days, and that overhead does not fit a request that lives for two seconds. The decision calls run inline in the application service or the gateway, in the same way as the embedding call and the vector search.
What about agentic RAG, where the LLM itself decides when and where to retrieve? The sources are then tools, which are more and more often exposed through the Model Context Protocol (MCP). MCP standardizes how an agent reaches a source. The choice of the tool stays a decision, and the generative model makes it in an extra turn. Anthropic reports that agents typically use about four times more tokens than chat interactions, and multi-agent systems about fifteen times more. A decision model can take the routine choices before the agent starts, such as picking the right one of twenty MCP tools for a question. The open-ended planning stays with the LLM.
The Escalation Path Brings in People and Audits
Every decision below its confidence band needs a place to go. In the ingestion path, a document with an uncertain sensitivity label waits for a reviewer before it enters the index. In the query path, the user cannot wait for a person. The answer goes out with a safe default, such as a narrower source set, a flagged claim or a plain “I don’t know”. The case lands in a review queue.
Here the orchestrator returns. It creates the review case, assigns it, waits with a deadline and records who decided what. The decisions of the reviewers become labels, and the labels become the evaluation set for the next model version.
Confidence Bands per Decision Follow the Cost of a Wrong Answer
The Thoughtworks article illustrates confidence routing with one set of bands for the whole pipeline: automatic above 0.9, an LLM between 0.6 and 0.9, and a person below. This is a good start for a pilot. In production, the threshold of each decision follows the cost of a wrong answer. The picture below shows illustrative bands for six decisions.

The cost of a wrong answer differs per decision:
- Routing and rewrite. A wrong route costs one more retrieval or a slightly worse context. The automatic band is wide, and the fallback is an LLM call.
- Reranking. A wrong score costs answer quality. The band is medium, and close calls go to a fixed rule.
- Context sufficiency. A false “enough” produces a confident answer from thin context. Google researchers found that models often answer although the context is insufficient and they should abstain. Gemma gave incorrect answers to 10.2 percent of questions without any context and to 66.1 percent with insufficient context. The fallback is a second retrieval round or an abstention.
- Grounding. A false “supported” lets an unsupported claim reach the user. The band is narrow, and the fallback flags or removes the claim.
- Sensitivity. A document wrongly labelled as not sensitive can show up in answers for people who must never see it. Only a very high probability for “not sensitive” passes automatically, and the uncertain middle goes to a person before indexing.
Two rules follow. The bands are configuration of the pipeline, versioned with it and pinned to a model version, because a new model version shifts the probabilities. And the bands are asymmetric wherever the two possible errors cost different amounts, which one global threshold cannot express.
What a Decision Log Needs to Count as an Audit Trail
A decision log is a data set with one record for every model decision. Each record holds the question, the probabilities the model returned and the action the pipeline took. It is separate from the application log and from the execution history of an orchestrator. The component that calls the model writes the record. In the ingestion and escalation paths this is the workflow task, and in the query path it is the application service or the gateway. A common setup sends each record as an event to a Kafka topic and stores it in a table in the lakehouse or a database. Auditors and data scientists query it there.
The Fields of a Decision Log Entry
With this log, every answer of the system can be traced back to the decisions that shaped it. An entry that holds up in an audit carries at least these fields:
- Reference to the state. A hash or identifier of the document, question, chunk or claim, so the input can be found again without copying it into the log.
- The question and its options. The question text and the full option set, because options change over time.
- Model and version. The model name and the pinned version. An alias such as “latest” does not belong in an audit trail.
- Probabilities. One per option. The margin between the top two is often the most useful number in a review.
- Band and version. The band configuration that applied.
- Outcome. Automatic, LLM fallback, human review or safe default.
- Approver. For escalated cases, who decided, when and whether the person overruled the model.
- Drift signal. The rolling distribution of probabilities per question and an alert when it moves.
With these fields, a team can answer questions a normal application log cannot. Why did the system show this document to this user? Did the probabilities shift after the last model update, and who adjusted the bands? The same log feeds process mining, as described in my post on decision models in workflow orchestration and process intelligence. It also helps with regulations that ask for logging and human oversight of automated decisions.
When a Decision Model Is the Wrong Tool in a RAG Pipeline
A decision model is NOT the answer to every call in the pipeline. Five situations need something else:
- Pure relevance ranking. A cross-encoder reranker is trained for this one question and is hard to beat on cost. A decision model earns its place in ranking only when the questions go beyond relevance, such as freshness or source trust, and then often only on the shortlist.
- Stable, high-volume labels. A fixed document taxonomy with millions of labelled documents is a classic machine learning problem. A model trained on your own labels beats a general decision model on cost and on the last few percent of accuracy. The decision log is the fastest way to collect these labels.
- A justification in text. A decision model returns a choice and a probability. When the pipeline has to explain why it flagged a claim or withheld an answer, the step needs an LLM.
- Long or hostile input. TypeSafe’s own documentation notes that accuracy falls as the state grows with content unrelated to the decision. The model also does not treat the state as hostile by default. Keep the state small and focused, and guard decisions on user-supplied documents with other checks.
- Data that must stay in your own infrastructure. A hosted decision model is the wrong tool when the state must not leave the company, because every call sends the document or the question to the vendor. The alternatives are an open-weight decision model that you run and calibrate yourself, or a classifier or LLM that already runs in your environment.
How to Introduce Decision Models Into a RAG Pipeline Without a Rebuild
Nobody should rebuild a working RAG pipeline around a model category that is a few months old. The entry point is one decision with high volume and low risk, run in shadow mode next to the call that makes this decision now.
Query routing is the usual candidate. It runs on every request, a wrong route costs little, and the existing LLM routing call already produces a label to compare against. The shadow setup adds a parallel decision call, logs both answers with their probabilities, and changes nothing in the user-facing path. After enough traffic, the log shows where the two agree and where the decision model is confidently wrong. The bands come from this comparison. Then the confident share switches over, and the LLM call stays as the fallback for the uncertain middle.
The next decisions follow in order of risk. Sufficiency checks come second, because user feedback and abstentions provide comparison data. Sensitivity labels need the review path first. Grounding checks per claim come last, because they multiply the number of calls. Once the interface, the bands, the log and the review queue exist for routing, every further decision is a configuration change plus an evaluation run.
What Decision Models Change in a RAG Pipeline
A RAG pipeline is a decision pipeline with one generation step. Seen by output type, most model calls choose from known options, and this is where decision models pay off. A decision model answers them with a probability in milliseconds. Specialized models such as rerankers keep the jobs they do best, and the generative LLM writes the answer and handles the hard cases. The architecture around the models lasts longer than any of them. It consists of three paths with their own runtimes, confidence bands per decision, and a log of every decision.
The price of a single decision keeps falling, and most vendors speak the same request schema. In my view, the lasting work and the lasting budget move to the evaluation set, the bands and the decision log. These decide whether a pipeline can switch models without losing control.
This post continues my series on decision models. The earlier posts cover:
- How System One models like Jev change enterprise AI architecture
- Why Jev fits Apache Kafka and Flink better than LLMs
- Jev in workflow orchestration and process intelligence
- How decision models became a new AI category after Jev
To follow this work across data integration, workflow orchestration, process intelligence, and trusted agentic AI, subscribe to the newsletter and connect on LinkedIn.