Every orchestration vendor now has an agentic AI story. The business process platforms pitch it. The data pipeline tools pitch it. Each promises that its engine can pick the right model for the work, weighing cost, compliance, and performance as it runs. What the pitches skip is the question that decides whether multi-model AI orchestration survives production: which layer of your stack chooses which model, and on what grounds.
A previous article on multi-model AI strategy made the case that enterprises need more than one model, for availability and model churn, the cost of running agents, regulation and compliance, sovereignty and jurisdiction, and picking the right model for the task. Deciding to run several models is the easy part. The harder question follows: where does the choice between those models actually get made?
What Is Multi-Model AI Orchestration?
Multi-model AI orchestration means governing which AI models are allowed for each unit of work, and executing every call within that allowed set. The governing half is a business decision. The executing half is an engineering problem. The two belong in different layers of the stack.
Take an insurance claim as an example. It arrives with photos and scanned documents. A pipeline ingests them, classifies the images, extracts text, enriches the file with policy data, and scores it for fraud, much of it high-volume batch and streaming work. An adjuster then reviews the file and approves or denies it, and a payout follows. One model will not serve all of that well. The photo classification, the document extraction, the fraud score, and the adjuster’s decision support are different tasks with different cost, sensitivity, and accuracy needs.
The distinction that runs through this article is simple. The platform router optimizes within a set of models. The orchestration layer defines the set. I call this principle decide high, route low.
The Claims Process Was Orchestrated Before AI Arrived
Walk through the insurance claim three times: without AI, with AI models embedded in the steps, and with an agent running parts of the process.
Before any AI, the process is already orchestrated. Rules route claims by type and value. Approval gates hold a payout until a human signs off. Exceptions escalate. Residency rules already say which documents may go to a third-party service and which must stay in-house. A decision gate is doing real work here, choosing paths and enforcing boundaries, with no model in sight.
Add AI, and models slot into steps that already existed. An extraction model reads the documents. A vision service classifies the damage photos. A model scores the file for fraud. A copilot drafts the adjuster’s recommendation. Every insertion raises the same new question: which model, running where.
Add agentic AI, and an agent runs intake to recommendation on its own, calling tools and selecting models step by step. The payout still passes through a human and a compliance check.
Notice what stayed constant across all three stages. AI did not introduce orchestration to this process. It added one new decision, which model, to a gate that was already governing the work. The vendors pitching agentic model selection are slotting that decision into governance machinery they already owned, and they are right to. The open question is not whether to orchestrate the model choice. It is where the choice belongs.
Model Selection Is a Decision, Not an API Call
What makes choosing a model a decision-gate problem is that it depends on context the request alone does not carry. The cost ceiling for this workload. Whether the data is special-category and has to stay in region. The security boundary the step runs in. How regulated the outcome is. Whether a smaller purpose-built model fits the task better than a frontier generalist.
A decision gate evaluates each unit of work against rules and either selects a path or escalates it. Model selection fits that shape, and it has to be auditable. Where an automated system shapes a decision about a person, regulators increasingly expect you to show how it reached its result. The EU AI Act makes that explicit for its named high-risk uses. Those include credit scoring, and the risk assessment and pricing of life and health insurance. Even outside those categories, regulated industries carry audit and anti-discrimination obligations that come to the same thing: a decision affecting a customer must be explainable after the fact. “Which model produced this recommendation, and why was it permitted for this case” is a question you answer during an audit, not a detail buried in a proxy log. The decision belongs where the business context and the audit trail already live.
Two Lanes of Multi-Model AI Orchestration: Business Process and Data Pipeline
The model decision shows up in two orchestration layers that sit at the same level of the stack with different owners.
Business process orchestration governs work that mixes humans and systems, with portable process models, approval gates, and compliance-grade audit trails. Appian, Camunda, and Pega are representative examples, and the users are process architects and compliance teams. Pipeline orchestration coordinates event-driven, data-intensive work with version control, retries, and execution observability. Apache Airflow and Prefect, which acquired Dagster recently, are representative examples, and the users are data and platform engineers.
Both now embed model selection, and each has a legitimate claim, because each owns part of the claim process. The approval decision and its audit trail live in the business process lane. The high-volume document extraction, enrichment, and fraud scoring live in the data pipeline lane. Choosing which lane owns a flow follows ownership and governance: who builds and maintains it, and who has to read and verify it afterward. Many enterprises run both, with events passing between them. The two lanes are different tools for different parts of the same process. I covered when to use each in detail in a dedicated comparison of data integration and workflow orchestration.
Converging Toward Unified Orchestration
The two lanes are not the only ones converging. Agent frameworks are heading the same way: LangGraph added checkpoint-based durable execution and human-in-the-loop, and CrewAI added persistence to its flows. Whether that makes them orchestration platforms or simply more workflow engines to operate is an open question, and either answer strengthens the case for a shared layer above them. For real durable distributed execution at the code level, Temporal remains the reference, and it is extending toward agentic workloads from that end.
Integration platforms are moving in from another direction: Workato, a long-time iPaaS leader, now positions itself as an agentic orchestration platform with one control plane across integration, automation, and AI.
And a newer class of platforms runs data pipelines, workflows, applications, and business processes on one engine with one audit trail, an approach the market has begun to call unified orchestration. Kestra is a representative example.
In practice these platforms coexist: unified orchestration typically grows alongside the established lanes and gateways rather than replacing them, even if consolidation shrinks the count over time. The model decision is a preview of why that consolidation is attractive: a decision gate that spans these lanes sees the full context, while a gate confined to one lane governs only half the claim.
AI Gateways and Model Routers: The Layer Underneath
Below the orchestration lanes sits a layer that can also pick a model: routers from the cloud AI platforms and independent AI gateways. The cloud routers select from their own catalog. Amazon Bedrock routes within a model family and predicts which member gives adequate quality at lower cost. Microsoft Foundry (formerly Azure AI Foundry) deploys a trained model router over a configured pool, which can now include Anthropic’s Claude models alongside the OpenAI lineup. Google’s Vertex AI Model Optimizer does the same within the Gemini family, and only for text. Independent AI gateways such as LiteLLM and Portkey widen the catalog, putting one OpenAI-compatible endpoint in front of many providers and adding failover, caching, and budget controls.
This layer became a product category of its own in a single week of July, and the multi-model AI strategy article covers that wave in full, including the caveat that several of these routers come from companies with model ambitions of their own. The better this layer gets, the more valuable the decision gate above it becomes, because even the smartest router needs someone to define the set it optimizes within, and that definition draws on process context, jurisdiction, and budget that no prompt-level product can see.
Two Limits of Prompt-Level Routing
Two limits matter. The cloud routers are catalog-bound. A router inside one cloud can only choose what that cloud offers, so it cannot reach a sovereign EU model or a self-hosted model that lives outside its catalog, which is exactly the kind of option a sovereignty or residency requirement forces onto the table. Microsoft even encourages teams to treat the router’s model subset as a compliance gate that a security team approves before routing begins. It is decide high, route low in miniature, and it confirms the pattern rather than replacing it, because the subset only governs what sits in Microsoft’s catalog. And every router at this layer chooses at the prompt level. It sees a request, not the process step behind it. It does not know that this attachment is a privileged document, that this step runs under a particular jurisdiction, or that this workload draws on a budget another team owns. Microsoft’s router reports which model it chose but not why it chose it. For trading cost against quality on a single call, that opacity is fine. For a regulated decision, the opacity is the problem.
How Slack Routes Models in Production
Slack’s routing layer is a well-documented version of this pattern, and Slack built it in-house rather than buying it. I covered why Slack added Google Vertex AI alongside AWS Bedrock in the strategy article. The mechanics are the part that belongs here: a circuit breaker watches time to first token, error rates, and latency per endpoint, reroutes traffic automatically, and ramps back gradually as a degraded endpoint recovers (Slack Engineering). The detail that matters most sits one step earlier. A new model enters Slack’s rotation only after security and compliance verification clears it, and the router then experiments and fails over within that cleared set. Slack never wrote the words, but that is decide high, route low, running in production.
Decide High, Route Low: Separating the Decision from the Routing
The phrase “which model” actually covers two different choices.
The first is a decision. Which models are allowed for this unit of work, given the cost ceiling, the jurisdiction, the data sensitivity, the modality, and how regulated the outcome is. It needs context and produces an audit trail, so it stays high, in the decision gate.
The second is routing. Within the allowed set, which model handles this specific call, and what happens when it errors or slows down. Failover, load-balancing, and cheap-first optimization can drop low, to the gateway or platform router.
A single step in the claims pipeline holds both. The damage photos go to a purpose-built vision service running as batch SaaS, because it is economical at volume and that data clears GDPR for the flow. A general chat model is the wrong tool for that step anyway. The legal documents stay on local infrastructure, because they carry privileged content that should not leave the environment. Modality, residency, and the choice between batch and streaming are context decisions, and they belong in the gate. Once the gate has narrowed legal extraction to a set of local models, failing over among them is routing the gateway can own.
The anti-pattern is handing the decision to an opaque router that cannot explain itself, then discovering during an audit that no layer recorded why a privileged document went where it went.
How the Layers Combine in Production
Real architectures use more than one layer at once. In the claim, the business process orchestrator owns the human approval and the audit trail. It calls the pipeline for the event-driven extraction, enrichment, and scoring. The pipeline calls a gateway for the model executions it has been cleared to delegate. Events carry the work between the lanes, so neither is tightly coupled to the other. The transport is typically an event streaming platform such as Apache Kafka when the lanes need a replayable history, or a message broker such as RabbitMQ or IBM MQ when transient delivery is enough. The trade-offs between those messaging and streaming options have their own article.
The shape stays consistent: decide high, route low, and let each layer own the part of the problem it has the context for. The layers describe responsibilities, not a product count. One platform can host both the gate and the routing, and the converged platforms discussed earlier increasingly do. What must not collapse is the separation itself: the decision stays explicit, recorded, and above the routing, whether the two live in one tool or five.
The model layer is also distinct from the agent protocols around it. When the agent works the intake stage, it reaches its tools through the Model Context Protocol (MCP) and, where it hands off to another agent, coordinates through A2A, both now under neutral foundation governance. Neither routes a model, and the gateway does neither of their jobs. They are separate layers in one architecture. Confusing them is how a team ends up expecting a gateway to enforce a policy it was never given the context to see.
Mapping the Five Drivers of a Multi-Model AI Strategy to Layers
The article on multi-model AI strategy set out five reasons to run more than one model. Each resolves to a layer, and two of them split across both.
Availability and model churn delegate down. Keeping a workload running when a provider degrades or retires a version is failover within an allowed set, which is what gateways do well. Cost splits: the budget and the ceiling are a policy decision up high, while cheap-first execution within that ceiling is routing down low. Regulation and sovereignty stay high, because only the orchestration layer knows the data class and the jurisdiction. Model fit splits by stakes: routine extraction routes to whatever model performs adequately, while the model behind a regulated recommendation is pinned by decision rather than selected by a router.
The five drivers stay the same. The work is knowing which layer enforces each one.
The Decision Is the Strategy
Deciding to run several models is easy to put on a slide. The strategy becomes real when the model choice is explicit, governed, placed at the layer that holds the context, and combined across layers when the work demands it.
The orchestration and decision layer is the durable part of the architecture. The model underneath it stays replaceable, which is the whole point of running more than one. The pattern matches the broader Trinity: data integration keeps the gate fed with current data, process intelligence governs the decision and records it, and trusted agentic AI acts within the boundaries the gate sets.
Each Trinity layer has its own landscape report: the Data Integration Landscape 2026, the Process Intelligence Landscape 2026, and the Trusted Agentic AI Landscape Q3 2026, which maps the vendor dimension behind this article.
My advice: map your model decisions to layers before a vendor’s router makes them for you. A router will make the decision if you let it. It just cannot tell you why, and at some point an auditor will ask.
To follow this work across data integration, process intelligence, and trusted agentic AI, subscribe to the newsletter and connect on LinkedIn.
