More than 150,000 organizations run Apache Kafka. In banks, automakers, telcos and retailers it is the nervous system: the place where orders, sensor readings, payments and status changes arrive as events, in order, replayable, decoupled from whoever consumes them.
The layer above it usually looks nothing like that. Data teams run a Python scheduler bound to its own database. IT operations runs a job scheduler bought in a different decade. Everyone else writes a small consumer service, one per reactive use case. Each has its own retry logic and its own deployment. The event backbone is event-driven and most of the orchestration layer around it is not.
Event-driven workflow orchestration on Kafka means coordinating what runs when an event arrives: which steps execute, in which order, under which conditions, and how the system recovers when one of them fails. Kafka delivers the message. The orchestrator decides what the business does about it.
This article covers both directions. An orchestrator can run its own coordination on the Kafka cluster an enterprise already operates. And it can turn Kafka topics into triggers for workflows that reach far beyond the streaming platform.
How Enterprises Orchestrate Around Kafka Instead of On It
Most enterprises start with cron jobs and glue code, then move to one of three heavier workarounds: a database-bound scheduler, a hand-written consumer service per use case, or a workflow engine built on stream processing. Each one rebuilds something the backbone already provides.
Database-Bound Schedulers Poll While the Backbone Pushes
Most orchestrators coordinate through a relational database. The scheduler writes, the workers poll, and the database becomes the point where everything contends. Under heavy execution volume the coordination layer turns into the bottleneck.
Kafka sits right next to it, built for exactly this kind of message passing, operated by a team that already knows how to scale it. The orchestrator cannot use it. The most reliable piece of infrastructure in the building is invisible to the layer that needs it most.
Every Reactive Use Case Becomes Another Consumer Service
Reacting to a topic normally means writing a consumer. Connection handling, deserialization, offset management, retries, backoff, alerting, a container, a pipeline, a runbook. Then the same thing again for the next topic.
A year later nobody remembers which service owns which consumer group. A downstream load fails and the whole batch gets re-consumed, because the retry boundary sits around the consumer rather than around the step that broke. The code is small. The operational surface is not.
Building the Workflow Engine in Stream Processing Is the Hardest Workaround
A third group went further and built the workflow engine itself on the streaming layer. Salesforce built a CI workflow engine on compacted Kafka topics. Swisscom’s Custodigit platform implemented the SAGA pattern with Kafka Streams. A SAGA splits one business transaction into a chain of local steps, and every step carries a compensating action that undoes it if a later step fails. Custodigit coded that chain and its compensation logic in Kafka Streams to orchestrate crypto custody transactions across brokers, exchanges and blockchains. I covered these architectures in detail in my article on Apache Kafka as a workflow and orchestration engine.
The pattern works, and these teams shipped regulated, production-grade systems. But look at what they had to build by hand: a state machine in application code, retry and compensation logic per step, deployment pipelines for every workflow change, and custom tooling for visibility. There is no workflow UI, no human task, no declarative definition. Every improvement to the engine competes with the business logic for the same engineering capacity. Salesforce archived its Junction Workflow project on GitHub in 2025.
Stream processing remains the right tool when the coordination stays inside the stream: high-throughput, low-latency logic between streaming microservices, embedded in the application. The workaround begins where the workflow leaves the stream. An incident ticket, an approval, a work order in a legacy system, a step that must retry on its own boundary. I wrote about the wider pattern in when not to use stream processing. A workflow orchestrator is told what to do when an event arrives. It does not transform a flow of events. Building an orchestrator on a stream processing framework, whether Kafka Streams, Apache Flink or anything else, solves the wrong problem.
Running the Orchestration Queue on an Existing Kafka Cluster
Kestra is an open source unified orchestration platform under the Apache 2.0 license, the same permissive license as Apache Kafka. It is the platform I use in this article to make the architecture concrete. Its engine is event-driven at the core, in the open source edition as much as in Kestra Enterprise, and the queue it uses for internal coordination is a deployment choice rather than a fixed dependency. Teams that already build on event-driven systems like Kafka and Flink will recognize the model.
The queue and the repository are configured separately. The queue carries messages between the executor, the scheduler and the workers. The repository holds execution records, metadata and flow definitions. Moving the queue onto Kafka puts the orchestrator’s own coordination on the message bus the platform team already runs, monitors and scales.
The value has nothing to do with benchmarks. The cluster is already in production. The SRE skills, the monitoring, the capacity planning and the disaster recovery procedure exist along with it. Adopting the orchestration layer does not mean adopting a new piece of stateful infrastructure with its own failure modes and its own on-call knowledge.
In my 2023 article I wrote that I wished Camunda had built Zeebe on Kafka instead of investing in its own replicated log. The architecture described in this section is exactly that idea.
Kafka as the Queue Without Stream Processing in the Engine
Running on Kafka is not the same as being built with Kafka Streams. Kestra has one implementation of the executor, the scheduler and the indexer, whichever backend is configured underneath. The engine contains no stream processing framework, no state stores and no rebalancing behavior to operate. An earlier generation of the engine ran on Kafka Streams, and the team took it out in Kestra 2.0 because the operational cost outweighed the benefit. I explored the reasoning in why Kestra 2.0 removed Kafka Streams from its core. Kestra went through the same evolution as the teams in the first section, just inside one product.
Only minimal information travels over the queue. The repository is the source of truth, and task outputs load on demand rather than streaming through the queue on every execution. Topics stay small. Kafka does what a broker is good at, and no distributed stream processing framework sits inside the orchestrator.
When PostgreSQL Is Enough for the Orchestration Backend
A single PostgreSQL database serving as queue and repository is the simplest way to get started with Kestra, and it is sufficient for many production deployments. The event-driven engine is not an Enterprise feature. Flows react to events on PostgreSQL exactly as they do on Kafka, and a plain database backend handles more than most teams need. Executions typically complete in milliseconds to seconds, and thousands of executions per minute run on a plain PostgreSQL backend.
Move the queue when latency or scale demands it, or when a regulator demands an availability property the database cannot give. Kafka, Redis and AMQP as queue backends are part of Kestra Enterprise, while the open source edition runs on PostgreSQL or MySQL with the full plugin catalog. Nothing in a workflow changes when the queue moves. The flow definitions stay identical.
Deploying Kestra Workers Next to the Kafka Brokers and the Data
In Kestra’s architecture, workers are the only components that touch private data sources, and they need no database access. Each worker holds a single outbound connection to the Controller, the control-plane component that brokers all worker communication.
This changes where orchestration can run. A worker can sit in another region, in a different cloud, in an on-premises data center next to a Kafka cluster that never leaves the building, or in a restricted network that permits outbound traffic and blocks every inbound connection. The data plane stays in your own data center or cloud account while the control plane runs wherever it makes sense, including as a managed service.
For regulated industries this is the difference between an orchestration project that gets approved and one that stalls in a security review. Payment events, patient records and production telemetry never leave the zone they are allowed to live in.
Event-Driven Workflow Triggers from Kafka Topics
Running the orchestration queue on Kafka is one half of the story. The other half is the plugin layer, where a Kafka topic becomes a workflow trigger and the hand-written consumer service disappears.
One Workflow Execution per Kafka Record
Four patterns cover almost every workflow that reacts to Kafka events:
- Realtime trigger: one execution per record, the moment it arrives. The trigger is the consumer. Connection handling, offsets, retries and alerting live in the platform instead of in another hand-written service.
- Interval trigger: batched pulls on a schedule, for roll-ups, reconciliations and warehouse loads.
- Consume task: read from a topic in the middle of a flow, for example to pull the latest reference events before a load runs.
- Produce task: publish results and status events back to a topic, so every downstream consumer sees the outcome of a workflow as a fact on the backbone.
Kafka hands at-least-once delivery to the workflow layer. A crash at the wrong moment produces a duplicate execution rather than a lost event, which is the right default for operational work. Design the flow idempotent, so an incident keyed on the alert id gets created once no matter how often the same event arrives. The retry boundary sits around the single step that failed, not around the whole consumer, so a failed downstream call never forces a topic to be re-consumed.
When One Execution per Record Is the Wrong Pattern
The boundary between stream processing and workflow orchestration from the first section runs in both directions. A topic with hundreds of thousands of events per second does not belong in front of a realtime trigger in the orchestration engine. Nobody needs a workflow execution per sensor reading. Filter and aggregate that stream with Apache Flink or Kafka Streams, publish the condensed signal to its own topic, and let the signal trigger the workflow.
One side runs continuous computation over many events. The other side coordinates a business reaction to one event that matters. Stream processing and workflow orchestration complement each other on the same Kafka backbone. The same distinction holds one level up, as I described in data integration vs workflow orchestration.
Example: A Machine Downtime Workflow Across Kafka, ServiceNow and RabbitMQ
A plant publishes machine alerts to a Kafka topic. A critical alert should open an incident in ServiceNow, wait for a maintenance planner to approve, release a work order to the plant’s RabbitMQ broker, and publish a status event back to Kafka so every downstream consumer knows what happened. A hand-written consumer for this chain is exactly the operational surface described earlier. As a flow definition it looks like this:
id: machine_downtime
namespace: production.plant
triggers:
- id: downtime_event
type: io.kestra.plugin.kafka.RealtimeTrigger
topic: machine-alerts
groupId: machine-downtime
properties:
bootstrap.servers: "{{ secret('KAFKA_BOOTSTRAP') }}"
keyDeserializer: STRING
valueDeserializer: JSON
tasks:
- id: handle_critical
type: io.kestra.plugin.core.flow.If
condition: "{{ trigger.value.severity == 'CRITICAL' }}"
then:
- id: create_incident
type: io.kestra.plugin.servicenow.Post
domain: "{{ secret('SNOW_DOMAIN') }}"
username: "{{ secret('SNOW_USERNAME') }}"
password: "{{ secret('SNOW_PASSWORD') }}"
table: incident
data:
short_description: "Downtime on {{ trigger.value.machineId }}"
- id: approval
type: io.kestra.plugin.core.flow.Pause
onResume:
- id: approved
type: BOOL
description: Release the work order
- id: work_order
type: io.kestra.plugin.amqp.Publish
host: "{{ secret('AMQP_HOST') }}"
exchange: maintenance
routingKey: work-orders
serdeType: JSON
from:
- data: "{{ trigger.value }}"
- id: status_event
type: io.kestra.plugin.kafka.Produce
properties:
bootstrap.servers: "{{ secret('KAFKA_BOOTSTRAP') }}"
topic: machine-status
keySerializer: STRING
valueSerializer: JSON
from:
key: "{{ trigger.value.machineId }}"
value:
incident: "{{ outputs.create_incident.result.sys_id }}"
state: WORK_ORDER_RELEASED
The listing is trimmed to the structure. Credentials come from the secret store, and the rejection path on the approval result is omitted for readability.
Facts travel on Kafka. Commands travel on AMQP. The alert and the status event are facts on the backbone, available to every consumer. The work order is a command for exactly one receiver, so it goes to a queue. The task speaks the AMQP protocol, and RabbitMQ serves as the broker on the receiving side in this plant.
The workflow keeps the state in between: the incident id from ServiceNow, the pending approval, the retry on the incident call if ServiceNow is briefly unavailable. A rejected approval still publishes a status event, because downstream consumers care about the outcome either way. None of this lives in application code, a database the team has to operate, or a runbook.
Orchestrating Kafka Platform Operations with the Same Control Plane
The workflows so far react to business events. The same control plane also operates the streaming platform underneath. Connector lifecycle in Kafka Connect, rollout of Apache Flink jobs, topic provisioning with schema checks against the Schema Registry, certificate rotation for mTLS, maintenance windows with rolling restarts, and failover rehearsal for disaster recovery. Each is a workflow with steps, approvals and retries, not a collection of shell scripts and runbooks.
Kestra models the pieces these workflows touch as Assets: topics, certificates and datasets with locks and lineage. Two maintenance flows cannot rotate the same certificate at the same time, and a topic’s lineage shows which flows read and write it. The CMDB stays the system of record. The orchestrator holds the resource model it needs to execute safely.
Unified Orchestration Above the Kafka Event Backbone
Event-driven workflow orchestration on Kafka is one entry point into a broader architecture. Unified orchestration means one control plane for data, infrastructure, applications and business processes. AI agents are governed across all four domains. The machine downtime flow already crosses three of them: a data event, a business approval, a command into an application queue. The platform operations in the previous section cover infrastructure.
I explored what this looks like in three industries:
- unified orchestration in manufacturing from shop floor to cloud
- unified orchestration in financial services across the bank
- unified orchestration in telecom across OSS, BSS, and IT
The backbone stays what it is. Facts are published once on Kafka and consumed by many. Flows are triggered by those facts and publish their results as new facts. Orchestration completes the event-driven architecture rather than replacing it.
A Governed Agent Step in an Event-Driven Flow
AI agents are becoming a new kind of consumer on the event backbone. An event arrives, an agent step reasons about it and calls a tool over MCP, a human approves the proposed action, and the workflow writes the result into the system of record. The agent is one step in a defined flow and carries the same boundaries as every other step: namespace-scoped access, versioned flow definitions, and an audit log on every call. A rejected proposal ends the flow with no action taken, and the rejection is recorded.
This separates a governed agent from an autonomous one wired directly to a topic. The event-driven pattern stays identical. Only the step type changes. My Trusted Agentic AI Landscape covers the vendors and frameworks in this space.
Event-Driven Workflow Orchestration on the Kafka Cluster You Already Run
Kafka delivers the message in order, replayable and decoupled. The orchestration decision is a separate concern, and most tools that make it cannot touch the backbone. A database-bound scheduler polls past it, a consumer service multiplies around it, and a hand-built engine on Kafka Streams eats the engineering budget of a platform team.
An event-driven orchestrator closes the gap from both sides. The internal queue runs on the Kafka cluster an enterprise already operates, or on plain PostgreSQL where that is enough. The engine stays event-driven on both. Topics become triggers, and the reaction to an event becomes a defined workflow with retries, human decisions and an audit trail instead of another service nobody remembers owning. The streaming platform itself becomes something the orchestrator operates.
In 2023 I argued that Kafka could serve as the workflow engine. The better answer is an orchestrator that treats Kafka as what it is: the most reliable piece of infrastructure in the building, and the natural backbone for event-driven workflow orchestration.
To follow this work across data integration, workflow orchestration, process intelligence, and trusted agentic AI, subscribe to the newsletter and connect on LinkedIn.
