Stream processing is one of the most valuable capabilities in modern enterprise architecture, and one of the most frequently misapplied. Knowing when not to use stream processing is the harder skill, and it is the question that rarely gets asked. Four years ago I wrote When NOT to Use Apache Kafka, still one of my most-read posts, because architects appreciate real boundaries more than another success story. This is the same treatment for stream processing. The short answer: it is the wrong choice when a caller is waiting for a response, when state has to be queried and corrected directly, or when the work needs locks rather than transformation.
I spent more than nine years at Confluent, focused on event streaming and stream processing, explaining the value and the use cases to engineering teams, business stakeholders, and executives. I recently joined Kestra as Global Field CTO. One of the first architectural decisions I learned about was that Kestra 2.0 removes Kafka Streams from the core of the product and rebuilds the messaging layer around plain queueing.
That triggered an old memory.
Confluent Control Center, the management and monitoring tool for Confluent Platform, struggled with scale and startup time for years. In May 2025, Confluent shipped a rearchitected version centered on Prometheus.
Two companies that understand event streaming as well as anyone both simplified an internal core. Neither concluded that stream processing is bad. Both concluded it was the wrong architecture for that particular job.
When is stream processing the right choice?
Stream processing continuously computes results over unbounded streams of events as they arrive, rather than answering questions about data that has already landed.
The family includes Apache Flink, Kafka Streams, and Spark Structured Streaming. They differ in deployment model and API, and they share a core design: continuous computation over a partitioned flow of events, with state derived from that flow. The work is initiated by data arriving, not by anyone asking for a result.
The capabilities below run from the simplest to the most demanding, and every example is a production deployment rather than slideware.
- Stateless transformation and enrichment. Each event is handled on its own, with no memory of what came before. Filtering, cleansing, format conversion, and lookup enrichment applied once in the streaming layer then serve every downstream consumer, rather than each team rebuilding the same logic inside its own warehouse. This is a first-class use of the technology rather than a lesser one, and the full pattern is in The Shift Left Architecture 2.0. The caveat is that the simplest cases need no framework at all, as I argued in Apache Flink: Overkill for Simple, Stateless Stream Processing and ETL?
- Joins across live sources. Two or more streams are correlated as they flow, which requires holding recent state from each side. CARIAD connects millions of Volkswagen Group vehicles through an event-driven platform, joining telemetry with reference data for predictive maintenance and fleet management.
- Windowed aggregation. Events are grouped into time ranges and summarized continuously. Siemens Healthineers streams machine data from CT scanner production lines and aggregates it to predict component wear before a line stops. The wider approach is in Shift Left Architecture at Siemens.
- Sub-second decisioning where delay costs money. The value of the answer decays within seconds, so nothing that runs on a schedule can deliver it. Mobility platforms including FREE NOW (Lyft), Grab, and Uber block GPS spoofing and payment abuse as it happens, because a fraud decision made an hour later is a loss already booked.
- Complex event processing. The signal is a pattern across several events rather than any single one, often within the same stream and sometimes across streams. That includes absence detection, where the pattern is an expected event that never arrives within its time window. Etihad Airways runs airline operations on Kafka and Flink, where the valuable alert is often a connecting bag that was never scanned. I covered the capability in depth in Complex Event Processing with Apache Flink.
- Continuously maintained tables. Flink keeps results current in table storage like Apache Paimon or Fluss for real-time analytics at extreme scale. More in Data Streaming Meets Lakehouse.
I have spent years arguing for these use cases and I still would. Nothing below walks that back.
Two products that rebuilt their core without a stream processing framework
Both examples are software vendors, and in both cases the product itself was built on a stream processing framework that turned out not to fit the job it was doing.
Confluent moved Control Center’s metrics pipeline from Kafka Streams to Prometheus
Confluent Control Center (C3) is the commercial web interface for managing and monitoring Confluent Platform clusters, connectors, and topics. It shipped in 2016. Its metrics pipeline used Kafka Streams, fed by client-side monitoring interceptors, with local state in RocksDB.
Confluent’s documentation said so plainly: the Stream Monitoring functionality of Control Center is implemented as a Kafka Streams application. The system requirements page recommended at least 32 GB of RAM, 8 cores, and 300 GB of storage, preferably SSDs, because Control Center relies on local state in RocksDB. Sizing guidance like that tells you a lot about an architecture.
Confluent published the results of the rewrite. Startup time dropped to roughly one minute, down from between 15 and 50 minutes. Supported partitions went from about 120,000 to 400,000. Metrics freshness improved from over five minutes to two or three. The separate Kafka cluster previously required for metrics storage disappeared, and the monitoring interceptors went with it. Confluent’s documentation now files them under Control Center Legacy.
A cold start of 15 to 50 minutes on a monitoring tool is not a tuning problem. It is the cost of rebuilding derived state, and no amount of configuration makes that cost disappear. Confluent’s engineers clearly understood this, which is why the fix was a new architecture rather than another round of tuning.
Confluent’s announcement focuses on the new architecture, not the old one. The numbers make the point on their own.
Confluent Platform 8.3, released in July 2026, continued in the same direction: Flink operations moved into a dedicated Confluent Manager for Apache Flink UI, and Kafka Streams and client monitoring moved into Unified Stream Manager (USM). USM collects client metrics directly from the brokers, with no client-side instrumentation. The console that monitors Kafka Streams applications now gets its data without running anything inside them, a clean separation the old interceptor design could not offer.
Kestra rebuilt its messaging layer around plain queueing
Kestra is an open source unified orchestration platform under the Apache 2.0 license: one control plane to automate and coordinate workflows across data, infrastructure, applications, and business processes. Its founding team published their reasoning for 2.0. Their engineering post describes Kafka Streams as a streaming framework rather than a queue, notes that the team had used nearly every feature it offers, and lands on an argument I find more persuasive than any benchmark: when a critical part of your infrastructure can only be maintained by a handful of people, you have a risk, not a foundation.
The workload made the mismatch sharper, not milder. Orchestration is stateful processing pushed to the extreme. The whole point is to keep state on every execution and every task, so state is not a byproduct of the computation, it is the deliverable. And orchestration coordinates dependencies across executions, where one execution triggers or waits for another. With Kafka partitioning, the event that unblocks a waiting execution lives in a partition unrelated to the one currently being processed, and correlating the two is work the framework never makes natural.
One point is easy to misread: Kestra 2.0 is still event-driven. Nothing about the execution model became synchronous or batch. What changed is that the event-driven design no longer depends on a stream processing framework. The same architecture now runs on a relational database, a message broker, or a distributed log, and the queue and the data layer are chosen independently. Two very different products reached the same conclusion.
Continuous processing, queries, and calls are three different architectures
The two rewrites look like technology swaps. They are the same architectural correction, and it applies well beyond Kafka Streams.
Three patterns show up in every enterprise architecture, and they are not interchangeable:
- Stream processing is continuous and driven by data arrival. Events arrive, computation runs, results flow onward.
- A database query is pull-based and on demand. A caller asks a specific question at a specific moment and expects an answer shaped by that question.
- An API call is request and response. A caller asks the system to do something and waits to learn whether it worked.
Both rewrites happened because the internal use case was mostly the second and third pattern, running on infrastructure designed for the first. A monitoring interface asks questions. A workflow orchestrator is told what to do. Neither is primarily a flow of events to be transformed.
Latency separates the patterns too, and not in the direction most people assume. Stream processing engines are not tuned for per-message latency. In Kafka Streams, records are fetched in batches and results are held by commit intervals and caching before they become visible, which shows up most at low throughput. A plain queue consumer processes message by message and reaches very low latency per message. Kestra measured exactly this while validating its 2.0 backends: Redis and AMQP delivered consistently low per-message latency, and so did Kafka at low throughput.
Why vendors struggle to merge the three patterns
Plenty of products have tried to collapse this distinction. Streaming databases promise one system that ingests events continuously and answers queries on demand, and ksqlDB put SQL on top of Kafka Streams for the same reason. Some of these products are technically impressive, and none has reached the adoption of a conventional database or a conventional stream processor.
Product quality is not the reason. A query returns a result. A continuous query returns a result that keeps changing and sometimes retracts what it said a minute ago. A team has to change how it thinks before it gets the first useful outcome, and most teams will not sign up for that on the strength of a product demo.
What operations teams inherit with derived state
A state of record is the authoritative, mutable, directly queryable version of data. Derived state is a computed view that can be rebuilt from an underlying log.
Stream processing engines hold derived state by design. In Kafka Streams it lives in local RocksDB stores backed by changelog topics. In Flink it lives in operator state captured by checkpoints. The mechanism differs, but the properties are the same in both.
Derived state is partitioned by key and co-located with the operator that computes it. Retrieval works by key or key range. Ad hoc queries are not part of the model. Both ecosystems tried to soften this, and neither treats it as a first-class path: Kafka Streams has Interactive Queries, and Flink’s equivalent queryable state was deprecated for lack of demand.
What this costs in production
Loïc Mathieu, a Kestra engineer who worked on both the old and the new architecture, put the cost in three points: RocksDB state stores get expensive once you hold a lot of data. Rebalancing is costly. And the framework itself is complex enough that not everyone on the team had good knowledge of it.
The operational consequences follow directly. Operators cannot inspect or repair state. There is no UPDATE statement against a changelog topic or a checkpoint, so when production state goes bad, the only fix is a rebuild. The service cannot serve requests until that rebuild finishes, and the more state there is, the longer it takes. Control Center’s startup time was this property, visible from the outside.
Kestra hit the same wall from a different direction. Loïc recalls a customer deployment on Kubernetes where startup approached 45 minutes, because the execution state store sat on non-persistent storage and had to be rebuilt in full on every restart. Persistent volumes avoid that, which is exactly the point: the architecture works only as long as every deployment gets the storage assumption right.
Florian Hussonnois, another Kestra engineer, makes the point that lands hardest: during customer upgrades the team routinely spent more time explaining Kafka Streams internals such as state store restoration than explaining Kestra itself.
None of this is a defect. Derived state is exactly right when state is a means to a computed result. It is the wrong choice when state is the product.
What developers and data engineers have to relearn
Operational cost is the visible part. The larger cost is that stateful stream processing asks developers to learn a model that does not resemble anything else they use.
Most developers have deep, transferable intuition for two things: a request that returns a response, and a table that answers a query. Stateful stream processing shares almost none of it. Event time and processing time diverge. Watermarks decide when a window closes. Late data arrives after the answer was already published. Results retract and revise.
State has a lifecycle someone has to design deliberately. Expiration mechanisms exist, such as state time-to-live (TTL) in Flink and retention on windowed stores in Kafka Streams, but they only help once someone configures them and reasons through what expiring state means for correctness.
A SQL interface does not solve this. SQL makes the syntax familiar and leaves the semantics unchanged. A developer can write a continuous SQL query that looks correct, passes review, and then behaves unpredictably the first time data arrives late or out of order.
Data engineering is a separate world again. The mental model there is tables, DAGs, and scheduled runs, and a large share of that work does not need continuous processing at all. Building dimensional models, backfilling history, and reconciling against source of record are bounded jobs against bounded inputs. Incremental batch handles them well and fails in ways the team already knows how to debug. I worked through where that line falls in Kafka vs Flink vs Spark: Do You Really Need Real-Time?
Processing guarantees are not database transactions
This is where architects most often talk past each other, and the confusion is not specific to Kafka.
Modern stream processors offer strong correctness guarantees. Kafka Streams commits offsets, state updates, and output records atomically through exactly-once semantics. Flink achieves the equivalent through checkpointing and two-phase commit sinks. Both are mature and run in production at serious scale.
Those guarantees answer one question: was this event reflected in the result exactly once, despite failures?
They do not answer the questions a database transaction answers. There is no lock to take. No isolation level across unrelated entities. No arbitrary read-modify-write spanning multiple keys. A system that must guarantee only one execution of a given process runs at a time needs a mutual exclusion primitive, and a processing guarantee is not one.
Loïc named this as one of the hardest parts of the old Kestra architecture: implementing flow concurrency was very challenging given the lack of locks. The workaround needed one state store with a prefix query just to count running executions, and a second state store with another prefix query to pop a queued execution once a running one finished. Getting the algorithm right took multiple attempts. In a database, the same requirement is a constraint and a lock.
I made a version of this argument in 2022 in Analytics vs. Transactions in Data Streaming with Apache Kafka. The trade-offs run between streaming transaction APIs, stateful orchestration in a separate application, and classic two-phase commit as implemented by systems like IBM MQ and Oracle Database. The storage side of the same question is in Can Apache Kafka Replace a Database?
The engine matters less than the fit
Kafka Streams and Apache Flink are both excellent, and I have written at length about why they are a match made in heaven rather than rivals. Flink runs as its own cluster with checkpointing, rescaling, and mature SQL, which changes the operational picture considerably compared to a library embedded in an application. But the engine comparison is the second question. The first is whether continuous stream processing fits the workload at all, and if the answer is no, then choosing between engines means choosing between two wrong answers with different operational profiles.
Where the backbone ends and the processing layer begins
Event-driven architecture has become a foundation of modern enterprise integration, and the direction is broadly vendor-independent. Not everyone converged on it as the center. What did happen is that almost everyone now supports it alongside request-response: open source projects, cloud-native services, commercial streaming platforms, and modernized iPaaS and ESB products.
The same shift reached the application layer. Salesforce and SAP both expose eventing and change data capture interfaces today, in addition to their web service APIs, and agent-facing interfaces based on MCP are arriving next. Enterprises no longer have to poll a SaaS platform to find out that something changed. I map the vendors and the paradigms in the Data Integration Landscape 2026.
Stream processing sits on top of that foundation. It is a complementary capability for specific workloads, not a prerequisite for an event-driven architecture. Plenty of organizations run a healthy event-driven core with no stream processing engine anywhere in it, and they are not doing it wrong.
How much processing belongs in the broker?
Brokers are not dumb pipes either, though capabilities vary more than most people assume. Routing is universal. Server-side filtering and schema enforcement are product-specific, and Apache Kafka offers neither: consumers read everything in their partitions, and schema validation comes from a commercial distribution rather than the broker itself.
Kafka is deliberately simple at the broker. That design came out of LinkedIn, where the requirement was moving enormous volumes cheaply, and keeping the broker out of the data’s way is what made the throughput possible. As Kafka spread into enterprise applications, transactional workloads, and deployments where scale is not the constraint, the requirements shifted, and people started asking why the broker cannot filter or validate on their behalf.
The direction of travel is toward more, not less. Confluent Platform 8.3 introduced centralized enforcement of data quality, schema validation, and encryption at the gateway with no client changes, and Confluent frames the shift in its own words as moving from dumb pipes to a smart data plane. That is a reasonable place to put governance. It is not a reason to put business logic there.
Where a broker does support this work, it overlaps with stateless stream processing, and both choices are defensible. The broker means fewer moving parts and enforcement nobody can bypass. A stream processor means separation of concerns, ordinary testing and versioning, portability across brokers, and no application logic competing for broker CPU. Shift Left favors the stream processor, because logic applied once in the streaming layer serves every consumer downstream.
The boundary that is not negotiable sits further along. Stateless work has several possible homes, and stateful work has one: joins, windows, and aggregation across events ideally use a stream processor, and no broker roadmap changes that. I worked through the distinction in Stateless vs. Stateful Stream Processing with Kafka Streams and Apache Flink. Keep governance in the backbone and business logic out of it, or you rebuild the ESB bottleneck the industry spent fifteen years dismantling. I went deeper on the governance side in Policy Enforcement and Data Quality for Apache Kafka with Schema Registry.
Why a framework is harder to reverse than a broker
A stream processing framework is not a component you swap out later. It shapes topology, partitioning, state, and failure handling across the entire application. Adopting one is an architectural commitment, and reversing it is a rewrite.
Messaging is an interface. The same application logic runs over IBM MQ, Kafka, RabbitMQ, Redis, or even a plain database table used as a queue. Decoupling services through queues has worked for decades and nothing about it is exciting, which is exactly what you want in the layer everything else depends on.
Kestra 2.0 illustrates the difference rather than proving it. Removing the streaming framework from the core is what made AMQP, JDBC, Kafka, and Redis backends possible behind thin adapters. The change shows up at the high end. Kafka was never required to run Kestra, since a JDBC backend existed long before 2.0, but the highly available architecture did depend on it. Now it does not, so an organization unwilling to operate Kafka has several supported paths to high availability, and one already running Kafka can keep it. The architecture stayed event-driven either way.
The trade-offs across messaging protocols are in When to Use AMQP, JMS, Kafka, or MQTT.
Where eating your own dog food breaks down
Running your own product in production is a good practice, and I want to be clear about that before criticizing it.
Vendors who dogfood find the bugs customers would otherwise find. They feel the operational pain their users feel. They ship better defaults. Confluent building Control Center on Kafka Streams and Kestra building its executor on Kafka Streams both came from the right instinct.
The failure mode is specific. Dogfooding stops validating and starts distorting when a vendor uses its product for a workload no customer would use it for.
Confluent’s own engineers show what the alternative looks like. When they designed the Confluent Cloud telemetry pipeline, they evaluated ClickHouse, Druid, and Pinot against stated requirements: a hundredfold increase in data and query load, sub-second latency on high-cardinality metrics, and native time-series support. They chose Apache Druid, and Druid now powers Confluent Cloud monitoring dashboards, Stream Lineage, the Metrics API, and internal billing.
So Confluent faced the same requirement twice and answered it the same way both times. Druid in the cloud, Prometheus in self-managed Control Center. Kafka moves the data in both architectures, and a purpose-built store answers the questions in both.
The test is simple: dogfood the use case you sell. A monitoring interface with a queryable UI is not the workload Kafka Streams is built and sold for, and using it there meant living with the mismatch for years.
When is stream processing the wrong choice?
Six signals point away from a stream processing framework.
- The latency requirement sits outside the middle band. Stream processing pays off between seconds and low milliseconds. If one report reads the result once a day, or a dashboard refreshes every few minutes, continuous computation buys nothing. If every single message needs the lowest possible latency, a plain queue is faster, because it hands over each message as it arrives rather than through commit intervals and caching.
- The interaction is request and response. Something calls in and waits for an answer. Those are API semantics, and a stream processor is a poor way to deliver them.
- State is the product, not a byproduct. It has to be queried on demand, corrected by hand when it goes wrong, and audited. You are describing a database.
- Restart speed determines availability, and the state is large. Rebuilding derived state takes as long as it takes, and the service is unavailable until it finishes.
- The work needs locks or isolation across entities. Coordination and mutual exclusion are database primitives. Processing guarantees do not provide them.
- Only a few people can maintain it, and hiring will not change that. Both rewrites in this post cite some version of this, and teams routinely discount it until someone leaves.
These are not a scorecard. Each one deserves an explicit answer before you commit, because the commitment is expensive to reverse.
Why Kafka still fits even when stream processing does not
Stream processing solves real problems that nothing else solves as well. Continuous computation over event streams remains underused in most enterprises, and I will keep saying so.
It is also not a substitute for a database, an analytics engine, or a queue. Confluent reached that conclusion for its own telemetry and its own monitoring interface. Kestra reached it for its own executor. Kestra wrote up the reasoning in detail, and Confluent published the numbers and the new architecture. Neither had to say anything publicly.
One clarification, because “use messaging instead” gets misread as “use Kafka less.” Kafka remains an outstanding choice for this kind of architecture, and often the best one. Unlike a classic message queue, Kafka keeps events after they are delivered and lets any consumer replay them from any point. Producers and consumers are therefore decoupled in time as well as in space, a new consumer can read the full history without asking anyone to resend it, and the enterprise keeps one copy of the truth instead of a point-to-point mesh.
High availability and horizontal scale are the other reasons, and both rest on the same two primitives: partitioning and replication. Few messaging systems offer both as maturely as Kafka does. Choosing Kafka as the backbone and choosing a stream processing framework for your application logic are two separate decisions. The first is usually easy. The second deserves the scrutiny in this post.
Most architects reason fluently about queues, APIs, and databases. Their fluency is not a limitation to design around. It is information about where additional complexity pays for itself, and where it only accumulates.
Choose the architecture that fits the workload, not the one you already know best. The two companies in this post did exactly that, and both are better for it.
Stay informed about the latest on data integration, process intelligence and workflow orchestration, and trusted agentic AI by subscribing to my newsletter and following me on LinkedIn.
