Every year I publish the data streaming trends after the landscape. This edition is different in two ways. It carries a quarter in its name, because the market moved faster in the first half of 2026 than in the previous two years together. And it names ten trends instead of six, because the changes did not stay inside the platform.
Data streaming moves and processes data continuously as events happen. Apache Kafka is the protocol most of this market now implements. Data streaming in 2026 is no longer a question of whether to stream. The questions are what belongs in the stream, where it runs, and who controls it. The ten trends below follow that shift, grouped into what is happening in the market, what is changing in the architecture, and what good practice looks like now.
This post builds on the Data Streaming Landscape Q3 2026, which maps the vendors on two axes: the workload a platform serves and the operating model it leads with. The trends explain why those axes exist.
The December 2025 edition named six trends for 2026: consolidation, diskless Kafka and Iceberg, real-time analytics in the stream, zero data loss, regional deployments, and streaming for agentic AI. All six held, and several landed faster than expected. Two changed shape. Diskless and Iceberg split into separate trends because they now move on different clocks. Real-time analytics in the stream turned into a broader question about which engine runs the processing. Three trends are new: diverging protocol implementations, governance across platform boundaries, and the discipline to say no.
Data Streaming Trends in the Market: Vendors, Ownership, and Jurisdiction
1. Consolidation and migration tooling in every direction
Consolidation was the defining event of the data streaming market this year. IBM closed the Confluent acquisition in March, Redis absorbed Decodable, CoreWeave took Bufstream in-house, and Redpanda bought Oxla. The other half of the trend is migration tooling, which became a competitive weapon on every side.
Confluent released KCP, a free open-source CLI. It started as MSK-to-Confluent-Cloud discovery and now covers the whole path: cluster inventory and cost assessment, Terraform for the target environment, and Cluster Linking for the data. Since April KCP also includes a Kafka proxy that redirects clients to the new cluster with a bootstrap change.
Redpanda’s Shadowing does the opposite direction, in-broker replication of topics, schemas, offsets, and ACLs out of Confluent and other Kafka services.
Amazon turned MSK Replicator into a migration product. It has replicated from external Kafka clusters into MSK Express brokers since April and into Standard brokers since July, with bidirectional consumer offset sync so producers and consumers move in any order. AWS also runs a dedicated MSK migration page that names Confluent Platform, Aiven, Redpanda, WarpStream, and AutoMQ as the clusters to migrate from.
The lesson for architects is that switching cost is falling on the protocol layer and rising everywhere else. Moving topics between two Kafka-protocol platforms is now a scripted exercise. Moving connectors, governance, stream processing jobs, and operational tooling is not. The real lock-in of a data streaming platform sits there today. Every migration story a vendor tells you is also a story about what did not migrate.
2. Data sovereignty and deployment control as table stakes
Sovereignty stopped being a European regulatory topic and became a first-order axis of the landscape. Every serious vendor now leads with where its software runs. Confluent Private Cloud targets regulated industries. Aiven and Redpanda offer BYOC, Axual and Cloudera run self-managed and air-gapped, Oracle puts managed Kafka inside Dedicated Region and Cloud@Customer, and Microsoft partners with Mistral on European-operated compute. Self-managed deployment is recovering ground it lost a decade ago, and the reasons are jurisdiction and continuity rather than cost.
The event that made this universal did not happen in data infrastructure. On June 12, 2026 the US Commerce Department directed Anthropic to block access to its two newest models for any foreign national, inside or outside the United States. Anthropic cannot verify nationality per account, so it disabled both models for every customer worldwide. Commerce lifted the controls on June 30 and access returned the next day. A capability that enterprises had already built into products was switched off for nearly three weeks by a decision none of them were part of. Every board that watched that episode asked the same question about the rest of the stack, including the platform that carries the operational data.
The sovereignty chapter of the landscape covers the deployment spectrum in detail. Fully managed brings the vendor’s jurisdiction with it. BYOC differs by vendor in what stops when the control plane is unreachable. Only self-managed and air-gapped put the software fully under your control.
3. Diverging Kafka protocol implementations
The Kafka protocol won the interoperability layer. In 2025 I called this the democratization of the protocol. In 2026 the consequence shows: compatibility claims multiply while semantics and ecosystem depth differ in production. Confluent’s Kora and Freight, Redpanda’s R1, WarpStream, AutoMQ, and Azure Event Hubs re-implement the protocol on their own architectures. Snowflake Datastream speaks the wire protocol without Kafka underneath. Databricks Zerobus accepts producers only. Google Cloud, Amazon, and Oracle sell managed Apache Kafka next to their proprietary streaming services, which is the strongest evidence that the protocol is the standard, not the product.
What differs is transactions, compaction semantics, exactly-once guarantees, Kafka Streams and Connect support, and the tooling around them. Two systems that both pass a client compatibility test can behave differently under a rebalance storm or a broker failure. My June post on why Databricks and Snowflake speak the Kafka protocol draws the line between using the API to feed a lakehouse and running Kafka as the operational backbone. Evaluate the implementation, not the checkbox.
Architecture Trends: How the Streaming Platform Itself Is Changing
4. Diskless Kafka and object-storage-first architectures
Diskless Kafka moved from vendor experiment to community direction. On March 2, 2026 the Apache Kafka community accepted KIP-1150, Diskless Topics. Acceptance settles the debate about where Kafka’s storage layer is heading: object storage as the durability layer for active data, brokers as stateless coordinators, and a per-topic choice between classic disk-backed and diskless.
It does not put a diskless topic into a Kafka release. KIP-1150 is an umbrella proposal, and the implementation KIPs 1163 to 1165 were still under discussion in mid-2026. Aiven’s Inkless fork is the only implementation of that upstream design running today, and Aiven says it will retire the fork once the code lands in the project.
Products built on their own designs are further along. WarpStream and Confluent Freight (both IBM since March), AutoMQ, Bufstream (CoreWeave), and Aiven Inkless all ship object-storage-first topics now. The gap between shipping products and the open-source standard is the practical point. If you need object-storage economics for high-volume, latency-tolerant workloads today, you buy a product and accept its implementation. If your posture is upstream-only, you wait years, and tiered storage (KIP-405) remains the available option for cold data. The economics are covered in The Rise of Diskless Kafka. The 2026 update is that the community agreed on the destination while the vendors are already there.
5. Streaming meets the lakehouse with Apache Iceberg
The direction is the same at every vendor: the topic becomes the table. Confluent Tableflow writes Iceberg and Delta, with upserts, both formats on one topic, and a dead letter queue for malformed records. StreamNative Ursa writes Iceberg and Delta, Snowflake Datastream lands topics as governed tables, Databricks Zerobus takes Kafka producers into the lakehouse, and Apache Fluss tiers into Paimon and Iceberg. Confluent’s Q2 release added schema IDs in message headers and Materialized Tables for Flink so that governed topics are lakehouse-ready by default.
In July the first hyperscalers joined. Amazon MSK data delivery to streaming tables for Apache Iceberg is generally available. A topic on Express brokers materializes continuously as an Iceberg table on S3 Tables, with inline compaction against the small-file problem and no connector, Flink job, or custom consumer in between. AWS puts the saving at up to 60 percent on ingestion against a self-managed pipeline and up to 30 percent on downstream queries from better file sizing.
The reality in most enterprises is still a connector and two copies, and that is often the better pattern. Kafka holds the raw event log for replay and the operational consumers that need low latency. Iceberg holds the governed tables for analytics. Zero-copy is rare for the same reason zero-ETL turned out to be rare: the work is not the copy, it is schematization, type conversion, schema evolution that stays compatible in both directions, and data quality applied before the write.
Read the documentation of the managed products and that work is still there, only relocated. The MSK channel needs a Glue Schema Registry schema to convert records and a dead letter queue for the ones it cannot. It partitions by time only. It does not backfill, so nothing produced before you switched it on ever arrives. Each channel creates its own table, delivery into an existing Iceberg table is not supported, and the table it creates belongs to the service: query it, but do not evolve its schema or write to it. Tableflow ships its own dead letter queue for the same reason, and its documentation describes the same list of jobs.
None of that argues against either product. It is what the boundary costs when someone else operates it. My November post Data Streaming Meets Lakehouse covers those challenges. Through 2027 the products will make the one-copy path easier, and the operational event log will still belong outside the analytics perimeter.
6. Operational maturity from disaster recovery to Queues for Kafka
Last year’s trend was zero data loss and disaster recovery. That requirement stayed, and the list around it grew. Synchronous multi-region replication for RPO zero, which I covered last year for WarpStream and Confluent, is now a standard question in every enterprise evaluation. The offset and ACL fidelity that makes Redpanda’s Shadowing a migration tool also makes a migration as verifiable as a failover, and buyers now test for both.
The newest item on that list can retire the separate message broker that sat next to Kafka, at least for one pattern. Queues for Kafka (KIP-932) is generally available with Apache Kafka 4.2, in Confluent Cloud and Confluent Platform 8.2. Java clients ship today and Python is in preview, and Spring for Apache Kafka 4.1 supports it in production. Share groups let more consumers than partitions process a topic with per-message acknowledgment, which is the capability that sent teams to RabbitMQ, ActiveMQ, or SQS for work queues in the first place. The work queue is the pattern that comes home. Strict ordering, exactly-once semantics with transactions, request-reply, and legacy protocols like JMS or AMQP are out of scope in the GA release, which is why I wrote a separate post on when not to use Queues for Kafka. A dedicated broker still earns its place in plenty of architectures.
The pattern behind these features is the same: buyers stopped asking about throughput and started asking about behavior under failure, migration, and consumer scaling. Feature lists matter less than documented behavior under those conditions.
7. Stream processing beyond Flink SQL
Flink SQL is the default entry point for stream processing, and for good reason. Three developments show it is not the whole story. Spark Structured Streaming’s real-time mode is generally available. Sub-second latency arrives in the engine most analytical teams already run, which matters because the Analytical and Self-Managed quadrant of the landscape runs on Spark far more than on Flink. The Flink DataStream API remains the choice for complex stateful logic in Java, where SQL becomes unreadable or impossible: custom state, timers, side outputs, exactly-once sinks with business logic in the middle.
The third development is the return of complex event processing (CEP). Pattern detection across sequences of events, with absence detection and time windows, is exactly what SQL expresses poorly and what Flink CEP and MATCH_RECOGNIZE express well. The use case that brought it back is agentic AI: an agent should act on a confirmed pattern, not on every raw event. I wrote about what Flink CEP is and when not to use it and about Flink CEP as the pattern layer for autonomous decisions in April. The legacy CEP vendors are gone; the capability came back inside the stream processor.
8. Cross-platform governance with catalogs and Kafka gateways
Every streaming platform now governs itself well: schema registry, lineage inside the platform, access control, data quality rules. What none of them does is govern across platforms, and consolidation widened that gap. A typical enterprise runs several Kafka-protocol platforms side by side, a Flink or Spark engine, one or two lakehouses, and a set of SaaS systems. Each of them shows its neighbors the surface and nothing underneath.
Two layers close the gap. Platform-independent catalogs and open lineage standards are the first. Snowflake shipped an OpenLineage API in public preview alongside metadata connectors that pull lineage from Postgres, SQL Server, dbt, and BI tools into one catalog. Confluent’s Tableflow registers tables in Snowflake Open Catalog. Apache Polaris, Unity Catalog, and Snowflake Open Catalog compete for the catalog position, and none of them is neutral yet.
Kafka gateways and proxies are the second layer. They sit in front of any Kafka-protocol cluster and route, filter, encrypt, and rewrite traffic without touching clients: Confluent Cloud Gateway, Kroxylicious from Red Hat, and Conduktor. The landscape’s governance chapter covers both layers. The trend through 2027 is that governance becomes an architecture layer you own rather than a feature of one platform.
Practice Trends: What Good Architecture Looks Like in 2026
9. Real-time context for agentic AI
Streaming for AI has two roles, streaming agents and the context engine, and both became products this year. Confluent’s Real-Time Context Engine and Streaming Agents are generally available. The release includes a managed MCP server and a Snowflake Cortex integration, and the Q3 update added usage and cost metrics. Redis launched Iris in May, a real-time context engine built on the Decodable acquisition: Redis Data Integration keeps the context fresh with change data capture, and the Context Retriever exposes it to agents through MCP. Snowflake sells the same idea inside its perimeter. MCP endpoints are appearing across streaming platforms.
The architecture pattern behind all of them is old and correct: continuously materialize the current state of the business from the event stream and serve it at low latency to whatever needs it. Agents made the pattern urgent because an agent approving a payment or changing an order acts on that state, and stale context is the most common cause of a confident wrong action.
Two cautions belong here. First, the context engine is a serving layer, not the system of record. The event log and the governed tables remain yours, which is what keeps the model and the agent platform replaceable. Second, “real-time context” is becoming a marketing term as fast as “zero-copy” did. Ask how fresh, from which sources, and under whose governance. The Trusted Agentic AI Landscape Q3 2026 covers the vendor side of that question.
10. When not to stream: the discipline to say no
The last trend is a sign of a market that grew up. Not every workload needs a stream processor, and knowing when batch or a queue wins is now a mark of good architecture rather than a lack of ambition. A nightly reconciliation job does not improve by becoming a Flink pipeline. A request that needs an answer now is a synchronous call. A work queue with per-message acknowledgment is a queue, and with Queues for Kafka it can live on the same platform without becoming a stream processing problem. I wrote the long version in When NOT to Use Stream Processing and in Kafka vs Flink vs Spark: Do You Really Need Real-Time?
Cost is what makes this a trend. Agentic workloads, diskless economics, and usage-based pricing put a price on every event processed. Teams that streamed everything in 2023 because the platform was there are now right-sizing. The platforms that make right-sizing easy keep the workloads that belong. Queues, tiered storage, and clear guidance on where streaming does not fit are what that looks like.
What the Ten Trends Mean for Your Architecture
Read together, the ten trends describe one shift. The platform question is largely answered: the Kafka protocol is the interface, Flink and Spark are the processing engines, Iceberg is the analytical storage format, and diskless is the storage direction. The open decisions moved up and out: which workloads belong in the stream, where it runs and under whose jurisdiction, and which layers you own so that everything else stays replaceable. This shift is why the landscape changed its axes this year, and why sovereignty and control appear in more than half of these trends.
Data streaming is the event-driven integration pillar of what I call the Trinity of modern data architecture. Process intelligence turns that data into an understanding of how the organization actually runs, and trusted agentic AI acts on it. Every one of the ten trends makes the streaming pillar cheaper, more controllable, or more useful to the other two. The vendors will keep moving. The full vendor-by-vendor picture is in the Data Streaming Landscape Q3 2026, free to read with no registration.
To follow this work across data integration, workflow orchestration, process intelligence, and trusted agentic AI, subscribe to the newsletter and connect on LinkedIn.
