Real-time is the most oversold word in data infrastructure.
Every vendor in the Kafka vs Flink vs Spark debate leads with it. Milliseconds. Faster engines. Benchmark charts where their product wins. The pitch is always the same, and it plays on a fear: fall behind on latency and you fall behind on the business.
Here is what the pitch leaves out: Most enterprise use cases do not need milliseconds. Many are fine with seconds. Plenty still run on request-response APIs or nightly batch, and that is acceptable. The one place that needs deterministic, hard real-time is OT, and these frameworks cannot serve it.
So who runs the speed race? The commercial vendors around the projects, not the projects themselves. Apache Kafka, Apache Flink, and Apache Spark are open source, built by large communities solving real problems. The faster marketing comes from the commercial vendors around them. The community ships capability. The vendors sell speed.
The real question behind Kafka vs Flink vs Spark is not latency. It is the SLA.
Do you really need real-time?
Ask four questions before you shortlist a single tool.
- What latency does the business actually require, as a number tied to a use case? Not “fast,” but 50 milliseconds, or two seconds, or ten minutes.
- Is this operational or analytical? A payment is not a dashboard.
- What is the SLA for data loss and uptime? Can you drop a record, or does the business break if you do?
- Where does the data live and get consumed? The edge, on-premise, one cloud region, or many?
Most teams cannot answer these, because nobody defined real-time for the use case. Until you can, latency benchmarks are noise.
Real-time is a spectrum, not a number
Hard real-time is a deterministic system with guaranteed response times and zero tolerance for latency spikes. That is OT: engine control, flight control, a collaborative robot on a factory line. These systems run on C, C++, or Rust in embedded environments. Kafka, Flink, and Spark are not hard real-time systems and never will be. This is a category boundary, not a flaw. I covered it in Apache Kafka is not hard real-time.
Everything else is soft real-time, and it splits into three tiers.
Critical real-time means microseconds. Trading engines at a stock exchange. This is not Kafka, Flink, or Spark. It needs specialized, co-located, often proprietary technology.
Low-latency real-time means tens to hundreds of milliseconds. Fraud detection in instant payments. Sensor analytics in manufacturing. End-to-end correlation in ride-hailing. This is the sweet spot for data streaming.
Near real-time means seconds. Streaming ETL into a warehouse. Regulatory reporting. This is still real-time enough for the business, and it covers a large share of production workloads.
Most enterprise use cases live in the low-latency or near real-time tier. The millisecond pitch targets a tier most workloads never reach. Even a single business shows this. At Nasdaq, the matching engine runs in the critical microsecond tier, but the surveillance, analytics, and reporting around it run in low-latency and near real-time, and Kafka fits those parts, not the matching engine. One use case, several latency tiers, one pipeline. I broke the Nasdaq trading architecture down in Apache Kafka is NOT real real-time.
Operational or analytical: the estate decides, not the latency
The distinction that matters more than latency is the estate, the operational side of the architecture versus the analytical side. Operational systems run the business and need transactions, uptime, zero data loss, and disaster recovery. Analytical systems make sense of the business and need throughput, cost efficiency, and freshness.
The operational side sits close to systems like the mainframe, Oracle, SAP, and Salesforce. The SLA here is about survival, not speed. If a payment is lost, latency was never the problem. The analytical side powers BI, reporting, and machine learning. The SLA here is about scale and cost.
As a rough mapping: Kafka and Flink fit operational workloads. Spark, Databricks, and Snowflake fit analytical workloads. Both matter, with different SLAs. Decide which side you are on first, then pick the tool. That decision drives more than any benchmark. And when a use case genuinely needs both at once, a unified engine like Apache Fluss can serve real-time analytics in one place.
Kafka vs Flink vs Spark: the speed race nobody won
The history of the speed race is instructive, because the fastest tool rarely won.
Pulsar is the clearest example. StreamNative ran benchmark campaigns showing Pulsar with lower latency and higher throughput than Kafka in specific setups. Pulsar won the benchmarks. It lost the market. Back in 2020 I went through the Pulsar vs Kafka claims one myth at a time, and the conclusion still holds today: the headline benchmarks were setup-dependent, and maturity, ecosystem, and community decided the outcome. Adoption stalled on architectural complexity and a thin ecosystem, while Kafka won on tooling, integrations, and support. Speed was never the deciding factor.
Redpanda made a similar bet with strong engineering. It rewrote the broker in C++ with a thread-per-core design and no JVM, pitched on raw low latency and operational simplicity. The market moved a different way. The cost conversation shifted to object storage economics with diskless designs like WarpStream and AutoMQ, and in March 2026 the Apache Kafka community accepted KIP-1150, the umbrella proposal for diskless topics that write directly to object storage. Redpanda read the room. In October 2025 it acquired the Oxla SQL engine and launched the Agentic Data Plane, then added an AI Gateway and governance in February 2026. The headline story moved from fastest broker to agentic enterprise data. The low-latency niche is real, for trading and fraud. It was not the growth engine.
Spark is getting faster too. Real-Time Mode was introduced with Apache Spark 4.1, stateless queries first, and reached general availability on Databricks in 2026. It breaks the microbatch floor with longer epochs, concurrent stages, and a streaming shuffle, reaching sub-second latency. Databricks markets it as faster than Flink for feature engineering, and as the reason you no longer need a second engine. The demand is concrete: feature stores and fraud scoring want fresher data.
Databricks and Snowflake speak Kafka at the ingestion layer
Data ingestion is the other front. Alongside the faster engine, Databricks now speaks the Kafka wire protocol through ZeroBus, and Snowflake announced Datastream, a Kafka-compatible service. In both cases existing Kafka producers stream straight into the lakehouse with a config change and no new code. Two separate parts of the platform, the engine and the ingestion layer, both aimed at the same goal: fresher data without leaving the lakehouse. That looks like the analytical vendors walking into the operational estate. It is not. Speaking the Kafka protocol for lakehouse ingestion is not running Kafka as an event-driven architecture, a distinction I made in Why Databricks and Snowflake Speak the Kafka Protocol. The lakehouse gets a continuous feed of the events. Kafka still runs the live business. Analytical vendors want the operational moment, but they are buying the feed, not the backbone.
The pattern across all of them is consistent. Raw speed rarely wins the deal. Ecosystem, SLAs, and fit win the deal.
Benchmarketing: every vendor wins its own benchmark
Benchmarketing is the practice of publishing a benchmark designed so the vendor wins. Most vendors does it. The setup is specific: chosen hardware, a favorable workload shape, tuned durability settings, and the percentile that flatters the result. Change one variable and the ranking flips. These numbers describe the vendor’s lab, not your production.
Take the MakeMyTrip case study from Databricks, which is strong engineering and a real use case. The headline is a P50 latency of 44 milliseconds for personalized last-searched hotels, down from over a second. Read one line further and the P99 is around 500 milliseconds, and the pipeline still runs Kafka for ingestion, Aerospike for state, and Redis for serving underneath the single-engine narrative. It is a personalization feature that lifts click-through, not a payment that demands exactly-once and zero data loss. The median headline is not the SLA. The tail and the estate are.
The fix is simple. Define your use cases and SLAs first. Then test the shortlist against those numbers, on your data, on your infrastructure. In many cases every tool will be fast enough, and the decision comes down to operations, cost, and ecosystem.
Does agentic AI need milliseconds?
Agentic AI raises the stakes for data, but not the way the marketing suggests.
An agent needs fresh, consistent context. Stale data is the failure mode. An AI initiative built on yesterday’s data fails from the start. That part is true and it matters.
Now measure the loop. An agent makes an LLM call, runs tool calls, retrieves context, sometimes waits for a human, and reasons across several steps. End to end, that is hundreds of milliseconds to several seconds. Shaving the data layer from 50 milliseconds to 5 milliseconds is invisible next to a two-second model call. Real-time context, yes. Microseconds, almost never. Hundreds of milliseconds or a few seconds is acceptable for most agentic use cases. Freshness and consistency beat raw speed here, the same lesson as everywhere else.
Erste Group is one of the largest banks in Central and Eastern Europe. They were clear about why they chose Kafka. Real-time messaging was not the requirement. Data consistency across systems was. That is what made Kafka the right fit. I covered it in this Erste Group case study. The same holds for AI: fresh and consistent beats fast.
When you really need real-time, use fewer tools
If a use case does need low-latency real-time analytics, complexity is the enemy, not latency.
The classic stack stitches together a broker, a stream processor, a serving store, a lakehouse, and a set of sync jobs. Kafka plus Spark or Flink plus Redis plus Iceberg plus glue. Every boundary between systems is a place where data diverges and freshness drifts.
A newer class of engines takes the opposite path. Instead of stitching systems together, they fold streaming storage, fast lookups, and lakehouse tiering into one layer. Apache Fluss is the clearest example, an incubating project, Flink-native with Spark support coming, created by the Flink SQL lead at Alibaba Cloud and supported by Ververica. Others push toward the same convergence from different angles, including RisingWave, Confluent Tableflow, and the Iceberg-plus-streaming direction the whole market is moving in.
Fluss unifies streaming storage, fast lookups, and lakehouse tiering in one system. It offers sub-second freshness, columnar streaming on Apache Arrow, primary-key tables, and native tiering to Iceberg and Paimon, with a union read across hot and cold data. The processing and analytics still run on Flink or a query engine on top. If the use case needs real-time analytics, one engine can replace five systems. It is early, so treat it as a direction, not a finished answer. The principle stands. Fewer moving parts, fewer places to fail.
Latency is physics: edge, on-premise, cloud, and air-gapped
You cannot optimize away the speed of light in a benchmark.
A cloud round trip adds latency no engine can remove. Real-time close to the source, in a factory, a vehicle, or a store, needs the edge or on-premise, not a hop to a distant region. Air-gapped environments in defense, OT, and critical infrastructure cannot use the cloud at all. This is where cloud-only analytical platforms hit a wall, and where Kafka and Flink at the edge, on-premise, or in hybrid setups fit the operational estate. Deployment location is part of the SLA. It rarely appears on a benchmark chart.
Kafka vs Flink vs Spark? Start with the SLA, not the speed!
Do not follow the real-time hype. Start with the business requirement, define the SLA, pick the estate, then choose Kafka vs Flink vs Spark vs something else. The order matters. Reverse it and you buy a benchmark instead of an outcome.
Look at the data integration landscape and the answer becomes clear. An event-driven architecture at the core is valuable. It creates consistency and decoupling that batch and point-to-point APIs cannot. But request-response APIs and batch are still there, and that is fine. The ideal is event-driven everywhere. The real world is a mix, and good enough is a legitimate target. The same spectrum, event-driven at the core with batch and request-response around it, shows up in workflow orchestration too.
What matters is not real-time. It is data consistency and the right SLA for the estate. Choose Kafka, Flink, Spark, or something else on that basis. For deeper context, see the Data Streaming Landscape 2026.
