Spark Archives - Kai Waehner

Durable Execution Engine with Restate Temporal DBOS vs Stream Processing with Kafka Streams Apache Flink Spark Structured Streaming

1.6K views
8 minute read

The Rise of the Durable Execution Engine (Temporal, Restate) in an Event-driven Architecture (Apache Kafka)

ByKai Waehner
5. June 2025

Durable execution engines like Temporal and Restate are redefining how developers orchestrate long-running, stateful workflows in distributed systems. Unlike traditional BPM tools focused on human-centric tasks, these engines automate machine-to-machine processes with built-in durability, retries, and fault-tolerant coordination. When integrated with event-driven platforms like Apache Kafka, they enable scalable, resilient architectures—handling complex business logic such as order processing, fraud detection, and multi-step transactions. This blog explores their capabilities, differences from stream processing tools like Apache Flink, Kafka Streams or Spark Structured Streaming, and the emerging role they play in modern enterprise infrastructure.

Confluent and Databricks for Data Integration and Stream Processing

1.9K views
10 minute read

Confluent Data Streaming Platform vs. Databricks Data Intelligence Platform for Data Integration and Processing

ByKai Waehner
5. May 2025

This blog explores how Confluent and Databricks address data integration and processing in modern architectures. Confluent provides real-time, event-driven pipelines connecting operational systems, APIs, and batch sources with consistent, governed data flows. Databricks specializes in large-scale batch processing, data enrichment, and AI model development. Together, they offer a unified approach that bridges operational and analytical workloads. Key topics include ingestion patterns, the role of Tableflow, the shift-left architecture for earlier data validation, and real-world examples like Uniper’s energy trading platform powered by Confluent and Databricks.

Data Streaming and Lakehouse - Comparison of Confluent with Apache Kafka and Flink and Databricks with Spark

1.9K views
9 minute read

The Past, Present, and Future of Confluent (The Kafka Company) and Databricks (The Spark Company)

ByKai Waehner
2. May 2025

Confluent and Databricks have redefined modern data architectures, growing beyond their Kafka and Spark roots. Confluent drives real-time operational workloads; Databricks powers analytical and AI-driven applications. As operational and analytical boundaries blur, native integrations like Tableflow and Delta Lake unify streaming and batch processing across hybrid and multi-cloud environments. This blog explores the platforms’ evolution and how, together, they enable enterprises to build scalable, data-driven architectures. The Michelin success story shows how combining real-time data and AI unlocks innovation and resilience.

Fraud Prevention in Mobility Services with Data Streaming using Apache Kafka and Flink with AI Machine Learning

2.4K views
7 minute read

Fraud Detection in Mobility Services (Ride-Hailing, Food Delivery) with Data Streaming using Apache Kafka and Flink

ByKai Waehner
28. April 2025

Mobility services like Uber, Grab, and FREE NOW (Lyft) rely on real-time data to power seamless trips, deliveries, and payments. But this real-time nature also opens the door to sophisticated fraud schemes—ranging from GPS spoofing to payment abuse and fake accounts. Traditional fraud detection methods fall short in speed and adaptability. By using Apache Kafka and Apache Flink, leading mobility platforms now detect and block fraud as it happens, protecting their revenue, users, and trust. This blog explores how real-time data streaming is transforming fraud prevention across the mobility industry.

10.9K views
21 minute read

The Data Streaming Landscape 2025

ByKai Waehner
4. December 2024
1 share

Data streaming is a new software category. It has grown from niche adoption to becoming a fundamental part of modern data architecture, leveraging open source technologies like Apache Kafka and Flink. With real-time data processing transforming industries, the ecosystem of tools, platforms, and cloud services has evolved significantly. This blog post explores the data streaming landscape of 2025, analyzing key players, trends, and market dynamics shaping this space.

Data Streaming Landscape 2023 with Apache Kafka Flink and much more

9.9K views
13 minute read

The Data Streaming Landscape 2023

ByKai Waehner
21. December 2022
1 share

Data streaming is a new software category to process data in motion. Apache Kafka is the de facto standard used by over 100,000 organizations. Plenty of vendors offer Kafka platforms and cloud services. Many complementary stream processing engines like Apache Flink and SaaS offerings have emerged. And competitive technologies like Pulsar and Redpanda try to get market share. This blog post explores the data streaming landscape of 2023 to summarize existing solutions and market trends.

11.0K views
5 minute read

The Heart of the Data Mesh Beats Real-Time with Apache Kafka

ByKai Waehner
28. July 2022
1 share

If there were a buzzword of the hour, it would undoubtedly be “data mesh”! This new architectural paradigm unlocks analytic and transactional data at scale and enables rapid access to an ever-growing number of distributed domain datasets for various usage scenarios. The data mesh addresses the most common weaknesses of the traditional centralized data lake or data platform architecture. And the heart of a decentralized data mesh infrastructure must be real-time, reliable, and scalable. Learn how the de facto standard for data streaming, Apache Kafka, plays a crucial role in building a data mesh.

Kappa Architecture vs Lambda Architecture for Apache Kafka Pulsar Data Lakes

47.6K views
17 minute read

Kappa Architecture is Mainstream Replacing Lambda

ByKai Waehner
23. September 2021
4 shares

Real-time data beats slow data. That’s true for almost every use case. Nevertheless, enterprise architects build new infrastructures with the Lambda architecture that includes separate batch and real-time layers. This blog post explores why a single real-time pipeline, called Kappa architecture, is the better fit. Real-world examples from companies such as Disney, Shopify, Uber, and Twitter explore the benefits of Kappa but also show how batch processing fits into this discussion positively without the need for Lambda.