Lambda Architecture Archives

Apache Kafka Transactions API vs Big Data Lake and Batch Analytics

3.8K views
9 minute read

Analytics vs. Transactions in Data Streaming with Apache Kafka

ByKai Waehner
9. March 2022
1 share
No comments

Workloads for analytics and transactions have very unlike characteristics and requirements. Many people think that Apache Kafka is not built for transactions and should only be used for big data analytics. This blog post explores when and how to use Kafka in resilient, mission-critical architectures and when to use the built-in Transaction API.

Kappa Architecture vs Lambda Architecture for Apache Kafka Pulsar Data Lakes

21.3K views
17 minute read

Kappa Architecture is Mainstream Replacing Lambda

ByKai Waehner
23. September 2021
4 shares
4 comments

Real-time data beats slow data. That’s true for almost every use case. Nevertheless, enterprise architects build new infrastructures with the Lambda architecture that includes separate batch and real-time layers. This blog post explores why a single real-time pipeline, called Kappa architecture, is the better fit. Real-world examples from companies such as Disney, Shopify, Uber, and Twitter explore the benefits of Kappa but also show how batch processing fits into this discussion positively without the need for Lambda.