Data at Rest Archives

Lakehouse and Data Streaming - Competitor or Complementary

2.9K views
12 minute read

How Microsoft Fabric Lakehouse Complements Data Streaming (Apache Kafka, Flink, et al.)

ByKai Waehner
12. October 2024
No comments

In today’s data-driven world, understanding data at rest versus data in motion is crucial for businesses. Data streaming frameworks like Apache Kafka and Apache Flink enable real-time data processing. Meanwhile, lakehouses like Snowflake, Databricks, and Microsoft Fabric excel in long-term data storage and detailed analysis, perfect for reports and AI training. This blog post delves into how these technologies complement each other in enterprise architecture.

Kappa Architecture vs Lambda Architecture for Apache Kafka Pulsar Data Lakes

33.0K views
17 minute read

Kappa Architecture is Mainstream Replacing Lambda

ByKai Waehner
23. September 2021
4 shares
4 comments

Real-time data beats slow data. That’s true for almost every use case. Nevertheless, enterprise architects build new infrastructures with the Lambda architecture that includes separate batch and real-time layers. This blog post explores why a single real-time pipeline, called Kappa architecture, is the better fit. Real-world examples from companies such as Disney, Shopify, Uber, and Twitter explore the benefits of Kappa but also show how batch processing fits into this discussion positively without the need for Lambda.