Cost Archives - Kai Waehner

The Rise of Diskless Apache Kafka with Object Storage and No Brokers

670 views
8 minute read

The Rise of Diskless Kafka: Rethinking Brokers, Storage, and the Kafka Protocol

ByKai Waehner
11. August 2025

Apache Kafka has evolved from a data lake pipeline into the backbone of real-time transactional systems. The shift from broker-based storage to Tiered Storage and now to Diskless Kafka using cloud object storage redefines Kafka’s role. This blog explores the business value, technical architecture, and use cases of running Kafka without brokers, using the Kafka protocol as the foundation for scalable, cost-efficient event streaming.

Shift Left Architecture with Confluent Data Streaming and Databricks Lakehouse Medallion

3.5K views
9 minute read

Shift Left Architecture for AI and Analytics with Confluent and Databricks

ByKai Waehner
9. May 2025

Confluent and Databricks enable a modern data architecture that unifies real-time streaming and lakehouse analytics. By combining shift-left principles with the structured layers of the Medallion Architecture, teams can improve data quality, reduce pipeline complexity, and accelerate insights for both operational and analytical workloads. Technologies like Apache Kafka, Flink, and Delta Lake form the backbone of scalable, AI-ready pipelines across cloud and hybrid environments.