Explore Kai Waehner’s latest articles on Apache Kafka, Apache Flink, real-time data streaming, data integration, process intelligence, agentic AI, and modern enterprise architecture. The blog covers practical use cases, technology trends, conference insights, and hands-on perspectives for teams building scalable, future-ready data platforms.
Enterprises run separate tools for IT scheduling, data pipelines, business processes, and infrastructure. None talk to each other. Modernization and agentic AI are forcing them into one platform. Here is why I joined Kestra as Global…
Nine years at Confluent: from a Silicon Valley startup with 100 people to an $11 billion IBM acquisition. A personal reflection on the Apache Kafka and data streaming journey and what the team accomplished together….
XML, JSON, and YAML were built for different jobs in different eras. This post covers where each came from, how they compare, and where each one still wins, including why YAML borrows JSON Schema and how schemas became the contract…
Databricks and Snowflake now speak the Kafka protocol. But using the Kafka API to feed a lakehouse is very different from running Kafka as the event-driven backbone of the enterprise. Here is why the two are complementary, not the same….
ERP vendor selection for manufacturing is not a product decision. It is a strategic bet on fit, total cost, and which AI future your vendor is building toward. Three German manufacturers chose SAP, ams.erp, and VlexPlus. All three were…
Process intelligence is not a single tool. It combines process mining, process orchestration, and a decision gate into one architecture that shows how processes really run, governs what happens next, and keeps automation and AI inside…
ERP modernization fails when the technology leads and the process work follows. Three German manufacturers ran their migrations differently, on SAP, ams.erp, and VlexPlus. The structural lessons are the same. This post covers what…
Most organizations start their data governance journey by asking how to track where data comes from and where it goes. They quickly discover a harder question: why can none of their existing tools answer that across all systems?…
Every enterprise is being told to go agentic. Meanwhile, the platforms holding your most critical business data are tightening control over how AI agents can access it. SAP made that move explicitly. Other software vendors are doing…
AI agents fail in production when they are connected directly to raw event streams. Flink CEP is the missing layer between your data streams and your Agentic AI architecture: it detects meaningful event sequences in real time, reduces…
Complex Event Processing is the most underused capability in Apache Flink. It detects meaningful event sequences in real time, fires only when a pattern is confirmed, and even catches events that never arrive. This guide covers what…
MCP, REST/HTTP APIs, and Apache Kafka are not alternatives. They solve different problems at different layers of the architecture. This article maps the decision: what each technology is built for, where the boundaries are, and where…
The Enterprise Agentic AI Landscape 2026 maps every major AI vendor across two dimensions that matter most: how much you trust their AI, and how much lock-in you accept. An independent, vendor-neutral analysis covering Anthropic,…
Agentic AI without governed processes is fast but ungoverned. Event-driven integration without process intelligence moves data but not decisions. Process intelligence without live data automates the wrong outcomes. The fix is a…
Two toolchains, two skill sets, two CI/CD pipelines — that has been the reality for data engineers working across batch and streaming. dbt extending to Apache Flink changes that equation. One workflow, one tool, one engineering team…
The Shift Left Architecture moves data integration logic into an event-driven architecture where governed data products are built once and served to multiple consumers. The original pattern covered two interfaces: operational via…
The Ultimate Fighting Championship (UFC) held Fight Night London on March 21, 2026, at The O2 Arena. It was a great event inside the octagon. But a 1,200 USD VIP ticket raised real questions about premium customer experience, and about…
Dashboards are a popular way to make streaming data visible and useful, but they are not always the right solution. This blog post explains when dashboards make sense for Apache Kafka data and when other approaches like automation,…
Mobile World Congress (MWC) 2026 highlights the shift from batch systems to real time data streaming in telecom. AI and agentic automation, network APIs, sovereign cloud, autonomous networks, and 5G monetization all depend on…
This blog post explores how data streaming transforms airline operations by enabling real-time visibility, faster decision-making, and improved customer experience. Using Qantas as a leading example, it highlights how a modern data…
The second edition of The Ultimate Data Streaming Guide is now available as a free eBook. It includes over 70 use cases, over 20 customer stories, a detailed use case / customer matrix, and a stronger focus on AI topics like GenAI and…
Apache Kafka has long been the foundation for real-time data streaming. With the release of Queues for Kafka (QfK) in Apache Kafka 4.2, it now also supports native queuing, eliminating the need for separate message queue systems for…
Diskless Kafka is transforming how fintech and financial services organizations handle observability and log analytics. By using the Kafka protocol with cloud-native object storage, companies like Robinhood reduce infrastructure costs…
Rivian and Volkswagen, through their joint venture RV Tech, process high-frequency telemetry from connected vehicles using Apache Kafka and Apache Flink. With a shift left architecture and smart filtering, they reduced data volume by…
Airlines face constant pressure to deliver reliable service while managing complex operations and rising customer expectations. This blog post explores how Etihad Airways uses real-time data streaming with Apache Kafka and Flink to…
Running Apache Flink on a mainframe may sound surprising, but it is already happening and for good reason. As modern mainframes like IBM z17 evolve to support Linux, Kubernetes, and AI workloads, they are becoming a powerful platform…
Financial services companies are moving from batch processing to real time data flow. A data streaming platform enables financial institutions to connect systems, process events instantly, and power AI, fraud prevention, and customer…
Each year brings new momentum to the data streaming space. In 2026, six key trends stand out. Platforms and vendors are consolidating. Diskless Kafka and Apache Iceberg are reshaping storage. Real-time analytics is moving into the…