Latest Insights on Data Streaming, AI, and Enterprise Architecture
Explore Kai Waehner’s latest articles on Apache Kafka, Apache Flink, real-time data streaming, data integration, process intelligence, agentic AI, and modern enterprise architecture. The blog covers practical use cases, technology trends, conference insights, and hands-on perspectives for teams building scalable, future-ready data platforms.
Why I Joined Kestra: Enterprise Workflow Orchestration for the Agentic AI Era
Enterprises run separate tools for IT scheduling, data pipelines, business processes, and infrastructure. None talk to each other. Modernization and agentic AI are forcing them into one platform. Here is why I joined Kestra as Global…
My Confluent Chapter: From Apache Kafka Startup to $11 Billion IBM Acquisition
Nine years at Confluent: from a Silicon Valley startup with 100 people to an $11 billion IBM acquisition. A personal reflection on the Apache Kafka and data streaming journey and what the team accomplished together….
YAML vs XML vs JSON: History, Trade-offs, and Where Each Wins in the Age of Agentic AI
XML, JSON, and YAML were built for different jobs in different eras. This post covers where each came from, how they compare, and where each one still wins, including why YAML borrows JSON Schema and how schemas became the contract…
Why Databricks and Snowflake Speak the Kafka Protocol: Ingestion vs. Architecture
Databricks and Snowflake now speak the Kafka protocol. But using the Kafka API to feed a lakehouse is very different from running Kafka as the event-driven backbone of the enterprise. Here is why the two are complementary, not the same….
Choosing an ERP for Manufacturing: How AI Is Reshaping the Vendor Landscape
ERP vendor selection for manufacturing is not a product decision. It is a strategic bet on fit, total cost, and which AI future your vendor is building toward. Three German manufacturers chose SAP, ams.erp, and VlexPlus. All three were…
Process Intelligence Explained: Mining, Orchestration, and the Decision Gate
Process intelligence is not a single tool. It combines process mining, process orchestration, and a decision gate into one architecture that shows how processes really run, governs what happens next, and keeps automation and AI inside…
ERP Migration to SAP S/4HANA and Beyond: Lessons Learned from German Manufacturing
ERP modernization fails when the technology leads and the process work follows. Three German manufacturers ran their migrations differently, on SAP, ams.erp, and VlexPlus. The structural lessons are the same. This post covers what…
Beyond Enterprise Data Lineage: The Case for a Platform-Independent Data Catalog
Most organizations start their data governance journey by asking how to track where data comes from and where it goes. They quickly discover a harder question: why can none of their existing tools answer that across all systems?…
Data Ownership in the Age of Agentic AI: Why SAP’s API Policy Forces a Data Integration Reckoning for Every Enterprise
Every enterprise is being told to go agentic. Meanwhile, the platforms holding your most critical business data are tightening control over how AI agents can access it. SAP made that move explicitly. Other software vendors are doing…
Flink CEP and Agentic AI: Real-Time Pattern Detection as the Foundation for Autonomous Decisions
AI agents fail in production when they are connected directly to raw event streams. Flink CEP is the missing layer between your data streams and your Agentic AI architecture: it detects meaningful event sequences in real time, reduces…
Complex Event Processing (CEP) with Apache Flink: What It Is and When (Not) to Use It
Complex Event Processing is the most underused capability in Apache Flink. It detects meaningful event sequences in real time, fires only when a pattern is confirmed, and even catches events that never arrive. This guide covers what…
MCP vs. REST/HTTP API vs. Kafka: The Architect’s Guide to Agentic AI Integration
MCP, REST/HTTP APIs, and Apache Kafka are not alternatives. They solve different problems at different layers of the architecture. This article maps the decision: what each technology is built for, where the boundaries are, and where…
Enterprise Agentic AI Landscape 2026: Trust, Flexibility, and Vendor Lock-in
The Enterprise Agentic AI Landscape 2026 maps every major AI vendor across two dimensions that matter most: how much you trust their AI, and how much lock-in you accept. An independent, vendor-neutral analysis covering Anthropic,…
The Trinity of Modern Data Architecture: Process Intelligence, Event-Driven Integration, and Trusted Agentic AI
Agentic AI without governed processes is fast but ungoverned. Event-driven integration without process intelligence moves data but not decisions. Process intelligence without live data automates the wrong outcomes. The fix is a…
dbt Meets Apache Flink: One Workflow for Data Engineers on Snowflake, BigQuery, Databricks, and Confluent
Two toolchains, two skill sets, two CI/CD pipelines — that has been the reality for data engineers working across batch and streaming. dbt extending to Apache Flink changes that equation. One workflow, one tool, one engineering team…
The Shift Left Architecture 2.0: Operational, Analytical and AI Interfaces for Real-Time Data Products
The Shift Left Architecture moves data integration logic into an event-driven architecture where governed data products are built once and served to multiple consumers. The original pattern covered two interfaces: operational via…
UFC VIP Experience Worth the Price? Fan Review. Business Perspective. Tech Vision.
The Ultimate Fighting Championship (UFC) held Fight Night London on March 21, 2026, at The O2 Arena. It was a great event inside the octagon. But a 1,200 USD VIP ticket raised real questions about premium customer experience, and about…
Dashboards and Queries for Apache Kafka: Operational, Explorative, and the Role of the Context Engine
Dashboards are a popular way to make streaming data visible and useful, but they are not always the right solution. This blog post explains when dashboards make sense for Apache Kafka data and when other approaches like automation,…
Data Streaming at MWC 2026: How Apache Kafka, Flink and Agentic AI Power Telecom Trends
Mobile World Congress (MWC) 2026 highlights the shift from batch systems to real time data streaming in telecom. AI and agentic automation, network APIs, sovereign cloud, autonomous networks, and 5G monetization all depend on…
From Takeoff to Touchdown: Real-Time Aviation with Data Streaming at Qantas
This blog post explores how data streaming transforms airline operations by enabling real-time visibility, faster decision-making, and improved customer experience. Using Qantas as a leading example, it highlights how a modern data…
The Ultimate Data Streaming Guide is Back – Second Edition of the Book and Industry Editions Now Available
The second edition of The Ultimate Data Streaming Guide is now available as a free eBook. It includes over 70 use cases, over 20 customer stories, a detailed use case / customer matrix, and a stronger focus on AI topics like GenAI and…
When (Not) to Use Queues for Kafka?
Apache Kafka has long been the foundation for real-time data streaming. With the release of Queues for Kafka (QfK) in Apache Kafka 4.2, it now also supports native queuing, eliminating the need for separate message queue systems for…
Diskless Kafka at FinTech Robinhood for Cost-Efficient Log Analytics and Observability
Diskless Kafka is transforming how fintech and financial services organizations handle observability and log analytics. By using the Kafka protocol with cloud-native object storage, companies like Robinhood reduce infrastructure costs…
Shift Left in Automotive: Real-Time Intelligence from Vehicle Telemetry with Data Streaming at Rivian
Rivian and Volkswagen, through their joint venture RV Tech, process high-frequency telemetry from connected vehicles using Apache Kafka and Apache Flink. With a shift left architecture and smart filtering, they reduced data volume by…
Etihad Airways Makes Airline Operations Real-Time with Data Streaming
Airlines face constant pressure to deliver reliable service while managing complex operations and rising customer expectations. This blog post explores how Etihad Airways uses real-time data streaming with Apache Kafka and Flink to…
Stream Processing on the Mainframe with Apache Flink: Genius or a Glitch in the Matrix?
Running Apache Flink on a mainframe may sound surprising, but it is already happening and for good reason. As modern mainframes like IBM z17 evolve to support Linux, Kubernetes, and AI workloads, they are becoming a powerful platform…
10 FinTech Predictions That Depend on Real Time Data Streaming
Financial services companies are moving from batch processing to real time data flow. A data streaming platform enables financial institutions to connect systems, process events instantly, and power AI, fraud prevention, and customer…
Top Trends for Data Streaming with Apache Kafka and Flink in 2026
Each year brings new momentum to the data streaming space. In 2026, six key trends stand out. Platforms and vendors are consolidating. Diskless Kafka and Apache Iceberg are reshaping storage. Real-time analytics is moving into the…