Browsing Category

Stream Processing

46 posts
Stream Processing on the IBM Mainframe with Apache Flink - Genius or a Glitch in the Matrix
Read More
  • 357 views
  • 6 minute read

Stream Processing on the Mainframe with Apache Flink: Genius or a Glitch in the Matrix?

Running Apache Flink on a mainframe may sound surprising, but it is already happening and for good reason. As modern mainframes like IBM z17 evolve to support Linux, Kubernetes, and AI workloads, they are becoming a powerful platform for real-time stream processing. This blog explores why enterprises are deploying Apache Flink on IBM LinuxONE, how it works in practice, and what business value it brings. With Kafka providing the data backbone, Flink enables intelligent processing close to where business-critical data lives. The result is a modern hybrid architecture that connects core systems with cloud-based innovation without needing to fully migrate off the mainframe.
Read More
0
0
0
The Future of Data Streaming with Apache Flink for Agentic AI Supporting A2A and MCP
Read More
  • 10.3K views
  • 8 minute read

The Future of Data Streaming with Apache Flink for Agentic AI

Agentic AI is moving into production. Autonomous, tool-using, goal-driven systems that need real-time data and context. Apache Kafka and Flink provide the event-driven foundation to run these agents at scale. With the new Flink Agents project (FLIP-531), Flink will natively support long-running, system-triggered AI agents integrated with LLMs, tools, and emerging protocols like MCP and A2A. This marks a major step toward reliable, enterprise-grade Agentic AI.
Read More
0
0
0
Confluent and Databricks for Data Integration and Stream Processing
Read More
  • 10.4K views
  • 10 minute read

Confluent Data Streaming Platform vs. Databricks Data Intelligence Platform for Data Integration and Processing

This blog explores how Confluent and Databricks address data integration and processing in modern architectures. Confluent provides real-time, event-driven pipelines connecting operational systems, APIs, and batch sources with consistent, governed data flows. Databricks specializes in large-scale batch processing, data enrichment, and AI model development. Together, they offer a unified approach that bridges operational and analytical workloads. Key topics include ingestion patterns, the role of Tableflow, the shift-left architecture for earlier data validation, and real-world examples like Uniper’s energy trading platform powered by Confluent and Databricks.
Read More
0
0
0
Apache Flink - Overkill for Simple Stateless Stream Processing
Read More
  • 12.7K views
  • 8 minute read

Apache Flink: Overkill for Simple, Stateless Stream Processing and ETL?

Discover when Apache Flink is the right tool for your stream processing needs. Explore its role in stateful and stateless processing, the advantages of serverless Flink SaaS solutions like Confluent Cloud, and how it supports advanced analytics and real-time data integration together with Apache Kafka. Dive into the trade-offs, deployment options, and strategies for leveraging Flink effectively across cloud, on-premise, and edge environments, and when to use Kafka Streams or Single Message Transforms (SMT) within Kafka Connect for ETL instead of Flink.
Read More
0
0
0
Stateless and Stateful Stream Processing with Kafka Streams and Apache Flink
Read More
  • 12.8K views
  • 8 minute read

Stateless vs. Stateful Stream Processing with Kafka Streams and Apache Flink

The rise of stream processing has changed how we handle and act on data. While traditional databases, data lakes, and warehouses are effective for many batch-based use cases, they fall short in scenarios demanding low latency, scalability, and real-time decision-making. This post explores the key concepts of stateless and stateful stream processing, using Kafka Streams and Apache Flink as examples.
Read More
0
0
0
Data Streaming Trends for 2025 - Leading with Apache Kafka and Flink
Read More
  • 30.3K views
  • 18 minute read

Top Trends for Data Streaming with Apache Kafka and Flink in 2025

Apache Kafka and Apache Flink are leading open-source frameworks for data streaming that serve as the foundation for cloud services, enabling organizations to unlock the potential of real-time data. Over recent years, trends have shifted from batch-based data processing to real-time analytics, scalable cloud-native architectures, and improved data governance powered by these technologies. Looking ahead to 2025, the data streaming ecosystem is set to undergo even greater changes. Here are the top trends shaping the future of data streaming for businesses.
Read More
0
0
0
Dynamic Pricing with Data Streaming using Apache Kafka and Flink
Read More
  • 11.4K views
  • 8 minute read

A New Era in Dynamic Pricing: Real-Time Data Streaming with Apache Kafka and Flink

In the age of digitization, the concept of pricing is no longer fixed or manual. Instead, companies increasingly use dynamic pricing — a flexible model that adjusts prices based on real-time market changes to enable real-time responsiveness, giving companies the tools they need to respond instantly to demand, competitor prices, and customer behaviors. This blog post explores the fundamentals of dynamic pricing, its link to data streaming, and real-world examples across different industries such as retail, logistics, gaming and the energy sector.
Read More
0
0
0
The Past Present and Future of Stream Processing
Read More
  • 30.7K views
  • 12 minute read

The Past, Present and Future of Stream Processing

Stream processing has existed for decades. The adoption grows with open source frameworks like Apache Kafka and Flink in combination with fully managed cloud services. This blog post explores the past, present and future of stream processing, including the relation of machine learning and GenAI, streaming databases, and the integration between data streaming and data lakes with Apache Iceberg.
Read More
0
0
0
Read More
  • 14.1K views
  • 10 minute read

Top 5 Trends for Data Streaming with Kafka and Flink in 2024

Do you wonder about my predicted TOP 5 data streaming trends with Apache Kafka and Flink in 2024 to set data in motion? Discover new technology trends and best practices for event-driven architectures, including data sharing, data contracts, serverless stream processing, multi-cloud architectures, and GenAI.
Read More
0
0
1
Python Kafka Quix Streams and Flink for Open Source Stream Processing
Read More
  • 11.9K views
  • 12 minute read

Quix Streams – Stream Processing with Kafka and Python

Over 100,000 organizations use Apache Kafka for data streaming. However, there is a problem: The broad ecosystem lacks a mature client framework and managed cloud service for Python data engineers. Quix Streams is a new technology on the market trying to close this gap. This blog post discusses this Python library, its place in the Kafka ecosystem, and when to use it instead of Apache Flink or other Python- or SQL-based substitutes.
Read More
0
0
0