Browsing Category

Apache Kafka

231 posts
Mainframe Modernization and Integration with Data Streaming using Apache Kafka IBM MQ IIDR CDC Precisely Qlik
Read More
  • 430 views
  • 14 minute read

Mainframe Integration with Data Streaming: Architecture, Business Value, Real-World Success

The mainframe is evolving—not fading. With cloud-native features, AI acceleration, and quantum-safe encryption, platforms like IBM z16 and z17 remain central to critical industries. But modern demands require real-time data access and system agility. Apache Kafka and Flink make this possible by streaming data bi-directionally between DB2, IMS, and MQ and cloud analytics platforms. This enables event-driven architectures without disrupting core systems. This post outlines proven strategies—offloading, integration, and replacement—and includes real-world examples across industries. The result: lower costs, faster innovation, and smarter use of legacy systems.
Read More
0
0
0
How OpenAI Uses Apache Kafka and Flink for GenAI and Agentic AI
Read More
  • 15.4K views
  • 5 minute read

How OpenAI uses Apache Kafka and Flink for GenAI

OpenAI revealed how it builds and scales the real-time data streaming infrastructure that powers its GenAI systems, including ChatGPT, at the Current 2025 conference in London. This blog post summarizes the role of Apache Kafka and Apache Flink in OpenAI’s architecture—enabling near-instant data processing, continuous feedback loops, and scalable coordination across model training and applications. From simplified Kafka consumption to multi-region Flink pipelines, OpenAI’s sessions showed why real-time data infrastructure is essential for both generative and agentic AI.
Read More
0
0
0
Durable Execution Engine with Restate Temporal DBOS vs Stream Processing with Kafka Streams Apache Flink Spark Structured Streaming
Read More
  • 1.8K views
  • 8 minute read

­­The Rise of the Durable Execution Engine (Temporal, Restate) in an Event-driven Architecture (Apache Kafka)

Durable execution engines like Temporal and Restate are redefining how developers orchestrate long-running, stateful workflows in distributed systems. Unlike traditional BPM tools focused on human-centric tasks, these engines automate machine-to-machine processes with built-in durability, retries, and fault-tolerant coordination. When integrated with event-driven platforms like Apache Kafka, they enable scalable, resilient architectures—handling complex business logic such as order processing, fraud detection, and multi-step transactions. This blog explores their capabilities, differences from stream processing tools like Apache Flink, Kafka Streams or Spark Structured Streaming, and the emerging role they play in modern enterprise infrastructure.
Read More
0
0
0
How Penske Logistics Transforms Fleet Intelligence with Kafka and AI
Read More
  • 781 views
  • 7 minute read

How Penske Logistics Transforms Fleet Intelligence with Data Streaming and AI

Real-time visibility has become essential in logistics. As supply chains grow more complex, providers must shift from delayed, batch-based systems to event-driven architectures. Data Streaming technologies like Apache Kafka and Apache Flink enable this shift by allowing continuous processing of data from telematics, inventory systems, and customer interactions. Penske Logistics is leading the way—using Confluent’s platform to stream and process 190 million IoT messages daily. This powers predictive maintenance, faster roadside assistance, and higher fleet uptime. The result: smarter operations, improved service, and a scalable foundation for the future of logistics.
Read More
0
0
0
Agentic AI with Apache Kafka as Event Broker Combined with MCP and A2A Protocol
Read More
  • 6.0K views
  • 7 minute read

Agentic AI with the Agent2Agent Protocol (A2A) and MCP using Apache Kafka as Event Broker

Agentic AI is emerging as a powerful pattern for building autonomous, intelligent, and collaborative systems. To move beyond isolated models and task-based automation, enterprises need a scalable integration architecture that supports real-time interaction, coordination, and decision-making across agents and services. This blog explores how the combination of Apache Kafka, Model Context Protocol (MCP), and Google’s Agent2Agent (A2A) protocol forms the foundation for Agentic AI in production. By replacing point-to-point APIs with event-driven communication as the integration layer, enterprises can achieve decoupling, flexibility, and observability—unlocking the full potential of AI agents in modern enterprise environments.
Read More
0
0
1
Real Time Gaming with Apache Kafka Powers Dream11 Fantasy Sports
Read More
  • 1.0K views
  • 4 minute read

Powering Fantasy Sports at Scale: How Dream11 Uses Apache Kafka for Real-Time Gaming

Fantasy sports has evolved into a data-driven, real-time digital industry with high stakes and massive user engagement. At the heart of this transformation is Dream11, India’s leading fantasy sports platform, which relies on Apache Kafka to deliver instant updates, seamless gameplay, and trustworthy user experiences for over 230 million fans. This blog post explores how Dream11 leverages Kafka to meet extreme traffic demands, scale infrastructure efficiently, and maintain real-time responsiveness—even during the busiest moments of live sports.
Read More
0
0
0
Shift Left Architecture with Confluent Data Streaming and Databricks Lakehouse Medallion
Read More
  • 1.7K views
  • 9 minute read

Shift Left Architecture for AI and Analytics with Confluent and Databricks

Confluent and Databricks enable a modern data architecture that unifies real-time streaming and lakehouse analytics. By combining shift-left principles with the structured layers of the Medallion Architecture, teams can improve data quality, reduce pipeline complexity, and accelerate insights for both operational and analytical workloads. Technologies like Apache Kafka, Flink, and Delta Lake form the backbone of scalable, AI-ready pipelines across cloud and hybrid environments.
Read More
0
0
0
Confluent and Databricks for Data Integration and Stream Processing
Read More
  • 1.9K views
  • 10 minute read

Confluent Data Streaming Platform vs. Databricks Data Intelligence Platform for Data Integration and Processing

This blog explores how Confluent and Databricks address data integration and processing in modern architectures. Confluent provides real-time, event-driven pipelines connecting operational systems, APIs, and batch sources with consistent, governed data flows. Databricks specializes in large-scale batch processing, data enrichment, and AI model development. Together, they offer a unified approach that bridges operational and analytical workloads. Key topics include ingestion patterns, the role of Tableflow, the shift-left architecture for earlier data validation, and real-world examples like Uniper’s energy trading platform powered by Confluent and Databricks.
Read More
0
0
0
Data Sharing for MVNO Growth and Beyond with Data Streaming in the Telco Industry
Read More
  • 1.4K views
  • 8 minute read

Real-Time Data Sharing in the Telco Industry for MVNO Growth and Beyond with Data Streaming

The telecommunications industry is transforming rapidly as Telcos expand partnerships with MVNOs, IoT platforms, and enterprise customers. Traditional batch-driven architectures can no longer meet the demands for real-time, secure, and flexible data access. This blog explores how real-time data streaming technologies like Apache Kafka and Flink, combined with hybrid cloud architectures, enable Telcos to build trusted, scalable data ecosystems. It covers the key components of a modern data sharing platform, critical use cases across the Telco value chain, and how policy-driven governance and tailored data products drive new business opportunities, operational excellence, and regulatory compliance. Mastering real-time data sharing positions Telcos to turn raw events into strategic advantage faster and more securely than ever before.
Read More
0
0
0
Fraud Prevention in Mobility Services with Data Streaming using Apache Kafka and Flink with AI Machine Learning
Read More
  • 2.5K views
  • 7 minute read

Fraud Detection in Mobility Services (Ride-Hailing, Food Delivery) with Data Streaming using Apache Kafka and Flink

Mobility services like Uber, Grab, and FREE NOW (Lyft) rely on real-time data to power seamless trips, deliveries, and payments. But this real-time nature also opens the door to sophisticated fraud schemes—ranging from GPS spoofing to payment abuse and fake accounts. Traditional fraud detection methods fall short in speed and adaptability. By using Apache Kafka and Apache Flink, leading mobility platforms now detect and block fraud as it happens, protecting their revenue, users, and trust. This blog explores how real-time data streaming is transforming fraud prevention across the mobility industry.
Read More
0
0
0