Browsing Tag

Data Streaming

30 posts
Data Lineage for Data Streaming with OpenLineage Apache Kafka and Flink
Read More
  • 381 views
  • 11 minute read

Open Standards for Data Lineage: OpenLineage for Batch AND Streaming

One of the greatest wishes of companies is end-to-end visibility in their operational and analytical workflows. Where does data come from? Where does it go? To whom am I giving access to? How can I track data quality issues? The capability to follow the data flow to answer these questions is called data lineage. This blog post explores market trends, efforts to provide an open standard with OpenLineage, and how data governance solutions from vendors such as IBM, Google, Confluent and Collibra help fulfil the enterprise-wide data governance needs of most companies, including data streaming technologies such as Apache Kafka and Flink.
Read More
0
0
0
The Past Present and Future of Stream Processing
Read More
  • 16.1K views
  • 12 minute read

The Past, Present and Future of Stream Processing

Stream processing has existed for decades. The adoption grows with open source frameworks like Apache Kafka and Flink in combination with fully managed cloud services. This blog post explores the past, present and future of stream processing, including the relation of machine learning and GenAI, streaming databases, and the integration between data streaming and data lakes with Apache Iceberg.
Read More
0
0
0
GenAI Demo with Kafka, Flink, LangChain and OpenAI
Read More
  • 3.4K views
  • 10 minute read

GenAI Demo with Kafka, Flink, LangChain and OpenAI

Generative AI (GenAI) enables automation and innovation across industries. This blog post explores a simple but powerful architecture and demo for the combination of Python, and LangChain with OpenAI LLM, Apache Kafka for event streaming and data integration, and Apache Flink for stream processing. The use case shows how data streaming and GenAI help to correlate data from Salesforce CRM, searching for lead information in public datasets like Google and LinkedIn, and recommending ice-breaker conversations for sales reps.
Read More
0
0
0
Real Time Customer Loyalty and Reward Platform with Apache Kafka
Read More
  • 1.3K views
  • 13 minute read

Customer Loyalty and Rewards Platform with Apache Kafka

Loyalty and rewards platforms are crucial for customer retention and revenue growth for many enterprises across industries. Apache Kafka provides context-specific real-time data and consistency across all applications and databases for a modern and flexible enterprise architecture. This blog post looks at case studies from Albertsons (retail), Globe Telecom (telco), Virgin Australia (aviation), Disney+ Hotstar (sports and gaming), and Porsche (automotive) to explain the value of data streaming for improving the customer loyalty.
Read More
0
0
0
Read More
  • 9.9K views
  • 12 minute read

Apache Kafka + Vector Database + LLM = Real-Time GenAI

Generative AI (GenAI) enables advanced AI use cases and innovation but also changes how the enterprise architecture looks like. Large Language Models (LLM), Vector Databases, and Retrieval Augmentation Generation (RAG) require new data integration patterns. Data streaming with Apache Kafka and Apache Flink processes incoming data sets in real-time at scale, connects various platforms, and enables decoupled data products.
Read More
0
0
0
Data Streaming with Apache Kafka at Airlines - Lufthansa Case Study
Read More
  • 3.7K views
  • 5 minute read

How Lufthansa uses Apache Kafka for Middleware and Analytics

Aviation and travel are notoriously vulnerable to social, economic, and political events, as well as the ever-changing expectations of consumers. The coronavirus was just a piece of the challenge. This post explores how Lufthansa leverages data streaming powered by Apache Kafka as cloud-native middleware for mission-critical data integration projects and as data fabric for AI/machine learning scenarios such as real-time predictions in fleet management. An interactive conversation with Lufthansa as an on-demand video is added at the end as a highlight if you want to learn more.
Read More
0
0
0
The State of Data Streaming for Energy and Utilities in 2023
Read More
  • 2.0K views
  • 8 minute read

The State of Data Streaming for Energy & Utilities in 2023

The evolution of utility infrastructure, energy distribution, customer services, and new business models requires real-time end-to-end visibility, reliable and intuitive B2B and B2C communication, and integration with pioneering technologies like 5G for low latency or augmented reality for innovation. I look at trends in the utilities sector to explore how data streaming helps as a business enabler, including customer stories from SunPower, 50hertz, Powerledger, and more. A complete slide deck and on-demand video recording are included.
Read More
0
0
0
The State of Data Streaming for the Public Sector in 2023
Read More
  • 1.5K views
  • 5 minute read

The State of Data Streaming for the Public Sector in 2023

This blog post explores the state of data streaming for the public sector and government in 2023. Data streaming provides consistency across all layers and allows integrating and correlating data in real-time at any scale. I look at public sector trends to explore how Apache Kafka helps as a business enabler, including case studies from the US Department of Defense (DoD), NASA, Deutsche Bahn (German Railway), and others. A complete slide deck and on-demand video recording are included.
Read More
0
0
0
Data Streaming with Apache Kafka as Data Fabric for GenAI
Read More
  • 4.2K views
  • 12 minute read

Apache Kafka as Mission Critical Data Fabric for GenAI

Apache Kafka serves thousands of enterprises as the mission-critical and scalable real-time data fabric for machine learning infrastructures. The evolution of Generative AI (GenAI) with large language models (LLM) like ChatGPT changed how people think about intelligent software and automation. This blog post explains the relation between data streaming and GenAI and shows the enormous opportunities and some early adopters of GenAI beyond the buzz.
Read More
0
0
0