Use Cases and Architectures for Apache Kafka across Industries

Kafka Examples Use Cases and Architectures

Event Streaming is happening all over the world. This blog post explores real-life examples across industries for use cases and architectures leveraging Apache Kafka. Learn about architectures for real-world deployments from Audi, BMW, Disney, Generali, Paypal, Tesla, Unity, Walmart, William Hill, and more. Use cases include fraud detection, mainframe offloading, predictive maintenance, cybersecurity, edge computing, track&trace, live betting, and much more.

Kafka Examples Use Cases and Architectures

The following sections show a few of the use cases and architectures. Check out the slide deck and video recording at the end for all examples and the architectures from the companies mentioned above.

Use Cases for Event Streaming with Apache Kafka

Apache Kafka is an event streaming platform. It provides messaging, persistence, data integration, and data processing capabilities. High scalability for millions of messages per second, high availability including backward-compatibility and rolling upgrades for mission-critical workloads, and cloud-native features are some of the capabilities.

Event Streaming and Event Driven Architecture for a Smart City with Apache Kafka

Hence, the number of different use cases is almost endless. If you learn one thing from the examples in this blog post, then remember that Kafka is not just a messaging system! While data ingestion into a Hadoop data lake was the first prominent use case, this implies <5% of actual Kafka deployments. Business applications, streaming ETL middleware, real-time analytics, and edge/hybrid scenarios are some of the other examples:

Use Case Examples for Apache Kafka

 

Examples: SIEM, Streaming Machine Learning, Stateful Stream Processing

The following covers a few architectures and use cases. The presentation afterward goes into much more detail and examples from various companies about these and other use cases from various industries:

Modernized security information and event management (SIEM)

SIEM and cybersecurity are getting more important across industries. Kafka is used as open and scalable data integration and processing platform. Often combined with other SIEM solutions:

Modernized security information and event management (SIEM) with Kafka Splunk QRadar Arcsight

Streaming Model Training without the Need for Another Data Lake

Kafka and Machine Learning are a great combination for data integration, data processing, model training, model deployment, online monitoring, and other ML tasks. The most recent innovation is discussed in detail at the latest Kafka Summit: Simplified Kafka ML architecture for streaming model training without the need for another data lake like HDFS, S3, or Spark:

Streaming Ingestion and Model Training with Kafka without another Data Lake like Hadoop S3 Spark

Stateful Stream Processing and Streaming Analytics with Kafka Streams / ksqlDB

Data aggregation and correlation at scale in real-time are key concepts for building innovative business applications and adding business value. The following is one example of doing continuous calculations for betting. It includes a synthetic delay to “adjust the live betting odds”:

Stateful Stream Processing and Streaming Analytics with Apache Kafka

While this is a controversial example, it shows the power of stateful streaming processing very well. I am sure you already have ideas on how to apply this to your industry.

Slides – Use Cases and Architectures for Kafka

Here are the slides from my presentation about Kafka examples across industries:

Click on the button to load the content from www.slideshare.net.

Load content

Video Recording – Examples of Kafka Deployments Across Industries

Here is the video recording with all the use cases and examples from various companies across the globe and industries:

Use Cases and Examples for Apache Kafka

Use Cases and Examples for Event Streaming with Apache Kafka Exist in Every Industry

Kafka is used everywhere across industries for event streaming, data processing, data integration, and building business applications / microservices. It is deployed successfully in mission-critical deployments at scale at silicon valley tech giants, startups, and traditional enterprises. Scenarios include cloud, multi-cloud, hybrid, and edge infrastructures.

What are your experiences with Apache Kafka and its ecosystem for event streaming? Which use cases and architectures did you deploy? What are your status quo and future strategy? Let’s connect on LinkedIn and discuss it!

Dont‘ miss my next post. Subscribe!

We don’t spam! Read our privacy policy for more info.
If you have issues with the registration, please try a private browser tab / incognito mode. If it doesn't help, write me: kontakt@kai-waehner.de

Leave a Reply
You May Also Like
How to do Error Handling in Data Streaming
Read More

Error Handling via Dead Letter Queue in Apache Kafka

Recognizing and handling errors is essential for any reliable data streaming pipeline. This blog post explores best practices for implementing error handling using a Dead Letter Queue in Apache Kafka infrastructure. The options include a custom implementation, Kafka Streams, Kafka Connect, the Spring framework, and the Parallel Consumer. Real-world case studies show how Uber, CrowdStrike, Santander Bank, and Robinhood build reliable real-time error handling at an extreme scale.
Read More