Kai WaehnerApache Kafka and MQTT are a perfect combination for many Industrial IoT use cases. This blog series covers the pros and cons of both technologies. Various use cases across industries, including connected vehicles, manufacturing, mobility services, and smart city are explored. The examples use different architectures, including lightweight edge scenarios, hybrid integrations, and serverless cloud solutions. This post is part three: Manufacturing, Industrial IoT, and Industry 4.0.
Apache Kafka + MQTT Blog Series
The first blog post explores the relation between MQTT and Apache Kafka. Afterward, the other four blog posts discuss various use cases, architectures, and reference deployments.
- Part 1 – Overview: Relation between Kafka and MQTT, pros and cons, architectures
- Part 2 – Connected Vehicles: MQTT and Kafka in a private cloud on Kubernetes; use case: remote control and command of a car
- Part 3 – Manufacturing (THIS POST): MQTT and Kafka at the edge in a smart factory; use case: Bidirectional OT-IT integration with Sparkplug between PLCs, IoT Gateways, Data Historian, MES, ERP, Data Lake, etc.
- Part 4 – Mobility Services: MQTT and Kafka leveraging serverless cloud infrastructure; use case: Traffic jam prediction service using machine learning
- Part 5 – Smart City: MQTT at the edge connected to fully-managed Kafka in the public cloud; use case: Intelligent traffic routing by combining and correlating 3rd party services
Subscribe to my newsletter to get updates immediately after the publication. Besides, I will also update the above list with direct links to this blog series’s posts as soon as published.
Use Case: Manufacturing 4.0 and Industrial IoT with Kafka
The following list shows different examples where Kafka is used as a strategic platform for various manufacturing use cases to implement Industry 4.0 initiatives:
- Track&Trace / Production Control / Plant Logistics
- Quality Assurance / Yield Management
- Predictive Maintenance
- Supply Chain Management
- Cybersecurity
- Servitization leveraging Digital Twins
- Additive Manufacturing
- Augmented Reality
- Many more…
I already covered this topic in detail recently. Hence, check out the blog post “Apache Kafka for Manufacturing and Industrial IoT“. That post includes a detailed slide deck and video recording.
Let’s specific examples for Kafka and MQTT in Manufacturing 4.0 in the following sections.
Architecture: Smart Factory and Industry 4.0 with Kafka
The following diagram shows the architecture of a smart factory. Both MQTT and Kafka infrastructure are deployed at the edge in the factory for security, latency, and cost reasons:
This example connects to modern PLCs and other gateways via MQTT and Sparkplug B. The benefit of using this technology is a lightweight communication protocol and open standard.
An OT middleware such as OSIsoft PI is required if legacy and proprietary protocols such as Modbus or Siemens S7 need to be integrated. Most plants today are brownfield. Hence both proprietary integration platforms and open MQTT or OPC-UA integration must communicate with Kafka from the OT side. “Apache Kafka as Data Historian” explores the integration options in more detail.
The IT components (such as SAP ERP) and the integration platforms (HiveMQ, Confluent) run in the factory. Obviously, many other architectures are possible if latency and security allow it. Check out “Building a Smart Factory with Apache Kafka and 5G Campus Networks” for a hybrid cloud architecture. Additionally, most smart factories are not completely independent from the central IT world running in a remote data center or public cloud. Various “architecture patterns for distributed, hybrid, edge and global Apache Kafka deployments” exist to replicate data bi-directionally between smart factories and data centers.
Example: MQTT for Critical Manufacturing @ Daimler
Manufacturing processes in the automotive industry cannot go down. Hence, Daimler built a Vehicle Diagnostic System (VDS) to efficiently share information between test devices on the factory floor and enterprise IT systems. The VDS fulfills some core functionality in the manufacturing process for E/E components, such as calibrating sensors controlled by an ECU, flashing new firmware, personalizing the key to the car, and testing to make sure each ECU works properly.
MQTT works in bad networks. Therefore, test devices behave properly even if the network connection is dropped and reconnected.
The system is rolled out to 24 factories around the world. 10,000 testing devices are connected. The devices generate 470 million messages/month.
The complete case study from HiveMQ explores the use case in more detail.
Example: Kafka in the Cloud for Business Critical Supply Chain Operations @ Baader
BAADER is a worldwide manufacturer of innovative machinery for the food processing industry. They run an IoT-based and data-driven food value chain on Confluent Cloud:
The Kafka-based infrastructure provides a single source of truth across the factories and regions across the food value chain. Business-critical operations are available 24/7 for tracking, calculations, alerts, etc.
The event streaming platform runs on Confluent Cloud. Hence, Baader can focus on building new innovative business applications. The serverless Kafka infrastructure provides mission-critical SLAs and consumption-based pricing for all required capabilities: Messaging, storage, data integration, and data processing.
MQTT provides connectivity to machines and GPS data from vehicles at the edge. Kafka Connect connectors integrate MQTT and other IT systems such as Elasticsearch, MongoDB, and AWS S3. ksqlDB processes the data in motion continuously. Stream processing or streaming analytics are other terms for this concept.
Kafka + MQTT = Manufacturing 4.0
In conclusion, Apache Kafka and MQTT are a perfect combination for Manufacturing, Industrial IoT, and Industry 4.0.
Follow the blog series to learn about use cases such as connected vehicles, manufacturing, mobility services, and smart city. Every blog post also includes real-world deployments from companies across industries. It is key to understand the different architectural options to make the right choice for your project.
What are your experiences and plans in IoT projects? What use case and architecture did you implement? Let’s connect on LinkedIn and discuss it! Stay informed about new blog posts by subscribing to my newsletter.
