Browsing Tag

Data Integration

12 posts
Enterprise Application Integration with Confliuent and Databricks for Oracle SAP Salesforce Servicenow et al
Read More
  • 795 views
  • 12 minute read

Databricks and Confluent in the World of Enterprise Software (with SAP as Example)

Enterprise data lives in complex ecosystems—SAP, Oracle, Salesforce, ServiceNow, IBM Mainframes, and more. This article explores how Confluent and Databricks integrate with SAP to bridge operational and analytical workloads in real time. It outlines architectural patterns, trade-offs, and use cases like supply chain optimization, predictive maintenance, and financial reporting, showing how modern data streaming unlocks agility, reuse, and AI-readiness across even the most SAP-centric environments.
Read More
0
0
0
Confluent and Databricks for Data Integration and Stream Processing
Read More
  • 1.2K views
  • 10 minute read

Confluent Data Streaming Platform vs. Databricks Data Intelligence Platform for Data Integration and Processing

This blog explores how Confluent and Databricks address data integration and processing in modern architectures. Confluent provides real-time, event-driven pipelines connecting operational systems, APIs, and batch sources with consistent, governed data flows. Databricks specializes in large-scale batch processing, data enrichment, and AI model development. Together, they offer a unified approach that bridges operational and analytical workloads. Key topics include ingestion patterns, the role of Tableflow, the shift-left architecture for earlier data validation, and real-world examples like Uniper’s energy trading platform powered by Confluent and Databricks.
Read More
0
0
0
Electric Vehicle (EV) Charging - Automotive and ESG with Data Streaming at Virta
Read More
  • 1.4K views
  • 8 minute read

Virta’s Electric Vehicle (EV) Charging Platform with Real-Time Data Streaming: Scalability for Large Charging Businesses

The rise of Electric Vehicles (EVs) demands a scalable, efficient charging network—but challenges like fluctuating demand, complex billing, and real-time availability updates must be addressed. Virta, a global leader in smart EV charging, is tackling these issues with real-time data streaming. By leveraging Apache Kafka and Confluent Cloud, Virta enhances energy distribution, enables predictive maintenance, and supports dynamic pricing. This approach optimizes operations, improves user experience, and drives sustainability. Discover how real-time data streaming is shaping the future of EV charging and enabling intelligent, scalable infrastructure.
Read More
0
0
0
Apache Flink - Overkill for Simple Stateless Stream Processing
Read More
  • 4.3K views
  • 8 minute read

Apache Flink: Overkill for Simple, Stateless Stream Processing and ETL?

Discover when Apache Flink is the right tool for your stream processing needs. Explore its role in stateful and stateless processing, the advantages of serverless Flink SaaS solutions like Confluent Cloud, and how it supports advanced analytics and real-time data integration together with Apache Kafka. Dive into the trade-offs, deployment options, and strategies for leveraging Flink effectively across cloud, on-premise, and edge environments, and when to use Kafka Streams or Single Message Transforms (SMT) within Kafka Connect for ETL instead of Flink.
Read More
0
0
0
Virgin Australia Journey with Apache Kafka - Innovation in the Airline and Aviation Industry
Read More
  • 2.1K views
  • 5 minute read

Virgin Australia’s Journey with Apache Kafka: Driving Innovation in the Airline Industry

Data streaming with Apache Kafka and Flink is transforming the airline industry, enabling real-time efficiency and exceptional customer experiences. Virgin Australia exemplifies this innovation to modernize its Flight State Engine and overhaul its loyalty program. By embracing event-driven architecture, the airline has improved operational reliability and personalized services, setting a benchmark for aviation digitalization.
Read More
0
0
0
Airport and Airlines Digitalization with Data Streaming using Apache Kafka and Flink
Read More
  • 5.7K views
  • 10 minute read

The Digitalization of Airport and Airlines with IoT and Data Streaming using Kafka and Flink

The vision for a digitalized airport includes seamless passenger experiences, optimized operations, consistent integration with airlines and retail stores, and enhanced security through the use of advanced technologies like IoT, AI, and real-time data analytics. This blog post shows the relevance of data streaming with Apache Kafka and Flink in the aviation industry to enable data-driven business process automation and innovation while modernizing the IT infrastructure with cloud-native hybrid cloud architecture.
Read More
0
0
0
Snowflake and Apache Kafka Data Integration Anti Patterns Zero Reverse ETL
Read More
  • 6.4K views
  • 9 minute read

Snowflake Integration Patterns: Zero ETL and Reverse ETL vs. Apache Kafka

Snowflake is a leading cloud-native data warehouse. Integration patterns include batch data integration, Zero ETL and near real-time data ingestion with Apache Kafka. This blog post explores the different approaches and discovers its trade-offs. Following industry recommendations, it is suggested to avoid anti-patterns like Reverse ETL and instead use data streaming to enhance the flexibility, scalability, and maintainability of enterprise architecture.
Read More
0
0
0
Read More
  • 27.1K views
  • 12 minute read

Apache Kafka + Vector Database + LLM = Real-Time GenAI

Generative AI (GenAI) enables advanced AI use cases and innovation but also changes how the enterprise architecture looks like. Large Language Models (LLM), Vector Databases, and Retrieval Augmentation Generation (RAG) require new data integration patterns. Data streaming with Apache Kafka and Apache Flink processes incoming data sets in real-time at scale, connects various platforms, and enables decoupled data products.
Read More
0
0
4
Data Streaming with Apache Kafka at Airlines - Lufthansa Case Study
Read More
  • 7.4K views
  • 5 minute read

How Lufthansa uses Apache Kafka for Middleware and Analytics

Aviation and travel are notoriously vulnerable to social, economic, and political events, as well as the ever-changing expectations of consumers. The coronavirus was just a piece of the challenge. This post explores how Lufthansa leverages data streaming powered by Apache Kafka as cloud-native middleware for mission-critical data integration projects and as data fabric for AI/machine learning scenarios such as real-time predictions in fleet management. An interactive conversation with Lufthansa as an on-demand video is added at the end as a highlight if you want to learn more.
Read More
0
0
0
Disaster Recovery and Resiliency with Apache Kafka Data Streaming
Read More
  • 3.6K views
  • 3 minute read

Disaster Recovery with Kafka across the Edge and Hybrid Cloud (QCon Talk)

I spoke at QCon London in April 2022 about building disaster recovery and resilient real-time enterprise architectures with Apache Kafka. This blog post summarizes the use cases, architectures, and real-world examples. The slide deck and video recording of the presentation is included as well.
Read More
0
0
0