Browsing Tag

Data Quality

5 posts
Read More
  • 2.7K views
  • 8 minute read

Beyond Enterprise Data Lineage: The Case for a Platform-Independent Data Catalog

Most organizations start their data governance journey by asking how to track where data comes from and where it goes. They quickly discover a harder question: why can none of their existing tools answer that across all systems? Vendor-specific lineage tools like Confluent, Snowflake Horizon, and Databricks Unity Catalog each do a good job within their platform boundary. The problem is the boundary. Enterprise-wide lineage requires a platform-independent catalog layer that integrates everything and is owned by none of the platforms it connects.
Read More
0
0
0
Shift Left Architecture at Siemens with Stream Processing using Apache Kafka and Flink
Read More
  • 11.8K views
  • 6 minute read

Shift Left Architecture at Siemens: Real-Time Innovation in Manufacturing and Logistics with Data Streaming

Industrial enterprises face increasing pressure to move faster, automate more, and adapt to constant change—without compromising reliability. Siemens Digital Industries addresses this challenge by combining real-time data streaming, modular design, and Shift Left principles to modernize manufacturing and logistics. This blog outlines how technologies like Apache Kafka, Apache Flink, and Confluent Cloud support scalable, event-driven architectures. A real-world example from Siemens’ Modular Intralogistics Platform illustrates how this approach improves data quality, system responsiveness, and operational agility.
Read More
0
0
0
Read More
  • 21.1K views
  • 5 minute read

The Top 20 Problems with Batch Processing (and How to Fix Them with Data Streaming)

Batch processing introduces delays, complexity, and data quality issues that modern businesses can no longer afford. This article outlines the most common problems with batch workflows—ranging from outdated insights to compliance risks—and illustrates each with real-world examples. It also highlights how real-time data streaming offers a more reliable, scalable, and future-proof alternative.
Read More
0
0
0
Data Lineage for Data Streaming with OpenLineage Apache Kafka and Flink
Read More
  • 36.2K views
  • 11 minute read

Open Standards for Data Lineage: OpenLineage for Batch AND Streaming

One of the greatest wishes of companies is end-to-end visibility in their operational and analytical workflows. Where does data come from? Where does it go? To whom am I giving access to? How can I track data quality issues? The capability to follow the data flow to answer these questions is called data lineage. This blog post explores market trends, efforts to provide an open standard with OpenLineage, and how data governance solutions from vendors such as IBM, Google, Confluent and Collibra help fulfil the enterprise-wide data governance needs of most companies, including data streaming technologies such as Apache Kafka and Flink.
Read More
0
0
0