Browsing Category

Data Streaming

49 posts
Snowflake with Apache Kafka and Iceberg Connector
Read More
  • 739 views
  • 8 minute read

Snowflake Data Integration Options for Apache Kafka (including Iceberg)

The integration between Apache Kafka and Snowflake is often cumbersome. Options include near real-time ingestion with a Kafka Connect connector, batch ingestion from large files, or leveraging a standard table format like Apache Iceberg. This blog post explores the alternatives and discusses its trade-offs. The end shows how data streaming helps with hybrid architectures where data needs to be ingested from the private data center into Snowflake in the public cloud.
Read More
0
0
0
Snowflake and Apache Kafka Data Integration Anti Patterns Zero Reverse ETL
Read More
  • 981 views
  • 9 minute read

Snowflake Integration Patterns: Zero ETL and Reverse ETL vs. Apache Kafka

Snowflake is a leading cloud-native data warehouse. Integration patterns include batch data integration, Zero ETL and near real-time data ingestion with Apache Kafka. This blog post explores the different approaches and discovers its trade-offs. Following industry recommendations, it is suggested to avoid anti-patterns like Reverse ETL and instead use data streaming to enhance the flexibility, scalability, and maintainability of enterprise architecture.
Read More
0
0
0
Google Apache Kafka for BigQuery GCP Cloud Service
Read More
  • 1.0K views
  • 7 minute read

When (Not) to Choose Google Apache Kafka for BigQuery?

Google announced its Apache Kafka for BigQuery cloud service at its conference Google Cloud Next 2024 in Las Vegas. Welcome to the data streaming club joining Amazon, Microsoft, IBM, Oracle, Confluent, and others. This blog post explores this new managed Kafka offering for GCP, reviews the current status of the data streaming landscape, and shares some criteria to evaluate when Kafka in general and Google Apache Kafka in particular should (not) be used.
Read More
0
0
0
When NOT to use Apache Kafka
Read More
  • 794 views
  • 3 minute read

When NOT to Use Apache Kafka? (Lightboard Video)

Apache Kafka is the de facto standard for data streaming to process data in motion. With its significant adoption growth across all industries, I get a very valid question every week: When NOT to use Apache Kafka? What limitations does the event streaming platform have? When does Kafka simply not provide the needed capabilities? How to qualify Kafka out as it is not the right tool for the job? This blog post contains a lightboard video that gives you a twenty-minute explanation of the DOs and DONTs.
Read More
0
0
0
ESG and Sustainability powered by Data Streaming with Apache Kafka and Flink
Read More
  • 1.2K views
  • 13 minute read

Green Data, Clean Insights: How Kafka and Flink Power ESG Transformations

This blog post explores the synergy between Environmental, Social, and Governance (ESG) principles and Kafka and Flink’s real-time data processing capabilities, unveiling a powerful alliance that transforms intentions into impactful change. Beyond just buzzwords, real-world deployments architectures across industries show the value of data streaming for better ESG ratings.
Read More
0
0
0
SAP Datasphere and Apache Kafka as Data Fabric for ERP Integration
Read More
  • 2.2K views
  • 12 minute read

SAP Datasphere and Apache Kafka as Data Fabric for S/4HANA ERP Integration

SAP is the leading ERP solution across industries around the world. Data integration with other data platforms, applications, databases, and APIs is one of the hardest challenges in the IT and software landscape. This blog post explores how SAP Datasphere in conjunction with the data streaming platform Apache Kafka enables a reliable, scalable and open data fabric for connecting SAP business objects of ECC and S/4HANA ERP with other real-time, batch, or request-response interfaces.
Read More
0
0
0
Data Streaming Landscape 2024 around Kafka Flink and Cloud
Read More
  • 4.9K views
  • 21 minute read

The Data Streaming Landscape 2024

The research company Forrester defines data streaming platforms as a new software category in a new Forrester Wave. Apache Kafka is the de facto standard used by over 100,000 organizations. Plenty of vendors offer Kafka platforms and cloud services. Many complementary open source stream processing frameworks like Apache Flink and related cloud offerings emerged. And competitive technologies like Pulsar, Redpanda, or WarpStream try to get market share leveraging the Kafka protocol. This blog post explores the data streaming landscape of 2024 to summarize existing solutions and market trends. The end of the article gives an outlook to potential new entrants in 2025.
Read More
0
0
0
Tiered Storage for Apache Kafka - Use Cases Architecture Benefits.png
Read More
  • 5.2K views
  • 10 minute read

Why Tiered Storage for Apache Kafka is a BIG THING…

Apache Kafka added Tiered Storage to separate compute and storage. The capability enables more scalable, reliable and cost-efficient enterprise architectures. This blog post explores the architecture, use cases, benefits, and a case study for storing Petabytes of data in the Kafka commit log. The end discusses why Tiered Storage does NOT replace other databases and how Apache Iceberg might change future Kafka architectures even more.
Read More
0
0
0
Read More
  • 3.3K views
  • 10 minute read

Top 5 Trends for Data Streaming with Kafka and Flink in 2024

Do you wonder about my predicted TOP 5 data streaming trends with Apache Kafka and Flink in 2024 to set data in motion? Discover new technology trends and best practices for event-driven architectures, including data sharing, data contracts, serverless stream processing, multi-cloud architectures, and GenAI.
Read More
0
0
0
Dish Wireless Cloud-native 5G Telco Network powered by Data Streaming with Apache Kafka
Read More
  • 1.6K views
  • 8 minute read

How Apache Kafka helps Dish Wireless building cloud-native 5G Telco Infrastructure

5G telco infrastructure provides the basic foundations of data movement and increasingly unlocks new capabilities for low latency and critical SLAs. Real-time data processing with data streaming using Apache Kafka enables innovation across industries. This blog post explores the success story of Dish Wireless and its cloud-native standalone 5G infrastructure leveraging data streaming.
Read More
0
0
0