Browsing Tag

data lake

12 posts
The Heart of the Data Mesh Beats Real Time with Apache Kafka
Read More
  • 233 views
  • 5 minute read

The Heart of the Data Mesh Beats Real-Time with Apache Kafka

If there were a buzzword of the hour, it would undoubtedly be “data mesh”! This new architectural paradigm unlocks analytic and transactional data at scale and enables rapid access to an ever-growing number of distributed domain datasets for various usage scenarios. The data mesh addresses the most common weaknesses of the traditional centralized data lake or data platform architecture. And the heart of a decentralized data mesh infrastructure must be real-time, reliable, and scalable. Learn how the de facto standard for data streaming, Apache Kafka, plays a crucial role in building a data mesh.
Read More
1
0
0
Best Practices for Data Analytics with AWS Azure Googel BigQuery Spark Kafka Confluent Databricks
Read More
  • 117 views
  • 10 minute read

Best Practices for Building a Cloud-Native Data Warehouse or Data Lake

The concepts and architectures of a data warehouse, a data lake, and data streaming are complementary to solving business problems. Unfortunately, the underlying technologies are often misunderstood, overused for monolithic and inflexible architectures, and pitched for wrong use cases by vendors. Let’s explore this dilemma in a blog series. This is part 5: Best Practices for Building a Cloud-Native Data Warehouse or Data Lake.
Read More
0
0
0
Case Studies for Cloud Native Analytics with Data Warehouse Data Lake Data Streaming Lakehouse
Read More
  • 119 views
  • 7 minute read

Case Studies: Cloud-native Data Streaming for Data Warehouse Modernization

The concepts and architectures of a data warehouse, a data lake, and data streaming are complementary to solving business problems. Unfortunately, the underlying technologies are often misunderstood, overused for monolithic and inflexible architectures, and pitched for wrong use cases by vendors. Let’s explore this dilemma in a blog series. This is part 4: Case Studies for cloud-native data streaming and data warehouses.
Read More
0
0
0
Data Warehouse and Data Lake Modernization with Data Streaming
Read More
  • 144 views
  • 9 minute read

Data Warehouse and Data Lake Modernization: From Legacy On-Premise to Cloud-Native Infrastructure

The concepts and architectures of a data warehouse, a data lake, and data streaming are complementary to solving business problems. Unfortunately, the underlying technologies are often misunderstood, overused for monolithic and inflexible architectures, and pitched for wrong use cases by vendors. Let’s explore this dilemma in a blog series. This is part 3: Data Warehouse Modernization: From Legacy On-Premise to Cloud-Native Infrastructure.
Read More
0
0
0
Data Warehouse vs Data Lake vs Data Streaming Comparison
Read More
  • 409 views
  • 9 minute read

Data Warehouse vs. Data Lake vs. Data Streaming – Friends, Enemies, Frenemies?

The concepts and architectures of a data warehouse, a data lake, and data streaming are complementary to solving business problems. Unfortunately, the underlying technologies are often misunderstood, overused for monolithic and inflexible architectures, and pitched for wrong use cases by vendors. Let’s explore this dilemma in a blog series. This is part 1: Data Warehouse vs. Data Lake vs. Data Streaming – Friends, Enemies, Frenemies?
Read More
0
0
0
When not to use Apache Kafka
Read More
  • 2.1K views
  • 18 minute read

When NOT to use Apache Kafka?

Apache Kafka is the de facto standard for event streaming to process data in motion. This blog post explores when NOT to use Apache Kafka. What use cases are not a good fit for Kafka? What limitations does Kafka have? How to qualify Kafka out as it is not the right tool for the job?
Read More
1
0
0
Stream Exchange for Data Sharing with Apache Kafka in a Data Mesh
Read More
  • 1.0K views
  • 10 minute read

Streaming Data Exchange with Kafka and a Data Mesh in Motion

Data Mesh is a new architecture paradigm that gets a lot of buzzes these days. This blog post looks into this principle deeper to explore why no single technology is the perfect fit to build a  Data Mesh. Examples show why an open and scalable decentralized real-time platform like Apache Kafka is often the heart of the Data Mesh infrastructure, complemented by many other data platforms to solve business problems.
Read More
0
0
0
Kappa Architecture vs Lambda Architecture for Apache Kafka Pulsar Data Lakes
Read More
  • 2.6K views
  • 17 minute read

Kappa Architecture is Mainstream Replacing Lambda

Real-time data beats slow data. That’s true for almost every use case. Nevertheless, enterprise architects build new infrastructures with the Lambda architecture that includes separate batch and real-time layers. This blog post explores why a single real-time pipeline, called Kappa architecture, is the better fit. Real-world examples from companies such as Disney, Shopify, Uber, and Twitter explore the benefits of Kappa but also show how batch processing fits into this discussion positively without the need for Lambda.
Read More
13
0
3
Serverless Kafka for Data in Motion as Rescue for Data at Rest in the Data Lake
Read More
  • 1.4K views
  • 12 minute read

Serverless Kafka in a Cloud-native Data Lake Architecture

Apache Kafka became the de facto standard for processing data in motion. Kafka is open, flexible, and scalable. Unfortunately, the latter makes operations a challenge for many teams. Ideally, teams can use a serverless Kafka SaaS offering to focus on business logic. However, hybrid scenarios require a cloud-native platform that provides automated and elastic tooling to reduce the operations burden. This blog post explores how to leverage cloud-native and serverless Kafka offerings in a hybrid cloud architecture. We start from the perspective of data at rest with a data lake and explore its relation to data in motion with Kafka.
Read More
1
0
0