In today's data-driven world, real-time data processing and analytics have become crucial for businesses to stay competitive. Apache Hudi (Hadoop Upserts and Incremental) is an open-source data management framework that provides efficient data ingest...
Recently I read this article where Discord migrated its messages cluster from Cassandra to ScyllaDB, it reduced message latencies from 200 milliseconds to 5 milliseconds, which got me intrigued to explore ScyllaDB.How Discord Migrated Trillions of Me...
Advanced SQL is a powerful tool that allows you to retrieve, analyze, and manipulate large amounts of data in a structured and efficient way
Idempotency is an important concept in data engineering, particularly when working with distributed systems or databases. In simple terms, an operation is said to be idempotent if running it multiple times has the same effect as running it once. This...
A data lake is a central repository that allows you to store all your structured and unstructured data at any scale. You can store your data as-is, without having to first structure the data, and run different types of analytics—from dashboards and v...
dbt (data build tool) is an open-source command-line tool that helps data analysts and data engineers automate the process of transforming and loading data from various sources into a data warehouse. In this tutorial, we will be setting up dbt with S...
