Pinned
Advanced SQL - The next frontier
Advanced SQL is a powerful tool that allows you to retrieve, analyze, and manipulate large amounts of data in a structured and efficient way
Jan 12, 20235 min read444
Command Palette
Search for a command to run...
Latest articles
Apache Hudi: A Deep Dive with Python Code Examples
In today's data-driven world, real-time data processing and analytics have become crucial for businesses to stay competitive. Apache Hudi (Hadoop Upserts and Incremental) is an open-source data management framework that provides efficient data ingest...
Jun 8, 20245 min read1.9K
Exploring Large Language Models (LLMs) with Python: A Comprehensive Guide
Introduction Large Language Models (LLMs) have revolutionized the field of Natural Language Processing (NLP). These models, such as GPT-4, are designed to understand and generate human-like text. In this post, we will delve into how to work with LLMs...
Feb 15, 20246 min read147
Implementing Real-Time Credit Card Fraud Detection with Apache Flink on AWS
Credit card fraud is a significant concern for financial institutions, as it can lead to considerable monetary losses and damage customer trust. Real-time fraud detection systems are essential for identifying and preventing fraudulent transactions as...
Jan 5, 20244 min read243
Managing keys & environment variables in a python pipeline/app
In a production ETL (extract, transform, load) pipeline, it is often helpful to manage environment variables to store sensitive information, such as database credentials or API keys. This allows you to keep this sensitive information separate from yo...
Oct 31, 20233 min read100
Migrating from AWS Redshift to Google BigQuery: A Step-by-Step Guide
A Comprehensive Guide to Migrating from Redshift to BigQuery Migrating your data from Amazon Redshift to Google BigQuery can be a significant undertaking, but with careful planning and execution, it can lead to enhanced performance and scalability fo...
Aug 10, 20234 min read22
ScyllaDB - Getting started
Recently I read this article where Discord migrated its messages cluster from Cassandra to ScyllaDB, it reduced message latencies from 200 milliseconds to 5 milliseconds, which got me intrigued to explore ScyllaDB.How Discord Migrated Trillions of Me...
Mar 12, 20236 min read1.8K
Deploy your data pipelines with Github Actions
Automate, customize, and execute your software development workflows right in your repository with GitHub Actions. You can discover, create, and share actions to perform any job you'd like, including CI/CD, and combine actions in a completely customi...
Jan 29, 20234 min read389
Idempotency in Data pipelines - Overview
Idempotency is an important concept in data engineering, particularly when working with distributed systems or databases. In simple terms, an operation is said to be idempotent if running it multiple times has the same effect as running it once. This...
Jan 11, 20235 min read628