Pinned
Advanced SQL - The next frontier
Advanced SQL is a powerful tool that allows you to retrieve, analyze, and manipulate large amounts of data in a structured and efficient way
HD's blog - Data, DataOps, Observability, MLops
Staff Software Engineer, Risk Platform, working mostly on Fraud, Risk and compliance problems in Fintech Writing mostly about Risk, Fraud, Data and cloud, sometimes productivity and other musings.
Latest articles
Apache Hudi: A Deep Dive with Python Code Examples
In today's data-driven world, real-time data processing and analytics have become crucial for businesses to stay competitive. Apache Hudi (Hadoop Upserts and Incremental) is an open-source data management framework that provides efficient data ingest...
Exploring Large Language Models (LLMs) with Python: A Comprehensive Guide
Introduction Large Language Models (LLMs) have revolutionized the field of Natural Language Processing (NLP). These models, such as GPT-4, are designed to understand and generate human-like text. In this post, we will delve into how to work with LLMs...
Implementing Real-Time Credit Card Fraud Detection with Apache Flink on AWS
Credit card fraud is a significant concern for financial institutions, as it can lead to considerable monetary losses and damage customer trust. Real-time fraud detection systems are essential for identifying and preventing fraudulent transactions as...
Managing keys & environment variables in a python pipeline/app
In a production ETL (extract, transform, load) pipeline, it is often helpful to manage environment variables to store sensitive information, such as database credentials or API keys. This allows you to keep this sensitive information separate from yo...
Migrating from AWS Redshift to Google BigQuery: A Step-by-Step Guide
A Comprehensive Guide to Migrating from Redshift to BigQuery Migrating your data from Amazon Redshift to Google BigQuery can be a significant undertaking, but with careful planning and execution, it can lead to enhanced performance and scalability fo...
ScyllaDB - Getting started
Recently I read this article where Discord migrated its messages cluster from Cassandra to ScyllaDB, it reduced message latencies from 200 milliseconds to 5 milliseconds, which got me intrigued to explore ScyllaDB.How Discord Migrated Trillions of Me...
Deploy your data pipelines with Github Actions
Automate, customize, and execute your software development workflows right in your repository with GitHub Actions. You can discover, create, and share actions to perform any job you'd like, including CI/CD, and combine actions in a completely customi...
Idempotency in Data pipelines - Overview
Idempotency is an important concept in data engineering, particularly when working with distributed systems or databases. In simple terms, an operation is said to be idempotent if running it multiple times has the same effect as running it once. This...