List: Data | Curated by Siddique Ahmad

Apr 23, 2024

21 stories

Data
Netflix Technology Blog
Data Gateway — A Platform for Growing and Protecting the Data TierShahar Zimmerman, Vidhya Arvind, Joey Lynch, Vinay Chella
Apr 23, 2024
694
5
Apr 23, 2024
694
5
In
TDS Archive
by
Ciro Greco
Write-Audit-Publish for Data Lakes in Pure Python (no JVM)An open source implementation of WAP using Apache Iceberg, Lambdas, and Project Nessie all running entirely Python
Apr 12, 2024
334
2
Apr 12, 2024
334
2
Ong Xuan Hong
DataOps 03: Trino + DBT + Spark —  Everything Everywhere All at OnceI choose a lazy person to do a hard job. Because a lazy person will find an easy way to do it.
 — Bill Gates
Feb 22, 2023
255
2
Feb 22, 2023
255
2
Albert Wong
Open Source Alternatives to DataBricks SQL WarehouseDatabricks SQL Warehouse is a managed service within the Databricks platform that provides scalable SQL compute resources decoupled from…
Jan 5, 2024
65
Jan 5, 2024
65
In
StarRocks Engineering
by
StarRocks Engineering
Trip.com Chooses StarRocks over Trino to Query Data in Apache Hive (10x performance)Introduction
Dec 9, 2023
21
Dec 9, 2023
21
Jeremy Surget
What I learned after one year of building a Data Platform from scratchMy key learnings on building a Data platform, from the tech side to the business side
Nov 14, 2023
3.2K
49
Nov 14, 2023
3.2K
49
In
Dev Genius
by
Petrica Leuca
Data processing with Spark: time travelingThe utilities which enable us to accomplish ACID in Spark, do not come empty handed. They have many features and in this article I will…
Sep 28, 2022
59
1
Sep 28, 2022
59
1
In
InterviewNoodle
by
Mahesh Saini
DoorDash’s Write-Heavy Scalable Inventory Platform — System DesignAs DoorDash made the move from made-to-order restaurant delivery into the Convenience and Grocery (CnG) business, they had to find a way…
Oct 4, 2023
1.1K
12
Oct 4, 2023
1.1K
12
In
BlaBlaCar
by
Souhaib Guitouni
11 lessons learned managing a Data Platform team within a data meshIntroduction
Oct 10, 2023
439
7
Oct 10, 2023
439
7
Sairam Krish
Visualize parquet files with Apache Superset using Trino or PrestoSQLMany times, I like to visualize contents in formats like parquet, csv, json etc in Apache Superset. This article tries to provide a demo…
Dec 30, 2021
59
Dec 30, 2021
59
Mariusz Kujawski
Data Lakehouse vs Data Warehouse vs  Data Lake-Comparison of data platformsFor decades, data warehouses have been the dominant architectural approach for building data platforms in enterprises. However, with the…
Jul 24, 2023
390
4
Jul 24, 2023
390
4
In
TDS Archive
by
💡Mike Shakhomirov
How to Become a Data EngineerA shortcut for beginners in 2024
Oct 7, 2023
1.3K
15
Oct 7, 2023
1.3K
15
In
TDS Archive
by
Damian Gil
Mastering Customer Segmentation with LLMUnlock advanced customer segmentation techniques using LLMs, and improve your clustering models with advanced techniques
Sep 26, 2023
4.5K
36
Sep 26, 2023
4.5K
36
In
ProfitOptics
by
Jean-Georges Perrin
Data Contract 101A quick and not-so-dirty introduction to data contracts
Sep 10, 2023
365
1
Sep 10, 2023
365
1
In
AWS in Plain English
by
Apache SeaTunnel
Meet Apache SeaTunnel, a new Apache Top-Level Project!Introduction
Jun 1, 2023
67
1
Jun 1, 2023
67
1
Teepika R M
Materialized views in hiveTable vs View vs Materialized View
Aug 8, 2022
13
Aug 8, 2022
13
In
TDS Archive
by
Mahdi Karabiben
Writing design docs for data pipelinesExploring the what, why, and how of design docs for data components  —  and why they matter.
May 22, 2023
781
1
May 22, 2023
781
1
Vasileios Anagnostopoulos
Manipulating Delta Lake tables on MinIO with TrinoAn educational Delta Lakehouse
Mar 2, 2023
83
2
Mar 2, 2023
83
2
In
ITNEXT
by
Gary A. Stafford
Building Data Lakes on AWS with Kafka Connect, Debezium, Apicurio Registry, and Apache HudiLearn how to build a near real-time transactional data lake on AWS using a combination of Open Source Software (OSS) and AWS Services
Feb 28, 2023
174
3
Feb 28, 2023
174
3
In
Netflix TechBlog
by
Netflix Technology Blog
Auto-Diagnosis and Remediation in Netflix Data PlatformBy Vikram Srivastava and Marcelo Mayworm
Jan 14, 2022
567
1
Jan 14, 2022
567
1

Data

Data Gateway — A Platform for Growing and Protecting the Data Tier

Shahar Zimmerman, Vidhya Arvind, Joey Lynch, Vinay Chella

Write-Audit-Publish for Data Lakes in Pure Python (no JVM)

An open source implementation of WAP using Apache Iceberg, Lambdas, and Project Nessie all running entirely Python

DataOps 03: Trino + DBT + Spark — Everything Everywhere All at Once

I choose a lazy person to do a hard job. Because a lazy person will find an easy way to do it. — Bill Gates

Open Source Alternatives to DataBricks SQL Warehouse

Databricks SQL Warehouse is a managed service within the Databricks platform that provides scalable SQL compute resources decoupled from…

Trip.com Chooses StarRocks over Trino to Query Data in Apache Hive (10x performance)

Introduction

What I learned after one year of building a Data Platform from scratch

My key learnings on building a Data platform, from the tech side to the business side

Data processing with Spark: time traveling

The utilities which enable us to accomplish ACID in Spark, do not come empty handed. They have many features and in this article I will…

DoorDash’s Write-Heavy Scalable Inventory Platform — System Design

As DoorDash made the move from made-to-order restaurant delivery into the Convenience and Grocery (CnG) business, they had to find a way…

11 lessons learned managing a Data Platform team within a data mesh

Introduction

Visualize parquet files with Apache Superset using Trino or PrestoSQL

Many times, I like to visualize contents in formats like parquet, csv, json etc in Apache Superset. This article tries to provide a demo…

Data Lakehouse vs Data Warehouse vs Data Lake-Comparison of data platforms

For decades, data warehouses have been the dominant architectural approach for building data platforms in enterprises. However, with the…

How to Become a Data Engineer

A shortcut for beginners in 2024

Mastering Customer Segmentation with LLM

Unlock advanced customer segmentation techniques using LLMs, and improve your clustering models with advanced techniques

Data Contract 101

A quick and not-so-dirty introduction to data contracts

Meet Apache SeaTunnel, a new Apache Top-Level Project!

Introduction

Materialized views in hive

Table vs View vs Materialized View

Writing design docs for data pipelines

Exploring the what, why, and how of design docs for data components — and why they matter.

Manipulating Delta Lake tables on MinIO with Trino

An educational Delta Lakehouse

Building Data Lakes on AWS with Kafka Connect, Debezium, Apicurio Registry, and Apache Hudi

Learn how to build a near real-time transactional data lake on AWS using a combination of Open Source Software (OSS) and AWS Services

Auto-Diagnosis and Remediation in Netflix Data Platform

By Vikram Srivastava and Marcelo Mayworm

Siddique Ahmad