Tag: #data-engineering

← All posts

• Following 657,607 Links, and the Lifespan of a URL — Why Links Die of Connection Failure, Not 404

  A study published its results after following, in 2026, every one of roughly 655,000 shortened links created between 2009 and 2014. More important than the headline number is the composition of the failures: most dead li

  2026-08-14 · 6 min read #web#data-engineering#archival#reliability#url-design

• Open Source Worth Watching Right Now (5) Data and ML Pipelines

  A data pipeline is not something a single scheduler solves. Ingestion, transformation, orchestration, execution engines, the model lifecycle, and search stores have each become the territory of a different tool. This pos

  2026-08-12 · 5 min read #open-source#data-engineering#mlops#python#rust

• How to Read the 2026 Data Tooling Landscape — Layer-by-Layer Decisions, Not a Tool List

  On July 28, 2026, "A Data Tooling Landscape Guide for Developers" landed on GeekNews and picked up 51 points. It is a good map, but a map does not choose your destination for you. This post redraws the same landscape acr

  2026-07-31 · 16 min read #data-engineering#lakehouse#iceberg#duckdb#data-platform

• After Airflow 2 EOL — The Real Work List for Going 2 to 3, and Where 3.x Stands at 3.3

  Apache Airflow 2 passed EOL on April 22, 2026, and it no longer receives security patches. But the road from 2 to 3 is not a pip upgrade — it is an architecture change. Direct metadata-DB access from workers disappears,

  2026-07-17 · 15 min read #data-engineering#airflow#workflow-engine#migration

• Parquet 2.13.0 Adds nan_count and IEEE 754 Total Order — How Float Statistics Got Fixed After 3 Years and 3 Months

  Parquet's min/max statistics are the mechanism that lets query engines skip data. But on floating-point columns, this mechanism has been subtly broken for over 10 years — NaN returns false in every comparison, so it gets

  2026-07-16 · 17 min read #data-engineering#parquet#columnar-storage#apache-arrow

• Iceberg v3 Row Lineage: Row IDs Aren't Stored in the File

  Apache Iceberg format v3 writes row lineage — a stable identifier for every row — into the spec, and it isn't optional: any v3 table has it on unconditionally. But that value isn't stored in the data file — it's computed

  2026-07-16 · 18 min read #iceberg#lakehouse#table-format#data-engineering#spark

• DuckDB Gets a Client-Server Protocol — What Quack Changes and What It Doesn't

  On May 12, 2026, the DuckDB team announced Quack — a client-server protocol built on top of HTTP that lets two DuckDB instances become client and server to each other, so the same database can be read and written concurr

  2026-07-16 · 18 min read #database#duckdb#olap#analytics#data-engineering

• PySpark 4.2 Puts Python UDFs on Arrow by Default — the Story of 132 out of 225 Type-Coercion Cells Changing

  Apache Spark 4.2.0, released on July 14, 2026, flipped the default of spark.sql.execution.pythonUDF.arrow.enabled from false to true under SPARK-54555. Vendor blogs describe this as a faster columnar path without rewriti

  2026-07-16 · 16 min read #spark#python#arrow#data-engineering

• From Documents to a Knowledge Graph: An Honest Pipeline

  'Extract a knowledge graph from your documents' looks like a single LLM call in a demo. But turning a customer's documents into a graph you can actually query is a six-stage pipeline, and most of the cost and pain lives

  2026-07-15 · 10 min read #knowledge-graph#ai#llm#data-engineering

• The Complete Guide to LLM Training Data Preprocessing — From Web Crawls to Token Packing, with the Latest Papers

  Good models come from good data, and good data comes from a preprocessing pipeline. This post walks through the entire pretraining data process step by step — web crawl collection → text extraction → language identificat

  2026-07-09 · 7 min read #ai#llm#data-engineering#preprocessing#training

• DuckDB in Practice — The Era of Embedded Analytics That Ends on Your Laptop

  Why is DuckDB, often called the SQLite of OLAP, so loved? We start from the architecture and cover direct Parquet querying, hands-on examples of window functions, PIVOT, and ASOF JOIN, zero-copy integration with pandas,

  2026-06-12 · 14 min read #duckdb#database#olap#analytics#parquet

• Hadoop Ecosystem & Data Engineering 2026 Deep Dive - Hadoop, Spark, Flink, Trino, Iceberg, Delta Lake, Hudi, Airflow, dbt ♪ Listenable

  A complete look at the data engineering stack in May 2026. The truth behind the "Hadoop is dead" headline (HDFS is shrinking, but YARN and the lakehouse pattern survive), the conclusion of the Iceberg vs Delta Lake vs Hu

  2026-05-16 · 25 min read #hadoop#spark#flink#trino#presto

• Data Lakehouse & Modern Data Engineering 2026 — Iceberg / Delta / Hudi / Paimon / Tabular (Databricks acquisition) / Trino / Spark 4 / Flink 2 / DataFusion Deep Dive ♪ Listenable

  Data engineering in 2026 is no longer the era of "data warehouse vs data lake." Apache Iceberg emerged as the winner of the 2024-25 table format war — Netflix, Apple, LinkedIn, Stripe, Airbnb all converged on it. Databri

  2026-05-16 · 30 min read #data-engineering#data-lakehouse#apache-iceberg#delta-lake#apache-hudi

• Is Hadoop Dead? — The Evolution of the Big Data Stack, From Hadoop to Lakehouse (Spark, Iceberg, Delta, and Where Things Actually Stand in 2026) ♪ Listenable

  Is Hadoop dead? More precisely — it has been demoted from the default. The HDFS plus YARN plus MapReduce trio is no longer how anyone starts a new analytics platform. Spark replaced MapReduce, object storage replaced HDF

  2026-05-14 · 21 min read #hadoop#big-data#spark#iceberg#delta-lake

• Polars 1.x vs Pandas — End of an Era? A Modern DataFrame Deep Dive for 2026 ♪ Listenable

  Polars hit 1.x with a stable-API promise, Pandas 2.2 finally adopted PyArrow as a real backend, and the gravity of the dataframe world shifted. Is Polars actually replacing Pandas? This deep dive walks through the Rust p

  2026-05-14 · 20 min read #polars#pandas#dataframe#apache-arrow#duckdb

• Data Engineering Complete Guide — Lakehouse, Streaming, dbt, Orchestration, Data Mesh (Season 2 Ep 8, 2025)

  Data engineering is no longer "running ETL scripts." In 2025 a data engineer designs Lakehouse architectures (Iceberg, Delta, Hudi), combines Streaming (Flink, Kafka) with Batch (Spark), standardizes modeling with dbt, r

  2026-04-15 · 10 min read #data-engineering#lakehouse#iceberg#delta-lake#hudi

• Data Engineering Fundamentals — ETL, Data Warehouses, Streaming, and Data Lakes

  How do you build data pipelines? From ETL/ELT and data warehouses to streaming, batch processing, Spark, Kafka, and Airflow.

  2026-04-13 · 17 min read #data-engineering-aws#data-engineering#etl#data-warehouse#spark

• Data Engineering Pipeline Complete Guide 2025: ETL/ELT, Spark, Airflow, Real-Time Streaming

  Everything about data engineering! ETL vs ELT, Apache Spark (PySpark), Apache Airflow (DAG/Operator/Sensor), real-time streaming (Kafka+Flink), dbt (data transformation), data warehouses (BigQuery/Snowflake/Redshift), da

  2026-03-25 · 18 min read #data-engineering#etl#elt#spark#airflow

• Snowflake Data Engineer Career Guide: Mastering the Cloud Data Warehouse King ♪ Listenable

  A complete analysis of the Snowflake Data Engineer role. Snowflake architecture (compute-storage separation), Snowpark, Cortex AI, Iceberg Tables, dbt integration, streaming (Snowpipe), cost optimization — 20 interview q

  2026-03-23 · 24 min read #snowflake#data-engineering#data-warehouse#sql#dbt

• Databricks AI Engineer (FDE) Complete Guide: Spark, Unity Catalog, RAG to Customer Deployment ♪ Listenable

  A complete analysis of the Databricks AI Engineer (FDE) JD. From Spark/Delta Lake/Unity Catalog tech stack, Lakehouse architecture, RAG pipeline construction, to customer deployment skills — 25 interview questions and an

  2026-03-23 · 34 min read #databricks#fde#spark#delta-lake#unity-catalog