Tag: #spark

← All posts

• Iceberg v3 Row Lineage: Row IDs Aren't Stored in the File

  Apache Iceberg format v3 writes row lineage — a stable identifier for every row — into the spec, and it isn't optional: any v3 table has it on unconditionally. But that value isn't stored in the data file — it's computed

  2026-07-16 · 18 min read #iceberg#lakehouse#table-format#data-engineering#spark

• PySpark 4.2 Puts Python UDFs on Arrow by Default — the Story of 132 out of 225 Type-Coercion Cells Changing

  Apache Spark 4.2.0, released on July 14, 2026, flipped the default of spark.sql.execution.pythonUDF.arrow.enabled from false to true under SPARK-54555. Vendor blogs describe this as a faster columnar path without rewriti

  2026-07-16 · 16 min read #spark#python#arrow#data-engineering

• Modern Hadoop & Big Data Ecosystem 2026 Deep Dive - Hadoop 3.4 · Spark 4 · Hive 4 · Kafka 4 · Flink 2 · Iceberg · Trino ♪ Listenable

  In 2026 Hadoop and the broader big-data ecosystem are not dead — they have been relocated. Hadoop 3.4 Erasure Coding and Ozone, Spark 4 ANSI mode and Spark Connect, Hive 4 Iceberg integration, Kafka 4 with KRaft GA and t

  2026-05-16 · 26 min read #english#hadoop#spark#hive#kafka

• Modern Scala 2026 — Scala 3.6 / 3.7 / ZIO 2 / Cats Effect 3 / Pekko / Tapir / Mill / Scala CLI Deep Dive ♪ Listenable

  Everything about the 2026 Scala ecosystem — new features in Scala 3.6/3.7, Scala 4 preview, sbt 2.x vs Mill vs Scala CLI, ZIO 2 vs Cats Effect 3, the Pekko/Akka BSL drama, Play/http4s/Tapir, Scala.js/Native, Spark, Liche

  2026-05-16 · 26 min read #scala#scala-3#sbt#mill#scala-cli

• Hadoop Ecosystem & Data Engineering 2026 Deep Dive - Hadoop, Spark, Flink, Trino, Iceberg, Delta Lake, Hudi, Airflow, dbt ♪ Listenable

  A complete look at the data engineering stack in May 2026. The truth behind the "Hadoop is dead" headline (HDFS is shrinking, but YARN and the lakehouse pattern survive), the conclusion of the Iceberg vs Delta Lake vs Hu

  2026-05-16 · 25 min read #hadoop#spark#flink#trino#presto

• Data Lakehouse & Modern Data Engineering 2026 — Iceberg / Delta / Hudi / Paimon / Tabular (Databricks acquisition) / Trino / Spark 4 / Flink 2 / DataFusion Deep Dive ♪ Listenable

  Data engineering in 2026 is no longer the era of "data warehouse vs data lake." Apache Iceberg emerged as the winner of the 2024-25 table format war — Netflix, Apple, LinkedIn, Stripe, Airbnb all converged on it. Databri

  2026-05-16 · 30 min read #data-engineering#data-lakehouse#apache-iceberg#delta-lake#apache-hudi

• Is Hadoop Dead? — The Evolution of the Big Data Stack, From Hadoop to Lakehouse (Spark, Iceberg, Delta, and Where Things Actually Stand in 2026) ♪ Listenable

  Is Hadoop dead? More precisely — it has been demoted from the default. The HDFS plus YARN plus MapReduce trio is no longer how anyone starts a new analytics platform. Spark replaced MapReduce, object storage replaced HDF

  2026-05-14 · 21 min read #hadoop#big-data#spark#iceberg#delta-lake

• Columnar Storage Deep Dive 2025: Parquet, ORC, Apache Arrow, Dremel — Why Analytics DBs Are 10,000x Faster

  How Snowflake, BigQuery, and Spark query terabytes in seconds. A complete tour of Parquet Dremel structure, ORC, Apache Arrow in-memory format, RLE/Dictionary compression, and vectorized execution.

  2026-04-15 · 13 min read #columnar-storage#parquet#orc#apache-arrow#dremel

• Apache Spark Internals Deep Dive 2025: RDD, Catalyst Optimizer, Tungsten, Whole-Stage Codegen, Shuffle

  Why Spark runs the same query 100x faster than MapReduce. RDD to DataFrame/Dataset, Catalyst optimizer, Tungsten project, whole-stage code generation, and shuffle optimization — a complete internal analysis.

  2026-04-15 · 13 min read #spark#catalyst#tungsten#rdd#whole-stage-codegen

• Data Engineering Fundamentals — ETL, Data Warehouses, Streaming, and Data Lakes

  How do you build data pipelines? From ETL/ELT and data warehouses to streaming, batch processing, Spark, Kafka, and Airflow.

  2026-04-13 · 17 min read #data-engineering-aws#data-engineering#etl#data-warehouse#spark

• Data Engineering Pipeline Complete Guide 2025: ETL/ELT, Spark, Airflow, Real-Time Streaming

  Everything about data engineering! ETL vs ELT, Apache Spark (PySpark), Apache Airflow (DAG/Operator/Sensor), real-time streaming (Kafka+Flink), dbt (data transformation), data warehouses (BigQuery/Snowflake/Redshift), da

  2026-03-25 · 18 min read #data-engineering#etl#elt#spark#airflow

• Databricks AI Engineer (FDE) Complete Guide: Spark, Unity Catalog, RAG to Customer Deployment ♪ Listenable

  A complete analysis of the Databricks AI Engineer (FDE) JD. From Spark/Delta Lake/Unity Catalog tech stack, Lakehouse architecture, RAG pipeline construction, to customer deployment skills — 25 interview questions and an

  2026-03-23 · 34 min read #databricks#fde#spark#delta-lake#unity-catalog

• Apache Spark in Production: Performance Tuning, Shuffle, Skew, AQE, and Streaming Operations

  A practical production guide to Apache Spark covering the execution model that matters, shuffle and skew, AQE, cache strategy, and Structured Streaming operational checklists.

  2026-03-17 · 6 min read #spark#spark-tuning#data-engineering#shuffle#adaptive-query-execution

• Apache Iceberg Data Lakehouse Table Format Guide ♪ Listenable

  A comprehensive guide to Apache Iceberg table format for data lakehouse architecture, covering schema evolution, partitioning, time travel, and query optimization.

  2026-03-14 · 27 min read #apache-iceberg#data-lakehouse#big-data#hadoop#spark

• How to Install Spark on YARN

  Learn how to install Spark on YARN.

  2022-11-26 · 7 min read #hadoop#spark#yarn