Lakehouse Table Format — Understanding Apache Iceberg Through Its Metadata

Spark creates, Python writes, DuckDB reads — three engines on one table

Goal

Have pyiceberg write one more day to a table Spark created, have DuckDB and pyiceberg read that table, and have Spark again gather and aggregate the rows written by all the engines. Along the way, confirm the "stale pointer" trap of reading while holding on to an old metadata path, and how one engine's schema change looks to the other engines.

Why it matters

The value of a table format lies in not having to choose an engine. Even if you do batch with Spark, small loads with Python, and ad hoc analysis with DuckDB, they all see the same metadata and the same Parquet files. But this promise comes with conditions. Every engine has to find the "current" metadata through the same catalog, and they have to support the same spec features (format version, delete files, types). A tool that reads a metadata.json path you pass it directly is convenient, but it is pinned to the moment that path points to. If someone commits after that, you are reading an old table with no warning at all. Be careful with types too — Spark's TIMESTAMP is Iceberg's timestamptz, so Python code that gives timestamps without a time zone gets caught by the schema check. Conversely, changes that resolve by field ID, like renaming a column, are visible as they are to every engine.

Steps

With /root/ice/eng/spark.py (app ice-eng-spark), create lake.eng.orders split by days(order_ts) and commit the seven files from 2026-03-01 to 03-07 in one go.
With /root/ice/eng/py_append.py (pyiceberg), append 2026-03-08 to the same table.
With /root/ice/eng/duck.py (DuckDB), read the current metadata and write the amount totals by region to /root/ice/eng/out/duck_region.json.
With /root/ice/eng/py_read.py (pyiceberg), write the number of rows where region = 'seoul' and order_ts >= 2026-03-05 to /root/ice/eng/out/py_count.json.
Write down the current metadata path, then commit 2026-03-09 with /root/ice/eng/append.py (app ice-eng-append), and with /root/ice/eng/stale.py read the old path and the new path with DuckDB separately and write the results to /root/ice/eng/out/stale.json.
With /root/ice/eng/rename.py (app ice-eng-rename), rename region to area, and with /root/ice/eng/columns.py write the column names seen by pyiceberg and DuckDB to /root/ice/eng/out/rename.json.
With /root/ice/eng/daily.py (app ice-eng-daily), create the daily table of the number of orders and the amount totals, lake.eng.daily(d, orders, amount).
In /root/ice/eng/report.md, write three sections, ## 한 표, 세 엔진 ## 낡은 포인터 ## 이름 바꾸기 (keep these headings as written; they stand for one table and three engines, the stale pointer, and renaming).

Notes

DuckDB reads with iceberg_scan('<metadata.json 경로>') after con.execute("LOAD iceberg") (the placeholder stands for the metadata.json path). ice-loc eng.orders prints the path. The extension was put into the image in advance (there is no internet).
You can see who wrote a snapshot in the summary — Spark leaves engine-name: spark and an app-id, and pyiceberg does not (SELECT summary FROM lake.eng.orders.snapshots).
Common mistakes: in step 2, trying to append with a timestamp without a time zone and getting blocked by a schema mismatch, and in step 5, reading the old path after the commit so that the two paths become the same.
Official docs: Multi-Engine Support · pyiceberg — API · DuckDB — Iceberg extension · Spec — Primitive Types

Spark creates the table

Create /root/ice/eng/spark.py with the app name ice-eng-spark, have it create lake.eng.orders (six columns, PARTITIONED BY (days(order_ts)), 'format-version' = '2'), and load the seven files from 2026-03-01 to 03-07 with a single append().

The summary of this snapshot records spark as the engine-name. The grader checks the row count of the first commit and the engine that wrote it.

Python writes to the same table

With /root/ice/eng/py_append.py, read the 2026-03-08 file with pyarrow (amount int32, order_ts a timestamp in the UTC time zone) and append it with load_catalog("lake").load_table("eng.orders").append(…).

pyiceberg compares the pyarrow schema with the table schema before writing. If order_ts is a timestamp without a time zone, it rejects it saying it does not match timestamptz. pyiceberg computes the partition (days) values itself and records them in the manifest. The grader checks that the second commit added the row count of March 8 and that an engine other than Spark wrote it.

DuckDB reads

With /root/ice/eng/duck.py, put the path printed by ice-loc eng.orders into iceberg_scan() to compute sum(amount) by region, and write it to /root/ice/eng/out/duck_region.json as {"지역": 합계, …} (the placeholders in the code stand for the region and the total).

DuckDB starts from the single metadata.json file without going through a catalog and follows the manifests down. The files Spark wrote and the files pyiceberg wrote are in the same list, so both are read. The grader compares with the totals computed from the March 1–8 source.

pyiceberg reads with a condition

With /root/ice/eng/py_read.py, count the rows where region == 'seoul' and order_ts >= 2026-03-05T00:00:00+00:00 and write it to /root/ice/eng/out/py_count.json as {"rows": 정수} (the placeholder in the code stands for an integer).

pyiceberg's conditions first select files by the partition values (dates) and column statistics in the manifests, and then filter rows within the selected files. The grader compares with the value computed from the March 5–8 source.

The stale pointer — an old path is an old table

Write down the current path with OLD=$(ice-loc eng.orders), commit 2026-03-09 with /root/ice/eng/append.py (app ice-eng-append, date argument), and then read NEW=$(ice-loc eng.orders). Make /root/ice/eng/stale.py take the two paths as arguments, count the rows of each with DuckDB, and write them to /root/ice/eng/out/stale.json as {"old_path", "old_rows", "new_path", "new_rows"}.

A metadata file never changes once written. If you read with an old path, you get the table at that moment, whenever you read — useful for time travel, but for a dashboard that meant to read "now", it silently shows stale numbers. The grader checks that both paths are in the metadata history and that the row counts equal the total-records at that time.

One engine renames, another engine sees it

With /root/ice/eng/rename.py (app ice-eng-rename), run ALTER TABLE lake.eng.orders RENAME COLUMN region TO area, and with /root/ice/eng/columns.py write the column names of pyiceberg's tbl.schema() and the result column names of DuckDB's iceberg_scan() to /root/ice/eng/out/rename.json as {"pyiceberg": [...], "duckdb": [...]}.

Renaming changes only the schema in the metadata, and the files still hold the old name (region). An engine that matches by field ID reads the values of the old files with the new name. The grader checks that area is in both lists and region is not.

Spark gathers the rows of all the engines

Create /root/ice/eng/daily.py with the app name ice-eng-daily, group lake.eng.orders by date (to_date(order_ts)), and create the number of orders and the amount total as lake.eng.daily(d, orders, amount) (CREATE TABLE … AS SELECT).

The March 8 rows were written by pyiceberg and the rest by Spark. To the reading engine there is no difference. Date boundaries are cut in the session time zone (UTC). The grader compares with the daily values computed from the March 1–9 source.

Rules for attaching several engines to one table

In /root/ice/eng/report.md, write three sections, ## 한 표, 세 엔진 ## 낡은 포인터 ## 이름 바꾸기 (one table and three engines, the stale pointer, and renaming). In the second section, put the old_rows and new_rows from step 5 as numbers.

If you were attaching a new engine (for example an in-house BI tool) to your team, what would you check first — whether it goes through the catalog, which format versions and delete files it reads, and how it handles timestamptz?