TT Lab
Get started
Learn Learning paths Courses

Lakehouse Table Format — Understanding Apache Iceberg Through Its Metadata

Share one catalog between Spark and pyiceberg, rename a table, and bring a dropped one back

Continue in TT Lab

Goal

Have Spark and pyiceberg share one JDBC catalog (a single SQLite file), and confirm from the values before and after a commit that a commit is a change of one pointer slot in the catalog. See that renaming and dropping a table happen only in the catalog while the files stay as they are, and bring a dropped table back from a single metadata file.

Why it matters

The "real state" of an Iceberg table is in its metadata file, and the catalog is a small table that links one name to one file. But this small table is the referee for concurrent writes. When two writers commit at the same time, the catalog accepts only one of them under the condition "change the pointer only if it is still the one I read". If the catalog cannot do this atomic swap, the table breaks. When several engines look at the same catalog, a table created by Spark is continued by a Python job, and Spark reads the result again. On the other hand, accidents happen when you do not know that names, locations, and files are different layers. People delete the old path believing the files moved when the table was renamed, wait for DROP TABLE to free up space, or, conversely, give up on a table they dropped by mistake, thinking it is lost forever.

Steps

  1. With /root/ice/cat/make.py (app ice-cat-make), create lake.cat.orders (format-version 2) and commit the three files 2026-03-01, 03-02, and 03-03 in one go.
  2. With /root/ice/cat/list.py (pyiceberg, load_catalog("lake")), write the namespaces and the table list to /root/ice/cat/out/tables.json.
  3. With /root/ice/cat/customers.py (pyiceberg), read /data/ice/customers.csv, create lake.cat.customers, and load it.
  4. With /root/ice/cat/join.py (app ice-cat-join), join the two tables on customer_id and create the table lake.cat.tier_counts(tier, orders) with the number of orders per tier.
  5. Commit 2026-03-04 to lake.cat.orders, and write the path before the commit, the path after the commit, and the previous slot after the commit to /root/ice/cat/out/pointer.json.
  6. Rename lake.cat.tier_counts to lake.cat.tier_summary, and write the metadata paths before and after the rename to /root/ice/cat/out/rename.json.
  7. Drop lake.cat.tier_summary without PURGE, count the data files that remain, register lake.cat.tier_restored from the metadata file from just before the drop, and write the results to /root/ice/cat/out/restore.json.
  8. In /root/ice/cat/report.md, write three sections, ## 포인터 ## 이름과 위치 ## 지우기와 되살리기 (keep these headings as written; they stand for pointer, name and location, and dropping and restoring).

Notes

Spark creates a table — one commit

Create /root/ice/cat/make.py with the app name ice-cat-make, have it create the lake.cat namespace and lake.cat.orders (six columns, 'format-version' = '2'), and insert the three files 2026-03-01, 03-02, and 03-03 with a single append().

If you pass a list like spark.read.csv([경로1, 경로2, 경로3], …), it becomes one DataFrame (the three placeholders stand for the three file paths). One commit means one snapshot. The grader checks that the row count of the first snapshot is the sum of the three files and that the summary has the engine-name that Spark leaves.

See the same catalog with pyiceberg

Create /root/ice/cat/list.py with pyiceberg's load_catalog("lake"), have it write the namespaces and the table list to /root/ice/cat/out/tables.json as {"namespaces": ["cat", …], "tables": ["cat.orders", …]}, and run it with python3 list.py.

pyiceberg reads the lake catalog settings (type: sql, uri: sqlite:////root/ice/catalog.db) from ~/.pyiceberg.yaml. It opens the same file without a server, so the table Spark created shows up as it is. Names come out as tuples, so join them with a dot.

pyiceberg creates a table

With /root/ice/cat/customers.py, read /data/ice/customers.csv with pyarrow (signup_date as date32), create lake.cat.customers, and load it. Use overwrite() so the rows do not double when you run it again.

create_table_if_not_exists("cat.customers", schema=arrow_table.schema) converts a pyarrow schema into an Iceberg schema and assigns a field ID to each column. The summary of a snapshot written by pyiceberg has no engine-name, which Spark leaves — the grader uses that to see who wrote it.

Spark joins the tables of the two engines

Create /root/ice/cat/join.py with the app name ice-cat-join, join lake.cat.orders and lake.cat.customers on customer_id, and create the number of orders per tier (tier) as lake.cat.tier_counts (columns tier, orders) with CREATE TABLE … AS SELECT.

No matter who wrote it, an Iceberg table is metadata and Parquet of the same spec, so the engine does not care. The grader computes the number of orders per tier directly from the source CSVs and compares (orders has March 1–3 at this point).

One commit moves the pointer by one slot

Write down the current metadata path of lake.cat.orders, then commit 2026-03-04 with /root/ice/cat/append.py (date argument), read the path after the commit and the catalog's previous_metadata_location, and write them to /root/ice/cat/out/pointer.json as {"before", "after", "previous_after"}.

A commit finishes with a single conditional update in the catalog, "change to after only if the current value is before", once the new metadata file has been fully written. So after the commit, the previous slot must equal the path from before the commit. You can bundle shell variables into JSON with jq -n --arg.

The name exists only in the catalog

Run ALTER TABLE lake.cat.tier_counts RENAME TO cat.tier_summary (do not put the catalog lake in front of the new name), and write the metadata paths before (tier_counts) and after (tier_summary) the rename to /root/ice/cat/out/rename.json as {"before", "after"}.

Renaming in a JDBC catalog is an UPDATE that fixes the table_name of one row in iceberg_tables. The table location and the files stay the same, so the table with the new name still points to files under …/cat/tier_counts/. The grader checks that the two paths are the same and that the old name has disappeared from the catalog.

Bring a dropped table back from one metadata file

Write down the metadata path of lake.cat.tier_summary, run DROP TABLE without PURGE, count the Parquet files that remain in /root/ice/warehouse/cat/tier_counts/data, and bring it back with CALL lake.system.register_table(table => 'lake.cat.tier_restored', metadata_file => '<그 경로>') (the placeholder stands for that path). Write {"metadata_file", "files_left"} to /root/ice/cat/out/restore.json.

A DROP without PURGE deletes only one row in the catalog. Because the metadata file holds the schema, snapshots, and file lists in full, you can register it again in any catalog if you know just that path (this is the method used when moving catalogs or restoring from a backup). If you added PURGE, the files are deleted and cannot be restored.

What the catalog does and does not do

In /root/ice/cat/report.md, write three sections, ## 포인터 ## 이름과 위치 ## 지우기와 되살리기 (pointer, name and location, and dropping and restoring). In the third section, put the number of remaining files you counted in step 7 as a number.

Write down what you would have to move and what you could leave as it is when you change catalogs in production (for example JDBC → REST) or restore a table from a backup.