Lakehouse Table Format — Understanding Apache Iceberg Through Its Metadata

Create a table and walk from metadata.json down to its data files

Goal

Create an Iceberg table with Spark and commit one day of orders twice. Then walk down the tree by hand, without tools: catalog → metadata.json → manifest list → manifest → data file. Write the values you find at each layer to files, and the grader compares them with the actual metadata.

Why it matters

An Iceberg table is not a "directory" but a "list of files". The reader does not scan directories; it trusts only the list that metadata.json points to. So a commit is not about moving files — it is about writing a new list and changing one pointer in the catalog, and the moment that one pointer changes, every reader sees the new state. If you follow this structure by hand once, you can see that every later feature is a variation on the same principle. Time travel reads the list of an old snapshot, rollback turns the pointer back to an old snapshot, and compaction and expiration rewrite lists and delete files that nobody points to anymore. This tree is also the first thing you look at when something goes wrong. "Why can't I see this row?" almost always becomes "Is that file in the list of the current snapshot?"

Steps

With /root/ice/meta/create.py (app ice-meta-create), create the namespace lake.meta and the table lake.meta.orders. The columns are order_id STRING, customer_id STRING, region STRING, amount INT, status STRING, order_ts TIMESTAMP and the property is 'format-version' = '2'.
Make /root/ice/meta/load.py take a date as an argument and commit /data/ice/orders/<날짜>.csv to the table once (where the placeholder stands for the date argument), then run it with 2026-03-01.
Run the same script with 2026-03-02 so that there are two snapshots.
From iceberg_tables in the catalog /root/ice/catalog.db, read this table's metadata_location and previous_metadata_location and write them on two lines to /root/ice/meta/out/pointer.txt.
Extract the snapshot list from the current metadata.json and write it to /root/ice/meta/out/snapshots.json.
Decode the manifest list (Avro) of the current snapshot and write the list of manifests to /root/ice/meta/out/manifests.json.
Decode those manifests and write the list of data files that are live now to /root/ice/meta/out/datafiles.json.
In /root/ice/meta/report.md, write three sections, ## 포인터 ## 스냅샷 ## 파일 (keep these headings as written; they stand for pointer, snapshots, and files). Put the number of snapshots in the second section, and the number of manifests and the number of data files in the third.

Notes

Run the scripts like cd /root/ice/meta && spark-submit create.py. Spark takes 15–20 seconds to start.
ice-loc meta.orders prints the path of the current metadata.json that the catalog points to. ice-avro <파일> decodes Avro into one line of JSON per record (filter it with jq), where the placeholder stands for the file path.
The file: before a path is added by Java, and file:/// is added by Python. The grader treats both as the same path.
A common mistake: running load.py twice with the same date, which makes three snapshots. To start over, run spark-sql -e "DROP TABLE lake.meta.orders PURGE" and then begin at step 1 — you have to extract every value you wrote down again from the new table.
Official docs: Table Spec — Overview · Spark Getting Started · JDBC Catalog

Create the table — the first metadata with no snapshot

Create /root/ice/meta/create.py with the app name ice-meta-create, have it create the lake.meta namespace and the lake.meta.orders table (six columns, 'format-version' = '2'), and run it with spark-submit.

Creating the table produces one metadata file (00000-….metadata.json) and adds one row with its path to the catalog. Nothing has been committed yet, so there is no snapshot. The grader checks that the first metadata file has no snapshot and that the format version and the column names and types are correct.

First commit — one snapshot

Create /root/ice/meta/load.py with the app name ice-meta-load, have it read /data/ice/orders/<날짜>.csv for the date given as the argument with an explicit schema and insert it with writeTo("lake.meta.orders").append() (where the placeholder stands for the date). Run it with spark-submit load.py 2026-03-01.

One append is one commit, and one commit is one snapshot. The grader checks that the first snapshot was created by an append with no parent, and that added-records in the summary equals the number of rows in that day's file.

Second commit — a snapshot that points to its parent

Run spark-submit load.py 2026-03-02 with the same script so that there are exactly two snapshots.

The new snapshot points to the previous snapshot as its parent (parent-snapshot-id), and the sequence number goes up by one. The files of the first snapshot are not rewritten; only one new file is added. If you loaded the same date twice, drop the table with PURGE and start again from step 1.

Two slots of the catalog pointer

From iceberg_tables in /root/ice/catalog.db, read metadata_location and previous_metadata_location of the meta.orders row and write them on the first and second lines of /root/ice/meta/out/pointer.txt.

A JDBC catalog has just one row per table. A commit is a conditional UPDATE — "change to the new value only if the current value equals the one I read" — and the value before the change is left in the previous slot. You can print the two slots as two lines with sqlite3 -separator.

The snapshot list in metadata.json

From the current metadata.json (ice-loc meta.orders), build /root/ice/meta/out/snapshots.json in the shape {"current_snapshot_id": 정수, "snapshots": [{"snapshot_id", "parent_snapshot_id", "sequence_number", "manifest_list"}, …]} (the placeholder in the code stands for an integer).

Keys in metadata.json use hyphens (current-snapshot-id, parent-snapshot-id). In jq, read them with brackets like .["snapshot-id"]. A snapshot ID is a 19-digit integer and easy to get wrong when copied by hand — carry it over with jq as is.

The manifest list — the list of manifests

Decode the manifest_list file of the current snapshot with ice-avro and write it to /root/ice/meta/out/manifests.json as an array [{"manifest_path", "added_snapshot_id", "added_files_count", "existing_files_count"}, …].

The manifest list of the second snapshot contains the manifest created by the first commit again, unchanged. A new commit never rewrites an old manifest; it only points to it — that is why commits are cheap and old snapshots stay intact. Use added_snapshot_id to see which commit created a manifest.

The manifests — data files and row counts

Decode the manifests in manifests.json and write the data files of the entries whose status is not 2 (DELETED) to /root/ice/meta/out/datafiles.json as [{"file_path", "record_count"}, …].

One line (entry) of a manifest is a status (0 EXISTING, 1 ADDED, 2 DELETED) and a data_file struct. The reading engine decides which files to open by looking only at this list and the column statistics (lower and upper bounds) — it does not scan directories. The grader checks that the list matches exactly the live files of the current snapshot and that the sum of the row counts equals the number of rows in the table.

The tree on one page

In /root/ice/meta/report.md, write three sections, ## 포인터 ## 스냅샷 ## 파일 (pointer, snapshots, files). Put the number of snapshots in the second section as a number, and the number of manifests and the number of data files of the current snapshot in the third as numbers.

Write down, in order, which layer you would check first when someone says "the data I loaded yesterday is missing". Copy the numbers from your out/ files.