Add, rename, widen, drop and re-add columns — without rewriting a single file

Goal

Change the schema of an Iceberg table in four ways (add a column, rename, widen a type, and drop then re-add) and confirm that the old data files are read correctly even though they are never rewritten. Open a file directly to see that the secret is the field_id embedded in the Parquet file footer.

Why it matters

In a table that finds columns by name (most Hive-style Parquet tables), renaming is an accident. The old files have only the old name, so reading by the new name makes that column entirely null. In a format that finds columns by position (CSV), the moment you change the order, values go into the wrong columns. So people rewrite the whole table every time they change a schema, or cannot change it at all and live with a wrongly named column for years. Iceberg gives each column an unchanging ID and writes that ID when it writes the file. Reads match by ID, not by name. So renaming is one line of metadata, and even if you add a new column with the same name as a dropped one, it has a new ID, so the old values do not come back. A type can be changed only in a direction where no value is truncated (widening such as int → long).

Steps

1. With /root/ice/sch/base.py (app ice-sch-base), create lake.sch.orders (format-version 2) and load 2026-03-01.
2. With /root/ice/sch/add.py (app ice-sch-add), add a coupon STRING column and then load 2026-03-02. The coupon of that day's rows is 'SPRING' if amount >= 100000 and null otherwise.
3. With /root/ice/sch/rename.py (app ice-sch-rename), rename amount to amount_krw.
4. With /root/ice/sch/footer.py (pyarrow, pyiceberg), open one data file of the first commit and write the in-file name of field ID 4 to /root/ice/sch/out/footer.json.
5. With /root/ice/sch/widen.py (app ice-sch-widen), widen amount_krw to BIGINT, and write the name of the error condition raised when you try to narrow it back to INT to /root/ice/sch/out/narrow.txt.
6. With /root/ice/sch/readd.py, drop coupon and add it again with the same name, and write the old ID, the new ID, and the number of non-null values now to /root/ice/sch/out/readd.json.
7. With /root/ice/sch/history.py, write the number of schemas, the current schema ID, the number of snapshots, and the number of live data files to /root/ice/sch/out/history.json.
8. In /root/ice/sch/report.md, write three sections, ## 이름 바꾸기 ## 형 넓히기 ## 지웠다 다시 더하기 (keep these headings as written; they stand for renaming, widening a type, and dropping and re-adding).

Notes

• Spark's df.schema has no field IDs. You see the IDs in schemas[].fields[].id of metadata.json or in pyiceberg's tbl.schema().
• Read the Parquet footer with pyarrow.parquet.read_schema(경로) (the placeholder stands for the path), and the metadata[b"PARQUET:field_id"] of each field is the Iceberg field ID.
• A schema change does not create a snapshot. When this lab is over, there should still be only two snapshots (March 1 and 2).
• A common mistake: running the scripts of steps 1 and 2 again and ending up with three snapshots. If that happened, run spark-sql -e "DROP TABLE lake.sch.orders PURGE" and start again from step 1.
• Official docs: Evolution — Schema evolution · Spec — Schema Evolution · Spark DDL — ALTER TABLE

The table and the first commit — an ID for each column

Create /root/ice/sch/base.py with the app name ice-sch-base, have it create lake.sch and lake.sch.orders (six columns, 'format-version' = '2'), and load the 2026-03-01 file.

When you create a table, Iceberg numbers the columns with IDs starting at 1 (order_id 1 … order_ts 6). These IDs do not change even if a name changes. The grader looks at the IDs and names in the schema of the metadata and at the row count in the first snapshot.

Add a column — old files are read as null

Create /root/ice/sch/add.py with the app name ice-sch-add, run ALTER TABLE lake.sch.orders ADD COLUMN coupon STRING, and then load the 2026-03-02 file with coupon attached ('SPRING' if amount >= 100000, otherwise null).

The new column gets a new ID (7). The March 1 file has no ID 7, so the coupon of those rows is read as null — because the file was not rewritten. The grader opens the Parquet file written by the March 2 commit directly and checks that the values of the ID 7 column match the conditions of the source.

Rename — one line of metadata

Create /root/ice/sch/rename.py with the app name ice-sch-rename and run ALTER TABLE lake.sch.orders RENAME COLUMN amount TO amount_krw.

Renaming adds a new schema to the metadata (with only the name of ID 4 different). Even though the files of both days are unchanged, sum(amount_krw) comes out as the total of both days. The grader checks that the name of ID 4 changed and that the sum read by the new name equals the sum of the source.

The old name left in the Parquet footer

With /root/ice/sch/footer.py, open one data file of the first commit (March 1) with pyarrow.parquet.read_schema, find the in-file name of the field whose PARQUET:field_id is 4, and write it to /root/ice/sch/out/footer.json as {"file", "name_in_file", "field_id", "name_in_table"}.

A file keeps the name (amount) from the moment it was written. The table uses the current name (amount_krw). What links the two is the field_id in the footer. You can find a file of the first commit with pyiceberg's tbl.inspect.files(첫_스냅샷_ID) (the placeholder stands for the first snapshot ID); strip the leading file: from the path.

Types can only be widened

Create /root/ice/sch/widen.py with the app name ice-sch-widen, change amount_krw to BIGINT, then wrap the statement that tries to change it back to INT in a try and write the exception's getCondition() to the first line of /root/ice/sch/out/narrow.txt.

int → long truncates no value, so the old files (written as int) can be read as long as they are. The reverse could truncate values, so the spec does not allow it. The grader checks that the type of ID 4 is long and the name of the error condition.

If you drop a column and add it back with the same name

With /root/ice/sch/readd.py, run DROP COLUMN on coupon and then ADD COLUMN coupon STRING with the same name, and write the old ID, the new ID, and the current count(coupon) to /root/ice/sch/out/readd.json as {"old_id", "new_id", "non_null"}.

The ID of a dropped column is never reused. The new coupon gets a new ID, and the 'SPRING' values under ID 7 that remain in the March 2 file do not pair up with the new column and are invisible. In a table read by name, the old values would have come back to life. Read the IDs with pyiceberg's tbl.schemas() (history) and tbl.schema() (current).

Five schemas, two snapshots, files unchanged

With /root/ice/sch/history.py (pyiceberg), write the number of schemas, the current schema ID, the number of snapshots, and the number of live data files to /root/ice/sch/out/history.json as {"schemas", "current_schema_id", "snapshots", "data_files"}.

Each time you change the schema, one more schema piles up in the metadata, but snapshots do not grow, and the data files are exactly what the two commits wrote. The grader compares the four values with the metadata and also checks that the files that are live now are exactly the same as the files of the second snapshot (that no file was rewritten).

Schema change rules as team rules

In /root/ice/sch/report.md, write three sections, ## 이름 바꾸기 ## 형 넓히기 ## 지웠다 다시 더하기 (renaming, widening a type, and dropping and re-adding). In the third section, put the old ID and the new ID from step 6 as numbers.

If several teams read this table, write down which changes are fine to make at any time and which should be announced. Also write what breaks for consumers that do not read by field ID (scripts that open files directly).