Old files still read after a rename because of column IDs

In one line

Iceberg gives each column an unchanging field ID and writes that ID into the data files too. Because reads match columns by ID, not by name, adding, dropping, renaming, and reordering columns and widening types all finish with one line of metadata, and old files are not rewritten.

Why schema changes used to be accidents

In a data lake, changing a schema was a frightening thing for a long time. The cause lies in what you use to find a column inside a file.

In a table that finds columns by name (most Hive-style Parquet tables), if you rename amount to amount_krw, the old files have only a column called amount, so when you read by the new name that column is entirely null. Worse things happen too. If you drop coupon and a few months later create a new coupon with a different meaning, the old coupon values left in the old files come back to life and show up mixed into the new column.

A format that finds columns by position (CSV, order-based Hive tables) breaks in the opposite direction. If you drop the third column, the fourth column is pulled into the third position and names and values fall out of step.

The Evolution docs cite exactly these two failures. A format that tracks by name can bring back a dropped column if you reuse the name, and a format that tracks by position changes which column a name refers to when you drop a column. So teams either rewrote the whole table every time they changed a schema, or could not change it at all and lived with a wrong name for years.

How it works — matching by ID

When you create a table, each column gets an ID (order_id 1, customer_id 2, …). The writing engine also records that ID as field_id in the column definitions of the Parquet file. Schemas pile up in metadata as a list, each with its own schema-id, and a snapshot remembers the schema ID from when it was created.

The column projection rule is simple. The columns of a data file are selected by field ID. An ID that is in the table schema but not in the file is filled with null (unless there is a name mapping or a default value). From this one rule, all five kinds of change become safe.

Change	What happens in metadata	When reading an old file
Add	A field with a new ID is created	That ID is not there, so null
Rename	Only the name of the same ID changes	Linked by ID to the old name in the file, so the values stay
Drop	That ID is removed from the current schema	Even if the values remain in the file, they are not read
Reorder	Only the field order changes	Selected by ID, so values do not get mixed
Widen	The type changes in the wider direction	Values written with the narrower type are read widened

A column added again with the same name as a dropped column gets a new ID. The old files hold only values for the old ID, so nothing shows up in the new column. This is what the docs guarantee as "an added column never reads existing values of other columns".

Types can only be widened

The schema evolution section of the spec nails down three type changes allowed in format versions 1 and 2 — int → long, float → double, decimal(P, S) → decimal(P', S) (P' > P, widening only the precision). Version 3 adds a few more, such as date → timestamp. All of them go in the direction where no value is truncated. Turning a long back into an int is not allowed because large values already in files could be truncated, and the engine rejects that change.

What it looks like in the field

Fixing a wrongly named column. It is common for a table to live for years with a typo column like amout. In Iceberg, renaming is one line of metadata, so you can fix it the same day. You only need to tell consumers that open files directly (scripts that do not know field IDs) that they still see the old name.

An amount column overflowed int. The day comes when an amount in won exceeds 2.1 billion. Widening int → long happens right away without rewriting files. Conversely, a request to change a long to an int "to save space" is rightly rejected.

I dropped a column and recreated it, and the old values showed up mixed in. This was a common accident with old tables that were read by name. After moving to Iceberg, even if you recreate the column with the same name, the new column starts out empty.

What really matters in practice

• The ID is a column's identity, and the name is only a label. Renaming and reordering are always safe.
• A schema change does not rewrite data files. No snapshot is added either; one more schema just piles up in metadata.
• Types can only be widened. int → long, float → double, widening decimal precision.
• The ID of a dropped column does not come back. A column recreated with the same name is an empty new column.

What you will do in the next lab

You create a table and load March 1, add a coupon column, and then load March 2. After renaming amount to amount_krw, you open the footer of the Parquet file from the first commit directly with pyarrow and confirm that the old name amount and field_id 4 are still there. You write down the error condition raised when you widen amount_krw to BIGINT and then try to go back to INT, drop coupon and add it again to confirm the new ID and the empty values, and count that even though several schemas have piled up, the snapshots and data files are unchanged.