Switch a month-partitioned table to days — and leave the old files alone

Goal

Create hidden partitioning that splits a table with only a transform rule on a source column (order_ts), and measure with planning how many files and rows one condition ends up opening. While writing to the table, change month granularity to day granularity (partition evolution), and confirm from the metadata that old files stay under the old rule while only new files follow the new rule.

Why it matters

A Hive-style table makes the partition a column. You keep a separate column such as order_date, and both the writer and the reader have to know that column. If the reader filters only on order_ts, it cannot skip a single partition, and if the writer computes the time zone wrongly, rows land in the wrong partition. And to change the partitioning method, you have to rewrite the whole table. Iceberg makes the partition a rule. If you write only a source column and a transform, such as days(order_ts), the engine computes the partition value for each file and records it in the manifest, and the reader skips files with only an order_ts condition. Rules carry spec IDs and pile up in the metadata, so even if you change the rule, old files are still read under the old spec. The feeling that you have to rewrite them is the most common misunderstanding.

Steps

1. With /root/ice/part/month.py (app ice-part-month), create lake.part.orders with PARTITIONED BY (months(order_ts)) and commit the whole of March (31 files) in one go.
2. With /root/ice/part/plan1.py (pyiceberg), plan the scan for the condition that order_ts is the single day 2026-03-10 and write it to /root/ice/part/out/plan_march.json.
3. With /root/ice/part/evolve.py (app ice-part-evolve), change months(order_ts) to days(order_ts).
4. With /root/ice/part/april.py (app ice-part-april), commit the whole of April (30 files) in one go.
5. With /root/ice/part/plan2.py, plan the conditions for March 10 and April 10 and write them to /root/ice/part/out/plan.json.
6. With /root/ice/part/bucket.py (app ice-part-bucket), create lake.part.customers with PARTITIONED BY (bucket(4, customer_id)) and load the customers.
7. With /root/ice/part/specs.py, write the default spec ID of lake.part.orders and the number of data files per spec to /root/ice/part/out/specs.json.
8. In /root/ice/part/report.md, write three sections, ## 숨은 파티셔닝 ## 파티션 진화 ## 버킷 (keep these headings as written; they stand for hidden partitioning, partition evolution, and buckets).

Notes

• Partition values are in the manifests, not in directory names. In Spark you see them with SELECT spec_id, partition, record_count FROM lake.part.orders.files.
• pyiceberg's tbl.scan(row_filter=…).plan_files() selects the files to read from the manifests alone (partition values and column statistics), without opening files.
• order_ts is a timestamp with a time zone (timestamptz). In Python conditions, use ISO strings with +00:00 attached.
• A common mistake: rewriting March with rewrite_data_files after changing the partitioning — in this lab, leaving the old files as they are is the correct answer, and rewriting them lowers your grade. To undo, run DROP TABLE lake.part.orders PURGE and start again from step 1.
• Official docs: Partitioning · Evolution — Partition evolution · Spec — Partition Transforms · Spark DDL — REPLACE PARTITION FIELD

A table split by month — without a partition column

Create /root/ice/part/month.py with the app name ice-part-month, have it create lake.part.orders (six columns, 'format-version' = '2') with PARTITIONED BY (months(order_ts)), and load the 31 files /data/ice/orders/2026-03-*.csv with a single append().

months(order_ts) uses the number of months counted from January 1970 as the partition value. Check that no new column appears in the schema. The grader checks that the transform of spec 0 is month, that the first commit is the whole of March, and that the partition values of the files are all March 2026.

The files a one-day condition opens — the limit of month granularity

With /root/ice/part/plan1.py (pyiceberg), build a scan for the condition order_ts >= 2026-03-10T00:00:00+00:00 and < 2026-03-11T00:00:00+00:00, and write the number of planned files, the sum of the record_count of those files, and the number of rows that actually match to /root/ice/part/out/plan_march.json as {"files_planned", "records_scanned", "rows"} (the table has only March right now).

Even though the condition is on order_ts only, it is compared with the partition value (month), so files of other months drop out. But even if you want just one day, March 10, you have to read the March files for a whole month. The difference between records_scanned and rows is that cost. The grader plans the same scan again based on the first snapshot (when only March existed) and compares.

Partition evolution — only metadata changes

Create /root/ice/part/evolve.py with the app name ice-part-evolve and run ALTER TABLE lake.part.orders REPLACE PARTITION FIELD months(order_ts) WITH days(order_ts).

A new spec 1 is created and the default (default-spec-id) changes to 1. No snapshot is created and the data files stay as they are — only the files written from now on follow the new rule. The grader looks at the spec list, the default spec, and the number of snapshots.

The April commit — only new files get the new spec

Create /root/ice/part/april.py with the app name ice-part-april and load the 30 files /data/ice/orders/2026-04-*.csv with a single append(). Do not rewrite the March files.

The files of the new commit are recorded with spec_id 1, separately for each day. The March files stay with spec_id 0 (month) and live together in the same snapshot. The reading engine plans separately for each spec, so the result is the same even with the two rules mixed. The grader also checks that the March files keep the path and spec of the first commit.

The same one-day condition, a different cost

With /root/ice/part/plan2.py, plan the one-day conditions for March 10 and April 10 separately and write them to /root/ice/part/out/plan.json as {"march": {"files_planned", "records_scanned", "rows"}, "april": {…}}.

The two days have similar numbers of rows, but the number of rows you have to read differs greatly. For March it opens the month file as a whole, and for April it opens only that day's one file. The grader plans the same scan again with the current table and compares, and checks that the April side reads only that day's rows.

Columns with many distinct values go into buckets

Create /root/ice/part/bucket.py with the app name ice-part-bucket, have it create lake.part.customers(customer_id STRING, tier STRING, region STRING, signup_date DATE) with PARTITIONED BY (bucket(4, customer_id)), and load /data/ice/customers.csv.

bucket(N, column) uses the remainder of the value's 32-bit murmur3 hash divided by N as the partition value. Because the spec fixes the hash, any engine computes the same slot — the grader recomputes the slot of each customer with pyiceberg and compares it with the partition value of the file.

Two rules in one table

With /root/ice/part/specs.py (pyiceberg), write the default spec ID of lake.part.orders and the number of data files per spec to /root/ice/part/out/specs.json as {"default_spec_id": 정수, "files_by_spec": {"0": 정수, "1": 정수}} (the placeholders in the code stand for integers).

Count with the spec_id column of tbl.inspect.files() (content 0 is a data file). It is normal for files of two specs to be mixed in one snapshot.

Partition design as team rules

In /root/ice/part/report.md, write three sections, ## 숨은 파티셔닝 ## 파티션 진화 ## 버킷 (hidden partitioning, partition evolution, and buckets). In the second section, put the two records_scanned values for March and April from step 5 as numbers.

Was the original choice to split by month wrong, or did it stop fitting as the data grew? Also write what the reason should be if you ever have to rewrite the old March files.