One table, many engines — the spec is the contract and the catalog is the meeting point

In one line

An Iceberg table is metadata and data files written according to the spec, so Python can write to a table Spark created and DuckDB can read it. But this promise holds only when everyone finds the "current" metadata in the same catalog and understands the same spec features and types.

Why several engines

Spark is convenient for batch transformations, Python for small loads and check scripts, and DuckDB for analysts' ad hoc queries. In the past, each engine kept its own table and copied data. Copies always drift apart, and you end up arguing over which one is real. The Multi-Engine Support docs introduce Iceberg as an open standard that any processing engine can use. That is possible because a table is defined not by a file format but by metadata that the spec defines.

The same document also shows the conditions. Spark and Flink each have runtime jars for each engine version, and you cannot write with a version that is not on the support list. That is why the lab image of this course uses Spark 4.1.3, unlike the other Spark course (4.2) in the repository — the runtime of Iceberg 1.11.0 goes up to 4.1.

How it works — where they meet and where they drift

The catalog is where they meet. In the lab environment, Spark and pyiceberg open the same SQLite catalog. Both commit with a conditional swap, so they do not overwrite each other's commits. DuckDB's iceberg extension offers two ways. The way that points to metadata and reads it directly needs no catalog and is read-only, and to write as well you ATTACH a REST catalog.

A direct read is pinned to that moment. A metadata file never changes once written, so iceberg_scan('…/00003-….metadata.json') returns the table as it was then, whenever you run it. Even if someone commits after that, there is no warning. The DuckDB docs say the feature of guessing the "latest" version from file names can break ACID and is off by default. To read the present, you have to look up the current path in the catalog every time.

Where types drift. The primitive types of the spec distinguish timestamp without a time zone from timestamptz, which is stored in UTC. Spark's TIMESTAMP becomes timestamptz. So if you try to write a timestamp without a time zone to the same column from Python, pyiceberg rejects it saying the schema does not match (confirmed in the lab image). The right thing is to attach UTC to the timestamps. Remember too that date boundaries are cut in UTC.

Where features drift. A format version goes up when old readers cannot read a new feature correctly. The spec says you can keep writing with an old version to avoid features an engine has not implemented yet. Before using version 3 deletion vectors or new types, you must check that every engine reading that table supports them. An engine that does not know delete files returns deleted rows.

Where they fit well. Columns are selected by field ID, so if you rename a column in one engine, other engines also read the values of old files with the new name. Partition values are in the manifests, so whoever wrote them, files are skipped with the same condition.

Who wrote it — the snapshot summary

When there are several engines, the time comes when you need to know "where did this commit come from?". The snapshot summary is the clue. In the lab image, Spark leaves engine-name (spark), engine-version, and app-id in the summary, and pyiceberg 0.12 does not. Also remember that the summary is filled by the writing side, so it differs by engine.

What it looks like in the field

The dashboard froze at yesterday's numbers. Someone hard-coded a metadata.json path in the BI tool. The table is committed to every day, but that tool was reading the table of the first day. Make the tool go through the catalog, or change it to find the current path every time.

A Python load stopped with a schema error. The timestamps read from the CSV had no time zone. Attaching a time zone fixes it. You should take this error gratefully — if it had gone in silently, values off by nine hours would have been mixed in.

Attaching a new engine. First check whether it supports your catalog, which format versions and delete formats it reads, and how it handles timestamptz. If even one of the three drifts, results are silently wrong.

What really matters in practice

• One catalog, and the "current" from the catalog every time. If you hold on to a metadata path, it goes stale.
• Distinguish timestamp from timestamptz. Spark's TIMESTAMP is timestamptz.
• Set the format version and delete format to match the slowest engine.
• Changes that resolve by ID, like renaming, are visible as they are to every engine.

What you will do in the next lab

Spark creates a table split by date and loads a week, and pyiceberg writes one more day with timestamps that have a time zone. After computing totals by region with DuckDB and doing a conditional read with pyiceberg, you write down the old metadata path, have Spark commit one more day, and read the old path and the new path with DuckDB separately. You rename a column in Spark to check that the other two engines read with the new name, and finally have Spark gather the rows written by all three engines into a daily table.