Same Filename, Different Contents
In one line
The schema of a file someone else gives you changes without notice. So the receiving side has to freeze a fingerprint, pin down in code what breaks and what does not, and judge it automatically.
Why this was needed
A file with the same name arrives every Monday. For 12 weeks there was no problem. In week 13 the dashboard's sales go to 0. You open the file and one column name has changed from amount_krw to amount. The sender says, "we tidied up the field names." The reason they did not tell you is simple — on their side they did not know it was the kind of thing that would stop our pipeline.
There is a structural reason this keeps happening. The sender changed their own system's schema, and the receiver was using it as a contract. If there is no document between those two facts, a change always arrives quietly. On top of that, we cannot control the side that makes the change. With a production DB you can migrate while old and new schemas coexist, but with someone else's file there is no such room.
So there is only one thing you can do. Make the pipeline notice before anyone else that it has changed.
How it works
You extract the schema from the received file and freeze it as a fingerprint. What goes into the fingerprint is the column names and order, the type inferred for each column, and a sample of values. If you join just the names and types and hash them, you get one short fingerprint, and if that fingerprint differs from last week's, something has changed.
For type inference, you have to fix the yardstick first. After removing empty values, if what remains are all integer-shaped it is int, if it includes decimals it is float, if all are YYYY-MM-DD it is date, and anything else is str. This is inference, not declaration — so which yardstick you used has to be written in the code so that you know where to fix it when it turns out wrong later.
Next you compare with last week's fingerprint and classify what changed into five kinds.
| Kind | What you see | How you find out |
|---|---|---|
| Addition | A column that did not exist appears | The difference of the name sets |
| Removal | A column that existed is gone | The difference of the name sets |
| Rename | One vanished and one appeared | Whether the value samples of the vanished column and the new column overlap heavily |
| Type change | The type of the same name differs | Comparing the inferred types |
| Semantic change | You see nothing | The schema cannot catch it. It shows only in the value distribution |
Finding a rename by values is the key. By names alone it is two cases, one vanished and one appeared, but if the value samples are almost the same, the same column has just changed its name. With this judgment, the pipeline can answer "adding one mapping is enough."
The last line is the most important in this lab. A semantic change is never caught by a schema check. Even if the unit of the amount changes from won to thousand won, the column name and the type stay the same. All the checks pass, and only the sales become one thousandth.
What it looks like in the field
First, it stops because it is surprised by an unknown column. It is common for the sender to add one more column for their own needs, and there is no reason for our load to stop because of it. So the default of the rules is that an unknown column passes. Conversely, a required column that has disappeared stops the load. These two lines are the skeleton of the compatibility rules, and everything else sits somewhere in between.
Second, you do not distinguish required from optional. Without the distinction, every removal is treated with the same weight and nobody reads the warnings anymore. The list of required columns is decided by the business, not by the data.
Third, you treat a widened type and a broken type the same. An int becoming a float is usually a change to decimal notation, so calculations still work. An int becoming a str makes a sum turn into string concatenation or raises an exception. If you put the two at the same level, the warnings become useless.
Fourth, you do not look at the distribution. The only way to catch a semantic change is to compare the distribution of values with last week's. If the median jumps to three times or more, or falls to below a third, a person has to look. This is not evidence but a lead — it may be that large transactions really did bunch up that week. Handing the lead to a person is as far as the machine's share goes.
What really matters in practice
- Keep the fingerprint together with the file. To say "it differs from last week", last week has to be written down somewhere.
- Pin the compatibility rules down in code, not in a document. If they are only in a document, nobody reads them, and even if they do, the judgment differs from person to person.
- Get the list of required columns from the business. If the engineer decides alone, there are no grounds to change it later.
- Distribution monitoring produces leads. Notify rather than stop, and let a person make the judgment.
What you will do in the next lab
You run a reproducer that exports the same 30 orders over seven weeks while changing only the schema, to build an inbox, and then grow a schema tool, schema.py, one step at a time. You freeze the fingerprint, classify additions and removals, find renames by value samples, separate type changes, and declare the compatibility rules in contract.json so that pass, warn, and stop are judged automatically. In the end you catch, by value distribution, a column whose unit changed while its name and type stayed the same, and show in numbers what a schema check cannot see.