Schemas — Skip Validation and You Pay Twice Later
One-line summary
Start by assuming that all data coming in from outside is a string, and put it into a typed table only after explicitly validating and converting it.
Why this was needed
Raw data in practice is messy without exception. The same date comes in four formats, amounts carry currency symbols and thousands separators, status values vary in case and leading and trailing spaces, and required items are empty.
If you try to put this as is into a typed table, the load fails. Then two wrong responses commonly appear. Either you simply skip the failed rows (data disappears without anyone knowing), or you turn every column into text (you push the problem downstream).
How it works
The standard structure has three layers.
- Raw (staging) — all columns are strings. Kept as is. Never fixed.
- Cleansed (clean) — only rows that pass validation go in, with their types in place.
- Rejected (reject) — rows that did not pass go in along with the reason.
The core of this structure is the conservation law. The sum of the clean count and the reject count must equal the raw count exactly. If there is a row that is in neither, that is silently vanished data, and it is the most dangerous incident in a pipeline.
Leaving the reject reason is also non-negotiable. A row discarded without a reason can never be recovered by anyone later. Even if short, like "amount missing" or "email missing," always write it together.
Conversion rules must also be explicit. If there are several date formats, identify each format with a regular expression and apply a different parsing rule. If you leave it to automatic inference, 03/04/2025 is quietly wrong depending on whether it means March 4 or April 3.
Turning an empty string into 0 in an amount is also a common mistake. No value and 0 are different. If a payment amount is empty, that is not a payment of 0 won but missing information, and the moment you fill it with 0, that fact is erased.
What it looks like in the field
It helps to leave the schema contract as code. If you record the list of columns and types of the clean table in a separate table, the downstream pipeline can detect immediately when someone later changes a column type. A schema change is by nature an incident of the kind that happens quietly and shows up days later as strange numbers.
Format choice is also worth pointing out. CSV can be read anywhere but has no type information and its delimiter escaping is fragile. JSON can express nesting but is bulky. Column-oriented formats such as Parquet carry types and statistics together and compress well, which favors analytics workloads. A configuration that splits raw storage into CSV or JSON and reloading for analysis into a columnar format is common.
Rules for changing a schema safely
A pipeline's schema is a contract between the producer and the consumer. If only one side changes, the other side breaks, so you first decide in which direction it stays compatible.
| Compatibility direction | Meaning | Allowed changes |
|---|---|---|
| Backward | A new consumer reads old data | Deleting a field, adding a field with a default |
| Forward | An old consumer reads new data | Adding a field, deleting an optional field |
| Full | Both | Only adding or deleting optional fields with defaults |
In streaming, you use backward compatibility as the default. This is because you can upgrade the consumers first and the producers afterward. If you reverse the order, old consumers run into new data and break.
Adding a required field is always a breaking change. You add it as optional with a default, and once all producers fill it in, you then raise it to required. Splitting it into two steps is the standard practice.
File format decides performance
| Format | Structure | Where it fits | Caution |
|---|---|---|---|
| CSV | Row | Small amounts that people look at | No types. Encoding and delimiter hell |
| JSON Lines | Row | Loads with a fluid schema | Big and slow |
| Avro | Row | Streaming, events | Together with a schema registry |
| Parquet | Column | Analytical queries | Heavy to write. Unsuitable for small files |
The reason columnar (Parquet) is fast for analytics is that it reads only the columns it needs. Queries that use only 3 of 50 columns are common, and a row-oriented format reads all 50. On top of that, column-wise compression works well, so the size is smaller too.
However, Parquet is actually slower when there are many small files. You have to read the metadata of every file, and that can be larger than the actual data. Aim to bundle them to 128MB–1GB.
Partitions follow the query pattern
s3://lake/events/dt=2026-09-06/hour=14/part-0001.parquet
└─ 날짜로 자르는 질의가 대부분이면 이렇게
If you choose the wrong partition key, every query scans everything. Conversely, if you split too finely, the small-file problem arises. Calculate how many files come out per day and then decide.
Keeping the date as a single string like dt=2026-09-06 is usually more convenient than
splitting it into year=/month=/day=. It is easier to use range queries and the directory depth is shallower.
What to do in the next lab
You profile staging.orders_raw, which consists only of strings, to count its defects, normalize the dates, amounts, and statuses, split the load into a clean table and a reject table, and confirm that the conservation law holds.