Data Migration Is Its Own Track
Summary
Data migration is not something that starts after development ends but a separate track that runs alongside development, and data problems discovered in migration sometimes change the design.
Why this is a problem
If you push migration to the back half of the work, discovery comes late. And the problems that come out of migration are usually problems whose options depend on time. Suppose a column you defined as NOT NULL in the new schema is 30% empty in the legacy system. If you learn that 6 months ahead, you can fix the schema, agree on a default-value rule, or ask the business side to clean up. If you learn it 2 weeks before launch, only one option remains — just fill it in. And the values filled in that way remain as that system's data for years.
Migration is not a late-project task
A lesson that comes up repeatedly in next-generation project retrospectives.
Data migration must be managed as a separate track from the start of the project.
The reason is simple. Migration is not something that can start only after development ends; it must run in parallel with development. And data problems discovered during migration sometimes change the design.
What actually happens often:
- A column defined as
NOT NULLin the new schema is 30% empty in legacy - 47 legacy code values appear that do not map to the new code scheme
- In legacy, the same customer exists 3 times over, differing only in how the name is written
If you discover these 2 weeks before launch, you have no options. If you know 6 months ahead, you can fix the schema, agree on cleansing rules, or ask for business cleanup.
Three migration rehearsals
The migration plan of a large project includes several rehearsals.
1차 리허설 (D-90) : 절차 확인. 시간이 얼마나 걸리는가
2차 리허설 (D-30) : 데이터 품질 확인. 정제 규칙 검증
3차 리허설 (D-7) : 실전과 동일 조건. 시간·순서·인원 확정
The real output of a rehearsal is not data but the measured duration. Only when you can say "the migration takes 4 hours" as a measurement and not an estimate does the migration plan's timeline hold. And in most cases, you learn at the first rehearsal that it takes 2–3 times longer than expected.
The AS-IS → TO-BE mapping definition
This is the core deliverable of migration. You write it per column like this.
| TO-BE table | Column | AS-IS source | Transformation rule | If unmapped |
|---|---|---|---|---|
| CUSTOMER | CUST_NM | TB_CUST.NAME | TRIM, normalize whitespace |
Error |
| CUSTOMER | REG_DT | TB_CUST.REGDATE | YY/MM/DD → YYYYMMDD |
19000101 |
| CUSTOMER | GRADE_CD | TB_CUST.LEVEL | Refer to the code mapping table | 99 (other) |
| ORDERS | ORD_STS_CD | TB_ORD.STATUS | Refer to the code mapping table | Excluded from migration |
The "If unmapped" column is the key. When a value that cannot be mapped appears, do you treat it as an error, put in a default value, or exclude the row? If you do not decide this, developers handle it arbitrarily, and the result comes back as "the counts do not match."
And always keep the excluded rows as a list. "1,187 of 1,200 migrated, 13 excluded" is the result, and you must be able to answer what those 13 are.
What cleansing actually involves
Legacy data usually looks like this.
| Symptom | Example | Handling |
|---|---|---|
| Leading/trailing spaces | " 홍길동 " |
TRIM |
| Mixed date formats | 20260801, 2026-08-01, 26/08/01 |
Normalize to one |
| String NULL | The value is the text "NULL" or "null" |
To a real NULL |
| Formatted amounts | "1,250,000" |
Remove commas, to a number |
| Duplicates | Several rows with the same key | Set a rule and keep one (usually the latest modified date) |
| Code value mismatch | Retired codes, typos | Mapping table or the other code |
| Broken encoding | ????? |
Re-extract from the source |
The deduplication rule must be agreed with the business. "Keep the latest" seems natural, but there are cases where the latest row is actually incomplete. It is a business judgment, not a technical one.
Verification — in four layers
Post-migration verification is stacked from the bottom up. If even one is missing, a hole opens.
1. 스키마 검증 테이블·컬럼·타입·제약·인덱스가 의도대로 있는가
2. 데이터 검증 건수 · 합계 · NULL 개수 · 체크섬
3. 성능 검증 주요 쿼리의 실행계획과 응답시간
4. 애플리케이션 핵심 업무 시나리오 스모크 테스트
We emphasize again why checksums are used in item 2. Counts and totals alone cannot catch "one row went missing and another doubled." Comparing a hash of the sorted key list catches even that case.
-- 개념적으로: 키를 정렬해 이어 붙인 문자열의 해시
SELECT md5(group_concat(CUST_ID, ',' ORDER BY CUST_ID)) FROM CUSTOMER;
Verification cost and reliability differ as follows.
| Method | Cost | Reliability |
|---|---|---|
| Sample check | Low | Low to medium |
| Aggregates (counts, totals) | Low | Medium to high |
| Checksum | Medium | High |
| Full comparison | High | Very high (areas where integrity is essential, such as finance) |
The migration result confirmation
Do not finish with verbal confirmation. Leave it in a document.
## 이관 대상
CUSTOMER 원천 1,200건 → 이관 1,187건, 제외 13건
ORDERS 원천 45,320건 → 이관 45,320건, 제외 0건
## 제외 사유
코드 미매핑 9건 (목록: excluded_code.csv)
필수값 누락 4건 (목록: excluded_null.csv)
## 검증 결과
건수 일치 OK
금액 합계 일치 OK (원천 8,213,400,000 / 대상 8,213,400,000)
키 체크섬 일치 OK
## 재이관 대상
13건 - 업무 정리 후 D+3 재이관 예정, 담당 ○○○
## 확인
수행사 ___ 발주사 ___
With this document, when an inquiry like "the data is missing" comes after launch, you can answer "those 13 are excluded from migration and are scheduled for re-migration on D+3." Without it, you have to investigate from scratch.
The principle when migration fails
One last thing. This is the principle for when a problem occurs during migration.
Recovery that touches data is the last resort.
The order is this.
- Turn off the new path with a feature flag (if possible)
- Return traffic to the old system
- If that still does not work, schema rollback
- Last, data recovery (restore from backup, point-in-time recovery)
Step 4 takes the longest and is the riskiest. So preparing the first three steps in advance is the skill in a migration plan.
What it looks like in the field
The criterion for judging whether the migration is finished is also often off. "It ran without errors" does not mean it is finished. When you reconcile, it is common for counts to differ or amount totals not to match, and the real criterion is whether the difference is an explainable difference.
If deduplication reduced the count by 20, the counts of course do not match, and that is normal. Conversely, if 3 rows are missing for no reason, that is an incident. So a reconciliation sheet needs not a cell that forces the difference to 0 but a cell that records the difference and its reason.
Always keep the list of excluded rows too. Without this file, there is no way to answer the post-launch inquiry "our customer did not come over," and recounting at that point is impossible.