Writing a Cutover Checklist and Rollback Script
Goal
You create the plan, checklist, and backup/release/smoke/rollback scripts needed for a cutover (go-live), actually run them, and write the result report.
Why it matters
Cutover is the most compressed risk window in an SI project. At dawn, tired, several teams must move in a set order, and the time to go back is short. What supports judgment then are the rollback criteria decided in advance as numbers, and automated verification scripts that give the same answer even when your hands shake. A smoke test in which a person clicks through screens becomes less accurate the more tired that person is. So the essence of cutover preparation is not 'writing documents' but 'automating judgment in advance'.
Steps
- Create the
/root/deploydirectory and write/root/deploy/plan.md. It must have four h2 headings,## 이행 일시,## 대상,## 롤백 기준and## 담당자, and the body must contain the phrase롤백 판단 시한and a time. - Create
/root/deploy/checklist.csv. The first line isseq,phase,task,owner,expected,rollback. There are 8 or more data rows, and thephasevalues must include all of사전,이행and검증(pre-work, cutover, verification).seqis an integer increasing by 1 from 1, andexpected(the expected result) must not be empty. - Copy
/opt/lab/fixtures/si-process/app/config.propertiesto/root/deploy/app/config.properties. Then create/root/deploy/backup.sh. When run, it leaves a backup in the form/root/deploy/backup/config.properties.YYYYMMDDHHMM. After writing it, actually run it once. - Create
/root/deploy/release.sh. When run, it changes, in/root/deploy/app/config.properties, theapp.versionvalue to2.0.0, and appends, to/root/deploy/deploy.log, a line containing the stringRELEASE 2.0.0. After writing it, actually run it once. - Create
/root/deploy/smoke.sh. It takes one argument (the configuration file path) and exits with code 0 if theapp.versionvalue is not empty and theapp.db.urlvalue starts withjdbc:, and with a non-zero value otherwise. - Create
/root/deploy/rollback.sh. It takes two arguments (백업파일 대상파일, meaning the backup file and the target file) and restores the backup file to the target file. If the backup file does not exist, it must not touch the target file and must end with a non-zero exit code. - Create
/root/deploy/rollback-criteria.csv. The first line ismetric,threshold,window,action. There are 3 or more data rows,actionis one of롤백,관찰and유지(roll back, observe, keep), and at least one row must be롤백.thresholdmust contain a number andwindow(the observation window) must not be empty. - Write
/root/deploy/result.md. It must have four h2 headings,## 수행 결과,## 이슈,## 백업 위치and## 확인 사항, and the body must contain the actual backup file name created in step 3 and the deployed version2.0.0.
Notes
- You can make a 12-digit timestamp with
date +%Y%m%d%H%M. - To read a configuration value:
grep '^app.version=' 파일 | cut -d= -f2 - Common mistake 1: a rollback script that runs
cpwithout checking that the backup file exists. - Common mistake 2: writing the scripts but not running them, so the artifacts (backup file, deploy.log) don't exist.
- Common mistake 3: in
checklist.csv, leaving theexpectedcolumn empty. Without an expected result, it is not a checklist.
Write the cutover plan
Create the /root/deploy directory and write /root/deploy/plan.md.
It must have four h2 headings, ## 이행 일시, ## 대상, ## 롤백 기준 and ## 담당자, and
the body must contain the phrase 롤백 판단 시한 and a time.
The minimum requirements for a cutover plan are 'when / what / who / when to go back'. Write the rollback criteria as numbers, not feelings, so judgment doesn't waver at dawn.
Step-by-step checklist
Create /root/deploy/checklist.csv. The first line is
seq,phase,task,owner,expected,rollback.
There are 8 or more data rows, and the phase values must include all of 사전, 이행 and 검증 (pre-work, cutover, verification).
seq is an integer increasing by 1 from 1, and expected (the expected result) must not be empty.
Without an 'expected result' on each item, you cannot judge success. All three phases, pre-work, cutover and verification, must be present, and the sequence numbers must match the actual order of execution.
Write and run the configuration backup script
Copy /opt/lab/fixtures/si-process/app/config.properties to
/root/deploy/app/config.properties.
Then create /root/deploy/backup.sh. When run, it leaves a backup in the form
/root/deploy/backup/config.properties.YYYYMMDDHHMM.
After writing it, actually run it once.
If you put a timestamp in the backup file name, running it several times won't overwrite anything. Use the format specifiers of the date command. Also get into the habit of checking that the backup has the same content as the original.
Write and run the release script
Create /root/deploy/release.sh. When run, it changes, in
/root/deploy/app/config.properties, the app.version value to 2.0.0, and
appends, to /root/deploy/deploy.log, a line containing the string RELEASE 2.0.0.
After writing it, actually run it once.
If you use sed -i to change a configuration value, the original is lost even if it fails. That is why the backup was made first in the earlier step. Keep the deployment history by appending.
Smoke test script
Create /root/deploy/smoke.sh. It takes one argument (the configuration file path) and
exits with code 0 if the app.version value is not empty and the app.db.url value starts with jdbc:,
and with a non-zero value otherwise.
A smoke test is reusable only if it takes the target to check as an argument. It must exit with 0 when healthy and a non-zero value when not, so that it can be used in automation.
Rollback script
Create /root/deploy/rollback.sh. It takes two arguments (백업파일 대상파일, meaning the backup file and the target file) and
restores the backup file to the target file. If the backup file does not exist, it must not touch the target file and
must end with a non-zero exit code.
A rollback script is safe only if it works from arguments. If the target backup file does not exist, it must do nothing and fail immediately — accidents where a file is overwritten with an empty one at dawn really happen.
Rollback decision criteria table
Create /root/deploy/rollback-criteria.csv. The first line is
metric,threshold,window,action. There are 3 or more data rows,
action is one of 롤백, 관찰 and 유지 (roll back, observe, keep), and at least one row must be 롤백.
threshold must contain a number and window (the observation window) must not be empty.
Metric, threshold, observation window and action form one set. Don't write 'if there are many errors'; write something like 'if the error rate stays above 5% for 10 minutes'.
Cutover result report
Write /root/deploy/result.md. It must have four h2 headings, ## 수행 결과, ## 이슈,
## 백업 위치 and ## 확인 사항, and the body must contain the actual backup file name created in step 3 and the deployed version 2.0.0.
The value of a result report shows up during the stabilization period. Write down the backup file name and the deployed version exactly. It is the only document that can later answer 'what did we change at go-live?'