1.5.0 showed Accepted, but the warehouse had no orders

Goal

You build scripts that deploy with releases/VERSION and a current symlink, go through a health check and a smoke test, and on failure roll back to the previous version and record it. In the retention cleanup, protect current and the previous version.

Why it matters

A health check asks whether the process is alive, and a smoke test asks whether one business transaction goes all the way through. The customer's last deployment asked only the first question and ended, and lost orders all night. If you split versions into directories and switch with a single symlink, the rollback also becomes a single identical action. But if current disappears at the moment of the switch, an old process turns on the green light instead, or a cleanup script deletes the version you would go back to, an accident happens even with the structure in place. The grader makes, each time with a different version number, builds that are normal, break only in the smoke test, die as soon as they start, and start late, runs your scripts, and measures again not the log wording but the version that is actually responding and the symlink.

The expected time is 60 minutes. Extend with +time before the default session ends (up to 180 minutes). When the session ends, the files in /root disappear, so keep the scripts separately.

Steps

1. Unpack the build orderapp-1.4.0.tar.gz into /root/site/releases/1.4.0, and make the /root/site/current symlink point to that directory.
2. Create /root/release/switch.sh ROOT VERSION. Reject a version that does not exist, and switch in one go with no moment when current disappears.
3. Create /root/release/deploy.sh ROOT BUILD PORT. Make a normal build pass through unpacking, switching, replacing the old process, the health check (with a version check), a smoke order, and recording in deploy.log.
4. If deploy.sh fails the smoke test, make it roll current back to the previous version, bring that version up again, record rolled_back with reason smoke, and end with exit code 2.
5. For deploy.sh's health check, make a version that dies as soon as it starts roll back with health, and make a normal version that starts 2.5 seconds late be waited for and deployed. The wait upper limit is 10 seconds.
6. Create /root/release/prune.sh ROOT KEEP. Keep the KEEP most recent by mtime, but take current and the previous version out of the list to delete.
7. Check that even when you do four deployments (the last with a smoke failure) and a retention-1 cleanup in a row, the record, the remaining versions, and the responding version are right.
8. Deploy the customer's 1.5.0 with deploy.sh to /root/site (port 8480), look at the result, and record the incident in /root/release/incident.json.

Notes

• Materials: the execution contract /opt/lab/p1a-release/CONTRACT.md, the customer memo /opt/lab/p1a-release/README.md, and the builds /opt/lab/p1a-release/builds/
• Looking inside a build: tar -tzf build-file, tar -xzOf build-file VERSION
• Checking the symlink: readlink /root/site/current, ls -la /root/site
• Direct test: bash /root/release/deploy.sh /tmp/try /opt/lab/p1a-release/builds/orderapp-1.4.0.tar.gz 18480; cat /tmp/try/deploy.log
• Common mistakes: leaving out -n in ln -sf, not redirecting output to a file when starting the app (the script never ends), not checking the version in the health check, and cleaning up only by newest order.
• When you are done, pick out and stop an app you started for testing by the shared/app.pid of that deployment root.

Unpack the first version by hand and set current

Unpack build 1.4.0 into /root/site/releases/1.4.0 and make the /root/site/current symlink point there.

You decide where to unpack with tar's -C. current must be a link made with ln -s, not a directory copy. A relative-path link (releases/VERSION) does not break even if you move the whole deployment root.

A switch script that changes current in one go

Create /root/release/switch.sh ROOT VERSION. It must reject a version that does not exist with a nonzero code, and current must change with no moment when it disappears.

rename(2) atomically replaces an existing name. Create a temporary link in the same directory and move it over current. mv has an option that keeps it from putting it inside the destination even if the destination points to a directory. If you use ln, do not leave out -n.

Unpack, switch, start, and even the smoke test

Create /root/release/deploy.sh ROOT BUILD PORT, deploy a normal build, and leave one deployed line in deploy.log.

You read the version from the VERSION inside the tar. You take the old process down with shared/app.pid, and you start the new app in the background with nohup and output redirection. You check whether the version in the health check response is the new version, and whether the smoke order (SMOKE- prefix) is read back with GET. If you build the record with jq -cn, you avoid quoting mistakes.

If the smoke test breaks, roll back to the previous version

Make /root/release/deploy.sh, on a smoke failure, switch to the previous version and restart it, record rolled_back (reason smoke), and end with exit code 2.

You have to remember the version current pointed to before the switch as previous so that you know where to go back. Do the rollback with switch.sh too. Keep the failed releases/VERSION for investigation.

A dead version quickly, a late version by waiting

Make the health check of /root/release/deploy.sh wait up to 10 seconds, but roll back immediately with health if the process dies.

If you give up after one failure, you roll back a normal version that starts late. Conversely, there is no reason to wait 10 seconds for a process that is already dead. kill -0 PID sends no signal and only checks whether the process exists.

Do not delete the version you would go back to while cleaning up

Create /root/release/prune.sh ROOT KEEP. Delete everything except the KEEP most recent by mtime, but keep current and the previous version.

After a rollback, current is not the latest. The previous version is the previous of the last line in deploy.log whose result is deployed and whose version is current. ls -t lists in order of most recent modification time.

A record that is right even after four deployments and a cleanup

Make /root/release/deploy.sh and /root/release/prune.sh keep the record and the actual state in agreement even when you do four deployments (the last with a smoke failure) and a retention-1 cleanup in a row.

The previous of a rolled-back deployment is the version it went back to. If you clean up after that, the latest 1 (the failed version) plus current and the previous version must remain. Run it in a row yourself on a temporary deployment root.

Deploy the customer's 1.5.0 and leave an incident record

Deploy 1.5.0 to /root/site (port 8480) with deploy.sh, and record the result in /root/release/incident.json as failed_version, restored_version, reason, deploy_log_line, and health_passed.

deploy_log_line is the line number (from 1) of the rollback record in /root/site/deploy.log. health_passed records whether this version passed the health check — judge from app.log and the record's reason. The grader reconciles incident.json with deploy.log, current, and the original build.