When the incident hit, nobody knew what was running in production

Goal

You set up fixed environments, per-branch review environments, a job that closes environments, and a manual production deployment, and you actually run with gitlab-ci-local the deployment record left by commit SHA, a rollback chosen from that record, and a deployment freeze rule.

Why it matters

A deployment is safe only if it can be reverted. What went out where and when remains as an environment, and what went out must be identifiable by commit SHA, so that in an incident you can return to "the one from then" in a few minutes. Review environments are convenient, but without a job that closes them they pile up and become cost and exposure, and the approval of a production deployment can become either a gate or a decoration depending on the single value of allow_failure. A promise such as "we do not ship right now" during a freeze period is kept only when it is engraved into the configuration.

Steps

Make /root/glci-deploy a git repository (with .gitlab-ci-local/ in .gitignore), and in .gitlab-ci.yml put stages [deploy, cleanup], the global variable DEPLOY_LOG: /root/glci-deploy-history/deploys.log, and a job deploy-staging (deploy) that is created only on main. The environment has the name staging and the URL https://staging.example.com, and the script is echo "env=$CI_ENVIRONMENT_NAME url=$CI_ENVIRONMENT_URL". Commit on main and run.
Add a job review-app (deploy) that is created only on a branch other than main. The environment name is review/$CI_COMMIT_REF_SLUG, the URL is https://$CI_COMMIT_REF_SLUG.review.example.com, and the script is the same echo as in step 1. Commit. The grader switches to the Feature/Login-Page branch in a copy and checks that the name is review/feature-login-page and the URL is https://feature-login-page.review.example.com.
To the environment of review-app, add on_stop: stop-review and auto_stop_in: 1 day, and create a job stop-review (cleanup). It has action: stop on the same environment name, when: manual on a branch other than main, and the script echo "stopping $CI_ENVIRONMENT_NAME". Commit. On a feature branch, stop-review must be in the list as manual, and when run with --manual stop-review, stopping review/<슬러그> (where the placeholder stands for the slug) must be printed.
Add, created on main with when: manual and allow_failure: false, a job deploy-prod (deploy). The environment is production, the URL is https://www.example.com, and the script is echo "env=$CI_ENVIRONMENT_NAME". Commit. If you just run it, deploy-prod must not run, and if you give --manual deploy-prod, it must run.
Create /root/glci-deploy/scripts/deploy.sh <환경> (the placeholder stands for the environment). If ROLLBACK_TO is empty, it decides the tag as $CI_COMMIT_SHORT_SHA, appends to $DEPLOY_LOG the one line <환경> registry.example.com/app:<태그> release (environment, image with the tag, then the word release), and prints deployed .... Add bash scripts/deploy.sh staging and bash scripts/deploy.sh production to the end of the scripts of deploy-staging and deploy-prod, and commit. The grader deploys two commits in turn in a copy and checks that the record remains with a different tag for each commit.
Make deploy.sh, when it receives ROLLBACK_TO, re-ship that tag without creating a new tag. If that environment's history has no record of a shipment as registry.example.com/app:<ROLLBACK_TO>, it prints 이력에 없는 태그입니다 (which means "This tag is not in the history") and fails. The last word of the record line is rollback. The grader deploys twice in a copy and then performs a run that reverts to the first tag and a run that tries to revert to a tag that does not exist.
At the very front of deploy-prod's rules, put an entry with if: $CI_DEPLOY_FREEZE and when: never. Commit. If you look at the list with --variable CI_DEPLOY_FREEZE=1, deploy-prod must be absent, and without the variable it must be there as manual. deploy-staging must be there regardless of the freeze.

Notes

This VM has no GitLab server or runner, and gitlab-ci-local 4.75.1 interprets .gitlab-ci.yml by the same rules as GitLab and runs jobs with the shell. If you write image:, it tries to run with Docker, so do not use it. Protected variables, masking, CI_JOB_TOKEN, runner tags, and merge request pipeline creation are server features and are not reproduced here.
Run: at the repository root, gitlab-ci-local --shell-isolation --no-artifacts-to-source (a separate working directory per job, artifacts not written back to the repository), job list: gitlab-ci-local --list-csv-all, interpreted configuration: gitlab-ci-local --preview. gitlab-ci-local passes only files tracked by git to jobs, so git add a file after creating it. The grader copies the repository, commits all files, and runs it again with the same tool.
Differences in gitlab-ci-local (measured): it does not queue deployments to the same environment with resource_group. A manual job runs only when specified with --manual, and it does not block later stages. The environment screen, deployment history, protected-environment approval, and freeze period settings are server features.
The deployment record /root/glci-deploy-history/deploys.log is outside the repository. When the grader runs a copy, it writes separately with --variable DEPLOY_LOG=<임시 경로> (the placeholder stands for a temporary path).
Environments · Predefined CI/CD variables (CI_COMMIT_REF_SLUG, CI_DEPLOY_FREEZE) · Resource groups · CI/CD YAML syntax reference

Put a label on the deployment job

Make /root/glci-deploy a git repository (with .gitlab-ci-local/ in .gitignore), and in .gitlab-ci.yml put stages [deploy, cleanup], the global variable DEPLOY_LOG: /root/glci-deploy-history/deploys.log, and a job deploy-staging (deploy) that is created only on main. The environment has the name staging and the URL https://staging.example.com, and the script is echo "env=$CI_ENVIRONMENT_NAME url=$CI_ENVIRONMENT_URL". Commit on main and run.

A job with environment attached is recorded as a deployment in GitLab, and what went out and when remains on the environment screen. Inside the job, you can tell where it is going with CI_ENVIRONMENT_NAME and CI_ENVIRONMENT_URL.

A review environment made for each branch

Add a job review-app (deploy) that is created only on a branch other than main. The environment name is review/$CI_COMMIT_REF_SLUG, the URL is https://$CI_COMMIT_REF_SLUG.review.example.com, and the script is the same echo as in step 1. Commit. The grader switches to the Feature/Login-Page branch in a copy and checks that the name is review/feature-login-page and the URL is https://feature-login-page.review.example.com.

Branch names contain characters that cannot be used in host names, such as uppercase letters and slashes. CI_COMMIT_REF_SLUG is a value lowercased with disallowed characters replaced by hyphens, so it can be used in a URL or environment name as it is.

A review environment is paired with a job that closes it

To the environment of review-app, add on_stop: stop-review and auto_stop_in: 1 day, and create a job stop-review (cleanup). It has action: stop on the same environment name, when: manual on a branch other than main, and the script echo "stopping $CI_ENVIRONMENT_NAME". Commit. On a feature branch, stop-review must be in the list as manual, and when run with --manual stop-review, stopping review/<슬러그> (where the placeholder stands for the slug) must be printed.

Review environments pile up as many as there are branches. on_stop is the link "the job that closes this environment is that one", and when you delete or merge the branch, GitLab calls that job. auto_stop_in makes a forgotten environment close after time passes.

A person presses the production deployment

Add, created on main with when: manual and allow_failure: false, a job deploy-prod (deploy). The environment is production, the URL is https://www.example.com, and the script is echo "env=$CI_ENVIRONMENT_NAME". Commit. If you just run it, deploy-prod must not run, and if you give --manual deploy-prod, it must run.

A manual job with allow_failure: false leaves the pipeline 'blocked' in GitLab until someone presses it. If it is true, the pipeline ends in success even without anyone pressing it, and it cannot be an approval gate. gitlab-ci-local does not reproduce this blocking, so you check with the allowFailure column of the list.

Leave what went out, by commit SHA

Create /root/glci-deploy/scripts/deploy.sh <환경> (the placeholder stands for the environment). If ROLLBACK_TO is empty, it decides the tag as $CI_COMMIT_SHORT_SHA, appends to $DEPLOY_LOG the one line <환경> registry.example.com/app:<태그> release (environment, image with the tag, then the word release), and prints deployed .... Add bash scripts/deploy.sh staging and bash scripts/deploy.sh production to the end of the scripts of deploy-staging and deploy-prod, and commit. The grader deploys two commits in turn in a copy and checks that the record remains with a different tag for each commit.

If you deploy with a moving tag such as latest, then when an incident happens, nobody knows "what was it that went out then". A commit SHA is an identifier that ties source, build, and deployment into one line. Take the deployment record path as a variable so that it can be changed at run time.

A rollback is re-shipping a tag that is in the history

Make deploy.sh, when it receives ROLLBACK_TO, re-ship that tag without creating a new tag. If that environment's history has no record of a shipment as registry.example.com/app:<ROLLBACK_TO>, it prints 이력에 없는 태그입니다 (which means "This tag is not in the history") and fails. The last word of the record line is rollback. The grader deploys twice in a copy and then performs a run that reverts to the first tag and a run that tries to revert to a tag that does not exist.

A rollback is not "rebuild from the old source" but "re-ship an output that was already verified and went out". Blocking tags that are not in the history keeps an unverified image from going to production because of a single typo. At run time, pass it with --variable ROLLBACK_TO=<태그> (the placeholder stands for the tag).

During a deployment freeze, do not create the production deployment

At the very front of deploy-prod's rules, put an entry with if: $CI_DEPLOY_FREEZE and when: never. Commit. If you look at the list with --variable CI_DEPLOY_FREEZE=1, deploy-prod must be absent, and without the variable it must be there as manual. deploy-staging must be there regardless of the freeze.

If you set a freeze period on the project, GitLab fills in CI_DEPLOY_FREEZE during that time. Rules use only the first match, so the freeze rule must come before the main rule. It is a device that makes the pipeline refuse, instead of leaving the freeze to a person's memory.