The Real Reason It Worked in Dev and Not in Production

One-line summary

If you rebuild for each environment, each environment gets something different. Promoting the artifact built once, as it is, is the only fundamental solution to this problem.

Why this is needed

A great many places have a pipeline that looks like this.

dev 브랜치  → 빌드 → dev 배포
stage 브랜치 → 빌드 → stage 배포
main 브랜치  → 빌드 → prod 배포

It builds three times. You might think the result will be the same because the source is the same, but it is not. Between builds, things like these change.

• The base image tag moved (python:3.12 is not what it was yesterday)
• npm install pulled in a new patch of a transitive dependency
• The package versions in the apt repository went up
• The toolchain versions on the build machines differ

So a test that passed in dev means nothing for the prod image. This is because what you tested and what you deployed are different things.

Build once, promote many times

소스 커밋 → 빌드 1회 → 아티팩트(불변) ─┬→ dev 배포 → 테스트
                                      ├→ stage 배포 → 검증
                                      └→ prod 배포

The key is that the artifact is immutable. The very bytes you verified in dev go to prod. Then the variable "it differs because of the environment" disappears, and the only differences left are configuration and data — those we can control.

Promotion is not a rebuild but a change of reference.

# gitops/prod/kustomization.yaml
images:
  - name: registry/backend
    newTag: v0820-1500      # ← 이 한 줄을 고치는 커밋이 배포다

LabHub itself runs this way. It bakes the image once, raises the tag in the dev overlay and verifies it, and once it passes, raises the same tag in the prod overlay. The deployment record is the git log itself.

Keep configuration outside the artifact

To use the same artifact in several environments, you must pull out to the outside whatever differs per environment.

Item	Inside the image	Outside the image
Application code	✅
Runtime and libraries	✅
DB address, external API URL		✅ environment variables/ConfigMap
Credentials		✅ Secret
Log level, feature flags		✅ configuration

The moment you bake application-prod.yml into the image, that image becomes prod-only and the promotion model breaks.

What to identify it by

An artifact needs a name you can trace back.

• Commit SHA — the most precise. backend:a1b2c3d. You know at a glance which source it came from.
• Semantic version — easy for people to read. Attach it to releases.
• Digest — @sha256:.... A tag can be moved, but a digest never changes. Use this where you really must be sure (prod).

latest must not be used in the promotion model. You cannot tell which point in time's latest it is, so you cannot pin down the target to roll back to.

Put gates between promotions

빌드 → [단위 테스트] → dev → [통합 테스트] → stage → [수동 승인] → prod

The brackets in front of each arrow are gates. A gate must say pass/block through its exit code. If it prints "failed" in the log and returns exit 0, the pipeline simply passes by — this is a bug that is actually common.

Rollback is the reverse of promotion

If promotion is a commit that raises the tag, rollback is a commit that returns to the previous tag. It needs neither a rebuild nor an emergency patch. Because the previous artifact is still in the registry.

For this to be possible, you must not delete artifacts. If you design the retention policy to keep only "the most recent N", you cannot go back to versions older than that.

Where promotion actually goes wrong

The principle "build once and only promote" is simple, but as you try to keep it, it leaks in three places.

Tags move. You promoted myapp:v1.2.3, but if someone pushes again with the same tag, what you verified yesterday and what is deployed today are different things. Deploy by digest and use the tag only as a label for humans. If the registry has a tag immutability setting, turn it on too.

docker buildx imagetools inspect myapp:v1.2.3 --format '{{.Manifest.Digest}}'

Per-environment builds quietly come back to life. If even one thing like --build-arg ENV=prod gets in, at that moment the image tested in the development environment and the production image become different. The difference must exist only in configuration at run time. If you see an environment name in the build arguments, that is the signal.

Configuration gets into the image. If you put a default configuration file into the image for convenience, that value will someday be used in production. It is safer to inject configuration from outside and make it fail to start if it is missing. Running quietly on defaults is the worst.

Put gates between promotions, but do not measure the same thing twice. Unit tests once at build, integration tests once after the promotion to the development environment, load tests once after the promotion to staging. Pipelines that run the unit tests again in front of the production promotion are common, but that only spends time and tells you nothing new.

Undoing must be a re-promotion of the old digest. If you rebuild in order to roll back, what you rolled back to becomes different from what it was before because of dependencies that changed in the meantime. So you do not delete old artifacts and keep at least a few generations.

Record what is where. If which digest is in each environment right now is not visible on one screen, you spend time during an outage finding out "what is running in production".

What it looks like in the field

• "It worked in staging" → it was rebuilt and a different thing went out.
• When you try to roll back, the previous image is not there → the retention policy is too short.
• Only prod has a different configuration file in the image → promotion is impossible in the first place.

What to look for in the next check

In the lab that follows, you confirm by hand the claims of this text. You name a release from a commit, create the artifact only once in a repository that stands for the development environment, move it to staging and production by pointing at it with the digest rather than the tag, catch the moment a tag came to point to different content, and leave in one file which digest is in which environment. In the quiz after the lab, you judge the difference between build and promotion, the immutable digest, and the effect of external configuration and retention policy on whether rollback is possible.