The Field I Edited by Hand Disappeared on Upgrade

Goal

You create for yourself, and check, what happens after helm upgrade to values and labels you fixed by hand in the cluster, and compare the differences among the three options that carry over the previous user values, using the release records.

Why it matters

When upgrading, Helm 3 builds a patch from three things: the old manifest, the new manifest, and the live objects in the cluster. This one rule explains two phenomena that repeat in the field — the replica count you raised with kubectl scale during an outage reverts in the next deployment, and a label you attached in a hurry stays. This is because for fields the chart declares, the declaration wins, and fields the chart does not know about are left alone. There is one more rule on the values side. helm upgrade by default does not carry over the previous release's user values. So if a deployment script drops one --set, that value quietly returns to the chart default. --reuse-values solves this problem but leaves the values only inside the release, making them invisible in the repository. Which one to use should be decided not by taste but by "can the deployment be reproduced?", and for that you need to see once what the three options actually do.

Steps

Create a /root/hc-upgrade/ledger chart (name ledger, version 0.1.0). values.yaml has three values: replicas: 1, image: "registry.local/ledger:1.0.0", and extraLabel: "". templates/deployment.yaml is a Deployment named <릴리스이름>-ledger (release name), with app: ledger in metadata.labels, adding tier: <그 값> (the value of extraLabel) only when extraLabel is not empty. After installing it with the release name books, save helm get manifest books to /root/hc-upgrade/out/rev1-manifest.yaml.
With kubectl, raise the replica count of the books-ledger Deployment to 5 and attach the label owner=ops. Save that state to /root/hc-upgrade/out/before-upgrade.json as JSON with the three keys {"replicas": …, "owner": …, "tier": …} (null for a label that does not exist).
Without changing the chart or the values at all, run helm upgrade books /root/hc-upgrade/ledger. Then save JSON with the same three keys to /root/hc-upgrade/out/after-upgrade.json, and check which of the two things you fixed by hand survived and which reverted.
Upgrade with --set extraLabel=gold and save the state with the tier label attached to /root/hc-upgrade/out/label-added.json. Then upgrade again giving extraLabel as an empty string and save the state where the tier label has disappeared to /root/hc-upgrade/out/label-removed.json. Both files are JSON with the same three keys. Also look at what happens to the owner label at this point.
Upgrade with --set replicas=4 --set extraLabel=silver and save helm get values books -o json to /root/hc-upgrade/out/values-1.json. Then upgrade again giving only --set replicas=6 and save the result of the same command to /root/hc-upgrade/out/values-2.json. What happens to extraLabel is the answer.
Upgrade carrying over the previous release's user values and adding only --set extraLabel=bronze. Save the result of helm get values books -o json to /root/hc-upgrade/out/values-reuse.json. replicas must keep the 6 from the earlier step and extraLabel must be bronze.
First, upgrade with --set replicas=4 --set extraLabel=silver to set a baseline. From there, upgrade with --set replicas=7 plus the option that goes back to the chart defaults and then layers last time's user values on again, and save the result to /root/hc-upgrade/out/values-rtr.json. Then upgrade with --set replicas=9 plus the option that discards last time's values, and save the result to /root/hc-upgrade/out/values-reset.json.
Run a server-side preview with --set replicas=3 and save the output to /root/hc-upgrade/out/dryrun.yaml (the release must not change). Then do an actual upgrade with the same value plus --force, and save the state afterward to /root/hc-upgrade/out/after-force.json as JSON with the same three keys as in step 2 — check what happens to the owner label you attached by hand. Save helm history books -o json to /root/hc-upgrade/out/history.json, and in /root/hc-upgrade/out/merge-report.json write the five booleans manual_scale_kept, manual_label_kept, chart_removed_label_deleted, default_upgrade_reuses_values, and manual_label_survives_force and the number final_replicas. You read the values from the files you saved in the earlier steps.

Notes

helm get values <릴리스> -o json (release in the placeholder) shows only the values the user gave, and --all shows even the chart defaults
Pick out only the fields you need and compare them with kubectl get deploy <이름> -o json | jq '{...}' (name in the placeholder)
helm upgrade --dry-run=server sends it all the way to the API server for validation only and does not create a release
Common mistake: the next deployment reverts a value you raised with kubectl scale during an outage
Common mistake: one --set dropping out of a deployment script, so only that value returns to its default
Official documentation: https://helm.sh/docs/helm/helm_upgrade/ · https://helm.sh/docs/faq/changes_since_helm2/

Deploy the thing you will try to revert first

Create a /root/hc-upgrade/ledger chart (name ledger, version 0.1.0). values.yaml has three values: replicas: 1, image: "registry.local/ledger:1.0.0", and extraLabel: "". templates/deployment.yaml is a Deployment named <릴리스이름>-ledger (release name), with app: ledger in metadata.labels, adding tier: <그 값> (the value of extraLabel) only when extraLabel is not empty. After installing it with the release name books, save helm get manifest books to /root/hc-upgrade/out/rev1-manifest.yaml.

On a kwok cluster Pods do not really run, but the Deployment object is created normally — this lab looks only at the field values of the object. Wrap the part that conditionally attaches the label in an {{- if .Values.extraLabel }} block, and use the leading - so no blank line is left when the condition is false.

Change it by hand in the cluster

With kubectl, raise the replica count of the books-ledger Deployment to 5 and attach the label owner=ops. Save that state to /root/hc-upgrade/out/before-upgrade.json as JSON with the three keys {"replicas": …, "owner": …, "tier": …} (null for a label that does not exist).

It is two commands: kubectl scale deploy/books-ledger --replicas=5 and kubectl label deploy/books-ledger owner=ops --overwrite. It is something often done during incident response, and the trouble arises in the next deployment. To pull it out as JSON, use kubectl get deploy books-ledger -o json | jq '{...}'.

Upgrade without changing the chart at all

Without changing the chart or the values at all, run helm upgrade books /root/hc-upgrade/ledger. Then save JSON with the same three keys to /root/hc-upgrade/out/after-upgrade.json, and check which of the two things you fixed by hand survived and which reverted.

Helm 3 builds a patch from three things: the old manifest, the new manifest, and the live objects in the cluster. For fields where the chart declares a value, even if the live object differs it is brought in line with the declaration. Fields the chart does not know about at all are left alone. You should be able to explain the result with these two rules.

A field removed from the chart is also deleted in the cluster

Upgrade with --set extraLabel=gold and save the state with the tier label attached to /root/hc-upgrade/out/label-added.json. Then upgrade again giving extraLabel as an empty string and save the state where the tier label has disappeared to /root/hc-upgrade/out/label-removed.json. Both files are JSON with the same three keys. Also look at what happens to the owner label at this point.

When the condition becomes false, that label is not in the new manifest. Comparing with the old manifest, Helm builds a patch that deletes the missing field. owner, on the other hand, is a field that was in neither manifest, so it is not a patch target. This is where the boundary between what the chart manages and what a person attached is drawn. Note: if you upgrade giving no values at all, Helm uses the previous release's user values as they are — so simply dropping --set does not make gold disappear. Try it yourself and check the difference.

An upgrade discards the previous user values

Upgrade with --set replicas=4 --set extraLabel=silver and save helm get values books -o json to /root/hc-upgrade/out/values-1.json. Then upgrade again giving only --set replicas=6 and save the result of the same command to /root/hc-upgrade/out/values-2.json. What happens to extraLabel is the answer.

helm get values shows only the values the user gave (to see even the chart defaults, use --all). By default behavior, an upgrade does not carry over the previous release's user values and uses only what you gave this time. This is why deployment scripts have to rewrite all the --set options every time.

Carry over last time's values and change just one value

Upgrade carrying over the previous release's user values and adding only --set extraLabel=bronze. Save the result of helm get values books -o json to /root/hc-upgrade/out/values-reuse.json. replicas must keep the 6 from the earlier step and extraLabel must be bronze.

There is a separate carry-over option (the one containing reuse in helm upgrade --help). It looks convenient but has a pitfall — the carried-over values are visible neither on the command line nor in the repository and exist only inside the release. So the older the release, the less anyone knows "what values it is running with now."

Put the three carry-over options side by side

First, upgrade with --set replicas=4 --set extraLabel=silver to set a baseline. From there, upgrade with --set replicas=7 plus the option that goes back to the chart defaults and then layers last time's user values on again, and save the result to /root/hc-upgrade/out/values-rtr.json. Then upgrade with --set replicas=9 plus the option that discards last time's values, and save the result to /root/hc-upgrade/out/values-reset.json.

The three options have similar names and are confusing — read the three side by side in helm upgrade --help. One carries over last time's values as they are, one goes back to the defaults and then layers last time's user values on again, and one discards last time's values altogether. You can tell them apart by the difference in the number of keys in the two saved files.

Preview it, apply it by replacement, and sort out the rules

Run a server-side preview with --set replicas=3 and save the output to /root/hc-upgrade/out/dryrun.yaml (the release must not change). Then do an actual upgrade with the same value plus --force, and save the state afterward to /root/hc-upgrade/out/after-force.json as JSON with the same three keys as in step 2 — check what happens to the owner label you attached by hand. Save helm history books -o json to /root/hc-upgrade/out/history.json, and in /root/hc-upgrade/out/merge-report.json write the five booleans manual_scale_kept, manual_label_kept, chart_removed_label_deleted, default_upgrade_reuses_values, and manual_label_survives_force and the number final_replicas. You read the values from the files you saved in the earlier steps.

--dry-run=server sends it all the way to the API server for validation only and does not create a release. --force applies by replacement instead of a patch — it is used when there is an immutable field that a patch cannot change, but because it is a replacement, the object is swapped wholesale for the new manifest. Think about what will then happen to what survived in step 3, and check. The five booleans come out directly if you compare steps 2 to 5 with the JSON you just saved.