You removed the old version and the objects vanished - the storage version migration procedure

Goal

Create a two-version CRD, raise the storage version, rewrite all objects, clean up status.storedVersions, and take down the old version, walking the migration procedure from start to finish. You will also see for yourself where the API server stops you along the way.

Why it matters

A CRD is an API contract, not code. To change even one field, you release a new version, give the people using the old version time to move, and only then can you delete it. The thing people most often forget at this point is rewriting the objects that are already stored. Changing the storage version changes only "the representation to be stored from now on," so the bytes already in etcd are still in the old representation. If you delete the old schema in that state, there is no way to read the stored objects. status.storedVersions is the record that exists to prevent exactly that incident, and the API server really does reject an attempt to remove a version that remains in this list. This lab has you run into that safeguard once, and then pass it in the correct order.

Steps

In /root/crd-version/tunnel-crd.yaml, write the CRD tunnels.net.labhub.io — group net.labhub.io, kind Tunnel, plural tunnels, and two versions. v1alpha1 has served: true and storage: true, gets deprecated: true and a deprecationWarning, and has only spec.endpoint (string) and spec.port (integer) in its schema. v1beta1 has served: true and storage: false, and in addition to those two fields has spec.mtu (integer, default: 1400). After applying, save the state of the two versions to /root/crd-version/versions-initial.txt as two lines in <버전> served=<참거짓> storage=<참거짓> form (the version, then true or false for served and storage).
Create the namespace crd-version and apply /root/crd-version/tunnel-east.yaml (name t-east, endpoint 10.30.0.11, port 4789) and /root/crd-version/tunnel-west.yaml (name t-west, endpoint 10.30.0.12, port 4789) as v1alpha1. Capture the warning that comes back when you apply, including standard error, and save it to /root/crd-version/deprecation-warning.txt, and save the status.storedVersions at that point to /root/crd-version/stored-initial.txt on one line.
Read the same object through the new version's window — save the output of kubectl -n crd-version get tunnels.v1beta1.net.labhub.io t-east -o yaml to /root/crd-version/as-v1beta1.yaml. And write two lines in /root/crd-version/conversion-note.txt — apiVersion=<읽은 apiVersion> (the apiVersion you read) and mtu=<spec.mtu 값, 없으면 none> (the spec.mtu value, or none if absent).
In /root/crd-version/tunnel-crd-beta-storage.yaml, copy the step 1 CRD but change storage of v1alpha1 to false and storage of v1beta1 to true, and apply it. After applying, save status.storedVersions to /root/crd-version/stored-after-flip.txt on one line.
Actually move the storage version — read every Tunnel in crd-version as v1beta1 and rewrite it as it is (kubectl get … -o json | kubectl replace -f -). After rewriting both objects, save the output of kubectl -n crd-version get tunnels.v1beta1.net.labhub.io -o yaml to /root/crd-version/after-rewrite.yaml.
Try to remove the old version from spec.versions — kubectl patch crd tunnels.net.labhub.io --type=json -p '[{"op":"remove","path":"/spec/versions/0"}]'. Collect the command's output and exit code in /root/crd-version/remove-blocked.txt — the first line is remove-rc=<종료 코드> (the exit code), and below it you paste the sentence the server produced exactly as it was.
Now that all objects are stored as v1beta1, clean up the record — use kubectl patch crd tunnels.net.labhub.io --subresource=status --type=merge to make status.storedVersions just ["v1beta1"]. Save the value after the cleanup to /root/crd-version/stored-pruned.txt on one line.
In /root/crd-version/tunnel-crd-retire.yaml, copy the step 4 CRD but change served of v1alpha1 to false, and apply it. Then run kubectl -n crd-version get tunnels.v1alpha1.net.labhub.io t-east and collect the result in /root/crd-version/retired.txt — the first line is get-rc=<종료 코드> (the exit code), and below it you paste the server sentence. Finally, create /root/crd-version/version-check.sh — it must print this CRD's served, storage, and stored, one line each, as <이름>=<값> (name=value), and if storedVersions has a value that is not the storage version, print PENDING <버전> (with the version) and exit with a nonzero code, and if not, print MIGRATED and exit with 0. Save that output to /root/crd-version/version-check.txt.

Notes

served is whether to accept requests at that version, and storage is whether to store at that version. Exactly one has storage.
To query a specific version, use the form kubectl get <복수형>.<버전>.<그룹> <이름> (plural name, version, group, and object name).
status.storedVersions is in the status window, so you fix it with kubectl patch … --subresource=status.
This environment has no way to run a webhook server, so conversion.strategy is covered only up to None.
Common mistake: changing only the storage version and not rewriting the existing objects.
Common mistake: hand-deleting storedVersions first and defeating the safeguard.
Reference: https://kubernetes.io/docs/tasks/extend-kubernetes/custom-resources/custom-resource-definition-versioning/
Reference: https://kubernetes.io/docs/tasks/extend-kubernetes/custom-resources/custom-resource-definitions/

Serve two versions of one type together

In /root/crd-version/tunnel-crd.yaml, write the CRD tunnels.net.labhub.io — group net.labhub.io, kind Tunnel, plural tunnels, and two versions. v1alpha1 has served: true and storage: true, gets deprecated: true and a deprecationWarning, and has only spec.endpoint (string) and spec.port (integer) in its schema. v1beta1 has served: true and storage: false, and in addition to those two fields has spec.mtu (integer, default: 1400). After applying, save the state of the two versions to /root/crd-version/versions-initial.txt as two lines in <버전> served=<참거짓> storage=<참거짓> form (the version, then true or false for served and storage).

The versions inside one CRD are different windows onto the same object, so storage happens in exactly one representation. That is why exactly one version must have storage true, and the rest keep only served on and remain as windows for reading and writing. deprecationWarning is the sentence returned to the client when it makes a request with that version.

Creating with a deprecated version returns a warning

Create the namespace crd-version and apply /root/crd-version/tunnel-east.yaml (name t-east, endpoint 10.30.0.11, port 4789) and /root/crd-version/tunnel-west.yaml (name t-west, endpoint 10.30.0.12, port 4789) as v1alpha1. Capture the warning that comes back when you apply, including standard error, and save it to /root/crd-version/deprecation-warning.txt, and save the status.storedVersions at that point to /root/crd-version/stored-initial.txt on one line.

The deprecation warning comes out on standard error, not standard output. storedVersions is a record in status, not spec, and it means "the versions that have ever actually been used to store objects of this CRD." If you have stored with only one version so far, try to predict what it will contain.

What changes when you read with the new version

Read the same object through the new version's window — save the output of kubectl -n crd-version get tunnels.v1beta1.net.labhub.io t-east -o yaml to /root/crd-version/as-v1beta1.yaml. And write two lines in /root/crd-version/conversion-note.txt — apiVersion=<읽은 apiVersion> (the apiVersion you read) and mtu=<spec.mtu 값, 없으면 none> (the spec.mtu value, or none if absent).

If you do not write conversion.strategy, it is None, and None leaves the stored bytes as they are and only changes the apiVersion field in what it shows. The v1beta1 schema has a field with a default, and defaults are filled in "at storage time." Think about which version this object is still stored in.

Raise the storage version

In /root/crd-version/tunnel-crd-beta-storage.yaml, copy the step 1 CRD but change storage of v1alpha1 to false and storage of v1beta1 to true, and apply it. After applying, save status.storedVersions to /root/crd-version/stored-after-flip.txt on one line.

Changing this one line changes only "the representation to use for storing from now on." The bytes of objects already in etcd stay as they are. That is why storedVersions does not shrink but grows instead — read again what that list means.

Rewrite every object once

Actually move the storage version — read every Tunnel in crd-version as v1beta1 and rewrite it as it is (kubectl get … -o json | kubectl replace -f -). After rewriting both objects, save the output of kubectl -n crd-version get tunnels.v1beta1.net.labhub.io -o yaml to /root/crd-version/after-rewrite.yaml.

The evidence that the rewrite really happened is the default that exists only in the new version. This is because the API server fills in that field the moment it writes in the storage version. In real production with thousands of objects, you do not run this job all at once but in batches — because every object causes an etcd write.

The API server blocks you: you cannot delete it yet

Try to remove the old version from spec.versions — kubectl patch crd tunnels.net.labhub.io --type=json -p '[{"op":"remove","path":"/spec/versions/0"}]'. Collect the command's output and exit code in /root/crd-version/remove-blocked.txt — the first line is remove-rc=<종료 코드> (the exit code), and below it you paste the sentence the server produced exactly as it was.

This rejection is not nagging but a safeguard. If you delete the schema of a version that remains in the list, there is no way to read the objects stored in that representation. Read exactly which field the error sentence points to — the place to fix is written right there.

Only after cleaning up the record does the door open

Now that all objects are stored as v1beta1, clean up the record — use kubectl patch crd tunnels.net.labhub.io --subresource=status --type=merge to make status.storedVersions just ["v1beta1"]. Save the value after the cleanup to /root/crd-version/stored-pruned.txt on one line.

This cleanup is the act of a person declaring "I have finished the rewrite." The API server does not count the rewrites for you — so if you skip step 5 and start here, it becomes an incident as it is. Status is a separate window, so an ordinary patch does not reach it.

Close the old window and prove the migration is complete

In /root/crd-version/tunnel-crd-retire.yaml, copy the step 4 CRD but change served of v1alpha1 to false, and apply it. Then run kubectl -n crd-version get tunnels.v1alpha1.net.labhub.io t-east and collect the result in /root/crd-version/retired.txt — the first line is get-rc=<종료 코드> (the exit code), and below it you paste the server sentence. Finally, create /root/crd-version/version-check.sh — it must print this CRD's served, storage, and stored, one line each, as <이름>=<값> (name=value), and if storedVersions has a value that is not the storage version, print PENDING <버전> (with the version) and exit with a nonzero code, and if not, print MIGRATED and exit with 0. Save that output to /root/crd-version/version-check.txt.

If you turn off served, that version's endpoint itself disappears, so a request sent with the old apiVersion gets the answer that the resource cannot be found. The stored objects are perfectly fine; they just cannot be seen through that window. Make the check script write only to standard output and not touch files directly — that way you get the same answer no matter how many times you run it.