What Is Stored and What Is Shown Are Not the Same

In one sentence

Multiple versions of a CRD are a device for showing the same bytes through different windows, and raising the storage version and actually migrating are different things. The record that connects the two is status.storedVersions.

Why this procedure was needed

Once you publish an API, it is hard to take back. Users have manifests committed to Git, and etcd holds objects stored in that representation. So when you need to change a field, what you do is not "fix it" but "release a new version."

There is one thing you must think about separately here. Serving (served) and storage (storage) are different axes.

Flag	Meaning	How many
served	Whether to accept requests at this version	Several
storage	Whether to write to etcd in this version's representation	Exactly one

Storage happens in only one representation. So if you query an object created as v1alpha1 as v1beta1, the stored bytes are taken out, converted to the requested version, and returned. What decides the conversion method is spec.conversion.strategy, and if you write nothing it is None.

None does not convert. It only changes the apiVersion field and shows the object as it is. It is a strategy usable only when field names are the same in both versions, which is why it is dangerous. Even if a field added in the new version has a default, that value is not visible. Defaults are filled in at storage time, and this object is still stored in the old representation. On screen the field looks absent, and a controller that trusts that state as it is makes a wrong judgment.

How it works

The migration procedure has four steps, and the order is everything.

1. 새 버전을 served 로 추가한다            (옛 버전도 그대로 제공)
2. storage 를 새 버전으로 옮긴다           (앞으로 저장할 표현만 바뀐다)
3. 기존 오브젝트를 전부 한 번씩 다시 쓴다   (여기서 실제 이전이 일어난다)
4. status.storedVersions 를 정리하고
   옛 버전을 served: false → 제거한다

Step 3 is the heart of this procedure. The moment you do step 2, status.storedVersions holds both the old and the new version. That list means "the representations that have actually been used to store objects of this CRD so far." Rewriting an object stores it in the new representation, but the API server does not count whether anything stored in the old representation remains. So the cleanup in step 4 is the act of a person declaring "I have finished the rewrite."

If you do not make that declaration, the API server keeps the last door locked. If you try to remove a version that remains in storedVersions from spec.versions, the request is rejected, and the error tells you exactly which field it is about. Without this safeguard, the moment you deleted the old schema, objects stored in that representation would become unreadable.

It is also good to know the two fields used when phasing out a version. If you add deprecated: true, clients get a warning when they make a request with that version, and you can set the sentence yourself with deprecationWarning. A warning does not block the request — it is a device that gives people time to move.

What you see in the field

First, the incident of skipping the rewrite. The most common form goes like this. You raise the storage version, and for a while there is no problem (because newly created objects use the new representation), and months later a cleanup task deletes the old version. At that moment the old objects vanish from the list. They are still in storage, but the schema to read them is gone. It is especially dangerous when someone has first hand-deleted storedVersions and bypassed the safeguard.

Second, a controller that is quietly wrong. In a CRD with strategy: None, you put a default on a new field and write the controller to change its behavior by reading that field. Newly created objects have the default filled in, while old objects are empty. The controller ends up treating objects of the same kind in two ways, and the difference depends only on when they were created. The rewrite removes this problem too.

Third, the practice when there are many objects. With tens of thousands of objects, you do not run the rewrite all at once. Every item is an etcd write, and every controller watching receives an event. Usually you run it in batches, during low-usage hours, counting progress as you go. Kubernetes has a separate API called StorageVersionMigration that does this for you (official guide). But the skeleton of the procedure is the same as what you are learning now, and only someone who has gone through it once by hand knows what that tool does for them. In this lab you check for yourself what changes at each step.

Limits of this lab environment

The lab Pod has no way to run a webhook server. So conversion.strategy: Webhook is not covered, and you check only within the scope of None. In other words, a migration where field names change cannot be tested in this environment. Instead, you prove with objects exactly what None does and does not do — before the rewrite, the default of the new field is not filled in, and after the rewrite it is. This difference is the evidence that the migration actually happened.

What you will do in the next lab

You create a CRD with two versions and receive a deprecation warning, then raise the storage version and rewrite all objects, proving the migration by the defaults being filled in. Then you get blocked by the API server when you try to delete the old version, and after cleaning up the record in the correct order, you close the old window. Finally, you build a check script that judges in one go whether the migration is finished.