What Actually Happens When You Delete Something

In one sentence

Kubernetes has no ledger that counts ownership relationships. A single ownerReferences line, in which a child points to its parent, is all there is, and the garbage collector acts by trusting that line. That is why writing that line wrongly causes a quiet failure.

Why this design was needed

An Operator takes one custom resource and creates several real resources. For one Site, for example, two ConfigMaps, one Service, and one Secret. Then when you delete the Site, who deletes those four?

Having the controller delete them directly looks simple, but it has a problem. If the parent is deleted while the controller is dead, the children remain forever. If you remove the controller entirely, those resources become garbage that nobody knows about. So Kubernetes left the cleanup not to the controller but to the garbage collector on the API server side, and had the information that collector reads written into the child's metadata.

metadata:
  ownerReferences:
    - apiVersion: own.labhub.io/v1
      kind: Site
      name: alpha
      uid: 8f1c...            # 진짜 열쇠는 이것
      controller: true        # 이 참조가 '관리자' 인가
      blockOwnerDeletion: true

Four fields are required — apiVersion, kind, name, and uid. And of these, the one actually used to find the parent is uid. The name is a value for humans to read, and the garbage collector confirms the parent's existence by uid.

How it works

If the uid is wrong, the child is deleted. From the collector's point of view, "no object has that uid" means the same as "the parent has already been deleted." So it cleans up the child within seconds. The typical path by which this property leads to an incident in the field goes like this — the controller cached the parent's uid, and in the meantime the parent was deleted and recreated with the same name. The new parent's uid is different. A child created with the old uid disappears as soon as it is created.

There is no ownership across namespaces. A namespaced child can have only a parent in the same namespace, and only a cluster-scoped object is the exception that can have children in any namespace. A reference that breaks the rule is not blocked at apply. Instead, when the collector later cleans up that child, it leaves a warning event called OwnerRefInvalidNamespace. Events disappear after one hour by default, so if you start investigating after that time, there is almost no way to know why the child vanished.

There are three kinds of deletion propagation.

--cascade	Parent	Children	When to use
background (default)	Disappears immediately	The collector deletes them afterward	Ordinary deletion
foreground	Disappears after all children are gone	Deleted first	When the cleanup order matters
orphan	Disappears immediately	Remain, with only the reference detached	When removing only the Operator

To express the waiting, foreground attaches a finalizer called foregroundDeletion to the parent. The parent remains with deletionTimestamp stamped, and it finally disappears only when every child with blockOwnerDeletion: true is gone. Instead of deleting the children, orphan detaches that owner reference from the children. If it did not detach it, a reference with no parent would remain and the child would disappear at the next cleanup.

A finalizer is a device that delays deletion, not one that blocks it. If there is even one finalizer, the delete request only stamps deletionTimestamp and ends. The object can still be queried and modified, but you cannot create a new one with the same name. In this interval the controller cleans up the outside world (releasing cloud resources, keeping a backup, closing connections) and, when done, removes its own finalizer from the list. The moment the list is empty, the object truly disappears.

What you see in the field

First, an object that will not get out of Terminating. The most common cause is that there is no controller to remove the finalizer — if you remove the Operator first and delete the CR afterward, this is exactly what happens. The thing people most often do then is hand-remove the finalizer, but that skips the cleanup, so the cloud resources remain and charges keep accruing. The order must be the opposite — clean up the CRs first and then remove the Operator.

Second, a namespace that stalls in Terminating. The cause is usually that some object inside it carries a finalizer. If you force-delete the namespace, only the records disappear while the objects inside are left uncleaned.

Third, prepare the diagnostic procedure in advance. When you run into a stuck object, there are three things to ask — when was deletionTimestamp stamped, which finalizers remain, and is the party that would remove that finalizer alive right now. If foregroundDeletion remains, one more question is added — which of the children with blockOwnerDeletion true have not yet disappeared. If you prepare a script that extracts these four things at once, you do not have to think from scratch during an incident.

Limits of this lab environment

The Pods in the kwok cluster are fake, so containers do not actually run. So the real work of cleaning up outside systems inside a finalizer (calling cloud APIs, releasing volumes) is only imitated. On the other hand, the garbage collector and deletion propagation themselves are handled by the real kube-controller-manager, so a child with a wrong uid disappearing, and a foreground delete leaving a finalizer and getting blocked, are exactly the real behavior.

What you will do in the next lab

You create a parent type called Site and create a child with a correct uid and one with a wrong uid side by side, to see what the garbage collector does. You check the warning event left by a cross-namespace reference, run each of the three --cascade modes, and prove the results with objects. Finally, using an object left stopped by foreground, you build a diagnostic script that extracts the stalled parent and the children blocking it at once.