Automating LLM Serving Deployment with Kubernetes CRDs and Operators

Status: Draft (Updated Oct 7, 2026, augmented with scale test measurements on Oct 9, 2026). Descriptions verified against official documentation form the basis of this post, and only **the isolated environment test in Section 8, along with "Direct kopf Operator Testing" (including scale testing) and "Direct LWS Testing" at the end**, are values directly verified by me. The rest has not yet been measured in practice. Please read "Verified Versions and Dates" and "Unverified Items" at the end of the post first. Writing Rule: Measured data is the baseline, and unmeasured items are marked as "Unmeasured".

Table of Contents

• 1. Why a Single Deployment Falls Short
• 2. API Shapes of LWS, KServe, and Dynamo
• 3. Core Patterns of Operators
• 4. Go (controller-runtime/kubebuilder) vs. Python (kopf)
• 5. CRD Schema Design Principles
• 6. Rollout Strategies
• 7. Conflicts with GitOps (Argo CD)
• 8. Behaviors Verified in an Isolated Environment
• 9. Try It Yourself
• 15. Common Misconceptions
• 10. Potential Interview Questions
• External References
• Reference Materials (Actually Opened)
  • Direct kopf Operator Testing
  • Direct LWS Testing
• Verified Versions and Dates (October 5, 2026)

LLM serving deployment involves a collection of requirements that Deployments cannot express: leaders and workers spanning multiple nodes, a preliminary weight-loading phase, rollouts that must respect ratios, GPU topologies, and scaling where the unit is a bundle of pods rather than a single pod. Kubernetes extends these through CRDs (Custom Resource Definitions) and operators. This post outlines the current shape of those APIs and the core patterns of operators, backed by behaviors verified in an isolated lab environment. I must clarify upfront that I do not have experience operating production-scale Go operators.

What You Can Explain in an Interview After Reading This Post

1. Why it is difficult to express LLM serving with Deployments, and what abstractions LeaderWorkerSet (LWS), KServe, and Dynamo each add.
2. Why reconciliation loop idempotency, level-triggered design, owner references and finalizers, status conditions, backoff, and leader election are designed the way they are.
3. CRD version transition procedures, rollout strategies, and how to resolve issues where Argo CD and operators fight over the same fields.

1. Why a Single Deployment Falls Short

• The unit of scaling is a group. When a single model replica spans GPUs across multiple nodes, the replica is a bundle of a leader pod and worker pods. Because collective communication breaks if a single pod dies, LWS sets the policy to recreate the entire group (RecreateGroupOnPodRestart) as the CRD default.
• The preliminary phase is long and expensive. As seen in Part 8, weight loading takes minutes. In single-GPU environments, people sometimes use strategy: Recreate because rolling updates that spin up new pods first require spare GPUs.
• Coordinated rollouts. Serving that separates prefill and decode must maintain the capacity ratio between the two roles throughout the update, and LWS's DisaggregatedSet is the API for this.
• Topology and gang scheduling. Pods in a group must be scheduled together within the same topology domain. Dynamo documentation guides users toward multi-node combinations of Grove and KAI Scheduler (default), or LWS and Volcano.

2. API Shapes of LWS, KServe, and Dynamo

KServe documentation states that InferenceService is left for traditional predictive models, while LLMInferenceService handles prefill/decode separation, multi-node setups, and prefix-aware routing (I opened the 0.20 page). Dynamo's CRD names and fields were read directly from the CRD YAML of the v1.5.0 tag.

 DynamoGraphDeployment (dgd)  ← 사용자가 선언: 컴포넌트 목록, 복제 수, 재시작 id
        │ 오퍼레이터가 만듦
        ▼
 DynamoComponentDeployment (dcd)  ← 컴포넌트마다, 이름에 worker hash 접미사
        │ 배포 모드에 따라
        ├─▶ Deployment (단일 노드)
        └─▶ Grove PodCliqueSet 또는 LWS (다중 노드, 갱 스케줄링)

3. Core Patterns of Operators

 API Server ─watch▶ Informer(로컬 캐시) ─▶ WorkQueue(키 중복 제거, 백오프) ─▶ Reconcile(키)
      ▲                                                                        │
      └────── 변경(자식 생성, status 기록) ◀── 관측 → 비교 → 행동 → 보고 ◀────┘

Level-triggered reconciliation. Comments in the controller-runtime source code (v0.25.2) note that reconciliation is driven by the actual state read from the apiserver or cache, rather than events. Request contains only the name and namespace without specifying what changed, so even when receiving a pod deletion event, the reconciliation function figures it out by reading "pod does not exist." The Good practices chapter of the Kubebuilder book also warns that writing logic per event can clog resources and require human intervention. kopf uses a model where handlers are called based on causes like creation, modification, or deletion, so changes to un-watched children must be supplemented with timers (Section 9). However, changes during downtime are handled in a level-triggered manner by listing all objects upon startup and processing only the final state.

Idempotency. Written as "make it exist in the desired state" rather than "create it." If it already matches, writing nothing leaves sub-object resourceVersion intact and prevents unnecessary watch events. This is essential because leader election can momentarily result in two leaders.

Owner references and finalizers. According to Kubernetes documentation, a child's ownerReferences points to the owner name and UID in the same namespace; cross-namespace references are prohibited by design, and the garbage collector leaves OwnerRefInvalidNamespace warning events. When a finalizer is present, a deletion request only sets deletionTimestamp , and actual deletion happens only after the controller finishes cleanup and removes the finalizer. The Kubebuilder book advises writing both finalizer addition and cleanup logic idempotently because deletions turn into updates.

Status and conditions. status.conditions and observedGeneration expose state in a standard format. Dynamo's DGD also places conditions, observedGeneration, state, rollingUpdate in status. Argo CD documentation advises that since CRD status formats vary, custom health checks are needed, and using observedGeneration prevents misjudgments before reconciliation completes.

Retry and backoff. If a reconciliation function returns an error, the workqueue re-queues it with exponential backoff; RequeueAfter requeues after a specified duration, and the Requeue field is deprecated. TerminalError does not requeue. "Not ready yet" is properly expressed as a delayed requeue rather than an error.

Leader election. The controller-runtime manager acquires a Lease via the LeaderElection, LeaderElectionID, LeaderElectionNamespace option (confirmed in Dynamo's main.go , and the LMCache operator installation YAML also contains a leader-election Role). Kubernetes documentation explains that Leases are used for node heartbeats and control plane leader election.

Webhook validation. Simple rules are sufficient with CRD schema CEL (x-kubernetes-validations), while complex rules and injections require webhooks. Dynamo documented a rule in validation webhooks that aggregates and returns independent errors with field.ErrorList , and the LMCache operator injects connection settings into vLLM pods using mutating webhooks while issuing webhook certificates via cert-manager.

4. Go (controller-runtime/kubebuilder) vs. Python (kopf)

This is my judgment. The Go ecosystem is the default for upstream-grade CRDs that require conversion webhooks and large-scale watching, whereas internal automation used by a couple of teams benefits most from the language the team knows best. Since I have no experience operating Go operators, the comparison above is the result of reading documentation and code.

5. CRD Schema Design Principles

• Keep specs small; use status for one-off commands. restartNow: true Imperative fields like restart.id become ambiguous in meaning after execution. Dynamo's DGD restarts when oldSelf.hasValue() || !has(self.restart) is changed, and CEL transition rules like
• leave restart empty at creation time. Put defaults in the schema. size LWS puts restartPolicy default 1, RecreateGroupOnPodRestart default LMCacheEngine in the CRD. Conversely, LMCache isolatedIPC 's
• varies by GPU vendor, so it has no static default and is calculated by the operator; documentation warns that operator upgrades can change default behavior and restart engine pods. Validate with CEL. minAvailable <= replicas Dynamo's DCD enforces immutability of minAvailable, type and LLMInferenceService with CEL, and KServe replicas forbids simultaneous specification of scaling and
• . Validation ratcheting that does not break existing objects has been stable since Kubernetes 1.33. servedVersioning and conversion. storage(serving requests) and None (expression written to etcd, exactly one) are separate axes. If schemas are identical, the conversion strategy services is sufficient, but if shapes change, a conversion webhook is required. Dynamo moved from v1alpha1 (deprecated) to v1beta1 (storage), where the components map became a CONVERSION.md list, and documented conversion webhooks, hub-and-spoke rules, and rules for preserving unrepresentable fields in sparse annotations in LLMInferenceService . KServe's None has no conversion settings in the CRD, so it defaults to kvCacheOffloading and StorageVersionMigration is present only in v1alpha2 (verified by comparing schemas). Regarding storage version migration, there is an API, but since Kubernetes documentation marked it as Beta starting in 1.35 while default activation status conflicted within the docs, I could not verify it. Dynamo borrows Cluster API's migrator.

6. Rollout Strategies

• LWS: Rolling updates occur at the group level, and maxUnavailable, maxSurge is based on the number of groups (neither can be 0). partition makes it a canary that rolls out only some ordinals to the new template, and it is not supported in Hash mode. maxSurge requires additional GPU groups. DisaggregatedSet maintains the capacity ratio between roles, rolls them out together, and does not support partitions.
• Dynamo: Places a hash of the worker inputs into the DGD annotation (nvidia.com/current-worker-hash-v2), and when the hash changes, creates a DCD with a new suffix to roll out, committing the annotation only after completion. There are also rules for migrating from v1 to v2 to ensure operator upgrades do not trigger rollouts. status.rollingUpdate supports only single-node, non-Grove deployments.
• KServe: canaryTrafficPercent Sends a portion of traffic to the new revision and remembers the last known good revision to roll back, while avoiding unhealthy revisions. This is supported only in serverless deployment mode.
• Model Version Switching and Blue-Green: The process of spinning up a new version as a separate group, verifying model loading and readiness pass, and then shifting traffic. This is an expression summarizing the common principles of the tools above and is not a quotation from a single document.

7. Conflicts with GitOps (Argo CD)

Conflicts occur when fields specified in Git are modified by an operator or HPA. replicas If you include this in Git, it fights with HPA (HPA scales it up, Argo reverts it, and HPA scales it up again). The remedies are as follows.

• ignoreDifferences: jsonPointers, jqPathExpressions, managedFieldsManagers to exclude it from comparisons. The system-wide setting is specified as argocd-cm of resource.customizations.ignoreDifferences.<그룹>_<종류> .
• RespectIgnoreDifferences=true: By default, this applies only to comparisons, and the desired state is applied as-is during synchronization. Therefore, this option is required if you also want to exclude it from synchronization. It only has an effect on already existing resources.
• Server-side diff (stable since v3.1.0): Compares against the server-side apply dry-run result, allowing admission controllers to participate in diff calculation. This is not performed when resources are created for the first time.
• Custom Health Checks (Lua): Reflects the status of the operator CR in the application health.

A common pitfall is health checks. In a structure where application health is determined by the state of child resources and synchronization waits for that health, if the built-in check evaluates a resource (e.g., a CronJob that failed once) as Degraded, synchronization may hang waiting for health and fail to apply new changes. Check rules can be solved by redefining them in Lua (I have not verified which version introduces this behavior). In ApplicationSet as well, turning off automation for child applications causes a conflict between two reconcilers where the parent template turns it back on; the solution is to narrow down the exception to a single field by name and path.

8. Behavior Verified in an Isolated Environment

The following content was verified with actual Kubernetes controllers in an isolated practice environment (a kwok cluster or personal VM) and is not a record of my production incidents.

• Owner Reference UID Error. A child with an incorrect UID is recognized by the garbage collector as having its parent already deleted, and is removed within seconds. If a controller caches the parent UID and the parent is recreated with the same name, a child created with the old UID disappears as soon as it is created. Cross-namespace references pass apply validation and leave only a warning event, which disappears by default in an hour. If you delete the operator first and then the CR, it gets stuck in Terminating; if you manually remove the finalizer, it skips external cleanup, so you must delete the CR first and then strip away the operator.
• Storage Version Migration. Under the strategy None , default values for new fields are populated upon saving, so they are not visible on old-style objects until rewritten, which can cause the controller to treat the same kind in two different ways. storedVersions I verified in a practice pod that if you remove the remaining version from spec.versions , the API server rejects requests. Conversion webhooks could not be tested because a webhook server could not be spun up.
• Reconciliation Loop. Go 컨트롤러를 실행할 수단이 없어 셸 스크립트 조정기로 관측, 비교, 행동, 보고와 멱등성(resourceVersion Grades immutability) and does not emulate informer caches and work queues. The leader election exercise demonstrates the risk of a window with dual leaders, as server-side apply reports conflicts if two instances attempt to write to the same field.

Limitations. The above describes the behavior of Kubernetes itself verified through practice, and is not incident response for a specific operator implementation or a Go production operator.

9. Try It Yourself

Performed entirely within a single local kind cluster. All commands explicitly specify the kind-op-lab context, so they do not touch other clusters. When finished, delete the entire cluster using the final line.

kind create cluster --name op-lab
kubectl --context kind-op-lab create ns op-lab
cat <<'EOF' | kubectl --context kind-op-lab apply -f -
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata: {name: modelcaches.labs.example.dev}
spec:
  group: labs.example.dev
  scope: Namespaced
  names: {kind: ModelCache, plural: modelcaches, singular: modelcache, shortNames: [mc]}
  versions:
  - name: v1
    served: true
    storage: true
    subresources: {status: {}}
    additionalPrinterColumns:
    - {name: Phase, type: string, jsonPath: .status.phase}
    schema:
      openAPIV3Schema:
        type: object
        properties:
          spec:
            type: object
            required: [model]
            properties:
              model: {type: string}
              replicas: {type: integer, default: 1, minimum: 0, maximum: 8}
            x-kubernetes-validations:
            - {rule: "self.model.size() > 0", message: "model must not be empty"}
          status:
            type: object
            x-kubernetes-preserve-unknown-fields: true
EOF

The minimal reconciliation loop (op.py) binds a single ConfigMap as an owner reference to a single CR. kopf is installed with pip install kopf kubernetes .

import kopf, kubernetes as k8s

@kopf.on.resume("labs.example.dev", "v1", "modelcaches")
@kopf.on.create("labs.example.dev", "v1", "modelcaches")
@kopf.on.update("labs.example.dev", "v1", "modelcaches")
@kopf.timer("labs.example.dev", "v1", "modelcaches", interval=30, idle=10)
def reconcile(spec, name, namespace, patch, **_):
    cm = {"apiVersion": "v1", "kind": "ConfigMap",
          "metadata": {"name": f"{name}-cfg"},
          "data": {"model": spec["model"], "replicas": str(spec.get("replicas", 1))}}
    kopf.adopt(cm)                      # ownerReferences: 이 CR의 이름과 uid
    api = k8s.client.CoreV1Api()
    try:
        api.create_namespaced_config_map(namespace, cm)
    except k8s.client.ApiException as e:
        if e.status != 409:
            raise
        api.patch_namespaced_config_map(f"{name}-cfg", namespace, cm)   # 멱등
    patch.status["phase"] = "Ready"

@kopf.on.delete("labs.example.dev", "v1", "modelcaches")
def cleanup(name, logger, **_):
    logger.info("%s: 외부 자원 정리 자리", name)     # 삭제 핸들러가 있으면 kopf 가 finalizer 를 붙인다

kubectl config current-context            # kind-op-lab 인지 확인(kind 가 만들면서 바꿉니다)
kopf run --standalone --namespace op-lab op.py &
kubectl --context kind-op-lab -n op-lab apply -f - <<'EOF'
apiVersion: labs.example.dev/v1
kind: ModelCache
metadata: {name: demo}
spec: {model: tiny-llm}
EOF
kubectl --context kind-op-lab -n op-lab get mc,cm -o wide
kubectl --context kind-op-lab -n op-lab get cm demo-cfg -o jsonpath='{.metadata.ownerReferences}'
# 에지 대 레벨 관찰: 자식을 지워도 CR 이벤트가 없으므로 타이머(30초 주기)가 되살립니다
kubectl --context kind-op-lab -n op-lab delete cm demo-cfg && sleep 40 && kubectl --context kind-op-lab -n op-lab get cm
# CEL 검증: spec.model 을 빈 문자열로 넣은 CR 을 적용하면 "model must not be empty" 로 거절됩니다
# 정리
kill %1; kind delete cluster --name op-lab

Things to try next: ownerReferences Check if a ConfigMap with a purposely mismatched UID gets deleted, and observe deletionTimestamp and the list of finalizers during CR deletion. Because this skeleton does not watch child changes, it is intended for learning and supplemented by a timer.

10. Potential Interview Questions

1. Why use LWS instead of Deployment for LLM serving? Because replicas are bundles of leaders and workers, requiring scaling, rolling updates, disaster recovery, and topology placement to be handled at the group level. Since collective communication breaks if a single pod fails, the default policy is to recreate the entire group.
2. Why is it okay even if a reconciliation loop misses an event? Because requests do not contain event details; instead, the current state is read every time to bridge the gap. Even if a controller crashes and restarts, running it from the beginning to finish only the remaining work requires all writes to be idempotent.
3. What is the difference between ownerReference and finalizer? The former is a mechanism where the child points to the parent and lets the garbage collector handle cleanup, and the child is deleted if the uid does not match. The latter is a mechanism that delays deletion for external resource cleanup; if removed without a controller, cleanup is skipped.
4. What is the procedure for promoting a CRD from v1alpha1 to v1beta1? Add the new version as served, attach a conversion webhook if necessary, migrate storage and rewrite all existing objects, then storedVersions clean up and turn off the old version. Strategy None is safe only when the field shapes are identical.
5. What if Argo CD reverts a field changed by the operator? Remove that field from Git, or if that is difficult, exclude it from comparison using ignoreDifferences and from synchronization using RespectIgnoreDifferences . Health reflects observedGeneration via a custom Lua check for the CR.

11. Common Misconceptions

• Reconciliation functions are event handlers: They are functions that only receive a key and re-read the current state.
• If there is leader election, it never runs in two places at once: There can momentarily be two instances.
• Removing a finalizer safely resolves Terminating: It skips external cleanup.
• Adding a field is always compatible: Under strategy None , old representations cannot have new default values.
• If Argo CD is green, the model is serving: CR health is only reflected if a custom check exists.

External Basis

References (actually opened)

• https://lws.sigs.k8s.io/docs/overview/ , /concepts/leaderworkerset/ , /concepts/leaderworkerset/failure-handling/ , /concepts/leaderworkerset/rollout-strategy/ , /concepts/disaggregatedset/ , /reference/disaggregatedset.v1/ , and https://github.com/kubernetes-sigs/lws (v0.11.1 CRD YAML)
• https://kserve.github.io/website/docs/model-serving/generative-inference/llmisvc/llmisvc-overview , /concepts/architecture/control-plane-llmisvc , /model-serving/predictive-inference/rollout-strategies/canary , and https://github.com/kserve/kserve (v0.21.0 CRD YAML)
• https://docs.nvidia.com/dynamo/latest/ , https://github.com/ai-dynamo/dynamo (v1.5.0: operator directory's README.md, api/CONVERSION.md, internal/dynamo/worker-hash.md, CRD YAML, docs/fern/pages/kubernetes/installation/multinode-orchestration.md)
• https://docs.lmcache.ai/mp/operator.html
• https://book.kubebuilder.io/reference/good-practices.html , /reference/using-finalizers.html , /multiversion-tutorial/tutorial.html
• https://github.com/kubernetes-sigs/controller-runtime (v0.25.2 pkg/reconcile/reconcile.go)
• https://kopf.readthedocs.io/en/stable/ , /peering/ , /errors/ , /continuity/ , /configuration/
• https://kubernetes.io/docs/concepts/extend-kubernetes/operator/ , /concepts/overview/working-with-objects/owners-dependents/ , /finalizers/ , /concepts/architecture/leases/ , /tasks/extend-kubernetes/custom-resources/custom-resource-definitions/ , /custom-resource-definition-versioning/ , /tasks/manage-kubernetes-objects/storage-version-migration/ , /reference/command-line-tools-reference/feature-gates/
• https://argo-cd.readthedocs.io/en/stable/user-guide/diffing/ , /sync-options/ , /diff-strategies/ , /operator-manual/health/

Direct testing of the kopf operator

Running the minimum reconciliation loop from Section 9 (op.py, unchanged) on a disposable single-node k3s (v1.36.5, 4 vCPU virtual machine) with kopf 1.44.6, I verified the pattern described in Section 3. A version with just one extra log line to count call counts (op2.py) was used for duplicate execution and error testing.

• The magnitude of recovery delay is determined by the implementation approach. The reason recovery for child deletion was scattered between 2.5–26.9 seconds is that kopf's CR handler is not awakened by the deletion of children it does not watch, recovering only via a timer (in this test, interval=30, idle=10). controller-runtime's Owns() immediately re-reconciles the owner upon child deletion events, so the delay is close to the watch propagation time (described in official documentation, not directly measured). Both approaches are level-based reconciliations that read the current state to bridge gaps, and tests confirmed the property that recovery occurs even if events are missed.
• Without an operator, the finalizer blocks deletion. It is normal behavior for CR deletion not to complete; you must delete the CR first or intentionally clean up the finalizer before removing the operator. This test also paused once during the cleanup phase for the same reason (running full CR deletion with the operator turned off).
• Duplicate execution (no leader election). Two operators simultaneously reconciled the same CR. Over 50 seconds, instance A did it 6 times and B 6 times, performing the same work at the same times. Enabling kopf's peering (KopfPeering) and setting priorities to 100 and 50 resulted in A doing it 6 times, B 2 times, and B stopping with Pausing operations in favour of logs. Gracefully shutting down the higher-priority instance A caused B to start working again within 0.2 seconds (handover delay on abnormal termination was not measured, though peering lifetime 60 seconds is estimated to be the upper limit).

Error handling and retry intervals. A CR configured to always fail its reconciliation function was left for 100 seconds while call timestamps were recorded (kopf defaults).

• The general exception retry interval of 60 seconds matches the kopf documentation default. "Not ready yet" should be expressed as TemporaryError to use a shorter interval.
• PermanentError stops only that handler. If a timer is attached to the same CR, it will continue to re-evaluate every 30 seconds with an incorrect spec, so permanent errors must be designed to remain as status conditions and be skipped in timers as well.
• Limitations. The initial scale test on a VM stopping at 20 CRs was caused not by disk or concurrency issues, but by the test script. After launching the operator as a background job in the same shell, it invoked wait without arguments every 20 items, waiting for the non-terminating operator. Results re-measured in a different environment are documented in "Scale Testing" below. Call times are log timestamps; direct comparisons with the Go operator were not performed.

Scale testing (10–300 CRs). Using a single-node kind cluster (Kubernetes v1.36.1) on Docker Desktop (10 VM CPUs, 8GB RAM) on a laptop (Apple M1 Max), the above op.py was run un-modified as an out-of-cluster process using kopf 1.44.6. Delay was measured from when the watcher first saw a CR until it received the first event carrying status.phase=Ready , showing the median of 3 runs after excluding 1 warm-up run per condition.

• The operator was not the bottleneck up to 300 items. Almost all of the total processing time was spent by kubectl submitting the CRs (3.57 seconds for 300 manifests, 3.61 seconds until final Ready), while individual CR delays had a median of 0.03–0.13 seconds regardless of count, with run maximums under 0.36 seconds.
• Operator CPU usage was proportional to the count at 7.5–8.6 milliseconds per CR. Average CPU usage during processing ranged from 31–64% of a single core, peaking at 55–85% on 0.1-second intervals. RSS went from ~100MiB idle to 101–109MiB at 100 CRs, and 111–115MiB at 300 CRs. When fully processed and only running 30-second timers, usage was 1.4–1.6% for 100 CRs.
• kubectl 을 20개씩 동시에 띄우는 쪽이 오히려 느렸습니다. 7.2 seconds versus 3.6 seconds for submitting 300 items. The cost of spawning a kubectl process is greater than a single request, and when 20 items arrive simultaneously they queue up inside the operator, making the median delay 2–4 times higher. What determined delay was the burstiness of arrivals rather than the total count.
• The cause of the 20-item stall on the initial VM was the test script. The script launched the operator with setsid nohup kopf run … & in the same shell and then invoked wait without arguments every 20 items, where argument-less wait waits for all children of that shell—namely, the non-terminating operator. Running the same structure again in a Linux container (bash 5.2, util-linux 2.41) and kind caused both to stall identically (stuck at 20 CRs for 60 seconds in kind), whereas restricting the targets to wait for to kubectl's PID prevented any blockage up to 300 items.
• Limitations. Other clusters running concurrently in the same Docker caused etcd latency warnings and API server restarts; runs with even a single warning during the generation phase were discarded (which is why parallel 300 had 2 runs). Single-node setup, running the operator as a pod, steady-state CPU for 300 items, and counts above 1000 were not measured (unmeasured).

Direct Testing of LWS

Installing LWS v0.11.1 on a disposable single-node k3s (v1.36.5, 4 vCPU, 4GiB virtual machine), group structures, recovery, rolling updates, and scaling were verified using busybox sleep pods. Since GPUs and models were not used, model loading times are excluded, and times were measured at 0.5-second intervals.

• If a single worker dies, the entire group is recreated(default policy). Force-deleting a worker in group 0 resulted in about 2.0 seconds later all 3 pods in the same group ( 1 leader and 2 workers) getting new UIDs , whereas the 3 pods in group 1 kept their UIDs. This reflects the structure where the loss of a single rank in multi-node inference renders the rest useless, handled via group-level restarts.

• restartPolicy: None causes only the deleted pod to be recreated(~1.9 seconds). In the same test, only that single worker received a new UID. Whether automatically reviving groups is beneficial or burdensome depends on model loading times, which this test could not demonstrate as there was no loading time (unmeasured).

• Rolling updates proceeded one group at a time, starting from the group with the highest ordinal. Modifying template values resulted in the demo-1 group becoming Pending with the new revision at second 1, demo-1 Running and demo-0 being replaced with the new revision at second 3, and all Running with the new revision at second 4 (8 seconds until observation loop ended). Without model loading, a group finished in a few seconds; in actual serving, it would take significantly longer as each group waits for the previous one to become ready (estimated).

• Scale subresource functions properly. when replicas were increased to 3, 9 pods became Running in 2.5 seconds, and when decreased to 1, demo-0, demo-0-1, demo-0-2 remained (groups with larger numbers were removed first).

• LWS does not perform gang scheduling. When creating two groups of size 4 (8 pods in total), each requesting 1 CPU, on a node with 0.8 available CPU, only one leader became Running while the remaining 7 were Insufficient cpu as Pending. Even when reducing the number of groups to 1, only the leader was Running while the 3 workers remained Pending. It appears the leader holds onto resources while waiting for the workers, which illustrates why gang scheduling tools (Kueue, Volcano) are needed. In the test of Part 10, both tools accepted only one job to avoid this deadlock.

• partition allows for a canary rollout. Across 4 groups (size: 2), when changing the template value to partition: 2 , only groups with an ordinal of 2 or higher (can-2, can-3) were updated to the new revision within 5 seconds, while ordinals 0 and 1 remained on the old revision (hash 7596). Subsequently, when lowering it to partition: 0 , the remaining two groups also transitioned to the new revision within 4 seconds. This matches the description in Section 7 (starting from groups with larger ordinals, updating only those greater than or equal to partition).

• maxSurge: 1 temporarily increases the number of groups by one before decreasing it. When changing the template across 4 groups, can-4(2 pods) were additionally created, bringing the total pod count from 8 to a maximum of 10, and returning to 8 once the replacement finished (taking about 9 seconds in total). maxUnavailable: 0 showed the same pattern, and even during the rollout, there were always 4 or more Running groups. Observations at 1 to 2-second intervals could not distinguish between maxUnavailable being 1 and being 0. The creation of additional groups means that in a real GPU environment, there must be enough spare GPUs for one group (here, they were sleep pods, so there was no cost).

• If both are 0, the admission webhook rejects the request. maxSurge: 0 When attempting to change from maxUnavailable: 0 to spec.rolloutStrategy.rollingUpdateConfiguration.maxUnavailable: Invalid value: 0: must not be 0 when maxSurge is 0 , it was blocked by a

error. sleep, the task performed by pods is busybox, the images are pre-pulled

, and there are no readiness probes. Group recovery due to node failures and DisaggregatedSet were not tested (unmeasured). Verified versions and dates (2026-10-05)

LWS v0.11.1 (2026-10-01), KServe v0.21.0 (2026-09-25, documentation is for version 0.20), Dynamo v1.5.0 (2026-09-21), controller-runtime v0.25.2 (2026-10-01), kubebuilder v4.16.0 (2026-09-10), kopf 1.44.6 (2026-06-03), Argo CD v3.5.3 (2026-09-14), LMCache v0.5.5, Kubernetes v1.37.1 (2026-09-23).

Things that were not verified: StorageVersionMigration default activation status of (descriptions conflict within the Kubernetes documentation), performance limits of kopf with over 1,000 CRs across multiple nodes (tested up to 300 CRs in the scale test above), details of controller unit testing tools (envtest), pros and cons of ValidatingAdmissionPolicy versus webhooks, differences between the latest KServe documentation and version 0.21.0, and the exact sub-resource creation paths for Dynamo's DGD deployment modes (diagrams were drawn based on CRD descriptions and documentation, but code paths were not traced). The kopf skeleton in Section 9 was written by me based on documentation and has not been validated by running it on an actual cluster.