What to Check Before Installing the Dynamo Operator: Webhook Scope, Two Installation Methods, and Import List

Status: Draft (2026-11-14), measured. Dynamo v1.6.2 (chart dynamo-platform-1.6.2, operator image kubernetes-operator:1.6.2) installed directly on Kubernetes v1.34 and Helm v3.18 on 2026-11-13. Resource counts were counted from the helm template output, and error messages are verbatim from actual output. The global method was only inspected, not installed. The latency that the global method's pod webhook introduces to pod creation in other namespaces was not measured (Unmeasured). Writing Conventions: Measured data is the default. Unmeasured items are marked as "Unmeasured", and unverified items are marked as "Unverified".

Table of Contents

• 1. What Wasn't There When Using Only vLLM
• 2. Global Method: Webhooks Intercept All Pod Creation
• 3. Webhooks Cannot Be Narrowed via Values
• 4. Three Things Missing When Using the Restricted Method Alone
• 5. Separating Namespaces Caused the Operator to Crash 161 Times
• 6. What to Bring into a Closed Network
• 7. Summary: What to Decide Before Installation
• References

1. What Wasn't There When Using Only vLLM

vLLM starts with just a Deployment and a Service. It does not create any cluster-scoped resources.

Dynamo is different. Workers and frontends are described using a single custom resource called DynamoGraphDeployment , and the operator reads it to create Deployments and Services. To do this, CRDs, the operator, admission webhooks, and their permissions must be placed in the cluster first. In a shared multi-team cluster, this should be evaluated even before engine performance.

Starting with the default values from the chart: in 1.6.2, etcd, NATS, Grove, the KAI scheduler, and snapshots are all disabled. This is because discovery defaults to the Kubernetes API, the request plane to TCP, and KV events to ZMQ. Although several documents in the repository still list NATS and etcd as prerequisites, we installed it without any of the four and successfully completed the measurements in Part 3.

2. Global Method: Webhooks Intercept All Pod Creation

The default installation uses a global method where a single operator oversees the entire cluster. Before installing, we rendered it with helm template and counted the generated resources (with the checkpoint feature disabled).

The eight webhooks in the global method have neither namespaceSelector nor objectSelector . Seven of them only watch Dynamo's custom resources, so they do not affect other teams. The remaining one is the problem.

mpodcheckpointrestore.kb.io   resources: pods   operations: CREATE
failurePolicy: Ignore         timeoutSeconds: 10

Every pod creation in the cluster passes through this webhook. Even with the checkpoint feature disabled (checkpoint.enabled: false), it is still registered. Because failurePolicy is Ignore , pod creation is not blocked even if the operator is down. However, in that case, the time the API server waits for a webhook response may be added to every pod creation. We could not measure what that latency actually is because we did not install the global method (Unmeasured).

We did not want to insert an experimental operator into other teams' pod creation paths. Therefore, we attempted to narrow the webhook scope to a test namespace.

3. Webhooks Cannot Be Narrowed via Values

The chart contains a value called webhook.namespaceSelector . If you specify a selector here and try to install, Helm rejects it.

VALIDATION ERROR: webhook.namespaceSelector must be empty. Helm manages admission
scope: cluster-wide webhooks cover every namespace and namespace-restricted
webhooks cover only their target namespace.

This means the webhook scope is determined by the installation method and cannot be customized by the user. The only remaining path is the namespace-restricted method (namespaceRestriction.enabled: true). When rendered with this method, all eight webhooks get the following selector attached.

namespaceSelector:
  matchLabels:
    kubernetes.io/metadata.name: <대상 네임스페이스>

The restricted method cannot be used together with automatic CRD updates. Enabling both will also cause rejection.

VALIDATION ERROR: namespaceRestriction.enabled=true is incompatible with
upgradeCRD=true. Set upgradeCRD=false so the namespaced release does not install
or update cluster-wide CRDs.

Furthermore, when deployed in restricted mode, the operator logs a prominent banner: "DEVELOPMENT AND TESTING ONLY: Namespace-restricted mode is not supported for production." The validation text in the chart also describes the restricted method as something to be "placed alongside a global operator for development and testing." The only officially supported path for production is the global method, and choosing it requires accepting the pod webhook discussed in Section 2.

4. Three Things Missing When Trying the Restricted Method Alone

When installing only the restricted method without the global operator, three things were missing because those tasks were handled by the global operator.

CRDs are missing. upgradeCRD: false Because deploy/operator/config/crd/bases/ is required, CRDs must be supplied manually. We retrieved six of them from the repository's kubectl apply . The largest file is 3,038,832 bytes. Client-side kubectl apply --server-side stores the entire applied content back into annotations, so it cannot exceed this size. We applied them using

. The service name pointed to by the conversion webhook is different. dynamo-system The conversion webhook in the original CRD points to dynamo-operator-webhook-service in the <릴리스 전체 이름>-webhook-service namespace. The name of the service created by the chart is dynamo-platform (or dynamo-platform-dynamo-operator-webhook-service if the release name is specified). Before applying the CRDs, these names and namespaces must be corrected to match the actual values.

The conversion webhook CA is not injected. Although webhook certificates are self-generated by the operator and stored in a Secret, the permission to inject that CA into the caBundle of the CRDs belongs exclusively to the global operator. After installation, we extracted ca.crt from the Secret and manually injected it into the four CRDs (dynamocomponentdeployments, dynamographdeploymentrequests, dynamographdeployments, dynamographdeploymentscalingadapters).

Only after filling in these three gaps could we create the v1beta1 resource.

5. Separating Namespaces Caused the Operator to Crash 161 Times

We placed the operator in dynamo-system and the workload in a different namespace. This is a common deployment pattern separating privileged processes from model servers. When using this layout in restricted mode, the operator could not survive for long.

failed to list *v1.Secret: secrets is forbidden: User "system:serviceaccount:
dynamo-system:dynamo-platform-dynamo-operator-controller-manager" cannot list
resource "secrets" in API group "" in the namespace "dynamo-system"
...
problem running manager  {"error": "failed to wait for cert-rotator caches to sync
kind source: *v1.Secret: timed out waiting for cache to be synced ..."}

The certificate rotator inside the operator watches Secrets in its own namespace. Watching requires list and watch permissions. The Role created by the chart in the release namespace looks like this.

rules:
- apiGroups: [""]
  resources: [secrets]
  resourceNames: [webhook-server-cert]
  verbs: [get, list, watch, update, patch]

resourceNames Rules restricting names via do not permit list operations without specifying a name. This is a known characteristic of Kubernetes RBAC. Consequently, cache synchronization fails to complete, and the process terminates itself once the timeout expires.

In the global method, a ClusterRole allows it, and in the restricted method where the release namespace and target namespace are the same, the Role in the target namespace allows listing Secrets, which masks this issue. It only surfaces when the two namespaces differ.

Here is what it looked like in practice:

• Controllers begin working about 35 seconds after the container starts. The process terminates at approximately 2 minutes and 30 seconds (one observed cycle: started at 15:17:27, terminated at 15:19:58).
• Kubernetes waits for 5 minutes before restarting it (CrashLoopBackOff upper limit).
• It restarted 161 times over 20 hours.
• During the approximately 2 minutes it remained alive, webhooks and controllers operated normally. A DynamoGraphDeployment submitted during that window turned into four Deployments and four Services within 25 seconds.
• Creation and modification of Dynamo resources would be rejected during the 5 minutes it was dead, because the validation webhook's failurePolicy is Fail . In this article, we waited for the alive state to apply changes and did not deliberately test submitting them while it was down (Unmeasured).
• Workers and frontends that were already running were unaffected. Some measurements in Part 3 ran while the operator was down with zero errors. This is because worker registration is handled directly by the DynamoWorkerMetadata resource used by the workers themselves, bypassing the operator.

Since the root cause is permissions, the fix is to add a Role in the release namespace that permits get·list·watch on Secrets. Alternatively, place the operator and workload in the same namespace. Since our measurements in this article were completed by waiting for the operator to be alive rather than adding the Role, neither method was applied and verified (Unmeasured).

Pods cycling between Running and CrashLoopBackOff can look normal at a quick glance. After installing the operator, you must check the restart count twice over a span of a few minutes.

6. What to Bring into a Closed Network

We retrieved four things from outside:

The runtime image includes vLLM 0.31.0, ai-dynamo 1.6.2, PyTorch 2.13.0+cu130, and NIXL 1.4.1. The load testing tool AIPerf is not in the runtime image and is only present in the frontend-only image.

We ran into three issues during the import process:

A 10GB image stalls other deployments on the same node. Pulling the runtime image took about 11 minutes per node. On nodes where the container runtime is configured to pull images sequentially, other images had to wait in line. In fact, another deployment that was supposed to start on the same node waited 12 minutes and 11 seconds, exceeded its progress deadline, and failed. Large images should be pre-pulled during off-peak deployment hours.

Fetching models by revision hash makes them undiscoverable offline. snapshot_download(..., revision="<커밋 해시>") Downloading via refs/main does not create HF_HUB_OFFLINE=1 in the cache. In that state, providing --model Qwen/Qwen3-0.6B via refs/main fails to resolve the repository name into a snapshot. After downloading, we noted the hash in the

file. Unpinned revisions are not recommended; using just a name might point to a different file next month. The model directory can be mounted as read-only. Permission denied Workers leave a

warning when trying to write a cache index file, but loading completes successfully.

When trying it out as a test, the restriction method does not affect other teams so it is safe, but it requires a lot of manual effort. When applying it to production, it is a global approach, and beforehand, you must agree with the cluster administrator on the fact that the pod webhook scope is the entire cluster. Either way, we confirmed that even if the operator dies, already running serving continues to work.

Reference

• Dynamo Repository v1.6.2: deploy/helm/charts/platform/components/operator/templates/ of _validation.tpl, webhook-rbac.yaml, manager-rbac.yaml, deploy/operator/cmd/main.go.
• Kubernetes documentation, Using RBAC Authorization resourceNames explanation.
• The values file and commands used for installation are in the repository's docs/experiments/2026-11-13-dynamo-adoption/ .