TT Lab
Blog

한국어 English 日本語 中文

Get started
Blog

Blog

Multi-tenant GPU Scheduling and Capacity Planning

Status: Draft (Updated Oct 7, 2026; reinforced with actual measurements of borrowing limits on Oct 9, 2026). The descriptions verified through official documentation and papers form the foundation, and **the three measurements at the end of Section 3 (gang scheduling comparison, cohort borrowing and preemption, borrowing limit and reclaim speed)** are values measured directly on a disposable single-node cluster. The remaining sections have not been measured yet (unmeasured). Please read "Verified Versions and Dates" and "What Was Not Verified" at the end of the post first. Writing rule: Measured data is the default, and anything that could not be measured is marked as "Unmeasured".

Table of Contents

To share a GPU cluster among multiple teams, three things are needed: quotas that guarantee each team's share while allowing idle resources to be borrowed, gang scheduling for jobs that must start together like distributed training, and preemption policies that push out batch jobs while protecting serving workloads. The default scheduler provides none of these directly. This post covers the design differences of supplementary mechanisms, isolation limits, and the sequence for calculating capacity.

What You Can Explain in an Interview After Reading This Post

  1. Why the default scheduler's GPU model (integer extended resources) falls short for gang scheduling and fair sharing, and the difference that Kueue is an admission layer while Volcano is a scheduler.
  2. What time-slicing, MIG, and MPS share versus what they fail to isolate, and choices based on trust levels.
  3. The sequence for calculating the number of GPUs backward from request rates and latency targets (throughput, concurrency, key-value (KV) cache, headroom, and failure resilience), plus the pitfalls of utilization metrics.

1. Limitations of the Default Scheduler

2. Kueue: Admission Layer

flowchart LR
  J["Job (queue-name 레이블)"] --> LQ["LocalQueue (네임스페이스)"]
  LQ --> CQ["ClusterQueue (쿼터, cohort)"]
  CQ -- "쿼터 예약과 승인" --> W["Workload 승인, 파드 생성"]
  W --> KS["kube-scheduler가 노드 배치"]
  KS --> DP["디바이스 플러그인 또는 DRA"]

Kueue documentation states that placing pods on nodes is handled by kube-scheduler, and autoscaling by cluster-autoscaler. Kueue decides when jobs start (pod creation) and when they stop.

Concept Role
ResourceFlavor Expresses node group characteristics (GPU model, etc.) via labels and tolerations
ClusterQueue Resource pool. Quotas (nominalQuota, borrowingLimit, lendingLimit), preemption, and fair-share rules
LocalQueue Point where jobs are submitted within a team namespace
cohort A collection of ClusterQueues that lend unused quotas to each other
apiVersion: kueue.x-k8s.io/v1beta2
kind: ClusterQueue
metadata: {name: team-a}
spec:
  cohortName: all-teams
  queueingStrategy: BestEffortFIFO
  preemption: {reclaimWithinCohort: LowerPriority, withinClusterQueue: LowerPriority}
  resourceGroups:
  - coveredResources: ["nvidia.com/gpu"]
    flavors:
    - name: gpu-pool
      resources: [{name: nvidia.com/gpu, nominalQuota: 8, borrowingLimit: 4}]

Jobs select a LocalQueue via the kueue.x-k8s.io/queue-name label, and priority is given via a separate WorkloadPriorityClass(label kueue.x-k8s.io/priority-class) decoupled from pods. Preemption has a priority-based default mode and a fair-sharing mode comparing Dominant Resource Share (DRS). The latest v0.20.0 (2026-09-30) no longer provides v1beta1and provides only v1beta2. It also supports Deployment, StatefulSet, and LeaderWorkerSet, allowing serving and training to be managed together. MultiKueue, for multi-cluster dispatch, is in beta.

waitForPodsReadyhalts jobs and returns them to the queue if all pods are not ready within a time limit. The documentation calls this a timeout-based implementation and notes that sequential admission can slow approvals. This post covers it separately from native gang scheduling.

3. Volcano and Selection Criteria

Volcano is a separate batch scheduler. Its PodGroupof minMemberis gang-based, and Queueis weight(nominal share), capability(upper limit), guarantee, reclaimable. The scheduler runs enqueue, allocate, preempt, reclaim, backfill actions and configures plugins like gang, priority, drf(Dominant Resource Fairness, DRF) proportion, binpack in a hierarchical structure within the configuration file. Gang-aware preemption entered alpha in v1.15.0, and the latest stable line is v1.15.3 (2026-09-30). NVIDIA's KAI Scheduler also positions itself as a separate scheduler featuring hierarchical queues, time-based fair sharing, and topology awareness (the latest version has not been verified).

Criteria Kueue Volcano
Location Admission controller, keeps kube-scheduler Separate scheduler
Gang Time-limited method PodGroup Guarantee minimum count
Fair sharing Adopts cohorts, DRS fair sharing Queue weight, DRF, proportion

If you want to keep the default scheduler and focus on team-based quotas and borrowing, Kueue is a candidate; if you want to strongly use gang and queueing within the scheduler, Volcano or KAI are candidates. The support scope for stacking the two on the same pod configuration has not been verified.

Measurement: When two jobs requiring 4 pods to start together compete

We directly measured whether the "per-pod decisions" from Section 1 actually cause a deadlock and how the two tools resolve it.

Environment and Method. On a disposable single-node k3s (v1.36.5) inside a 4 vCPU, 4GiB virtual machine, we advertised 8 extension resources example.com/gpu to the node status (there are no actual GPUs, and the pods only sleep do this). One job consists of 4 pods, and each pod requests 2 resources, so a single job uses all 8. When Jobs A and B are submitted simultaneously, demand is 16 and supply is 8. We measured the "time at which all 4 pods of a job reached Running," deleted the accepted job, and also measured the time it took for the waiting job to have all 4 of its pods start up (0-second pod termination grace period, 3 repetitions). We installed Kueue v0.20.0 and Volcano v1.15.3 using their respective manifests.

Method Result
Default scheduler (creates pods for A and B alternately) Only 2 for A and 2 for B run for 60 seconds and then stall. Neither job fills all 4, resulting in a deadlock(Reason for waiting: insufficient extension resources, no pods to preempt)
Kueue (submits jobs in a suspended state) Only one job is accepted and all 4 run within 2.1 seconds(all 3 trials), while the other job's pods are not created at all. When the accepted job is deleted, 3.0 to 3.6 seconds later, the 4 waiting job pods run
Volcano (minAvailable: 4) Only one job runs all 4 within 3.3 to 4.1 seconds , while the other job runs 0. When the accepted job is deleted, 3.6 to 4.2 seconds later, the 4 waiting job pods run

These values are only meaningful under the conditions of a single node, fake resources, 2 jobs, and lightweight containers with pre-cached images. As the number of nodes increases, scheduler throughput and location selection dictate the costs. Also, Kueue avoiding deadlock appears to be because the quota (8 resources) matched the actual available resources. Kueue documentation notes that deadlocks can occur if quotas do not match actual available resources. Production recovery time scales are on the order of minutes (Philly's 2-3 minute timeout and Kueue's 30-minute default timeout in external references below). The 2-4 seconds in this article are values for the normal path without contention.

Measurement 2: Cohort Borrowing, Return, and Priority Preemption

Under the above environment (single-node k3s, 8 extension resources, sleep jobs, 0-second termination grace period), we now submitted multiple single-pod (1 resource) jobs to check queue policies. Using Kueue v0.20.0, two teams (team-a, team-b) each have a quota of 4 and form a cohort where they can borrow each other's remaining quota.

Situation Result
reclaimWithinCohort: Never. team-a submits 8, borrowing team-b's share of 4 on top of its quota of 4 All 8 run within 3.7 seconds. Subsequently, when team-b demands its quota of 4, 0 run for 60 seconds(0 evicted jobs). As soon as the 4 team-a jobs finish, the 4 team-b jobs run 3.1 seconds later
reclaimWithinCohort: Any. Same situation team-a 8 run in 1.8 seconds. team-b 4 run within 4.2 seconds , team-a shrinks to 4, and 4 jobs are evicted (event Preempted to accommodate a workload). As soon as team-b finishes, team-a's 8 return 3.0 seconds later
withinClusterQueue: LowerPriority, quota 4, 4 low-priority items running The 2 high-priority ones ran within 3.0 seconds , leaving only 2 low-priority ones. The evicted jobs went back to a paused state and waited. Once the high-priority ones were cleared, the 4 low-priority ones returned after 3.0 seconds.
Volcano v1.15.3 default settings (actions: "enqueue, allocate, backfill"), two queues qa and qb (equal weights, reclaimable: true) when qa uses up all 8 and qb requests 4, 0 allocated for 45 seconds. This is because the default settings have reclaim no actions
In the same Volcano, adding actions to reclaim results in 4 qb jobs running within 10.3 seconds , and qa dropping to 4 (event Evict ... reclaim). Clearing qb causes qa's 8 to return after 1.8 seconds.
Number of jobs Default scheduler (restricted to 8 extended resources) Kueue (single queue, quota 8) Ideal value
100 55.0 seconds 74.0 seconds (+35%) Approx. 12 seconds
400 231.0 seconds Approx. 343 seconds (+48%) Approx. 50 seconds

Kueue's 400-job figure was measured in a separate test (completed steadily at about 1.3 jobs/sec) that took progress snapshots every 30 seconds. The initial test failed to get a value due to an error in how completed counts were tracked. Because both methods were 5 to 7 times the ideal value (when 8 resources never idle), what dominated the time was not the scheduler, but the cost of creating and deleting pods on this 4 vCPU VM, with Kueue adding about 0.2 to 0.3 seconds of admission processing per job (calculated). At small scale, this difference is buried in pod startup times, but it means that for workloads submitting dozens of short jobs per second, the queue layer could become a throughput bottleneck.

Actual Measurement 3: Borrowing Limit and Reclamation Speed

Measured with Kueue v0.20.0 on a kind single node (Kubernetes v1.36.1) on Docker Desktop on a Mac (Apple M1 Max, 8GB memory, 10 CPUs). Fake extended resources, sleep jobs, and a 0-second termination grace period are the same as in Actual Measurement 2. Placing team-a (quota 3) and team-b (quota 5) in a single cohort, we enabled reclamation with reclaimWithinCohort: Any . team-a submitted 10 jobs requiring 1 resource each, and then team-b requested its quota of 5.

team-a borrowing limit team-a max running Time for team-b to reclaim quota of 5 Evicted jobs
None 8 (quota 3 + borrowed 5) 6.4–7.7 seconds (median 6.8 seconds) 5
4 7 5.2–5.8 seconds 4
1 4 1.7–2.7 seconds (median 2.0 seconds) 1

4. Topology-Aware Batching

Proximity between NVLink within a node and InfiniBand (IB) between nodes determines training communication speed. Kueue's Topology Aware Scheduling (TAS) has been in beta since v0.14 and is enabled by default. Hierarchies like blocks and racks are defined as node labels in Topologyand connected via ResourceFlavor.spec.topologyName, while requirement levels are set using pod annotations podset-required-topologyor podset-preferred-topology. Documentation notes that memory usage increases because it tracks all pods and nodes. Volcano defines tiers using HyperNode(the documentation example API is topology.volcano.sh/v1alpha1) and applies hard or soft constraints to jobs. Multi-node NVLink is officially supported by the NVIDIA DRA driver's ComputeDomain.

5. Priority and Preemption Design

Grant guaranteed quotas and high priorities to serving, and low priorities and borrowed quotas to batch training. When serving reclaims its borrowed share (reclaimWithinCohort), batch jobs are pushed out, so batch jobs must endure interruptions via checkpoints. Reduce the termination grace period of victim pods to leave only the checkpoint saving time since it causes reschedule delays. Also, according to Kubernetes documentation, the scheduler cannot preempt pods using DRA resources.

6. Limitations of GPU Sharing and Isolation

Method Division basis Isolation
Time-slicing Time No memory or fault isolation, equal time distribution to all processes
MIG (Multi-Instance GPU) Hardware partitioning Memory and fault isolation, up to 7 instances
MPS (Multi-Process Service) Streaming Multiprocessor (SM) concurrent execution Fatal errors can affect other clients on the same GPU

Time-slicing inflates and advertises resources by a factor of replicas, and the DCGM (Data Center GPU Manager) exporter fails to link per-container metrics in this mode (GPU Operator documentation). MIG is supported on select GPUs starting from the Ampere generation (A100, H100, B200, etc.), with no consumer GPUs listed in the verified NVIDIA support list, and changing configurations requires user workloads to be absent from that GPU. MPS provides partial isolation when static SM partitioning is enabled starting from driver r610, and monitoring tools aggregate client usage into the MPS server process. Therefore, use MIG or dedicated nodes for untrusted tenants, and time-slicing or MPS for lightweight jobs within the same team. Kueue quotas are accounting devices, not security boundaries.

7. Current State of Dynamic Resource Allocation (DRA)

DRA lets you request devices by attributes (Common Expression Language (CEL) expressions) using ResourceClaim, DeviceClass, ResourceSlice. Feature gate data (checked 2026-10-05) is as follows.

Feature Stage
Core DRA stable (1.34, gate locked in 1.35, official page states "Stable since v1.35")
Priority lists, admin access stable (1.36)
Legacy extended resource name mapping, device taints stable (1.37)
Partitionable devices, consumable capacity, binding conditions beta (1.36)
Workload-scoped ResourceClaim beta (1.37, disabled by default)
Node allocatable resources, compatibility groups alpha

The NVIDIA DRA driver (kubernetes-sigs/dra-driver-nvidia-gpu, v0.5.0, 2026-08-19) README (main) refers to ComputeDomainas officially supported, notes that GPU allocation is not officially supported and is disabled by default in the chart. The documentation site lists MIG, time-slicing, and MPS as supported features but does not specify their stages, making it impossible to verify which is current. The Kueue v0.20 release notes include DRA device availability checking as an alpha gate, and Volcano v1.15 has incorporated DRA resource calculation into queue quotas.

8. Capacity Planning

λ(요청률), 입출력 길이, 지연 목표 -> 복제본 1개의 실측 처리량과 동시성
 -> 처리량 기준 대수, 동시성(Little 법칙) 기준 대수, 키-값(KV) 캐시 한도 확인
 -> 큰 쪽 선택 -> 여유율 -> 장애 대비(N+k) -> 병렬화(텐서 병렬, TP) 단위로 올림

This is an explanatory example. Baseline figures are measured in the author's experimental environment, where Qwen3.8-27B (NVFP4) achieves a total of 138 tok/s across 4 concurrent requests on a 24GB-class consumer GPU, with a KV cache of about 57,000 tokens. Assume a request rate of 0.5 req/s, 1,500 input tokens, and 400 output tokens.

  1. Throughput: 0.5 × 400 = 200 tok/s ÷ 138 = 1.45, so 2 units.
  2. Concurrency: At 138 ÷ 4 = 34.5 tok/s per request, response time is about 11.6 seconds, and by Little's Law, 0.5 × 11.6 ≈ 5.8 elements are in progress concurrently. With a limit of 4 concurrent requests per replica, at least 2 units (8 slots) are needed, and slot utilization is about 72%.
  3. KV check: 6 × (1,500 + 400) = 11,400 tokens, which is within 57,000.
  4. Add a 30% load fluctuation headroom and 1 unit for failure tolerance, totaling 3 units.

If a model spans multiple GPUs (tensor parallelism TP), the replication unit increases by the number of devices, so multiply the count by the TP size, and if a node dies, capacity decreases in replica units.

Pitfalls of Utilization Metrics: NVML (NVIDIA Management Library) utilization.gpuis the percentage of time during the sample period when at least one kernel was running, so it does not tell you how much the SM is actually used. Profiling metrics like DCGM's SM_ACTIVE, PIPE_TENSOR_ACTIVE are more granular. From a cost perspective, look at actual work time versus allocated GPU time (allocation rate) along with serving metrics (queue length, KV cache occupancy, throughput meeting target latency).

Common Pitfalls in Small-Scale Sharing

Since the author's experimental environment focuses on consumer GPUs (6–24GB) and does not use multi-node training or Kueue, this section generalizes pitfalls easily encountered in small-scale sharing rather than evidence of large-scale multitenancy.

Hands-on practice

This is read-only or ends in a temporary namespace. It was not executed and verified while writing this article.

# 읽기 전용: 노드별 GPU 광고량과 할당 상태
kubectl get node -o custom-columns='NAME:.metadata.name,GPU:.status.allocatable.nvidia\.com/gpu'
kubectl describe node NODE | grep -A8 "Allocated resources"

# 임시: 쿼터 0에서 거절, 쿼터 해제 후 Pending 사유 확인 (정리 포함)
pod() { cat <<EOF
apiVersion: v1
kind: Pod
metadata: {name: want-gpu}
spec:
  containers:
  - {name: c, image: busybox, command: ["sleep","60"], resources: {limits: {nvidia.com/gpu: $1}}}
EOF
}
kubectl create ns lab-gpu
kubectl -n lab-gpu create quota gpu-zero --hard=requests.nvidia.com/gpu=0
pod 1 | kubectl -n lab-gpu create --dry-run=server -f -
kubectl -n lab-gpu delete quota gpu-zero
pod 99 | kubectl -n lab-gpu create -f -
kubectl -n lab-gpu describe pod want-gpu | tail -8
kubectl delete ns lab-gpu

The first request should be rejected due to quota exceeded, and the second pod should remain in Pending state to report a GPU shortage as an event. Neither occupies the GPU. Perform Kueue practice only in disposable clusters (such as kind), not in clusters shared with others. The installation command (kubectl apply --server-side -f https://github.com/kubernetes-sigs/kueue/releases/download/v0.20.0/manifests.yaml) adds Custom Resource Definitions (CRDs) cluster-wide.

Interview-worthy questions

  1. What is the problem with running distributed training with the default scheduler? Pods may be scheduled incrementally and deadlock while holding only some GPUs. This is mitigated by Volcano PodGroup, the beta native PodGroup, or Kueue's timeout method, recognizing that the last one is a timeout-based implementation.
  2. How do you choose between Kueue and Volcano? Kueue is suitable when you want to add quotas, borrowing, and fair sharing while keeping kube-scheduler. Volcano is chosen when you want to heavily utilize gang scheduling, queues, and topology even if it means running a new scheduler.
  3. How to run batch training while protecting serving? Give serving guaranteed quotas and high priority, and let batch jobs borrow from the cohort. Since they are preempted upon reclamation, design checkpoints and termination grace periods.
  4. What are the GPU sharing methods and isolation limits? Time-slicing lacks memory and fault isolation, MIG provides hardware isolation but has limited GPU support, and MPS can propagate errors. Use MIG or dedicated nodes for untrusted tenants. DRA is moving toward expressing this based on attributes, and feature maturity varies.
  5. How do you calculate the number of GPUs from request rates and latency targets? Calculate throughput, concurrency using Little's Law, and KV cache limits respectively, pick the larger one, and scale up with headroom, N+k, and parallelism units. Do not just look at utilization utilization.gpu, but check SM activity and serving metrics together.

Common misconceptions

References

External Evidence

The scale and actual frequency of deadlocks, which could not be measured directly, were verified through the materials below. Paper figures were checked directly from the original PDFs, and figures with only summaries were not used.

Claim Source Verified Content
The problem of partial resource acquisition and waiting was actually handled by production schedulers. Jeon et al., USENIX ATC 2019, Analysis of Large-Scale Multi-Tenant GPU Clusters for DNN Training Workloads (arXiv 1901.05758) The Microsoft Philly scheduler releases acquired resources and retries after a 2-minute backoff if it cannot obtain all required GPUs within a 2-3 minute timeout. Tracking 96,260 jobs over 75 days showed that most waiting was due to fragmentation delay (97.9% for jobs over 8 GPUs). It did not measure the occurrence frequency of deadlocks.
Time-limited approaches are not true gang scheduling. Kueue KEP-349 (All-or-nothing), waitForPodsReady Documentation Deadlock can occur if quotas and available resources do not match, with a default timeout of 30 minutes, and upon timeout, backing off exponentially from 60 seconds up to a maximum of 3,600 seconds. It states that sequential admission sacrifices throughput to gain reliability.
Volcano's Gang Volcano Documentation gang plugin, PodGroup An all-or-nothing approach that schedules all pods only when the minimum running count is satisfied.
Scale of Queue Wait Times Kueue Repository Performance Test Configuration (test/performance/scheduler) With 30 queues and heavily oversubscribed quota load, the total was about 351 seconds (limit 425 seconds), and the average wait until admission was about 215 seconds for small jobs. This is the expected value from the configuration file, not the admission processing delay, so it cannot be compared with the 2-4 seconds in this article.
Large-Scale Throughput Google Cloud Blog, 130k-node GKE Cluster (2025-11-22) Pod creation and binding approx. 1,000/sec, 39,000 pods preempted in 93 seconds (vendor test).
Volcano's Scale Volcano Performance Tuning Guide (Updated 2026-05-26) In a Kind environment, 10,000 pods (10,000 single-pod jobs) took about 250 seconds, and about 180 seconds after webhook tuning (calculated as going from approx. 40 to 56 per second). It states that with a webhook timeout of 10 seconds, 98.7% of pod creation requests failed.
DRA Phase Kubernetes Feature Gate Documentation Core DRA stable phase. Table in Section 7 above.

Among external materials, none directly measured the occurrence frequency of gang deadlocks. "MLaaS in the Wild" (NSDI 2022) could not be opened because the PDF exceeded the size limit, and its figures were not used.

Verified Versions and Dates

As of 2026-10-05: Kubernetes 1.37.1, Kueue v0.20.0 (2026-09-30), Volcano v1.15.3 (2026-09-30), NVIDIA DRA driver v0.5.0 (2026-08-19). The author's experimental environment is Kubernetes 1.34.10, GPU Operator v26.3.3. Gang scheduling measurements were performed on a separate disposable k3s v1.36.5.

Things Not Verified: Comparison of Kueue and Volcano at scale with many nodes (tests used a single node and dozens of jobs or fewer), long-term behavior of prioritized queues, the latest version and maturity of KAI Scheduler, the support scope of configurations using Kueue and Volcano together, the GPU allocation feature stage of the NVIDIA DRA driver (README and documentation site differ), specific integration between Kueue TAS and NVLink labels, and the actual output of the above commands. Capacity calculations are arithmetic for explanatory purposes and should be replaced with load testing.

Sign in to like

Comments

No comments yet.

Sign in to leave a comment