A Knowledge Map for ML/AI Infrastructure and Kubernetes Engineers: With Numbers Measured Directly

Status: Draft (2026-10-07), map article. This article does not perform new measurements, but weaves together values measured directly by the authorand content verified through documentationfrom other articles in the same series into a single sheet. Next to each number, the article from which it was measured and the conditions under which it was measured are noted. All measurements were made using one or two small models, a single GPU (8GB class), and a disposable single-node cluster, so please focus on the shapes and principles rather than the absolute values. Items based on documentation or never measured are marked as "Documentation-based" or "Unmeasured". Authoring rule: Measured data is the default, and unmeasured items are marked as "Unmeasured".

Table of Contents

• 1. How to Read This Article
• 2. Single-Sheet Map: Eight Layers
• 3. Layer 1: GPUs and Hardware, Where Do Speed Limits Come From?
• 4. Layer 2: The Kubernetes GPU Stack
• 5. Layer 3: Scheduling and Multi-Tenancy
• 6. Layer 4: LLM Serving Engines, Understanding Through Numbers
• 7. Layer 5: Networking
• 8. Layer 6: Storage and Model Deployment
• 9. Layer 7: Observability and Troubleshooting
• 10. Layer 8: Controllers and Deployment Automation (Tools for Kubernetes Engineers)
• 11. How to Measure: Techniques Spanning All Layers
• 12. 4-Week Learning Path
• 13. Frequently Asked Questions in the Field and Interviews
• 14. What This Map Does Not Cover
• Reference: Articles Underlying This Map
• Date Verified

1. How to Read This Article

ML/AI infrastructure is not a single domain, but a field where multiple layers overlap. You cannot explain serving numbers without knowing GPU memory bandwidth, you cannot know why GPU jobs stall without knowing the Kubernetes scheduler, and you cannot fix either without knowing observability. This article summarizes "What you must know" for each layer and links to articles for deep reading.

There are two reading orders:

• ML/AI Infrastructure Engineers: Read sections 2 to 5 (hardware, stack, scheduling, serving) first, then move on to the measurement methods in section 9.
• Kubernetes Engineers: Read sections 3, 4, and 8 (stack, scheduling, controllers) first, and fill in how GPUs differ from general resources using sections 2 and 5.

2. Single-Sheet Map: Eight Layers

3. Layer 1: GPUs and Hardware, Where Do Speed Limits Come From?

Things to Know

1. Token generation is bottlenecked by memory bandwidth. Every time a single token is generated, the weights of all layers are read once, so 초당 토큰 수 ≤ 메모리 대역폭 ÷ 가중치 바이트it is memory bandwidth-bound.
2. Input (prompt) processing is compute-bound. When calculating many tokens at once, multiple tokens reuse the same weights.
3. VRAM is shared among weights, KV cache, activations, and CUDA graphs. What determines serving throughput is typically the share remaining for the KV cache.
4. Consumer GPUs and datacenter GPUs have different diagnostic features. ECC, row remapping, and profiling metrics may be absent.
5. The container's CUDA version cannot exceed the limit set by the host driver. If startup fails after uploading an image, look at the driver version first.

Directly Measured Numbers (Articles 21, 11)

Common Pitfalls

• Not comparing measured values against hardware limits. We realized an error called prefix cache hit when the input processing speed came out to be 20 times the GPU computation limit (Articles 14, 21).
• Mistaking PCIe generation drops in idle state for a downgrade. You must apply load and read again (Article 11).
• grep Xid Isolating the wrong line of errors. Catching a network card driver's chip revision line along with GPU driver errors that are not Xid. NVRM: Xid Narrow it down to the exact error (Article 11).

4. Layer 2: The Kubernetes GPU Stack

Things to Know

1. GPUs are extended resources. They are requested only as integers, requests and limits must be equal, and overcommitment and sharing are not allowed.
2. The stack builds from the bottom up. Host kernel module (driver), container runtime toolkit, device plugin (resource advertising), node labels (GFD), monitoring (DCGM exporter). The GPU Operator installs these all at once.
3. Sharing methods have different isolation levels. Time-slicing does not isolate memory, MIG partitions hardware only on supported GPUs, and software methods (such as HAMi) intercept CUDA calls (Documentation-based, Articles 3 and 10).
4. You can also pass GPUs entirely to virtual machines. In this case, the host cannot use the GPU, so nodes are divided between those for containers and those for virtual machines (Article 13).
5. Separating the responsibilities of the driver, toolkit, and operator speeds up failure analysis. Which layer failed depends on the symptom (nvidia-smi failure, Pod Pending, container start failure).

Common Pitfalls

• Installing the operator without deciding whether the host driver is installed by the operator or already present on the host. The two will conflict (Article 3).
• New CUDA images failing to start on old drivers. Indeed, a CUDA 13 image failed on a CUDA 12 series driver with an initialization error (Error 804) (author's test; general principles only in this article).
• GPU pods going Pending due to node memory request sums. Even if the GPU is empty, if general resources (memory) are lacking, the pod will not be scheduled. Read the reason for waiting message first.

5. Layer 3: Scheduling and Multi-Tenancy

Things to Know

1. The default scheduler decides on pods one by one. Jobs where multiple pods must start together (distributed training, multi-node inference) get slots for only some of them and end up waiting for each other.
2. Gang scheduling has two approaches. Kueue checks quotas per job and does not create pods until accepted, while Volcano uses a separate scheduler to allocate slots all at once only when the minimum number of pods in a pod group can be met.
3. Quotas, borrowing, returning, and preemption are design problems. Policies determine who can borrow remaining resources and who gets evicted when the owner returns.
4. LeaderWorkerSet provides group-level recovery. In multi-node serving, if one rank disappears, the rest become useless, so recreating the entire group is the default.

Directly Measured Numbers (Articles 10, 9)

Common Pitfalls

• Submitting large jobs without gang scheduling. Resources are held while waiting for each other, leaving the cluster idle.
• Measuring preemption and return times without knowing the termination grace period. If an evicted pod ignores termination signals, it runs through the full grace period (default 30 seconds) before ending, making the measured value equal to the grace period (actually experienced in Article 10).
• Believing that "submitting simultaneously with the same priority determines the order." Which job gets accepted first was not identical every time.

6. Layer 4: LLM Serving Engines, Understanding Through Numbers

This layer is the most frequently asked about and the most frequently measured. vLLM and SGLang were measured with a 1.7B model on a single 8GB-class GPU (Articles 5, 14, 15).

Things to Know

1. Continuous batching distributes the weight-reading cost of concurrent requests. Thus, increasing concurrency increases total throughput almost proportionally until it hits a wall at KV cache capacity.
2. The KV size per token is calculated by formula. 2 × 층 × KV 헤드 × head_dim × 바이트. The concurrency limit is estimated as KV 용량 ÷ 요청당 토큰.
3. Prefix caching skips input processing for common prefixes. It is the cheapest optimization for services that fix system prompts and documents at the beginning.
4. Chunked prefill determines how long a long input interrupts other users' tokens.
5. Quantization comes in two types. Weight quantization changes speed (bytes read), while KV quantization changes capacity (number of tokens held).
6. LoRA serves by placing small adapters on top of a single base model. The relationship between the number of simultaneously used adapters and slot counts determines performance.

Directly Measured Numbers

Rules of thumb from the field

• Set the concurrency limit close to the number of requests the KV cache can hold. Exceeding this limit does not increase throughput, only TTFT.
• Monitor TTFT at p95 and inter-token latency (ITL) at p99. Even if the median looks fine, the tail latency is large (ITL p99 is 223ms).
• Align default settings before comparing engines. Options with the same name can have different default values (chunk size, mixed chunks, memory settings).
• Route requests with overlapping prefixes to the same instance. The cache resides within the instance (Part 8, per documentation).

7. Layer 5: Network

What you need to know

1. As the number of services grows, iptables rules increase. Cilium's eBPF performs service load balancing using map lookups instead of rules.
2. Service meshes add proxy hops to every request. While negligible for LLM serving where TTFT is hundreds of milliseconds, it can become noticeable in responses streamed token-by-token or in short calls.
3. Inter-GPU communication is a separate layer. NCCL handles collective operations, and RDMA (InfiniBand or RoCE) bypasses the CPU between nodes.
4. A common issue with RDMA is configuration. Memory locking limit (memlock), MTU mismatch, GID selection, PFC/ECN.

Directly measured figures (Part 2, Part 7)

Common pitfalls

• Understanding "iptables is slow when there are many services" solely as datapath latency. In this environment, what made a big difference was the timing of updating rules all at once.
• Trusting tables where the same value appears repeatedly. Connection attempt limits have sometimes fixed measurement values to two specific numbers (be skeptical of the tool's resolution).

8. Layer 6: Storage and Model Deployment

What you need to know

1. Weight size is 파라미터 수 × 바이트. A 70B bf16 model is 140GB, so a shared cache or image strategy is needed to avoid downloading it on every node.
2. Cold start is the sum of image pulling, model downloading, loading, compiling, and warming up. In autoscaling design, this sum represents the time it takes for a new pod to start receiving traffic.
3. Offloading the KV cache outside the GPU (to CPU, disk, or remote) allows prefixes to be reused longer. However, it is only beneficial if the loading time is shorter than recalculation.
4. Weight encryption prevents storage leaks and tampering, but cannot protect memory during execution.

Directly measured figures (Part 8, Part 16, Part 14)

9. Layer 7: Observation and Troubleshooting

What You Need to Know

1. GPU metrics consist of three layers. Hardware (DCGM: utilization, power, temperature, clock throttling reasons, XID), server (queue, KV utilization, preemption count), service (TTFT, ITL, error rate).
2. Looking at utilization alone is misleading. A server bound by memory bandwidth may perform very little actual compute even when utilization is high.
3. XID is the error number left by the GPU driver in the kernel log. Regular accounts may have dmesg blocked, so they use journalctl -k instead.
4. Cilium and DCGM have many metrics, and cardinality is cost. Label combinations determine the number of time series (Parts 17 and 18).
5. Alerts should separate symptoms (TTFT tail latency) from causes (VRAM trend, clock throttling).
6. A signal that traces a single request is a trace. vLLM leaves queue, prefill, and decode times as attributes in a single span for each request, which alone allowed us to distinguish whether a slow request was due to queuing or computation. The overhead of enabling it was within the margin of error, and capturing slow requests required tail-based sampling rather than probabilistic sampling (Part 25).

Common Pitfalls

• Treating N/A and 0 as the same. Unsupported metrics appear as N/A rather than 0 (Part 11).
• Comparing memory numbers without realizing that KV capacity changes on every server restart. Compare using bytes per token (Part 14).

10. Layer 8: Controller and Deployment Automation (Tools for Kubernetes Engineers)

What You Need to Know

1. Level-based reconciliation. Instead of relying on events, it reads the current state, compares it with the desired state, and reconciles them. Even if an event is missed, the next reconciliation fixes it.
2. Idempotency. If the state already matches, it should write nothing.
3. Owner references and finalizers. Children are cleaned up when their owner is deleted, and finalizers allow deletion only after cleanup is complete.
4. CRDs are designed through schemas and validation. Simple rules use CEL, while complex rules use webhooks.
5. In GitOps, if another controller modifies fields defined in Git, conflicts occur. HPA and replicas, ignoreDifferences(Part 9).

Numbers measured directly (Part 9)

The actual operational behavior of the operator (recovery time upon deleting children, idempotency, finalizers, duplicate execution, retries) will be reflected in Part 9, following direct test results.

11. How to Measure: A Technology That Cuts Across All Layers

The most valuable part of this series, beyond the numbers, is Moments I doubted the numbers.

And make three things a habit: Different inputs for each request, Iteration and variance(Never jump to conclusions based on a single measurement), Record the environment and method next to the value.

12. 4-Week Learning Path

If studying alone, we recommend this sequence. The deliverable for each week is a table measured with your own hands.

Perform these only in disposable environments (k3s in a virtual machine, temporary pods) and delete them when finished. Never try them for the first time in a production cluster.

13. Frequently Asked Questions in the Field and Interviews

1. How do you estimate the upper bound of LLM token generation speed? Memory bandwidth ÷ weight bytes. We achieved 86–98% of the bandwidth measured across four models.
2. What is the concurrent request limit of the server? KV cache token count ÷ tokens per request. Exceeding this does not increase throughput and causes TTFT to spike by 8x.
3. When is prefix caching most effective? Services with long common prefixes. For 3,000 tokens, the time to first token dropped from 394 ms to 28 ms.
4. How can you reduce long inputs from interrupting other requests? Reduce the chunk size in chunked prefill and check the engine's default behavior (whether chunked prefill is mixed).
5. GPU tasks are stalled, yet the GPU is idle. Why? Either some pods secured spots without gang scheduling (resulting in a deadlock), or the request is Pending due to insufficient general resources (memory).
6. How do you implement gang scheduling? Kueue uses work-unit-based approval, while Volcano uses a minimum pod group count.
7. What are the GPU sharing methods? Time-slicing (no memory isolation), MIG (hardware partitioning, supported GPUs), and software-based interception (API interception).
8. The container fails to start. What about the CUDA image and driver? The CUDA version in the container cannot exceed the host driver's upper limit.
9. What is the latency cost of a service mesh? Roughly 2.2 ms per request in this environment, with encryption accounting for about 8% of that.
10. What should you watch out for when serving multiple LoRA adapters? The number of concurrently used adapters versus the slot count. If slots are insufficient, throughput drops by half.
11. What does encrypting weights protect? Storage leaks and tampering. It does not protect memory during execution.
12. What happens if an operator misses an event? Level-triggered reconciliation re-reads the current state to align it (revived by timers or resyncs).
13. What is the first thing to do when benchmark results look abnormal? Compare them against hardware limits.

14. What This Map Does Not Cover

• Training infrastructure. Distributed training (data parallelism, tensor/pipeline parallelism), checkpoints, data loaders, and fault recovery during training were not measured in this series (unmeasured).
• Datacenter GPUs, NVLink/NVSwitch, and InfiniBand physical hardware, along with scheduler throughput at a scale of hundreds of nodes.
• Large models (tens of billions of parameters or more) and multi-GPU servingnumbers. The serving numbers in this article are based on a 1.7B model.
• ML platforms outside of serving(feature stores, experiment management, pipeline orchestration).

Note: Articles That Formed the Basis of This Map

Part 2 (Hybrid GPU Clusters and Networks), Part 3 (GPU Operator), Part 4 (LLM Inference and vLLM), Part 5 (vLLM and SGLang), Part 6 (Distributed Inference), Part 7 (RDMA), Part 8 (Storage and KV Offloading), Part 9 (CRDs and Operators), Part 10 (Multi-Tenant Scheduling), Part 11 (GPU Failure Diagnostics), Part 13 (GPU Passthrough), Part 14 (vLLM Load and KV Cache), Part 15 (LoRA Serving), Part 16 (Weight Protection), Part 17 (DCGM Metrics), Part 18 (Cilium Metrics), Part 21 (Principles of Speed).

Verification Date

As of 2026-10-07. Numbers follow the "Verified Versions and Dates" section of each article (vLLM 0.31.0, SGLang 0.5.21, Kueue 0.20.0, Volcano 1.15.3, LeaderWorkerSet 0.11.1, etc.).