Tag: #gpu
Writing on GPUs, LLMs, MLOps, Kubernetes — and mindset · 69 posts
69 posts are tagged #gpu; the most recent was published on 2026-08-26.
Often tagged together: #nvidia 23 #kubernetes 22 #cuda 17 #2026-03 11 #deep-learning 10
Most read in the last 90 days:
A Three-Line Config File Killed Four GPUs for a Week — How containerd Drop-in Merging Really Works ♪ Listenable
The nodes stopped advertising GPUs. The nvidia runtime was written out cleanly in the config file, but it was nowhere to be found in containerd config dump. The culprit was a three-line registry config I had added a week
2026-08-26 · 12 min read #kubernetes#containerd#gpu#nvidia#troubleshootingvLLM Metrics — What to Chart and What to Alert On
The series vLLM exposes answer questions GPU metrics cannot: how many requests are running versus waiting right now, how full the KV cache is, how long until the first token. This post reads the official vLLM documentati
2026-08-12 · 7 min read #gpu#kubernetes#vllm#prometheus#observabilityMIG and Time-Slicing — Two Ways to Share One GPU
There are broadly two ways to put multiple workloads on one GPU: time-slicing, which divides time, and MIG, which divides hardware. Despite sounding similar, their isolation guarantees are nothing alike. This post works
2026-08-12 · 6 min read #gpu#kubernetes#mig#time-slicing#nvidiaDCGM Exporter — GPU Utilization Is Not What You Think It Is
DCGM Exporter is the standard path for exposing GPU telemetry in Prometheus format, but the utilization-style metric that ends up on nearly every dashboard does not measure what people expect. This post reads the default
2026-08-12 · 9 min read #gpu#kubernetes#dcgm#prometheus#observabilityA GPU Troubleshooting Playbook — Fix the Layers, Then Walk Down
The biggest waste in diagnosing GPU problems on Kubernetes is poking around without an order. A pod that will not schedule, a pod that runs but cannot see the GPU, a driver and toolkit version mismatch, memory exhaustion
2026-08-12 · 7 min read #gpu#kubernetes#troubleshooting#nvidia#gpu-operatorNVIDIA GPU Operator — The Six Pieces You Used to Install by Hand
Running GPUs on Kubernetes used to mean matching six pieces on every node by hand: the driver, the NVIDIA Container Toolkit, the device plugin, DCGM, GPU Feature Discovery, and Node Feature Discovery. The NVIDIA GPU Oper
2026-08-12 · 6 min read #gpu#kubernetes#gpu-operator#nvidia#dcgmGPU Serving SLOs and Alert Design — What to Promise and What to Page On
Putting an SLO on a GPU inference service starts with deciding which metric represents the user experience. Time to first token and throughput eat each other, so setting a target on only one guarantees the other collapse
2026-08-12 · 7 min read #gpu#kubernetes#slo#alerting#prometheusDevice Plugins and GPU Scheduling — Where nvidia.com/gpu Comes From
Kubernetes does not know what a GPU is. What makes a node advertise GPUs as a resource is a device plugin registered with the kubelet, and the name that falls out of it is the extended resource nvidia.com/gpu. This post
2026-08-12 · 6 min read #gpu#kubernetes#device-plugin#scheduling#nvidiaInside vLLM (7) — Deployment Tuning, Common Pitfalls, and OOM Triage
A practical order for tuning a vLLM deployment. Covers what gpumemoryutilization actually sets, when to use tensor parallelism versus pipeline parallelism, how to choose quantization and a KV cache data type, and a diagn
2026-08-12 · 10 min read #vllm#gpu#quantization#tensor-parallel#llmInside vLLM (2) — Why PagedAttention Splits the KV Cache Into Pages
Explains why PagedAttention splits the KV cache into fixed-size blocks, verified against the original paper (arXiv:2309.06180) and the official vLLM design documentation. Covers the internal and external fragmentation th
2026-08-12 · 7 min read #vllm#paged-attention#kv-cache#llm#gpuInside vLLM (1) — The Full Path From One Request to One Token
Follows the full path a single request takes through vLLM, from arrival to the first token out. Covers what the API server, scheduler, KV cache manager, worker, and sampler each do, and how the process layout splits afte
2026-08-12 · 6 min read #vllm#llm#inference#gpu#ai-platformGPU Compiler and Framework Landscape — One Problem, Turning a Graph into a Kernel, a Different Answer at Every Layer ♪ Listenable
This post puts NVCC and PTX, LLVM, MLIR, Triton, torch.compile, XLA, IREE, and TVM on one map. Different names, different owners, but they all solve the same problem: turning a computation graph into an executable kernel
2026-08-02 · 21 min read #gpu#compiler#mlir#triton#pytorchLabel Locality Scheduling in Ray 2.56 — Placement Groups Start Seeing NVLink Racks, Not Nodes
Ray 2.56.0, released on June 29, 2026, adds a domain-level scheduling layer to placement groups as an alpha feature. Until now every placement strategy — PACK, STRICTPACK and friends — operated strictly at node granulari
2026-07-17 · 11 min read #ray#gpu#scheduling#distributed-systemsHow Much VRAM to Run an LLM Locally — Compute It From Formulas, Not Tables ♪ Listenable
The answer to "how many GB does an 8B model need?" is not a table but two formulas. Weights are parameter count times bpw divided by 8; the KV cache is 2 times layers times KV heads times headdim times bytes times tokens
2026-07-17 · 24 min read #llm#quantization#kv-cache#local-llm#gpuTriton Gluon: The Language That Hand-Writes the Layouts the Compiler Used to Hide ♪ Listenable
Gluon is a low-level GPU language built on the same compiler stack as Triton — it hands kernel authors the layouts, shared memory, and warp specialization that Triton normally keeps hidden. Its reason for existing is cle
2026-07-16 · 19 min read #gpu#triton#kernel#compiler#performanceRust 1.97 Cut Off Pre-Volta GPUs — Inside the nvptx64 Baseline Bump
Rust 1.97 (July 9, 2026) raised the minimum requirements for the nvptx64-nvidia-cuda target to PTX ISA 7.0 (CUDA 11 driver or newer) and SM 7.0 (Volta or newer). Maxwell- and Pascal-generation GPUs, and CUDA 10 and older
2026-07-16 · 13 min read #rust#cuda#gpu#compiler#nvidiaRunning Small Models Hands-On with a Single RTX 5090 — microGPT, OCR, Music Generation
I SSHed into a single RTX 5090 (Blackwell, 32GB) and ran a trio of small models by hand. I trained a char-level GPT from scratch in 28 seconds (10.75M parameters, 1.17M tokens/s), pitted a dedicated OCR model (TrOCR) aga
2026-07-11 · 8 min read #pytorch#gpu#llm#ocr#hands-onWhy KubeVirt GPU Passthrough VMs Could Not Be Scheduled for 112 Days — A Real Cluster Postmortem
After a Rust operator delivered the diagnosis that "all 4 GPU nodes are NotReady," I traced the cause of that death all the way to the end on a real 8-node cluster (GPU Operator v25.3.0, KubeVirt v1.7.0). The real reason
2026-07-11 · 5 min read #kubevirt#gpu#kubernetes#nvidia#devopsDiffusion LLMs That Write CUDA Kernels — DICE and Why Parallel Generation Might Help
DICE, a February 2026 preprint, claims that diffusion large language models beat autoregressive models of the same size at generating CUDA kernels, setting a new state of the art. The core idea: instead of writing one to
2026-07-11 · 6 min read #ai#llm#diffusion#cuda#gpuBuilding a Kubernetes GPU Operator in Rust — Diagnosing a Real Cluster with kube-rs ♪ Listenable
Against a production 8-node homelab cluster (k8s v1.32.5), I used kube-rs to build and run a GPU operator in Rust myself. I defined a GpuInventory custom resource and launched two controllers (node scan → record CR statu
2026-07-11 · 6 min read #rust#kubernetes#operator#gpu#kube-rs