Tag: #troubleshooting
Writing on GPUs, LLMs, MLOps, Kubernetes — and mindset · 24 posts
24 posts are tagged #troubleshooting; the most recent was published on 2026-08-26.
Often tagged together: #kubernetes 9 #networking 9 #devops 6 #linux 6 #debugging 3
Most read in the last 90 days:
A Three-Line Config File Killed Four GPUs for a Week — How containerd Drop-in Merging Really Works ♪ Listenable
The nodes stopped advertising GPUs. The nvidia runtime was written out cleanly in the config file, but it was nowhere to be found in containerd config dump. The culprit was a three-line registry config I had added a week
2026-08-26 · 12 min read #kubernetes#containerd#gpu#nvidia#troubleshootingWhy Debugging Resists Replacement — The Craft of Elimination
Debugging runs in the opposite direction from writing code. Writing produces one possible thing; debugging erases possible causes until one is left. This piece covers why debugging sits on the expensive-to-verify side, h
2026-08-15 · 6 min read #career#skills#debugging#craft#troubleshootingA GPU Troubleshooting Playbook — Fix the Layers, Then Walk Down
The biggest waste in diagnosing GPU problems on Kubernetes is poking around without an order. A pod that will not schedule, a pod that runs but cannot see the GPU, a driver and toolkit version mismatch, memory exhaustion
2026-08-12 · 7 min read #gpu#kubernetes#troubleshooting#nvidia#gpu-operatorKorean Dev Blog Curation 2 — Incident Retrospectives and Troubleshooting, 12 Posts I Opened and Checked
The genre Korean developers write best is the incident retrospective. Twelve posts: p6spy silently defeating read/write datasource routing, why an HTTP timeout does not cover DNS resolution, a TCP half-close disguised as
2026-08-12 · 12 min read #curation#큐레이션#troubleshooting#postmortem#incidentFixing Kubernetes Pod Pending — How to Read the Rejection Reason the Scheduler Left Behind
When a Pod sits in Pending for minutes on end, the scheduler has already recorded the rejection reason in an event. We start with how to decode precisely that sentence beginning with 0/5 nodes are available, then rule ou
2026-07-26 · 14 min read #kubernetes#scheduler#pending#troubleshooting#autoscalingConnection refused vs. timeout — narrowing the cause at the TCP level
When a connection fails, the terminal hands back one of two messages. Connection refused means the peer returned an RST; timeout means nothing at all answered the SYN you sent. That one-line difference almost entirely de
2026-07-26 · 12 min read #network#tcp#troubleshooting#linux#kubernetesSpace Left but No space left on device — inode exhaustion, deleted open files, reserved blocks
df shows free space, yet a single file cannot be created and ENOSPC comes back. The cause is almost always one of five: inode exhaustion, files that were deleted but are still held open by a process, the reserved blocks
2026-07-26 · 17 min read #linux#filesystem#ext4#troubleshooting#storagedig works but the application fails — start by checking the DNS resolution order
A situation where dig answers normally but only the application cannot find the name is not a bug, it is structure. dig and nslookup bypass /etc/hosts and nsswitch.conf entirely and query the resolver directly. This post
2026-07-26 · 14 min read #network#dns#linux#kubernetes#troubleshootingDiagnosing Kubernetes CrashLoopBackOff by Cause — What to Look At When the Logs Are Empty
A start-to-finish walkthrough of the situation where a Pod is stuck in CrashLoopBackOff but kubectl logs prints nothing. It starts with the precise meaning of BackOff — not a cause, but a restart delay that grows from 10
2026-07-26 · 14 min read #kubernetes#crashloopbackoff#troubleshooting#kubectl#sreToo many open files solved for good — why raising ulimit does not help
Your server log fills with accept4 failed (24: Too many open files), you raise ulimit -n, and nothing changes. Behind this single symptom sit three independent limits: the per-process RLIMITNOFILE, the system-wide fs.fil
2026-07-26 · 15 min read #linux#systemd#file-descriptor#troubleshooting#containersWhy Load Average Is Not CPU Utilization — load average 24 With the CPU at 30%
Covers the situation where the load average from uptime is above 24 while the CPU in top does not even reach 30%. Unlike other Unix systems, the Linux load average counts not only runnable (R) tasks but also D state (uni
2026-07-26 · 14 min read #linux#performance#load-average#psi#troubleshootingKubernetes ImagePullBackOff and ErrImagePull Fully Dissected — Ending It With One Cause String
Breaks down, cause by cause, the situation where a pod passes through ErrImagePull and then sits in ImagePullBackOff. It starts from two points: that the two states are different stages of the same event, and that the an
2026-07-26 · 12 min read #kubernetes#imagepullbackoff#container-registry#troubleshooting#containerdWhen .gitignore is not working — the number one cause and a close reading of the pattern rules
You clearly wrote it in the ignore list, and yet the file keeps getting committed. The reason is almost always the same one: ignore rules do not apply to a file that is already tracked. This post covers the exact command
2026-07-26 · 12 min read #git#gitignore#troubleshooting#security#version-controlIngress Troubleshooting Playbook — From 502/504/404 to TLS
A practical playbook organizing the common Ingress symptoms (404, 502, 503, 504, TLS errors, redirect loops, 413) into symptom-based diagnostic trees. Each symptom comes with kubectl/curl/log inspection commands and fixe
2026-06-14 · 11 min read #ingress#kubernetes#nginx#troubleshooting#networkingSystematic Debugging — Finding Bugs by Reasoning, Not Guessing ♪ Listenable
Most debugging is undisciplined guessing. The core loop — reproduce, isolate, hypothesize, test, fix, verify — plus git bisect and binary search, how to read stack traces properly, the hard bug classes like heisenbugs an
2026-05-14 · 23 min read #debugging#methodology#engineering-craft#git-bisect#observabilityDNS Complete Guide: Name Servers, DDNS, nslookup — From Domain Registration to Troubleshooting ♪ Listenable
Everything about DNS in one post! How name servers work, practical domain registration and transfer guide, running home servers with DDNS, complete nslookup/dig usage, all DNS record types explained, DNS security (DNSSEC
2026-03-23 · 32 min read #dns#nameserver#ddns#nslookup#digsystemd Service Management, Unit File Configuration, and Troubleshooting
A practical guide to systemd service management, unit file configuration, and troubleshooting techniques for Linux system administrators.
2026-03-14 · 18 min read #systemd#linux#service-management#troubleshooting#devopsThe Complete Guide to TCP/IP Connection Debugging in Production
A practical guide covering TCP 3-way handshake, connection state analysis, SYN flood defense, packet capture with tcpdump/Wireshark, kernel tuning parameters, and real-world debugging scenarios.
2026-03-08 · 15 min read #networking#tcp#ip#troubleshooting#devopsThe Complete HTTP/HTTPS Troubleshooting Guide - Production Debugging Techniques ♪ Listenable
A comprehensive guide covering HTTP status code analysis, TLS handshake debugging, certificate chain validation, advanced curl techniques, 502/503/504 error resolution, CORS issues, redirect loops, HTTP/2 and HTTP/3 debu
2026-03-08 · 22 min read #networking#http#https#tls#troubleshootingComplete Guide to Cloud Network Architecture Troubleshooting — Practical Debugging for AWS, GCP, and Azure
Covers everything from VPC networking fundamentals, Security Group and NACL debugging, VPC Peering, Transit Gateway, Flow Logs analysis, NAT/IGW issues, cross-region connectivity, to DNS troubleshooting in cloud environm
2026-03-08 · 16 min read #networking#cloud#aws#troubleshooting#vpc