Air-Gapped GPU Driver Installation
The Shape of the Driver Package Set
In one line
The NVIDIA driver is not one package but a bundle in which four strands are intertwined: the kernel module, the user-space libraries, the utilities, and the display driver.
Why this was needed
It seems you need only download nvidia-driver-550, but that is a metapackage. It contains not a single actual file, only a list of dependencies. Follow that list and a dozen or so come out.
How it works
Four strands
nvidia-driver-550 (메타)
├─ nvidia-kernel-dkms-550 커널 모듈 소스 (DKMS 로 빌드)
│ ├─ dkms
│ ├─ nvidia-kernel-common-550
│ └─ linux-headers-* <- 커널 버전에 묶인다
├─ nvidia-utils-550 nvidia-smi 등
│ ├─ nvidia-compute-utils-550
│ └─ libnvidia-compute-550 CUDA 런타임 (Provides: libcuda1)
│ └─ libnvidia-cfg1-550
├─ libnvidia-gl-550 OpenGL/GLX/EGL
└─ xserver-xorg-video-nvidia-550 X.Org 드라이버
Three traps from the air-gapped point of view come out here.
1. linux-headers is tied to the kernel version. For DKMS to build the kernel module, it needs the headers of the running kernel. But when the kernel is updated, the name of the headers package changes too. If the kernel at import time and the kernel of the install target differ, the build fails. So checking uname -r on the target server before import is the first step.
2. Some dependencies are satisfied through Provides. libnvidia-compute-550 provides libcuda1. So even if a package requires libcuda1, there is no package named libcuda1 in the repository. It is common to wander around searching for "libcuda1 is missing" without knowing this relationship.
3. The container toolkit is a separate bundle.
nvidia-container-toolkit
├─ nvidia-container-toolkit-base nvidia-ctk, 런타임 바이너리
└─ libnvidia-container-tools nvidia-container-cli
└─ libnvidia-container1 공용 라이브러리
The driver and the toolkit have different repositories and different version schemes. The driver comes from the distribution repository or the NVIDIA repository, and the toolkit from NVIDIA's separate repository. Mistakes where only one side is prepared are frequent when importing.
How to resolve the graph
The tools on the apt side.
apt-get install --print-uris -y nvidia-driver-550 # 받을 URI 목록만
apt-get install --download-only -y nvidia-driver-550 # 받기만
apt-cache depends --recurse --no-recommends --no-suggests nvidia-driver-550
apt-rdepends nvidia-driver-550 # 별도 패키지
dpkg-deb -f pkg.deb Depends # 파일에서 직접 읽기
The closure check is the key. Is everything that every package in the bundle requires entirely inside the bundle? If even one points outside, installation stops inside. Building a repository and running apt-get install -s (simulation) is the surest check.
Producing the installation order
When you install one by one with dpkg -i, dependencies must come first. It is a problem of topological sort.
libnvidia-cfg1-550
libnvidia-compute-550
nvidia-compute-utils-550
nvidia-utils-550
libnvidia-gl-550
xserver-xorg-video-nvidia-550
nvidia-kernel-common-550
dkms
linux-headers-...
nvidia-kernel-dkms-550
nvidia-driver-550
If you can use apt, apt decides this order by itself, so you only need the list. But knowing how to produce the order is different from leaving it to apt — to read where it broke off when apt fails, you have to understand the graph.
What it looks like in the field
DKMS build failure. The import succeeded, but the kernel module build fails during installation. In the log, the headers are missing or the compiler version does not match. That is why GPU nodes often have kernel updates pinned with hold — when the kernel goes up, the driver has to be rebuilt, and in an air-gapped network that is yet another import.
Secure Boot. An unsigned kernel module is not loaded on a system with Secure Boot on. An additional MOK enrollment procedure is needed, and this requires physical console access.
What you will do in the next lab
From a source tree assumed to have been imported, you build the .deb files directly, parse the dependency graph, find the dependencies that are not resolved inside the bundle, and produce the installation order by topological sort.