What breaks before the driver: kernel headers

In one line

What breaks most often in a driver import is not the driver but the kernel headers. DKMS builds the module with a build tree of exactly the same version as the running kernel, and if the imported headers differ by even one character, it stops before compilation begins. On a server with Secure Boot on, the import procedure also includes the steps of signing the module thus built and enrolling that key in the firmware.

Why this was needed

The earlier modules practiced the driver's dependencies and installation order with dummy .deb files, because the real NVIDIA driver cannot be redistributed. But accidents in the field usually happen after that. "The driver package installed, but nvidia-smi says couldn't communicate with the NVIDIA driver" — the kernel module was not built. Follow the cause and it is one of these: the headers are missing, the headers you downloaded are linux-headers-generic (a metapackage pointing to the newest kernel) and so differ from the running kernel, or the module was built but Secure Boot refused the unsigned module.

How it works

The rules of DKMS. According to the DKMS documentation, you put the source at /usr/src/<PACKAGE_NAME>-<PACKAGE_VERSION>/, and the dkms.conf inside must have PACKAGE_NAME and PACKAGE_VERSION. If there are several modules, you need BUILT_MODULE_NAME[#] (without .ko), and DEST_MODULE_LOCATION[#] must start with /kernel, but on Ubuntu, Fedora, RHEL, and so on it is ignored (the module goes where the distribution decides). With AUTOINSTALL="yes", autoinstall builds again when a new kernel comes in. If you do not write MAKE[#], the make of the kernel build tree is used and KERNELRELEASE is attached. Current DKMS warns you not to use CLEAN.

The order of commands. dkms add registers the source, dkms build -k <커널> (the placeholder is the kernel) builds the module in /var/lib/dkms/<이름>/<버전>/<커널>/<아키텍처>/module/ (the placeholders are the name, the version, the kernel, and the architecture), and dkms install moves it into the kernel module tree. If the headers of the target kernel are missing, it stops with Your kernel headers for kernel <버전> cannot be found at /lib/modules/<버전>/build or /lib/modules/<버전>/source, as the documentation says (the exit code in the dkms source is 21, but dkms 3.0.11 on this lab VM ended the whole command with 1 — in a script, check that it is nonzero, not the code value). The shape of dkms status differs slightly by version — current versions print 이름/버전, 커널, 아키텍처: installed (the fields are the name/version, the kernel, and the architecture).

Headers exactly as the kernel string. Ubuntu's headers are split into linux-headers-<uname -r> (per flavor) and the linux-headers-<버전> (common) that it requires (the placeholder is the version). The import list must start from uname -r. linux-headers-generic is convenient, but it follows the newest kernel on the day of the download, so it easily diverges from the kernel of an air-gapped server.

vermagic. A module contains a vermagic string recording which kernel it was built for, and you can see it with modinfo -F vermagic. A module for a different kernel is not loaded. modprobe loads even the dependent modules using the list depmod made, while insmod loads just one file.

Secure Boot and MOK. According to the Ubuntu documentation, on a system with Secure Boot on, DKMS automatically signs a new module with an MOK (Machine Owner Key). The keys are /var/lib/shim-signed/mok/MOK.priv and MOK.der, and if they do not exist it creates them and files an enrollment request. Enrollment is complete only when you enter the password on the MokManager screen at the next boot — for an air-gapped server with remote access only, this means a console task must be put into the import schedule. mokutil --sb-state shows the state, and mokutil --import creates an enrollment request for a DER key. In enforcing mode, an unsigned module is refused with Key was rejected by service, and a kernel that does not enforce loads it while leaving module verification failed: signature and/or required key missing - tainting kernel.

What it looks like in the field

I measured this in this lab VM (Ubuntu 24.04, kernel 6.8.0-139-generic, dkms 3.0.11). When headers were missing and I asked for a build for a kernel that does not exist, the next line after Error! Your kernel headers for kernel 6.8.0-9999-generic cannot be found at ... was Please install the linux-headers-6.8.0-9999-generic package or use the --kernelsourcedir option. A module built with the imported headers went in compressed as /lib/modules/6.8.0-139-generic/updates/dkms/airgap_hello.ko.zst. The interesting part is the signature. On a VM that cannot use Secure Boot (mokutil --sb-state gives EFI variables are not supported on this system), DKMS still signed the module and the modinfo -F signer was <호스트이름> Secure Boot Module Signature key (the placeholder is the host name). Yet at load time the kernel left module verification failed: signature and/or required key missing - tainting kernel. The signature exists but that key is not in the list of keys the kernel trusts — in enforcing mode it would have been refused here, and that is why MOK enrollment is part of the import procedure.

If the kernel of an air-gapped GPU server is updated between imports, the driver is unchanged but from the next boot the module is gone. Even if AUTOINSTALL tries to build again for the new kernel, the headers of that kernel were not imported. The kernel package and the headers of that kernel must always come in the same import bundle. The VM in this lab cannot turn on Secure Boot, so the signing step is confirmed only from records — that limitation is written in the lab instructions as well. Also, in this VM's cloud image, the headers of the running kernel (6.8.0-139-generic on the day I measured) were installed from the start through linux-headers-virtual. An air-gapped server is usually not like that, so the lab preparation step removed those headers so that you start from a "server without headers".

What you will do in the next lab

You check the running kernel and, without installing, download the two header packages of that kernel and make a hash list. With the outside blocked, you install the headers from the imported .deb files, then add, build, and install a 20-line GPL module with DKMS and load it with a parameter. You record the vermagic, the signature, and the Secure Boot state, and ask for a build for a kernel without headers to capture DKMS's error.

Reference documents: dkms(8) · Ubuntu — UEFI Secure Boot · mokutil(1) · Kernel module signing · modprobe(8)