TT Lab
Get started
Learn Learning paths Courses

RHEL-Family Administration

Modularity and module_hotfixes

Continue in TT Lab

In one line

Modular RPM has an additional layer on top of ordinary RPM dependencies. Without that layer's information, a package is there as a file but cannot be found in queries.

Why this exists

On an air-gapped network, a report like this comes in.

ls /srv/repo/appstream/ | grep nodejs
# nodejs-18.20.4-1.module+el9.4.0+21212+d9e3c1f2.x86_64.rpm

dnf list available nodejs
# No matching Packages to list

The file is there. Yet dnf says it is not. And there is no error either.

How it works

Why this happens

The causation has five steps.

  1. You downloaded AppStream with dnf reposync but did not add --download-metadata
  2. All the package files came, but the module metadata did not
  3. createrepo_c makes only the ordinary metadata from the rpm files. Module data is not inside the rpm, so it cannot be made
  4. The inside dnf sees the modular RPMs yet does not know which stream they belong to
  5. As a result of filtering, those packages cannot be found

The RHEL 9 documentation describes module dependencies as "an additional layer on top of ordinary RPM dependencies" and explains that they "behave like virtual dependencies between repositories".

Differences by version

Item RHEL 8 RHEL 9 RHEL 10
Module delivery The core approach of AppStream From 9.1, as additional versions with shorter lifecycles No modules chapter in the official documentation
Default stream Yes No predefined default stream Not applicable
Install without specifying a stream The default stream is enabled automatically You must specify a stream Not applicable
Switching streams distro-sync → module reset → module enable → distro-sync One line, dnf module switch-to Not applicable

Going from RHEL 8 to RHEL 9, the weight of modules shrank greatly, and in RHEL 10 they have effectively disappeared. But RHEL 8/9 systems will remain for a long time.

Commands

dnf module list
dnf module list nodejs
dnf module info nodejs:20
dnf module enable nodejs:20
dnf module install nodejs:20/common
dnf module reset nodejs
dnf module switch-to nodejs:20      # RHEL 9
dnf module list --installed > state-modules.txt

reset initializes the enabled state. To change a stream, reset usually comes first.

Three solutions

Method 1 — download it together (the standard)

dnf reposync --repoid=...appstream-rpms --download-path=/srv/sync \
  --download-metadata --gpgcheck --arch=x86_64 --arch=noarch

In this case you must not run createrepo_c again after the import. If you do, you actually lose the module data.

Method 2 — dnf modulesync

While downloading the packages that belong to a module, it builds a repository that includes the module data.

dnf --destdir=/srv/bundle/nodejs modulesync nodejs:20/minimal --resolve

Method 3 — module_hotfixes=1 (a last resort)

[airgap-appstream]
name=RHEL 9 AppStream (airgap, no modular metadata)
baseurl=file:///srv/repo/appstream
enabled=1
gpgcheck=1
module_hotfixes=1
metadata_expire=-1

The definition in the documentation: "Disables module RPM filtering and makes all RPMs in the repository available. The default is False."

There is a price. Packages from different streams can be installed mixed together, and that combination is not one Red Hat has tested. That is why it is a "last resort".

When you use installroot

When you calculate against an empty root, you must specify the module platform ID as well.

dnf download --installroot=/var/tmp/airgap-root --releasever=9.4 \
  --setopt=module_platform_id=platform:el9 \
  --resolve --alldeps --destdir=/srv/bundle nodejs

If you do not specify it, that value comes from /etc/os-release of the empty root, but at creation time that file does not exist, so the value is empty and the module content drops out entirely without an error. Because it drops out silently, it is the hardest kind of incident to discover.

grep PLATFORM_ID /etc/os-release
# PLATFORM_ID="platform:el9"

What it looks like in the field

Four kinds of state records. If you leave these four before and after an import, investigation becomes much easier.

dnf module list --installed > state-modules.txt
rpm -qa --qf '%{NAME}\t%|EPOCH?{%{EPOCH}}:{0}|\t%{VERSION}\t%{RELEASE}\t%{ARCH}\n' | sort > state-packages.tsv
dnf history userinstalled > state-userinstalled.txt
dnf history list > state-history.txt

The third is especially useful. Only what the user explicitly installed comes out, so if you install just that list on another server, the dependencies follow automatically. It is far safer than pushing in the whole package list as it is.

What you will look at in the next check

This module ends with a quiz that tests your judgment on concepts. After telling apart the symptoms that appear when module metadata is missing from the scope of risk of module_hotfixes, you choose the repository replication method in the rh-createrepo and rh-airgap labs of the next module.