One mirror line and the JVM trust store

In one line

Maven's answer to air-gapped networks is to redirect every remote request to the internal repository with a single mirror entry in settings.xml. The internal repository can be a web server that simply lays files out at the specified paths, but the import list must include not only libraries but also the plugins used by the build, and if it is HTTPS, the JVM must trust that private CA.

Why this was needed

Java builds in an air-gapped network usually fail twice. The first time with Could not transfer artifact ... from/to central, and after you change the repository address to the internal one, with PKIX path building failed. To get past the second, many places bake verification-disabling options such as -Dmaven.wagon.http.ssl.insecure=true into CI. And it fails once more — on the mvn clean package that developers always type. That is because the import list was made with package alone.

How it works

Two places for settings.xml. According to the Maven documentation, there is a global ${maven.home}/conf/settings.xml and a user ${user.home}/.m2/settings.xml; if both exist they are merged, with the user one taking precedence. You can give a different file with -s and -gs.

mirror and mirrorOf. A <mirror> has id, mirrorOf, and url. For mirrorOf you can use * (all repositories), external:* (everything except localhost and file repositories), external:http:* (from 3.8.0), a list such as repo1,repo2, or an exclusion such as *,!repo1. When several mirrors match, the one whose id matches exactly comes first, and otherwise the one declared first wins. In an air-gapped network, you set mirrorOf to * so that every repository anyone has written into a POM is sent to the internal one. From 3.8.1, the global settings include a maven-default-http-blocker mirror that blocks external HTTP repositories, so if you stand up the internal repository over http you also have to wrestle with this block — it is better to use HTTPS.

A repository is a file layout. The path is the groupId with its dots replaced by slashes, followed by artifactId/version/artifactId-version.jar. So if on the connected side you build once with a new local repository (-Dmaven.repo.local=...) and put that directory on a web server, it becomes the internal repository. You use a new local repository to keep files downloaded earlier from getting mixed in and inflating the list, and, conversely, to keep things that were skipped because they already existed from dropping off the list.

_remote.repositories. In the local repository, a _remote.repositories file is created that records, for each file, from which repository id it came (the tracking file of Maven Resolver). The Resolver documentation says that even a file downloaded from R1 is treated as missing and downloaded again if the current build does not define R1. That is why you strip this file when moving to the internal repository, and conversely, by looking at this file in the air-gapped side's local repository, you can confirm whether it really came from the internal mirror.

The JVM's trust store. JSSE looks for a trust store in the order of the javax.net.ssl.trustStore property, jssecacerts, and cacerts. From JDK 9, keytool points directly at that store with the -cacerts option, and the initial password of cacerts given in the documentation is changeit. On Ubuntu, ca-certificates-java hooks into the operating system's update-ca-certificates and updates /etc/ssl/certs/java/cacerts as well. A JVM that uses its own lib/security/cacerts, such as an official JDK tarball, is not affected by this hook.

Translated to Nexus, it is the same structure to bundle a proxy that caches the central repository and a hosted repository that holds internal artifacts into a group, and write the address of that group in the url of the mirrorOf * entry.

What it looks like in the field

I measured this in this lab image (Maven 3.8.7, JDK 21). When I ran package on a project that uses a single gson with a new local repository, 51 jars and 20MB were collected, and most of that was plugins and their dependencies. If you do not write plugin versions in the POM, the default bindings of 3.8.7 (compiler 3.1 and so on) are used, and that version does not know maven.compiler.release, so the build broke on JDK 21. Pinning versions is also pinning the import list.

When I pointed the internal mirror to HTTPS, the build stopped with PKIX path building failed ... unable to find valid certification path to requested target. Once I put the root into cacerts with keytool, the same command passed, and in the _remote.repositories of the air-gapped side's local repository, gson-2.11.0.jar>airgap-internal= was recorded. When I tested the route of putting it in the operating system store, update-ca-certificates also put it into the JVM cacerts under the alias debian:airgap-os.pem.

The last trap was mvn clean package. Because the import list had been made with package, maven-clean-plugin 2.5 was missing and it failed with Could not find artifact. Even after I downloaded that plugin outside, put it into the internal repository, and ran again, it failed this time with was not found in ... during a previous attempt. This failure was cached in the local repository. That is because the 404 is recorded in the local repository and Maven does not ask again until the update interval has passed. Forcing it to ask again with -U made it pass. The import list must be made with every goal you will actually run, and after an additional import you need -U.

What you will do in the next lab

You build a project that uses gson with a new local repository to collect the files, and strip the tracking files to make /srv/maven. You issue a maven.airgap.internal certificate with a private CA, serve HTTPS with nginx, and redirect every request through settings.xml. After recording the PKIX error, you make the JVM trust the CA and get the air-gapped side's build to pass. Finally, you additionally import the clean plugin and finish with -U.

Reference documents: Settings Reference · Using Mirrors for Repositories · Maven 3.8.1 Release Notes · Repository Layout · Resolver Local Repository · keytool · JSSE Reference Guide · ca-certificates-java