There is more than one trust store

In one line

Nearly every internal mirror in an air-gapped network uses a certificate issued by a private CA. But there is not one trust store. The operating system, the JVM, Python's certifi, and Node's built-in list each have their own. Putting a CA in one place does not make everything trust it, so you must know in a table which tool looks at which store.

Why this was needed

You stood up the internal mirror over HTTPS and put the root certificate on the server. curl works. But a Python batch job dies with SSLError: CERTIFICATE_VERIFY_FAILED, a front-end build with UNABLE_TO_VERIFY_LEAF_SIGNATURE, and a Java build with PKIX path building failed. The common reaction is to switch verification off with verify=False, strict-ssl=false, or -k. Then nobody will know if someone inside the internal network poses as the mirror. An air-gapped network only has fewer attacks from outside; it is not a place without attacks from inside.

How it works

The CA and the server certificate. A root CA certificate must have basicConstraints = CA:TRUE to be qualified to sign other certificates (OpenSSL x509v3_config documentation). A server certificate must have the name to be connected to in subjectAltName as DNS:. RFC 9525 says that using commonName for name verification is no longer valid, and Go has not treated CN as a name since 1.15. A certificate with the name only in CN is now rejected by current tools as a name mismatch.

The operating system store. Debian and Ubuntu put it in /usr/local/share/ca-certificates/ and run update-ca-certificates. The manual says the extension must be .crt, and the result is /etc/ssl/certs/ca-certificates.crt, concatenated into one file. The command also runs the hooks in /etc/ca-certificates/update.d/, and Ubuntu's ca-certificates-java hangs a hook there that updates even the JVM store (/etc/ssl/certs/java/cacerts). The RHEL family puts it in /etc/pki/ca-trust/source/anchors/ and runs update-ca-trust extract.

Where each tool looks.

curl            빌드할 때 정해진 파일 저장소(우분투는 운영체제 묶음). --cacert·CURL_CA_BUNDLE 로 바꾼다
파이썬 ssl       OpenSSL 기본 경로 = 운영체제 묶음. SSL_CERT_FILE·SSL_CERT_DIR 로 바꾼다
requests        certifi 묶음. REQUESTS_CA_BUNDLE(없으면 CURL_CA_BUNDLE) 로 바꾼다. SSL_CERT_FILE 은 읽지 않는다
httpx           certifi 묶음. SSL_CERT_FILE·SSL_CERT_DIR 를 따른다
Node            빌드에 들어간 모질라 목록. NODE_EXTRA_CA_CERTS 로 더한다(프로세스 시작 때 한 번 읽는다)
Go              리눅스에서는 운영체제 묶음. SSL_CERT_FILE·SSL_CERT_DIR 로 바꾼다
JVM             cacerts(우분투는 운영체제 훅이 채운다). keytool 이나 javax.net.ssl.trustStore

For Node, the documentation adds several points. NODE_EXTRA_CA_CERTS is ignored in a process started with setuid, and if the file is missing or malformed it issues a single warning and moves on. --use-system-ca, which makes Node use the operating system store, appeared in v23.8.0 (on Linux from v23.9.0), and the environment variable NODE_USE_SYSTEM_CA=1, which does the same thing, arrived in v22.19.0 and v24.6.0. The node in this lab is 22.11, so neither exists — which is why you check the version first.

The chain. In the field, a private PKI often has an intermediate CA under the root, and the intermediate CA issues the server certificate. The client store has only the root, so the server must send its own certificate together with the intermediate CA certificate for the path to connect. If you install only the leaf certificate on the server, you get unable to get local issuer certificate. That is a mistake in the server's configuration, not the client's.

What it looks like in the field

These are the results measured in this lab Pod. Before the root was put in the operating system store, curl, Python urllib, requests, httpx, Node, and pip all failed certificate verification. After update-ca-certificates, curl, urllib, Go, and pip passed, while requests and httpx still failed with CERTIFICATE_VERIFY_FAILED and Node with UNABLE_TO_VERIFY_LEAF_SIGNATURE. Once REQUESTS_CA_BUNDLE, SSL_CERT_FILE, and NODE_EXTRA_CA_CERTS were given, all passed. pip passed because the pip in this image is a version patched by Ubuntu and reads the operating system bundle.

So in the field you gather the environment variables in one place (a file in /etc/profile.d/, or the unit's Environment for a service), and when you receive a new server you leave behind a table made by connecting once with each tool. "We put it in the OS, so it's done" is only half right.

What you will do in the next lab

You create a private root CA and a server certificate with a SAN, and serve repo.airgap.internal:8443 over HTTPS. You put it in the operating system store and gather the environment variables in /etc/profile.d/airgap-ca.sh so that Python and Node trust it. When you write in a table, for each tool, whether "the operating system store alone is enough", the grader measures again under the same conditions and compares. Finally, you serve a certificate issued by an intermediate CA together with its chain.

Reference documents: OpenSSL x509v3_config · RFC 9525 · update-ca-certificates(8) · RHEL: Using shared system certificates · requests: CA Certificates · httpx SSL · Node.js CLI (NODE_EXTRA_CA_CERTS) · Go crypto/x509 · curl SSL CA certificates