Renewal tools change files; servers serve memory

In one line

The renewal tool changes a file, while the server serves the certificate held in memory. Without a restart (reload) connecting the two, the disk has the new certificate while the browser sees the old one. That is why renewal monitoring has to look at the certificate the server actually serves, not at the file.

Why you need this

An alert arrived three days before expiry. The log says the renewal job already succeeded yesterday. Yet the browser still shows a certificate that expires in three days. The owner opens the file — it is indeed the new certificate. This contradiction cannot be resolved unless you know when the server reads the certificate. From nginx, Envoy, and JVM apps to Python's ssl module, most servers load the certificate into memory once, at startup. They do not read it again when the file changes. The renewal tool overwriting the file and the server using it are separate events.

The same kind of question came up in this repository too. We changed the configuration so that cert-manager would renew certificates automatically, but with that configuration the system had never gone through a renewal. To answer "it will probably renew periodically," we had to actually trigger one and then see whether the date on the certificate the browser receives changes. The evidence for the verdict was neither the Pod status nor the logs, but the date on the certificate that openssl s_client retrieved.

How it works

Looking at it through Python's ssl module, the structure shows itself directly. SSLContext.load_cert_chain(certfile, keyfile) reads the files at the moment of the call and keeps them in the context, and every handshake after that serves what the context holds. Overwriting the file does not tell the context. To make it read again, you have to restart the process or use a reload signal the server provides (a structure that watches for file changes, such as nginx's reload or Envoy's SDS). On Kubernetes, when cert-manager renews a Secret, the file mounted as a volume changes after a while, but reading that file is still up to the app — an app with no reload logic needs its Pod restarted.

The observation tool is openssl s_client. You do the handshake with -connect host:port, send SNI with -servername, and then pass the received certificate to x509 to read the serial, expiry date, and fingerprint.

$ echo | openssl s_client -connect 127.0.0.1:8443 -servername www.lab.internal 2>/dev/null \
    | openssl x509 -noout -serial -enddate
serial=2037629D1C3A75C63253D1BE593E399BA192FF71      ← 서버가 내보내는 것
$ openssl x509 -in /root/renew/live.crt -noout -serial
serial=2037629D1C3A75C63253D1BE593E399BA192FF72      ← 디스크에 놓인 것

If the two serials differ, it is an incident. A monitoring script only has to compare these two values, and expiry monitoring has to apply -checkend to the certificate received with s_client, not to the file on disk. If you apply it to the file, a false green light turns on saying "it was renewed, so we are safe."

Renewal itself is signing a new CSR with the same key. Only the serial and validity change, so even without rotating the key the certificate becomes a new version (rotating the key as well is safer, but that is not the subject of this module). If you keep the validity period short, renewal happens often and this kind of incident surfaces early — if you keep it long, it blows up once a year, after the owner has changed.

What it looks like in the field

The symptom always starts with "they say the renewal succeeded." There are three steps to check. First, fetch the certificate the server serves with s_client and look at the serial. Second, compare it with the file on disk. Third, if they differ, find the reload method — depending on the process, the answer is a reload signal, a Pod restart, or an SDS update from a sidecar. If there are several servers behind a load balancer, sometimes only one serves the old certificate, so you put each server's address directly into -connect and check them one at a time. When you first turn on renewal automation, do not wait for expiry; force a renewal once and confirm all the way to the server serving the new certificate. Automation can be trusted only once there is an actual issuance history.

What you will do in the next lab

You start a Python HTTPS server with a 3-day v1 certificate and read the serial with s_client. You issue a 30-day v2 with the same key and overwrite the file, then observe that the server, which has not been restarted, still serves v1. You write certdiff.sh, which compares disk and server, and servedcheck.sh, which looks at the expiry of the certificate the server serves, and have them checked in both directions against the grader's server. After the restart, you confirm that v2 is served and write the two serials in the report.