Production Defaults and Graceful Shutdown — Measured on Real Sockets

Goal

Growing a single /root/work/srv/server.mjs, you measure with a real socket the five things a server working behind a front end must decide for itself — input ranges, error responses, body size, time limits, keep-alive — and the order in which it ends on SIGTERM. The last step is an incident.

Why it matters

Handler tests have no socket, so they never meet slow clients, large bodies, the front end's connection reuse, or deployment signals. Yet production's 502s, memory blowups, and "errors that happen only during deploys" all come out of that gap. And with these settings, writing the value and having it actually take effect are different — in this lab you measure for yourself a time limit that cuts late because of the check interval, and a close() that leaves connections that finished their response open.

The grader starts your server.mjs itself on a free port (the PORT environment variable) to test it, and cleans it up when done. Even if the server you have running is old code, it does not affect grading. When trying it yourself, start it with PORT=8081 node /root/work/srv/server.mjs & and clean up with kill %1.

Steps

1. Listen on PORT and answer /healthz, /work (integers 0–5000 only), and 404
2. Even when /boom and /boom-async throw, 500 JSON, no secrets or stack in the body, and the process survives
3. A 64KiB limit on the POST /echo body — 413 from the header first, and while counting for chunked
4. Cut header dragging at 5 seconds and body dragging at 10 seconds, but do not cut a normal 3-second request
5. keepAliveTimeout longer than the front end's idle time (60 seconds)
6. SIGTERM — stop listening, close idle connections, finish in-flight ones, exit with 0, a 4-second deadline
7. Incident nd-deploy — a 502 on every deploy, with a different cause per service

Notes

• When handling the socket directly, try writing headers one line at a time with python3 -c 'import socket; ...' or node -e. curl corrects the headers for you.
• Common mistakes: putting err.message as is into the response, trusting only Content-Length and forgetting chunked, changing only the time limit values and leaving the check interval, waiting only for the server.close() callback.
• The session is not saved — when the Pod ends, /root/work disappears too.

A server that listens on PORT and filters input

Create /root/work/srv/server.mjs. Read the port from the environment variable PORT and listen on 127.0.0.1 (the grader picks a free port and starts it itself). GET /healthz answers 200 and {"status":"ok"}, and GET /work?ms=<n> waits n milliseconds and then answers 200 and {"done": n}. If ms is not an integer from 0 to 5000 (negative, a decimal, letters, empty), 400; a path that does not exist is a JSON 404. Every response is JSON.

Direction: every value the server receives comes from outside — if it waits for ms=99999999 as is, a single request holds the server. Tools: createServer and listen(port, host) from node:http, new URL(req.url, 'http://local').searchParams, process.env.PORT. Specifics: if you bundle the response into one function with writeHead(status, {content-type, content-length}) and end(body), adding one more header in later steps is easy. To start it yourself, PORT=8081 node server.mjs & and then curl -s 127.0.0.1:8081/work?ms=300.

Thrown errors pass on neither the process nor secrets

Add two paths that blow up on purpose. GET /boom does throw new Error('db password=hunter2 failed at /srv/app/db.js') in the handler, and GET /boom-async throws an error of the same shape after one await. Both requests must answer with 500 and {"error": "..."} JSON, and the response body must contain no error message, stack, or file path. Instead, record it on standard error (console.error), and the process must keep working.

Direction: when the promise returned by the request handler is rejected, someone must receive it — if nobody does, Node 22 ends the process. Tools: route(req, res).catch(...), res.headersSent, console.error(err.stack). Specifics: if the headers were already sent by the time you reach the catch, you cannot fix the response, so cut the connection (res.destroy()); otherwise send a 500 with fixed wording. The moment you put err.message in the response, this step fails — the message contains a password.

Limit the body size before receiving and while receiving

Add POST /echo. It receives a JSON body and returns it as 200 and {"echo": <received-json>}, with a body limit of 64KiB (65536 bytes). If the Content-Length header exceeds the limit, 413 without waiting for the body; for a Transfer-Encoding: chunked body that does not declare its size, count the bytes received and 413 the moment it exceeds the limit; broken JSON is 400. A 60KB body must be accepted.

Direction: the earlier you reject, the cheaper it is. If you reject only after receiving the whole body something you could have rejected from the headers alone, you have already given away memory and time. Tools: req.headers['content-length'], for await (const chunk of req), throwing an error class that carries a status code (for example HttpError) and sorting it out in the catch from step 02. Specifics: when sending 413, add Connection: close — the rest of the body keeps arriving on the same connection, and that connection cannot be used for the next request. Testing: with curl -s -H 'Content-Length: 10485760' -d '{}' ..., curl corrects the header, so write the header yourself with python3's socket, as the grader does.

Keep slow clients from holding connections

The server must cut a connection that drags without sending all its headers at about 5 seconds (headersTimeout), and a connection that sent its headers but drags its body at about 10 seconds for the whole request (requestTimeout). The grader sends one header line or one body byte every 0.5 seconds and measures when it gets cut, and at the same time checks that GET /work?ms=3000 ends with 200 (the time to produce a response is not the client's fault). Header dragging must be cut at 3–7.5 seconds, and body dragging at 8–13 seconds.

Direction: if it does not cut on time even though you set the values, suspect not the value but "when it checks". Tools: http.createServer({ headersTimeout, requestTimeout, connectionsCheckingInterval }, handler), python3's socket to send headers one line at a time while timing with time.time(). Specifics: Node looks at these two times once every connectionsCheckingInterval (default 30000ms). With a 5-second limit and a 30-second interval, it holds out for up to 35 seconds — set the interval to about 1 second.

keep-alive longer than the front end

There is a load balancer in front of this server, and its idle connection time is 60 seconds. Set the server's keepAliveTimeout longer than that (65000ms or more). The grader reads the value of the server object that actually received the request, and leaves a connection idle for 7 seconds and then sends a second request over the same connection (with the default of 5 seconds, it would already be closed). The time limits from the previous step (headersTimeout 5 seconds, requestTimeout 10 seconds) must stay as they are, and you must not add Connection: close to normal responses.

Direction: first decide which side should close the connection — at that moment the front end does not know that the server closed the connection. Tools: server.keepAliveTimeout, the Keep-Alive: timeout= response header, and curl -v to watch the same connection being reused (Re-using existing connection). Specifics: keepAliveTimeout and headersTimeout are different clocks — the former is the idle time after a response, the latter is the time to receive the headers of a new request. Do not loosen one because you raised the other.

End gracefully on SIGTERM

When it receives SIGTERM, it stops accepting new connections, closes idle keep-alive connections right away, answers in-flight requests to completion and then closes those connections too, and when everything is done ends by itself with exit code 0. If there is a request that does not finish within 4 seconds, it cuts the remaining connections and ends (3.5–5.2 seconds). The grader sends SIGTERM while holding one idle connection and one 1.5-second request, and also checks the deadline with an upload that drags its body (and never finishes on its own).

Direction: measure for yourself what server.close() closes and what it leaves — when the callback comes is the crux. Tools: process.on('SIGTERM', ...), server.close(cb), server.closeIdleConnections(), server.closeAllConnections(), the response header Connection: close, setTimeout(...).unref(). Specifics: close() closes only the connections idle at that moment. A keep-alive connection that was working then stays open after its response, so the callback never comes — during shutdown, add Connection: close to responses so the connection closes along with the response. The deadline timer must be unref'd so that it alone does not keep the process around.

Incident — a 502 on every deploy

Running incident nd-deploy starts the incident. Three services behind the front end (ndproxy, 127.0.0.1:8080), catalog, checkout, and profile, each run as two instances (-a and -b), and every time you deploy (ndctl deploy <service>) the front end produces 502s. For each service, fix the SIGTERM handling in /srv/nd-deploy/<service>/server.mjs, restart it (ndctl restart <instance>), and make all three services produce no 502 even during deploys. Do not touch the front end's configuration or process, and keep keep-alive. In /root/incident/nd-deploy/report.txt, write three lines catalog: <code>, checkout: <code>, and profile: <code>, and for each service one line of the records (/var/log/nd/deploy.log, proxy.log) that shows its cause, as evidence-catalog: <line>, evidence-checkout: <line>, and evidence-profile: <line>. The cause is set separately for each service, so the same cause may appear twice. If you are wrong three times with different cause codes, the incident is withdrawn and you must start it again. The code is one of the following: drain-too-short (the deadline is shorter than the in-flight requests), exit-on-sigterm (it ends right away on SIGTERM), idle-left-open (it leaves connections that finished their response open), keepalive-shorter (keep-alive is shorter than the front end's).

Direction: the 502 lines look the same for all three. What separates them is when and how the old process ended after SIGTERM — right away, a little later, or never on its own and then forcibly. Tools: the instance lines in /var/log/nd/deploy.log (exit= · after · SIGKILL), reused= in proxy.log, the instance logs, the process.on('SIGTERM' part of the service code, and measuring again with ndctl deploy <service> after fixing. Specifics: compare each service with the shutdown order you built in step 06 and find what is missing. The deployment tool's slow request is 800ms (at grading time it is changed per instance to 0.5 seconds and 1.3 seconds, to catch "wait a fixed time and then end"), and it SIGKILLs 3 seconds after SIGTERM. Avoiding it by turning off keep-alive or adding Connection: close to every response is not a fix (the grader looks at the front end's connection reuse).