TT Lab
Blog Get started
Learn Learning paths Courses

Linux Network Diagnosis

HTTP Diagnosis

Continue in TT Lab

Goal

You read an HTTP response along three axes, status code, headers, and timing, tie them together into a health check, and then tell apart "the monitor says FAIL but it works on my end" and "the name is supposedly right, yet there is a certificate error" using numbers and fields.

Why it matters

What is needed for a report that "responses are slow, or don't work" is not more dashboards but breaking a single request down into numbers. curl -w tells you the status code, the redirect count, and the time per segment separately. A health check must always have a maximum wait time; without it, the script stops forever when the other side does not respond, and such scripts pile up.

The two incidents at the end put those numbers to real use. On the monitor, all three are the single word FAIL, but in one the server's redirect rules go around in a loop, in another there is no virtual host matching the Host header, and in the third it is a proxy variable that exists only in the monitor process's environment: "who made the request, and with what environment" is half the diagnosis. In the certificate incident, you fix, without turning off verification, a connection that breaks because the SAN the client actually looks at differs, even though the subject (CN) is right.

Steps

Use /root/www as the document root and /root/http as the working directory.

  1. Put labhub-http-ok into /root/www/index.html, start an HTTP server with /root/www as its document root on 127.0.0.1:8000, and write its PID to /root/http/server_pid.txt.
  2. Write only the number of the status code for a request to a nonexistent path to /root/http/status404.txt.
  3. Receive only the response headers of /index.html and save them to /root/http/headers.txt.
  4. Save the three values time_namelookup, time_connect, and time_total of the request on one line separated by spaces to /root/http/timing.txt.
  5. Create the /root/www/sub directory, and write the status codes of http://127.0.0.1:8000/sub without following and with following redirects to /root/http/redirect.txt as no_follow=<code> and follow=<code>.
  6. Create /root/http/healthcheck.sh <URL>. For 2xx, ok code=<code> time=<seconds> and exit code 0; for non-2xx, down code=<code> and 1; for a connection failure, down … and 1. It decides within 10 seconds.
  7. Incident incident ln-http-monitor: the monitor sees three sites as FAIL in the same way. Make all three OK and report the cause for each site.
  8. Incident incident ln-http-tls: HTTPS breaks with a certificate error. Make it open without turning off verification, and report the cause.

Notes

Start a local HTTP server

Create /root/www/index.html containing labhub-http-ok, start an HTTP server with /root/www as its document root on 127.0.0.1:8000 in the background, and write its PID to /root/http/server_pid.txt.

python3's http.server module has options to choose the address to bind and the document root (-h). If you do not specify a document root, it serves the current directory.

Extract only the status code

Write only the number of the status code for a request to a nonexistent path to /root/http/status404.txt.

curl's write-out format (-w) has a variable that discards the body and prints only the response's number.

Receive only headers

Receive only the response headers of /index.html and save them to /root/http/headers.txt. The body must not be mixed in.

There is an HTTP method that returns only the headers without a body, and curl has an option to send that request.

Measure the time per segment

Save the three values time_namelookup, time_connect, and time_total of the request on one line separated by spaces to /root/http/timing.txt.

Put the three variables in -w's format string, joined by spaces, and output them on one line. The three values are cumulative times from the start of the request.

Follow redirects

Create the /root/www/sub directory and put one file in it. Write the status codes of requesting http://127.0.0.1:8000/sub without following redirects and with following them to /root/http/redirect.txt as the following two lines.

curl does not follow redirects by default. If you request a directory without a trailing slash, the server sends you to the address with the slash.

Health check script

Create /root/http/healthcheck.sh <URL>.

You must separate three cases: healthy, a response that came but is abnormal, and not reaching the server at all. Decide what code curl gives when it cannot reach the server, and after how many seconds to give up when the other side does not answer.

Incident: the monitor's three FAILs

After running incident ln-http-monitor, fix things so the monitor (/var/log/ln-web/monitor.log) sees all three of shop.lab.test, api.lab.test, and docs.lab.test as OK. You must not turn off the monitor or remove addresses from its list, and your fixes must remain in the configuration files so they persist across restarts. In /root/incident/ln-http-monitor/report.txt, write three lines, shop: <code>, api: <code>, and docs: <code>, plus evidence: <one observed output line>. Candidate codes: proxy-env (the requester's proxy environment variables), redirect-loop (redirects go around in a loop), server-down (the server does not respond), tls-name (the certificate's name is different), vhost-mismatch (no virtual host matches the Host header).

Direction: turn the single word FAIL into numbers. And "it works in my shell" means the requester is different. Tools: curl (-w, -H, -L), /proc, cat. Concretely: look at the numeric fields of the monitor log, a single response without following redirects, how the server picks a site by name, and the environment the monitor process actually has (/proc). The web configuration is read only once at startup, and the internal proxy policy itself must be respected.

Incident: the name is supposedly right, yet there is a certificate error

After running incident ln-http-tls, fix it so that https://secure.lab.test:9443/ opens with curl without turning off verification (-k is forbidden). The certificate must be signed by the internal CA (/etc/ln-web/tls/ca/) and must remain in the file that /etc/ln-web/tls.conf points to, so it persists across restarts. In /root/incident/ln-http-tls/report.txt, write cause: <code> and evidence: <one observed output line>. Candidate codes: ca-untrusted (the signing CA is not trusted), cert-expired (the certificate's validity period has passed), key-mismatch (the certificate and private key are not a pair), proxy-env (the requester's proxy environment variables), tls-name (the certificate's name differs from the name connected to).

Direction: curl's error message tells you at which TLS check it broke. The key is which field of the certificate that check looks at. Tools: curl -v, openssl s_client, openssl x509. Concretely: look at all the name-related fields of the certificate you actually received by connecting, not of the file, and recall which of them the client compares. When fixing it, first check who the issuer trusted by this server's clients is.