TT Lab
Blog Get started
Learn Learning paths Courses

Networking Fundamentals

Speak HTTP by hand

Continue in TT Lab

Goal

HTTP is not difficult. It is just exchanging agreed-upon text over TCP. You keep that agreement by hand, and at the end you take the faults of three sites that monitoring reported only as a single "FAIL", tell them apart by which field of the response went out of line, and fix them.

Why it matters

Because libraries and browsers keep the agreement for you, when something goes out of line you can no longer read what went out of line. Behind "the page won't load" hide completely different responses: a 200 answered by someone else's site, a 301 that goes around forever, a body whose end never comes. Only people who have read the status line, headers, and body by hand once can tell those differences apart in seconds.

Materials

/opt/lab/netplan/httpsrv.py   경로마다 일부러 다르게 대답하는 작은 서버 (인자: 포트)

Start this server on 127.0.0.1:8080 and keep it running.

Path How it answers
/ Announces the length with Content-Length
/chunked Sends in chunks without knowing the length
/slow Sends the first chunk and then pauses
/nolength With neither a length nor chunks, signals the end by closing the connection
/moved /seeother /keepme 301 · 303 · 307

Steps

  1. A request written by hand and the response for / → /root/http/01-raw.txt
  2. The results of HTTP/1.1 without Host and of HTTP/1.0, and why Host is mandatory → /root/http/02-nohost.txt
  3. Three ways of signaling the end of the body (chunked kept as the original) → /root/http/03-length.txt
  4. Two requests on one connection, and what reuse saves → /root/http/04-keepalive.txt
  5. The responses for 301, 303, and 307, and whether the method may be changed → /root/http/05-redirect.txt
  6. The per-segment times of / and /slow, and which segment grew → /root/http/06-timing.txt
  7. Incident nw-web-broken: fix the three sites through configuration only and record a cause code for each site in /root/incident/nw-web-broken/report.txt

Notes

Type the request by hand

Start the lab server /opt/lab/netplan/httpsrv.py on 127.0.0.1:8080, then, without a library, write the HTTP request string yourself over a TCP connection, receive the response for /, and save it to /root/http/01-raw.txt. Save both what you sent and what you received.

The server is a Python script that takes the port number as an argument; start it in the background. Then, in bash, opening /dev/tcp/127.0.0.1/8080 for reading and writing is itself a TCP connection (binding it to a numbered fd with exec is convenient). Write the request line GET / HTTP/1.1, the Host header, and Connection: close, and read from that fd.

Line endings must be \r\n, and there must be one more blank line at the end of the headers. That blank line is the signal that "the headers end here".

Try leaving out Host

Send an HTTP/1.1 request without the Host header, and send the same request as HTTP/1.0 too, and save the results to /root/http/02-nohost.txt. Write one line on why it became mandatory from 1.1.

In 1.1, 400 Bad Request is the rule. It is the specification, not the server's taste.

It became mandatory when hosting several sites on one address became possible. The server can tell which site a request was sent to only from Host.

Where does the body end

Receive each of /, /chunked, and /nolength, and save three ways of signaling the end of the body to /root/http/03-length.txt. Do not process the chunked one; save it as the original bytes exactly.

curl merges chunked for you automatically, so the original is not visible. Receive it directly with /dev/tcp.

Each chunk is preceded by a hexadecimal size line, and at the end a chunk of size 0 signals the end. Without that 0, the receiving side keeps waiting.

Twice on one connection

Keep one connection open, send two requests, receive the two responses, and save them to /root/http/04-keepalive.txt. Write one line on what is saved by reusing the connection.

If you leave out Connection: close, the server keeps the connection. After reading the first response for the length given in Content-Length, send the second request on the same fd.

If you do it in Python, it is easy to read exactly that length.

How do 301, 303, and 307 differ

Receive each of the three paths (/moved, /seeother, /keepme), save the responses to /root/http/05-redirect.txt, and write the differences among the three. In particular, write whether the method may be changed when requesting again.

All three give a Location, but the way of requesting again differs.

When a payment request meets a redirect, which of these it is separates an accident from normal operation.

Point at the slow segment with numbers

Measure each of / and /slow with curl's timing variables, save them to /root/http/06-timing.txt, and write which segment grew longer. Also write that these values are cumulative.

curl's -w option prints measurements in a format you define after the request finishes. Print the four times for name lookup, connection, first byte, and total (time_namelookup, time_connect, time_starttransfer, time_total) on one line in the form 이름=값 (name=value), and discard the body instead of showing it on screen.

The numbers are cumulative. If time_connect is 0.32, it means it took 0.32 seconds up to the connection, not that the connection itself took 0.32 seconds. You get the time per segment by subtracting.

/slow rests after sending the first byte. Then ttfb is fast and only total grows.

Incident: monitoring calls three sites FAIL in the same way

Running incident nw-web-broken makes three sites hosted on one web server (127.0.20.1:80), wiki.nw.test, blog.nw.test, and docs.nw.test, all FAIL in the same way in monitoring (nw-mon: does the first screen show that site's front page title, for example wiki 대문 (wiki front page), within 3 seconds?). Names and TCP connections are fine for all three. Make all three OK by fixing only the site configuration /etc/nw-web/sites/<wiki|blog|docs>.conf (the server rereads the configuration on every request). Modifying the page files (/srv/nw-web/), replacing the server at that address, or pointing the names at a different server is not a fix.

In /root/incident/nw-web-broken/report.txt, write one line <사이트>: <코드> (site: code) per site (three lines) and evidence: <curl 출력 한 줄> (one line of curl output). The candidate codes are body-length (the declared body length differs from the actual one), chunked-end (the end marker of chunked transfer is missing), host-header (the site cannot be picked by Host), redirect-loop (redirects never end), and slow-backend (the server produces the first byte late).

Direction: all three are in the application layer (HTTP). Different responses hide behind monitoring's single FAIL, so peel apart the status line, headers, and body separately for each site. The things you looked at by hand in the earlier steps (Host, the end of the body, 301, timing variables) come back exactly.

Tools: curl -v (the Host you sent and the headers you received), curl -L (follows redirects), and curl -w's %{http_code} %{num_redirects} %{size_download} %{time_starttransfer}.

Concretely: if it is 200 but not the front page, compare whose site the received body belongs to, and each configuration's server_name with the request's Host. If Maximum ... redirects followed appears, follow the Location header by hand one hop at a time and see whether the redirect lines point at each other. If it is cut off when time runs out, compare the "this many of that many bytes" in the error line with the response header's Content-Length; a late first byte (a large time_starttransfer) and an end that never comes are different causes.