Configuring nginx as a Reverse Proxy

Goal

You configure nginx as a reverse proxy and become able to set up forwarded headers, timeouts, body size, compression, static separation, access blocking and log format to a practical standard.

Why it matters

The six lines of proxy configuration should be remembered not by rote but by what happens when they are missing. If you don't pass Host, the absolute URLs the application builds become 127.0.0.1:8080 and payment callbacks don't come back. Without X-Forwarded-For, the access history is all the proxy IP, so there is nothing to say in an audit. Without X-Forwarded-Proto, a redirect infinite loop occurs. And if you don't know that the default of client_max_body_size is 1MB, you spend days on a file upload failure that leaves nothing in the backend log.

Steps

Preparation: start Tomcat and deploy /opt/lab/samples/labhub-1.0.0.war as /opt/tomcat/webapps/labhub.war. The nginx working directory is /root/ng.
Write /root/ng/nginx.conf and start nginx. Listen on port 8088 and proxy / to http://127.0.0.1:8080. http://127.0.0.1:8088/labhub/version must return 200.
Set four proxy headers and reload. Host, X-Real-IP, X-Forwarded-For, X-Forwarded-Proto. Check: the JSON from curl -s http://127.0.0.1:8088/labhub/echo must contain x-forwarded-for and x-forwarded-proto, and the host value must not be 127.0.0.1:8080.
Set proxy_connect_timeout 3s, proxy_send_timeout 30s and proxy_read_timeout 60s.
Set client_max_body_size to 20m and reload. A 5MB body POST must not be a 413, and a 25MB body POST must be a 413.
Set gzip on, gzip_min_length 1000 and gzip_vary on, and add to gzip_types the type application/json. The response of curl -H 'Accept-Encoding: gzip' -I http://127.0.0.1:8088/labhub/big.html must have Content-Encoding: gzip.
Create the file /root/ng/static/app.js (any content), and set it up to be served under the /assets/ path. That is, http://127.0.0.1:8088/assets/app.js must return 200, and the response headers must have Expires or Cache-Control.
Set it so that every request starting with /manager/ returns 403. http://127.0.0.1:8088/manager/html must return 403.
Define a log_format that includes the four variables $upstream_addr, $upstream_status, $upstream_response_time and $request_time, and leave the access log at /root/ng/logs/access.log. After the reload, send one request and the log must actually accumulate.

Notes

Start: nginx -c /root/ng/nginx.conf -p /root/ng
Syntax check: nginx -t -c /root/ng/nginx.conf -p /root/ng
Re-apply: nginx -s reload -c /root/ng/nginx.conf -p /root/ng
Sending a large body: head -c 26000000 /dev/zero | curl -s -o /dev/null -w '%{http_code}' --data-binary @- <URL>
Common mistake 1: adding and removing the slash after proxy_pass and producing 404s. If there is a URI part, the part matched by the location is cut off.
Common mistake 2: leaving out the events {} block. It is required in nginx.conf.
Common mistake 3: reloading without creating the log directory, so startup fails.

Minimal proxy configuration and startup

Write /root/ng/nginx.conf and start nginx. Listen on port 8088 and proxy / to http://127.0.0.1:8080. http://127.0.0.1:8088/labhub/version must return 200.

With nginx you can specify the configuration file with -c and the prefix (the base for relative paths) with -p. Remember that this environment cannot bind to ports below 1024. Get into the habit of checking syntax with -t before starting.

Set the forwarded headers

Set four proxy headers and reload. Host, X-Real-IP, X-Forwarded-For, X-Forwarded-Proto. Check: the JSON from curl -s http://127.0.0.1:8088/labhub/echo must contain x-forwarded-for and x-forwarded-proto, and the host value must not be 127.0.0.1:8080.

There is an endpoint that lets you check what the backend actually receives. See for yourself what Host the backend sees if you don't pass Host.

The three proxy timeouts

Set proxy_connect_timeout 3s, proxy_send_timeout 30s and proxy_read_timeout 60s.

The connect, send and read timeouts each measure a different stage. Think about which of these a 504 is most closely related to.

Request body size limit

Set client_max_body_size to 20m and reload. A 5MB body POST must not be a 413, and a 25MB body POST must be a 413.

When this limit is exceeded, nginx responds itself, so no trace is left in the backend log. You must know what the default is to catch a file upload failure quickly.

Response compression

Set gzip on, gzip_min_length 1000 and gzip_vary on, and add to gzip_types the type application/json. The response of curl -H 'Accept-Encoding: gzip' -I http://127.0.0.1:8088/labhub/big.html must have Content-Encoding: gzip.

Responses smaller than the minimum length are not compressed. You must also turn on one directive that prevents cache misbehavior.

Separate static files

Create the file /root/ng/static/app.js (any content), and set it up to be served under the /assets/ path. That is, http://127.0.0.1:8088/assets/app.js must return 200, and the response headers must have Expires or Cache-Control.

There is a directive you use when the location path and the actual directory name differ. Also give a cache expiry header.

Block the admin path

Set it so that every request starting with /manager/ returns 403. http://127.0.0.1:8088/manager/html must return 403.

If you block at the proxy, you can change the policy without deploying the application. Think about the priority of regex locations versus prefix locations.

Log format for incident analysis

Define a log_format that includes the four variables $upstream_addr, $upstream_status, $upstream_response_time and $request_time, and leave the access log at /root/ng/logs/access.log. After the reload, send one request and the log must actually accumulate.

You must leave both the time the backend took and the total time to distinguish 'the backend is slow' from 'the transfer is slow'. To leave a log entry, you have to actually send a request.