We turned on the proxy and the internal API died

Goal

You measure directly, with a fake proxy that records which route curl, Python urllib, and requests each take under the customer's internal proxy settings, and build settings, a connector, and a verdict tool with which all three tools take the same route.

Why it matters

Proxy environment variables are a convention, not a standard, so each tool interprets them differently. That is why, after putting in the customer's settings, the curl health check is green but only the Python collector gets 403 from the internal API. If you fix the settings by inference, it is easy to save one tool and kill another. This lab builds the habit of confirming "did that request arrive at the proxy" from a record.

Materials: /opt/lab/p1a-proxy/ — corpnet.py (the fake proxy and internal API), customer.env (the customer's settings), cases.tsv (18 cases), and connector.py (the connector with a problem). There is no internet. 127.0.0.2 and 127.0.0.3 are also this Pod itself (loopback), and names such as corp.example and saascorp.example do not resolve — a request that was trying to go direct fails at name resolution, but its absence from the proxy record is itself the evidence that it "went direct".

The expected time is 60 minutes. When the session ends, /root/proxy disappears, so keep the files you need separately.

Steps

1. Start an imitation of the internal network with python3 /opt/lab/p1a-proxy/corpnet.py up, and with customer.env applied, request http://127.0.0.2:8080/health once each with curl, urllib, and requests. Looking at the served_by and id of the response body, write three lines of 도구<TAB>PROXY|DIRECT<TAB>id (the placeholder is the tool name) into /root/proxy/repro.tsv.
2. Run the step 02 cases of cases.tsv (c01..c05, case of variable names and per-scheme variables) with the three tools and write them into /root/proxy/matrix.tsv as case<TAB>curl<TAB>urllib<TAB>requests.
3. Measure the step 03 cases (c06..c10, suffix, leading-dot, and asterisk domains) and add them to matrix.tsv.
4. Measure the step 04 cases (c11..c14, IP, CIDR, and port) and add them to matrix.tsv.
5. Measure the step 05 cases (c15..c18, a single asterisk and lowercase and uppercase precedence) and add them to matrix.tsv.
6. Create /root/proxy/fixed.env so that all three tools go direct to the four internal places (127.0.0.2:8080, 127.0.0.3:8080, localhost:8081, api.corp.example:8080) and go through the proxy http://127.0.0.1:3128 for the three outside places (saascorp.example over http and https, and updates.vendor.example).
7. Copy connector.py to /root/proxy/connector.py and fix it. It must keep using CONNECTOR_PROXY, but a URL that matches NO_PROXY must go direct, and it must not use the environment's HTTP_PROXY and HTTPS_PROXY.
8. Create /root/proxy/route_probe.py. python3 route_probe.py <env파일> URL... (the placeholder is the env file) must swap the proxy address in the settings file for a local sentinel and actually run the three tools, print url<TAB>curl<TAB>urllib<TAB>requests for each URL, and end with exit code 3 if the verdicts differ and 0 if they are all the same.

Notes

• Run with an environment given just once: env -i PATH=$PATH http_proxy=http://127.0.0.1:3128 curl -s http://127.0.0.2:8080/health — env -i clears the variables left in the shell.
• Looking at the records: tail -n 3 /root/proxy/logs/proxy.jsonl · tail -n 3 /root/proxy/logs/api.jsonl
• urllib throws a 403 as an exception (HTTPError). Read the body with read() of the exception object.
• The https cases arrive at the proxy as CONNECT. There is no body, so judge by the number of record lines.
• The grader does not use the corpnet you started. It starts the same imitation on its own port and measures again under the same conditions.
• Common mistakes: a proxy variable you exported in the shell getting mixed into the next case, and believing that *.corp.example will work as a wildcard.

Reproduce the failure with the customer settings and leave evidence

Start corpnet, send one request each from the three tools with customer.env, and write the tool, the route, and the id into /root/proxy/repro.tsv.

If the served_by of the response body is corp-proxy, it went through the proxy. The id must pair with a record line in logs, and which tool sent it is cross-checked with the User-Agent of the record. Export customer.env only in a subshell with set -a.

Measure uppercase HTTP_PROXY and ALL_PROXY

Run c01..c05 of cases.tsv with the three tools and write them into /root/proxy/matrix.tsv as PROXY/DIRECT.

For each case, make a clean environment with env -i, and check whether the number of lines in proxy.jsonl increased before and after the request. In the ENVIRONMENT section of the curl manual, look for the exception that applies only to http_proxy.

Suffix, leading-dot, and asterisk domains

Measure c06..c10 and add them to /root/proxy/matrix.tsv (do not delete the earlier case lines).

A request that was trying to go direct fails at name resolution. Judge not by whether it failed but by whether it arrived in the proxy record. saascorp.example is another company's name that ends with corp.example.

NO_PROXY with CIDR and ports

Measure c11..c14 and add them to /root/proxy/matrix.tsv.

The CIDR support appears with its version in the --noproxy explanation of the curl manual. The urllib documentation says a port can be attached to a no_proxy entry. For a tool the documentation does not speak about, you only know by measuring.

A single asterisk and lowercase precedence

Measure c15..c18 and add them to /root/proxy/matrix.tsv.

Compare when the asterisk is one entry in the list and when it is the whole list. Whether an empty lowercase variable is read as 'not set' or as 'an empty list' also differs per tool.

Rewrite the settings with the common denominator of the three tools

Create /root/proxy/fixed.env in which the three outside places use http://127.0.0.1:3128 as the proxy and the four internal places go direct.

From the table you filled in earlier, pick only the notations where all three tools gave the same answer. You have to decide four things: the case, the leading dot of the domain, what to use instead of CIDR, and whether to attach the port.

Fix the connector that uses the proxies argument

Copy /opt/lab/p1a-proxy/connector.py to /root/proxy/connector.py and fix it so that it respects NO_PROXY.

requests.utils has a function that tells you whether to bypass from a URL and a no_proxy string. The session's trust_env decides whether to use the environment variable proxies. If you hard-code the internal IPs in the code, it fails when graded with a different NO_PROXY.

A probe that measures each tool's route for any settings

Create /root/proxy/route_probe.py. It takes an env file and URLs, judges the three tools' routes by actually running them, prints a TSV, and ends with 3 if they differ and 0 if they are the same.

The customer proxy address is not reachable from this Pod. If you keep the decision rule (no_proxy) as it is and swap only the proxy value for a local sentinel address, you can judge by whether the sentinel received a connection. The grader tests with random domains and proxy addresses.