Two Authorizers, Two Envoys

Goal

Write authorization servers of both kinds, HTTP and gRPC, yourself in Python, start an Envoy that asks each, and measure where headers flow and what happens when the authorization server dies.

Why it matters

If you put authorization into the code of each service, you get as many different authorizations as there are services. If you gather the judgment in one place, then that one place sits on the path of every request — if you do not decide whether to block or let through when that server dies, in an outage you lose one of availability or security without noticing. Istio's CUSTOM authorization uses this filter behind the scenes, so the header rules and failure behavior you see here appear in the mesh exactly as they are.

Steps

In /root/envd-authz/authz_http.py, write an authorization server that listens on 127.0.0.1:9191 and start it in the background. If Authorization: Bearer alice-token, treat the user as alice, and if bob-token, as bob, and return 200 and the response header x-auth-user: <사용자> (the placeholder is the user); if the token is missing or an unknown value, return 403 and the response header x-deny-reason: missing-or-unknown-token. For each request, append one JSON line (path, user, tenant (the value of the x-tenant header) and allowed) to /root/envd-authz/authz-http.log.
Start the upstream python3 /opt/lab/envoy/echo.py 8091 (it returns the request it received as JSON), and write a configuration with admin port 9981 and listener 127.0.0.1:10081 to /root/envd-authz/envoy-http.yaml and start it (--base-id 11). Put envoy.filters.http.ext_authz at the very front of the HTTP filter chain, with stat_prefix: http_authz, making it ask the cluster authz_http (127.0.0.1:9191) through http_service. Pass the x-auth-user the authorization server gave to the upstream and x-deny-reason to the client with the rejection response, and failure_mode_allow is false.
Send three requests to listener 10081 and write the results in four lines to /root/envd-authz/03-http.txt — the status code of the request sent without a token as no_token=, the x-deny-reason value of that response as deny_reason=, the x-auth-user value the upstream received when bob-token was sent together with a forged header x-auth-user: mallory as spoof_upstream=, and the tenant value that a request with x-tenant: acme attached to alice-token left in the authorization server log as http_saw_tenant= (none if there is no value).
In /root/envd-authz/authz_grpc.py, implement envoy.service.auth.v3.Authorization/Check and start it on 127.0.0.1:9192 (run it with /opt/xds/bin/python3 — a virtual environment that has grpcio and Envoy's protobuf). The judgment is the same as in the HTTP version, except that if the path starts with /public, it is allowed without a token and the user is anonymous. If allowed, put the header x-auth-user in OkHttpResponse, and if denied, put status 403, the header x-deny-reason: grpc-missing-or-unknown-token and a body in DeniedHttpResponse. For each request, leave one JSON line of the same shape in /root/envd-authz/authz-grpc.log.
Write a second Envoy with admin port 9982 and listener 127.0.0.1:10082 to /root/envd-authz/envoy-grpc.yaml and start it (--base-id 12). ext_authz calls the cluster authz_grpc (127.0.0.1:9192, HTTP/2) through envoy_grpc of grpc_service with stat_prefix: grpc_authz, and turn on failure_mode_allow: true and failure_mode_allow_header_add: true. Leave the first Envoy as it is.
Add a route whose path is exactly /healthz at the very front of /root/envd-authz/envoy-grpc.yaml, returning 200 directly with the body ok (direct_response), and turn off ext_authz only on that route (disabled: true with ExtAuthzPerRoute in typed_per_filter_config). After you start that Envoy again, call /healthz three times without a token.
With both authorization servers stopped, send a request with alice-token attached to the first Envoy and a tokenless /orders request to the second Envoy, and in /root/envd-authz/07-failure.txt write three lines: http_on_error= (the status code of the first Envoy), grpc_on_error= (the status code of the second) and grpc_failure_header= (the value of x-envoy-auth-failure-mode-allowed the upstream received). After you write it, start the two authorization servers again.
In /root/envd-authz/08-report.md, write six lines — fail_closed_code= and fail_open_code= (the two codes from step 7), spoofed_header_reached_upstream= (yes if the forged mallory reached the upstream), http_authz_saw_tenant= and grpc_authz_saw_tenant= (yes if the x-tenant value was printed in each authorization server's log), and healthz_asked_authz= (yes if /healthz is in the gRPC authorization server's log) — and below them write what you learned in at least four lines. To check the gRPC-side value, send one request with alice-token and x-tenant: acme attached to the second Envoy.

Notes

Run the gRPC server with /opt/xds/bin/python3. This virtual environment has grpcio and the Envoy API protobuf (xds-protos). The system python3 does not have them.
You start two Envoys. Give different --base-id values (11 and 12) and add --concurrency 1. When you want to stop only one, use curl -X POST localhost:<관리포트>/quitquitquit (the placeholder is the admin port) — pkill -x envoy kills both.
Detach background processes from the shell with setsid --fork nohup <명령> > <로그> 2>&1 </dev/null (the placeholders are the command and the log). Otherwise they die along with the terminal when you close it and grading of the next step fails.
Wrap the first letter of a pkill -f pattern in brackets ('[a]uthz_http.py'). If you use it as it is, the shell that contains that string gets killed too.
Common mistake — not turning on HTTP/2 on the gRPC cluster. The second Envoy is configured to let requests through on failure, so every request passes and it is hard to notice the mistake.

Start the HTTP authorization server

In /root/envd-authz/authz_http.py, write an authorization server that listens on 127.0.0.1:9191 and start it in the background. If Authorization: Bearer alice-token, treat the user as alice, and if bob-token, as bob, and return 200 and the response header x-auth-user: <사용자> (the placeholder is the user); if the token is missing or an unknown value, return 403 and the response header x-deny-reason: missing-or-unknown-token. For each request, append one JSON line (path, user, tenant (the value of the x-tenant header) and allowed) to /root/envd-authz/authz-http.log.

In the HTTP mode of ext_authz, the authorization server is an ordinary web server. Envoy sends a request again with the original request's method and path and reads a 2xx as allowed and anything else as denied. So you have to make the same judgment for every method, not only GET. The standard library http.server is enough, and when you start it, detach it from the shell with setsid --fork nohup python3 … > 로그 2>&1 </dev/null (the placeholder is the log file). The grader sends this server three tokens directly (alice, bob and none).

Make Envoy ask the authorization server on every request

Start the upstream python3 /opt/lab/envoy/echo.py 8091 (it returns the request it received as JSON), and write a configuration with admin port 9981 and listener 127.0.0.1:10081 to /root/envd-authz/envoy-http.yaml and start it (--base-id 11). Put envoy.filters.http.ext_authz at the very front of the HTTP filter chain, with stat_prefix: http_authz, making it ask the cluster authz_http (127.0.0.1:9191) through http_service. Pass the x-auth-user the authorization server gave to the upstream and x-deny-reason to the client with the rejection response, and failure_mode_allow is false.

The filter order is the processing order. ext_authz must be before router so that a rejected request does not reach the upstream. Where to pass the authorization server's response headers is decided by the two lists in authorization_response — allowed_upstream_headers is attached to the upstream request when allowed, and allowed_client_headers is attached to the client response when denied. You start several Envoys in this Pod, so give different --base-id values, and to stop one, use curl -X POST localhost:9981/quitquitquit.

What went to the authorization server, and what went to the upstream

Send three requests to listener 10081 and write the results in four lines to /root/envd-authz/03-http.txt — the status code of the request sent without a token as no_token=, the x-deny-reason value of that response as deny_reason=, the x-auth-user value the upstream received when bob-token was sent together with a forged header x-auth-user: mallory as spoof_upstream=, and the tenant value that a request with x-tenant: acme attached to alice-token left in the authorization server log as http_saw_tenant= (none if there is no value).

What the upstream received is in headers of the JSON the echo returns. What the authorization server received is in the log you wrote. In the authorization request of the HTTP mode, only Host, Method, Path, Content-Length and Authorization are loaded by default, and other headers are passed only if you write them in allowed_headers. In the opposite direction (authorization server → upstream), allowed_upstream_headers overwrites a header with the same name — meaning that even if a client forges the identity header, the value the authorization server decided is what reaches the upstream.

Start the gRPC authorization server

In /root/envd-authz/authz_grpc.py, implement envoy.service.auth.v3.Authorization/Check and start it on 127.0.0.1:9192 (run it with /opt/xds/bin/python3 — a virtual environment that has grpcio and Envoy's protobuf). The judgment is the same as in the HTTP version, except that if the path starts with /public, it is allowed without a token and the user is anonymous. If allowed, put the header x-auth-user in OkHttpResponse, and if denied, put status 403, the header x-deny-reason: grpc-missing-or-unknown-token and a body in DeniedHttpResponse. For each request, leave one JSON line of the same shape in /root/envd-authz/authz-grpc.log.

In the gRPC mode, Envoy does not resend the request; it passes the attributes of the request in a CheckRequest. The headers are in request.attributes.request.http.headers (a map with lowercase names) and the path is in .path. The verdict is decided by CheckResponse.status.code (allowed if it is google.rpc.code_pb2.OK). The module paths are envoy.service.auth.v3.external_auth_pb2 and _pb2_grpc. The grader sends three Check calls directly to this server.

The second Envoy asks over gRPC, and lets requests through if there is no authorization server

Write a second Envoy with admin port 9982 and listener 127.0.0.1:10082 to /root/envd-authz/envoy-grpc.yaml and start it (--base-id 12). ext_authz calls the cluster authz_grpc (127.0.0.1:9192, HTTP/2) through envoy_grpc of grpc_service with stat_prefix: grpc_authz, and turn on failure_mode_allow: true and failure_mode_allow_header_add: true. Leave the first Envoy as it is.

gRPC runs only over HTTP/2. If you do not turn on HTTP/2 on the cluster, Envoy connects over HTTP/1.1 and fails, and that failure is counted as an "authorization server error" — this Envoy is configured to let requests through on failure, so every request passes and it is hard to notice that the configuration is wrong. Put explicit_http_config.http2_protocol_options of HttpProtocolOptions in the cluster's typed_extension_protocol_options.

The health check path does not ask the authorization server

Add a route whose path is exactly /healthz at the very front of /root/envd-authz/envoy-grpc.yaml, returning 200 directly with the body ok (direct_response), and turn off ext_authz only on that route (disabled: true with ExtAuthzPerRoute in typed_per_filter_config). After you start that Envoy again, call /healthz three times without a token.

A load balancer's health check does not know the token. If you apply authorization to every path, the health check gets a 403 and the load balancer takes a healthy proxy out. Turning the filter off is the job not of the filter configuration but of the route side — the mechanism that makes the same filter behave differently per path is typed_per_filter_config. A path that is turned off never goes to the authorization server, so /healthz should not remain in the authorization server log.

When the authorization server dies — the blocking side and the let-through side

With both authorization servers stopped, send a request with alice-token attached to the first Envoy and a tokenless /orders request to the second Envoy, and in /root/envd-authz/07-failure.txt write three lines: http_on_error= (the status code of the first Envoy), grpc_on_error= (the status code of the second) and grpc_failure_header= (the value of x-envoy-auth-failure-mode-allowed the upstream received). After you write it, start the two authorization servers again.

When the authorization server does not respond, Envoy takes one of two paths depending on the configuration. The blocking side (fail closed) returns status_on_error (403 by default), and the let-through side (fail open) sends the request on as it is, attaching a marker header if you want. Either way, ext_authz.<stat_prefix>.error in the statistics rises, and if it let the request through, failure_mode_allowed rises too. When you stop a server, if you write pkill -f authz_http.py, the shell that contains that string may die with it, so wrap the first letter in brackets, as in pkill -f '[a]uthz_http.py'.

Leave the reasoning for which way to fail

In /root/envd-authz/08-report.md, write six lines — fail_closed_code= and fail_open_code= (the two codes from step 7), spoofed_header_reached_upstream= (yes if the forged mallory reached the upstream), http_authz_saw_tenant= and grpc_authz_saw_tenant= (yes if the x-tenant value was printed in each authorization server's log), and healthz_asked_authz= (yes if /healthz is in the gRPC authorization server's log) — and below them write what you learned in at least four lines. To check the gRPC-side value, send one request with alice-token and x-tenant: acme attached to the second Envoy.

Copy the values not from memory but from the evidence files and the logs of the two authorization servers. In the explanation lines, write in your own words "if you choose the blocking side when the authorization server dies, what do you lose, and if you choose the let-through side, what do you lose". The grader reads the same logs and files and cross-checks them.