3,000 requests a second — and every one of them was an error page

Goal

You load-test a target whose status code is always 200 but whose body is an error, and count for yourself what a test that looks only at the summary misses. By changing connection reuse, response size, and the cache key, you judge whether the test conditions are representative of production, and at the end you build a checklist script that decides whether you can trust these results and set up a gate with the exit code.

Why it matters

The most dangerous sentence in a load test report is 'error rate 0%'. All the load generator knows is the status code, and an error contained inside a 200 is invisible to it. Moreover, the error path does no work, so it finishes quickly, and the more the test breaks, the better the latency numbers get. So a load test without verification is not a wrong answer but no answer at all. Even after attaching verification, one more question remains — are these test conditions representative of production? If you opened a new connection every time, you measured the TCP handshake; if the response was 1/64 of production, you never touched serialization and bandwidth; and if you hit only the same URL, the cache passed the test in your place. Pinning this judgment down with a script's exit code instead of leaving it to human eyes is the last step of this lab.

Steps

Start /opt/lab/lt/lt-test-validity/app.py on 127.0.0.1:8080 with DELAY=0.02 ERR_EVERY=3 SIZE=1024. Receive the access record in /root/lt-test-validity/01-access.log. After starting it, call /reset once, run hey -n 300 -c 10 'http://127.0.0.1:8080/api/order?id=1', and save the raw output to /root/lt-test-validity/01-summary.txt. Then write four lines, total=, status_200=, non_2xx=, and rps=, read from that summary, to /root/lt-test-validity/01-claim.txt. For rps, write the Requests/sec value of the summary as it is.
Count the sixth column (kind) of /root/lt-test-validity/01-access.log to create /root/lt-test-validity/02-truth.tsv. It has two lines, and each line has two tab-separated columns, ok <수> and error <수> (the kind, a tab, and the count; ok first). Then write three lines to /root/lt-test-validity/02-note.txt — error_ratio= is the ratio of error bodies as a percentage to one decimal place, hey_non2xx= is the number of non-2xx responses in the step 1 summary, and why= is, in at least 40 characters, why the two numbers differ this way.
Using the eighth column (dur_ms) of /root/lt-test-validity/01-access.log, compute the median for normal responses and for error responses separately and write them as two lines to /root/lt-test-validity/03-durations.tsv. Each line has two tab-separated columns, ok <중앙값> and error <중앙값> (the kind, a tab, and the median), with milliseconds to one decimal place. The median is the int(n/2)+1th value (counting from 1) of n values sorted in ascending order. Then write two lines to /root/lt-test-validity/03-note.txt: faster=<ok 또는 error> (ok or error) and effect=<이 성질이 시험 결과를 어느 쪽으로 밀어내는가, 40자 이상> (which way this property pushes the test results, in at least 40 characters).
Measure the same load twice with the target that has error injection turned off (ERR_EVERY=0 DELAY=0.02 SIZE=1024). Once with defaults (/root/lt-test-validity/04-on.txt, access record /root/lt-test-validity/04-on.log), and once with -disable-keepalive added (/root/lt-test-validity/04-off.txt, access record /root/lt-test-validity/04-off.log), and both times it is hey -n 200 -c 10 'http://127.0.0.1:8080/api/order?id=1'. Write two lines to /root/lt-test-validity/04-keepalive.tsv — on <연결 수> <rps> and off <연결 수> <rps> (the mode, the connection count, and rps), where rps is to two decimal places. The connection count is the number of distinct values in the second column of the access record. Then write three lines to /root/lt-test-validity/04-note.txt: use_run=<on 또는 off> (on or off), conn_ratio=<off 연결 수 ÷ on 연결 수, 소수 첫째 자리> (the off connection count divided by the on connection count, to one decimal place), and why=<50자 이상> (at least 50 characters). Assume that production clients use a connection pool.
Measure the same load (hey -n 200 -c 10 'http://127.0.0.1:8080/api/order?id=1', ERR_EVERY=0) twice, changing only the response size — SIZE=1024 goes to /root/lt-test-validity/05-1k.txt and SIZE=65536 goes to /root/lt-test-validity/05-64k.txt. Write two lines to /root/lt-test-validity/05-size.tsv. Each line has four tab-separated columns, <이름> <요청당 바이트> <rps> <초당 바이트> (name, bytes per request, rps, and bytes per second), the names are 1k and 64k, rps is to two decimal places, and bytes per second is bytes per request × rps rounded to an integer. You read bytes per request and rps from Size/request and Requests/sec of the summary. Then write three lines to /root/lt-test-validity/05-note.txt: bound_by=<latency 또는 bandwidth> (latency or bandwidth), bytes_ratio=<64k 초당 바이트 ÷ 1k 초당 바이트, 소수 첫째 자리> (the 64k bytes per second divided by the 1k bytes per second, to one decimal place), and why=<50자 이상> (at least 50 characters).
Measure twice with the target that has the cache turned on (CACHE=1 ERR_EVERY=0 DELAY=0.02 SIZE=1024). Once with a fixed key, 'http://127.0.0.1:8080/api/order?id=1' (/root/lt-test-validity/06-fixed.txt, record /root/lt-test-validity/06-fixed.log), and once with a rotating key, http://127.0.0.1:8080/api/order/rotate (/root/lt-test-validity/06-rotate.txt, record /root/lt-test-validity/06-rotate.log), and both times it is hey -n 200 -c 10. Start the second with ROTATE_KEYS=500. Write two lines to /root/lt-test-validity/06-cache.tsv. Each line has five tab-separated columns, <이름> <적중> <빗나감> <적중률> <rps> (name, hits, misses, hit rate, and rps), the names are fixed and rotate, the hit rate is a percentage to one decimal place, and rps is to two decimal places. Then write three lines to /root/lt-test-validity/06-note.txt: representative=<fixed 또는 rotate> (fixed or rotate), inflation=<fixed rps ÷ rotate rps, 소수 첫째 자리> (fixed rps divided by rotate rps, to one decimal place), and why=<50자 이상> (at least 50 characters).
Create /root/lt-test-validity/validate.sh. When called as bash validate.sh <hey 요약 파일> <접근 기록 파일> (the hey summary file and the access record file), it prints three lines — status <OK|FAIL> <값>, body <OK|FAIL> <값>, and volume <OK|FAIL> <값> (each with its value), with the name at the very beginning of the line. status is OK when the summary has 0 non-2xx responses, body is OK when the ratio of error bodies in the access record is under 1%, and volume is OK when the number of lines in the access record equals the total number of responses in the summary. It must exit with code 1 if even one is FAIL and with 0 if all are OK. The grader runs this script with four sets of inputs it made itself to confirm both passes and failures.
Judge the step 1 result with the checklist you built. Save the output of bash validate.sh 01-summary.txt 01-access.log to /root/lt-test-validity/08-gate-out.txt, and write four lines to /root/lt-test-validity/08-verdict.txt — gate_exit=<종료 코드> (the exit code), failed_check=<FAIL 이 난 검사 이름> (the name of the check that failed), trustworthy=<yes 또는 no> (yes or no), and fix=<이 시험을 다시 하려면 무엇을 고쳐야 하는가, 60자 이상> (what must be fixed to redo this test, in at least 60 characters).

Notes

The working directory is /root/lt-test-validity. If it does not exist, create it first.
The load target is /opt/lab/lt/lt-test-validity/app.py. The comment at the top of the file lists the environment variables, the paths, and the eight columns of the access record.
Containers cannot be started in this Pod (seccomp). Run the target by starting a Python standard library server directly on 127.0.0.1. Before starting it again, kill the previous one with pkill -f 'lt-test-validity/app.py'.
nproc reports the number of cores of the node, not of the Pod. The same goes for the default -cpus value that hey advises — that is why this lab's target spends time with time.sleep() rather than CPU.
Common mistake: not restarting the target before the second run, so the access record gets mixed with the previous run.
Common mistake: mixing -disable-compression and -disable-keepalive. The former turns off compression and the latter turns off connection reuse.
hey (rakyll/hey) · k6 checks · k6 thresholds · http.server · vegeta

Looking only at the summary, this test is perfect

Start /opt/lab/lt/lt-test-validity/app.py on 127.0.0.1:8080 with DELAY=0.02 ERR_EVERY=3 SIZE=1024. Receive the access record in /root/lt-test-validity/01-access.log. After starting it, call /reset once, run hey -n 300 -c 10 'http://127.0.0.1:8080/api/order?id=1', and save the raw output to /root/lt-test-validity/01-summary.txt. Then write four lines, total=, status_200=, non_2xx=, and rps=, read from that summary, to /root/lt-test-validity/01-claim.txt. For rps, write the Requests/sec value of the summary as it is.

The lines under Status code distribution in the summary have the shape [<코드>]<탭><수> responses. If you remove the brackets with awk, the first column is the code and the second is the count. To wait until the server comes up, send curl to /stats repeatedly. The numbers that come out in this step are numbers that will later turn out to be 'wrong'.

Count what the server actually returned

Count the sixth column (kind) of /root/lt-test-validity/01-access.log to create /root/lt-test-validity/02-truth.tsv. It has two lines, and each line has two tab-separated columns, ok <수> and error <수> (the kind, a tab, and the count; ok first). Then write three lines to /root/lt-test-validity/02-note.txt — error_ratio= is the ratio of error bodies as a percentage to one decimal place, hey_non2xx= is the number of non-2xx responses in the step 1 summary, and why= is, in at least 40 characters, why the two numbers differ this way.

Count with awk -F'\t' '$6=="ok"' 01-access.log | wc -l. The number of lines in the access record must equal the number of responses hey counted — if it differs, some requests were cut off midway. The status code is only one piece of the contract, and the body is the rest.

Errors are faster, so the numbers look better

Using the eighth column (dur_ms) of /root/lt-test-validity/01-access.log, compute the median for normal responses and for error responses separately and write them as two lines to /root/lt-test-validity/03-durations.tsv. Each line has two tab-separated columns, ok <중앙값> and error <중앙값> (the kind, a tab, and the median), with milliseconds to one decimal place. The median is the int(n/2)+1th value (counting from 1) of n values sorted in ascending order. Then write two lines to /root/lt-test-validity/03-note.txt: faster=<ok 또는 error> (ok or error) and effect=<이 성질이 시험 결과를 어느 쪽으로 밀어내는가, 40자 이상> (which way this property pushes the test results, in at least 40 characters).

Sort with awk -F'\t' '$6=="ok"{print $8}' 01-access.log | sort -n and then pick by line number. The error path touches neither the database nor the queue, so it finishes quickly. So the more errors are mixed in, the better the average and percentiles get, and the signal that the test is broken comes not as 'it got slower' but as 'it got faster'.

Whether you reuse connections changes half of the test

Measure the same load twice with the target that has error injection turned off (ERR_EVERY=0 DELAY=0.02 SIZE=1024). Once with defaults (/root/lt-test-validity/04-on.txt, access record /root/lt-test-validity/04-on.log), and once with -disable-keepalive added (/root/lt-test-validity/04-off.txt, access record /root/lt-test-validity/04-off.log), and both times it is hey -n 200 -c 10 'http://127.0.0.1:8080/api/order?id=1'. Write two lines to /root/lt-test-validity/04-keepalive.tsv — on <연결 수> <rps> and off <연결 수> <rps> (the mode, the connection count, and rps), where rps is to two decimal places. The connection count is the number of distinct values in the second column of the access record. Then write three lines to /root/lt-test-validity/04-note.txt: use_run=<on 또는 off> (on or off), conn_ratio=<off 연결 수 ÷ on 연결 수, 소수 첫째 자리> (the off connection count divided by the on connection count, to one decimal place), and why=<50자 이상> (at least 50 characters). Assume that production clients use a connection pool.

The second column of the access record is the number of the TCP connection that carried that request. Count with cut -f2 04-on.log | sort -u | wc -l. You must restart the target before the second run so the access records do not mix. The option name in hey is -disable-keepalive, not -disable-compression — the two turn off different things.

What changes if you make the response 64 times larger

Measure the same load (hey -n 200 -c 10 'http://127.0.0.1:8080/api/order?id=1', ERR_EVERY=0) twice, changing only the response size — SIZE=1024 goes to /root/lt-test-validity/05-1k.txt and SIZE=65536 goes to /root/lt-test-validity/05-64k.txt. Write two lines to /root/lt-test-validity/05-size.tsv. Each line has four tab-separated columns, <이름> <요청당 바이트> <rps> <초당 바이트> (name, bytes per request, rps, and bytes per second), the names are 1k and 64k, rps is to two decimal places, and bytes per second is bytes per request × rps rounded to an integer. You read bytes per request and rps from Size/request and Requests/sec of the summary. Then write three lines to /root/lt-test-validity/05-note.txt: bound_by=<latency 또는 bandwidth> (latency or bandwidth), bytes_ratio=<64k 초당 바이트 ÷ 1k 초당 바이트, 소수 첫째 자리> (the 64k bytes per second divided by the 1k bytes per second, to one decimal place), and why=<50자 이상> (at least 50 characters).

If requests per second stay almost the same while only bytes per second jump by tens of times, it means this target is bound not by bandwidth but by waiting time. If you look at throughput only in requests per second, you cannot see this difference. If the production response is 64 KB and you tested with 1 KB, that test never touched serialization and bandwidth.

If you hit only the same URL, the cache passes the test for you

Measure twice with the target that has the cache turned on (CACHE=1 ERR_EVERY=0 DELAY=0.02 SIZE=1024). Once with a fixed key, 'http://127.0.0.1:8080/api/order?id=1' (/root/lt-test-validity/06-fixed.txt, record /root/lt-test-validity/06-fixed.log), and once with a rotating key, http://127.0.0.1:8080/api/order/rotate (/root/lt-test-validity/06-rotate.txt, record /root/lt-test-validity/06-rotate.log), and both times it is hey -n 200 -c 10. Start the second with ROTATE_KEYS=500. Write two lines to /root/lt-test-validity/06-cache.tsv. Each line has five tab-separated columns, <이름> <적중> <빗나감> <적중률> <rps> (name, hits, misses, hit rate, and rps), the names are fixed and rotate, the hit rate is a percentage to one decimal place, and rps is to two decimal places. Then write three lines to /root/lt-test-validity/06-note.txt: representative=<fixed 또는 rotate> (fixed or rotate), inflation=<fixed rps ÷ rotate rps, 소수 첫째 자리> (fixed rps divided by rotate rps, to one decimal place), and why=<50자 이상> (at least 50 characters).

The seventh column of the access record is hit or miss. /api/order/rotate uses a different key for each request, so if the number of distinct keys is larger than the number of requests, there are no hits at all. If you test with a fixed key without knowing the production hit rate, the database behind the cache gets deployed without ever having received load.

Build a checklist that decides whether the result can be trusted

Create /root/lt-test-validity/validate.sh. When called as bash validate.sh <hey 요약 파일> <접근 기록 파일> (the hey summary file and the access record file), it prints three lines — status <OK|FAIL> <값>, body <OK|FAIL> <값>, and volume <OK|FAIL> <값> (each with its value), with the name at the very beginning of the line. status is OK when the summary has 0 non-2xx responses, body is OK when the ratio of error bodies in the access record is under 1%, and volume is OK when the number of lines in the access record equals the total number of responses in the summary. It must exit with code 1 if even one is FAIL and with 0 if all are OK. The grader runs this script with four sets of inputs it made itself to confirm both passes and failures.

You must run the three checks each and then issue the exit code only once at the end — if you exit midway, the remaining check lines are not printed. Compare decimals with awk's exit code, as in awk 'BEGIN{exit !(r < 1.0)}'. The reason the name goes at the very start of the line is that a pipeline, not a person, reads it.

Run that splendid step 1 result through the gate

Judge the step 1 result with the checklist you built. Save the output of bash validate.sh 01-summary.txt 01-access.log to /root/lt-test-validity/08-gate-out.txt, and write four lines to /root/lt-test-validity/08-verdict.txt — gate_exit=<종료 코드> (the exit code), failed_check=<FAIL 이 난 검사 이름> (the name of the check that failed), trustworthy=<yes 또는 no> (yes or no), and fix=<이 시험을 다시 하려면 무엇을 고쳐야 하는가, 60자 이상> (what must be fixed to redo this test, in at least 60 characters).

Looking only at the step 1 summary, non-2xx is 0 and the response count is right too. Even so, if the gate blocks it, which check blocked it is the conclusion of this lab. In fix, write 'what to verify by fixing the test' — changing the tool or attaching a layer that checks responses can both be answers.