Loki — A Log Store That Does Not Index Logs

I left retries on and went home; by morning the queue was full

Goal

You create the four reasons Loki rejects incoming requests yourself, and build a client that handles 400 and 429 differently, storing every line without losing a single one.

Why it matters

There are two kinds of rejection on the write side. Violations of label syntax, label count, or line length are 400s, and such lines are rejected forever no matter how many times they are resent. Exceeding the per-stream rate is a 429, and this one succeeds if you wait and resend. If you handle the two with the same code, one fills the queue and the other loses data. It is also important that the rate limit is per stream — if you merge labels to reduce cardinality, traffic concentrates on one stream and you get 429s, and if you split to avoid 429s, the number of streams grows. Striking a balance where the two forces pull in opposite directions is the judgment this course aims to teach.

Steps

In /root/lk-ingest-limits, start Loki and write date +%s in /root/lk-ingest-limits/anchor.txt. Then read five limits from http://localhost:3100/config and write them in /root/lk-ingest-limits/limits.tsv on five lines with no header, each with two tab-separated columns, <설정이름><탭><값> (the placeholders are the setting name, a tab, and the value). The order is per_stream_rate_limit, per_stream_rate_limit_burst, max_line_size, max_label_names_per_series, and unordered_writes.
Throw four pushes and write the status codes in /root/lk-ingest-limits/labels.tsv on four lines with no header, each with two tab-separated columns, <이름><탭><코드> (the placeholders are the name, a tab, and the code). The names and contents are, in order — ok (label app="probe"), digit (the label name is 1bad), dash (the label name is bad-name), and toomany (16 labels, with names l0 through l15). Use the reference time for all timestamps, and the body can be any short string.
Make and send one line that exceeds max_line_size from the step 1 table. You cannot pass it on the command line, so build the body as a file and send it with --data-binary @파일 (the placeholder is the file name). Write the result in /root/lk-ingest-limits/03-size.txt on three lines — code=<정수>, limit_bytes=<본문에 적힌 한도 바이트>, and sent_bytes=<보낸 줄의 길이> (the placeholders are the integer code, the limit in bytes written in the body, and the length of the line you sent).
Write /root/lk-ingest-limits/loki-low.yaml and start one more Loki on port 3300 (gRPC 9097). The storage path is under /tmp/lokilow, limits_config.per_stream_rate_limit is 4KB, and per_stream_rate_limit_burst is 8KB. Read two values from that server's /config and write them in /root/lk-ingest-limits/04-low.txt on two lines as rate=<값> and burst=<값> (the placeholders are the values).
Send 900-byte lines one after another to the Loki on port 3300 as a single stream ({"app":"flood"}) and get a 429. Write two lines in /root/lk-ingest-limits/05-429.txt — first_429=<처음 429 가 난 순번, 1부터> and limit_in_body=<본문에 적힌 한도 문자열> (the placeholders are the sequence number of the first 429, counting from 1, and the limit string written in the body). Then save the whole 429 response body in /root/lk-ingest-limits/429.body.
Explain the burst with the two values you read in steps 1 and 4 and the sequence number from step 5. Write four lines in /root/lk-ingest-limits/06-burst.txt — rate_bytes=<초당 허용 바이트 정수>, burst_bytes=<버스트 바이트 정수>, sent_bytes=<429 가 나기까지 보낸 본문 바이트 합>, and why=<공백 뺀 40자 이상 한 문장> (the placeholders are the allowed bytes per second as an integer, the burst bytes as an integer, the sum of body bytes sent until the 429, and one sentence of at least 40 characters excluding spaces).
Create /root/lk-ingest-limits/send.sh so that it sends 20 900-byte lines to port 3300 into the {"app":"keep"} stream, retrying with exponential backoff when it gets a 429 so that all of them are stored. For the timestamps, use the past 20 seconds back from the reference time, one second per line. Then write three lines in /root/lk-ingest-limits/07-send.txt — sent=20, stored=<저장된 줄 수>, and retries=<재시도 횟수> (the placeholders are the number of stored lines and the number of retries). You confirm the number of stored lines by querying.
Suppose a service emits 12,000 bytes per second. Calculate the minimum number of streams you must split it into to fall under the step 4 limit, and write four lines in /root/lk-ingest-limits/08-design.txt — throughput_bytes=12000, limit_bytes=<4단계 한도의 바이트>, streams_needed=<올림한 정수>, and tradeoff=<공백 뺀 60자 이상> (the placeholders are the step 4 limit in bytes, the integer rounded up, and at least 60 characters excluding spaces). In the last line, also write the price of adding streams.

Notes

The working directory is /root/lk-ingest-limits. You start the Loki with the default limits in step 1 and the second one with low limits in step 4.
A very long line cannot be passed as a command-line argument (Argument list too long). Build the body JSON as a file and send it with --data-binary @파일 (the placeholder is the file name).
If you send the same content with the same timestamp to the same stream, it is folded into a duplicate. When you send several lines, give each a different timestamp.
The second Loki dies from overlapping gRPC if you change only http_listen_port. The storage paths must not overlap either.
Common mistake: putting a retry on a 400. You must separate, in code, rejections you can fix from rejections you must wait out.
Request validation and rate limits · Configuration docs · HTTP API · Labels

Read the limits the server holds

In /root/lk-ingest-limits, start Loki and write date +%s in /root/lk-ingest-limits/anchor.txt. Then read five limits from http://localhost:3100/config and write them in /root/lk-ingest-limits/limits.tsv on five lines with no header, each with two tab-separated columns, <설정이름><탭><값> (the placeholders are the setting name, a tab, and the value). The order is per_stream_rate_limit, per_stream_rate_limit_burst, max_line_size, max_label_names_per_series, and unordered_writes.

They are all in the limits_config block. Write the value as the string the server printed it — it sometimes comes with a unit attached, like 3MB, and sometimes as a bare number. All five of these are used in the later steps.

Breaking the label name rules gets a 400

Throw four pushes and write the status codes in /root/lk-ingest-limits/labels.tsv on four lines with no header, each with two tab-separated columns, <이름><탭><코드> (the placeholders are the name, a tab, and the code). The names and contents are, in order — ok (label app="probe"), digit (the label name is 1bad), dash (the label name is bad-name), and toomany (16 labels, with names l0 through l15). Use the reference time for all timestamps, and the body can be any short string.

You can pull out just the code with curl -o /dev/null -w '%{http_code}'. Read a 400 body yourself at least once — it tells you which rule you broke, even the position. The upper limit on the number of labels is in the step 1 table.

If one line is too long, the whole request falls

Make and send one line that exceeds max_line_size from the step 1 table. You cannot pass it on the command line, so build the body as a file and send it with --data-binary @파일 (the placeholder is the file name). Write the result in /root/lk-ingest-limits/03-size.txt on three lines — code=<정수>, limit_bytes=<본문에 적힌 한도 바이트>, and sent_bytes=<보낸 줄의 길이> (the placeholders are the integer code, the limit in bytes written in the body, and the length of the line you sent).

Separately from /opt/lab/d5/gen.py, you may write a small Python script yourself to build the body JSON. The 400 body writes the limit and the sent length side by side in bytes. The reason this rejection is dangerous is that even the healthy lines carried in the same request fall along with it.

Start a second Loki with low limits

Write /root/lk-ingest-limits/loki-low.yaml and start one more Loki on port 3300 (gRPC 9097). The storage path is under /tmp/lokilow, limits_config.per_stream_rate_limit is 4KB, and per_stream_rate_limit_burst is 8KB. Read two values from that server's /config and write them in /root/lk-ingest-limits/04-low.txt on two lines as rate=<값> and burst=<값> (the placeholders are the values).

If you change only the HTTP port, the gRPC ports overlap and it dies — it is written as it is in the last line of the log file. The storage path must not overlap the first one's either. With the default limit (3MB/second) it is hard to produce a 429 in this lab, so we deliberately lower it.

Get a real 429

Send 900-byte lines one after another to the Loki on port 3300 as a single stream ({"app":"flood"}) and get a 429. Write two lines in /root/lk-ingest-limits/05-429.txt — first_429=<처음 429 가 난 순번, 1부터> and limit_in_body=<본문에 적힌 한도 문자열> (the placeholders are the sequence number of the first 429, counting from 1, and the limit string written in the body). Then save the whole 429 response body in /root/lk-ingest-limits/429.body.

The timestamp must differ every time (the same timestamp with the same content is folded into a duplicate). The body writes, in a sentence, which stream exceeded which limit. Which attempt it happens on is hard to know at a glance because of the burst — that is the next step.

Why did the first few get through

Explain the burst with the two values you read in steps 1 and 4 and the sequence number from step 5. Write four lines in /root/lk-ingest-limits/06-burst.txt — rate_bytes=<초당 허용 바이트 정수>, burst_bytes=<버스트 바이트 정수>, sent_bytes=<429 가 나기까지 보낸 본문 바이트 합>, and why=<공백 뺀 40자 이상 한 문장> (the placeholders are the allowed bytes per second as an integer, the burst bytes as an integer, the sum of body bytes sent until the 429, and one sentence of at least 40 characters excluding spaces).

4KB is 4096 bytes. Estimate the sum of bytes sent as the line length times the sequence number (leave out headers and labels and count only the body). Explain in numbers that, with a burst, traffic that is quiet and then surges passes for the first while.

Applied ① — lose not a single line with backoff retries

Create /root/lk-ingest-limits/send.sh so that it sends 20 900-byte lines to port 3300 into the {"app":"keep"} stream, retrying with exponential backoff when it gets a 429 so that all of them are stored. For the timestamps, use the past 20 seconds back from the reference time, one second per line. Then write three lines in /root/lk-ingest-limits/07-send.txt — sent=20, stored=<저장된 줄 수>, and retries=<재시도 횟수> (the placeholders are the number of stored lines and the number of retries). You confirm the number of stored lines by querying.

Backoff means increasing the wait time each time it fails (for example 1 second, 2 seconds, 4 seconds). Put a cap on it so it does not loop forever — it is a safeguard for when you mistake a 400 for a 429. To confirm storage, just query that stream around the reference time and count the lines.

Applied ② — a design that falls under the limit

Suppose a service emits 12,000 bytes per second. Calculate the minimum number of streams you must split it into to fall under the step 4 limit, and write four lines in /root/lk-ingest-limits/08-design.txt — throughput_bytes=12000, limit_bytes=<4단계 한도의 바이트>, streams_needed=<올림한 정수>, and tradeoff=<공백 뺀 60자 이상> (the placeholders are the step 4 limit in bytes, the integer rounded up, and at least 60 characters excluding spaces). In the last line, also write the price of adding streams.

The rate limit is per stream, so if you split, the limit grows accordingly. But adding streams means adding label combinations, and that comes with a cardinality cost. Write in the last line that the two forces pull in opposite directions.