Loki — A Log Store That Does Not Index Logs

We replayed two hours of logs and the screen showed nothing

Goal

You reproduce, in the real Loki inside the Pod, the phenomenon where a line pushed with a past timestamp is accepted but invisible, confirm directly on the server the setting that decides that boundary, and then bring the line back with a flush.

Why it matters

Loki's writes and reads take different roads. Incoming lines pile up in the ingester's open chunk, and they go up to storage only after the chunk is closed. A query asks storage, and if the range is recent it also asks the ingester, and the extent of that "recent" is query_ingesters_within (default 3 hours). So past data pushed in late during recovery work falls into the gap between the two roads — it is not yet in storage, and it is in the ingester but nobody asks. If you do not know this structure, you conclude "the push is lying" and end up fixing the wrong place.

Steps

In /root/lk-chunks, start Loki, write date +%s in /root/lk-chunks/anchor.txt, and then load the data with python3 /opt/lab/d5/gen.py chunks "$(cat anchor.txt)". This data lies within 30 minutes of the reference time. Query {app="fresh"} over a one-hour range and count the lines, also count the files under the storage directory /tmp/lokidata/chunks, and write them in /root/lk-chunks/01-boot.txt on two lines as lines=<정수> and chunk_files=<정수> (the placeholders are the integer counts).
Load the data from five hours before the reference time with python3 /opt/lab/d5/gen.py chunks-old "$(cat anchor.txt)". Then query {app="aged"} over a one-hour range around that time and count the lines, and write two lines in /root/lk-chunks/02-aged.txt — push_code=<HTTP 상태 코드> and lines=<정수> (the placeholders are the HTTP status code and the integer count).
Push one line each into four new streams and check whether they are visible — they are 1, 2, 4, and 5 hours before the reference time, and the labels are {"app":"b1h"}, b2h, b4h, and b5h. Write the result in /root/lk-chunks/boundary.tsv on four lines with no header, each with two tab-separated columns, <시간><탭><yes|no> (the placeholders are the hours, a tab, and yes or no; the hours are 1, 2, 4, and 5). Then find the setting that decides this boundary in the server's /config and write two lines in /root/lk-chunks/03-param.txt — param=<설정 이름> and value=<기본값 그대로> (the placeholders are the setting name and the default value as it is).
Record the before and after of a flush in one file. First measure the {app="aged"} line count and the number of files in /tmp/lokidata/chunks, call curl -XPOST http://localhost:3100/flush, and then measure the same two numbers again. Write four lines in /root/lk-chunks/04-flush.txt — aged_before=, chunks_before=, aged_after=, and chunks_after=.
In the server's /config, find the three settings that decide the conditions under which an open chunk is closed, and write them in /root/lk-chunks/chunkparams.tsv on three lines with no header, each with two tab-separated columns, <설정이름><탭><값> (the placeholders are the setting name, a tab, and the value). The order is chunk_idle_period, chunk_target_size, and max_chunk_age, and the values are exactly as the server printed them.
Check with the series API which streams are in the index now that the flush is done, and write two lines in /root/lk-chunks/06-series.txt — streams=<정수> and apps=<app 라벨 값들을 쉼표로, 사전순> (the placeholders are the integer count and the values of the app label, comma-separated, in alphabetical order). Give a query range of seven hours back from the reference time. Do not count the revive stream you will create in step 8 (it does not exist yet at this step).
Write four lines in /root/lk-chunks/runbook.txt. Each line starts with 1. through 4., and each line must contain one check command or value to check that you could actually type. The topic is the order of checks when you receive a report that "logs in a past range look empty." The four lines together must have at least 120 characters excluding spaces.
Push one line whose body contains the word revive-check with the label {"app":"revive"} at six hours before the reference time, make it visible with a query, and then write the body of that line on one line in /root/lk-chunks/08-revive.txt. The rest of the body is up to you.

Notes

The working directory is /root/lk-chunks. You start Loki yourself in step 1.
The data generator is /opt/lab/d5/gen.py and it uses two datasets, chunks and chunks-old. The grader does not read this file.
This lab's configuration has reject_old_samples: false, so old timestamps are accepted. The production default is on, so data that is too old is rejected at the acceptance stage.
The storage is the local file system (/tmp/lokidata). Only the path differs from production object storage; the principle is the same.
Common mistake: counting right after a flush. It is asynchronous and takes a few seconds — wait with a loop that has an upper bound until the number of files increases.
Common mistake: not giving start and end to the series API. The default range is short, so past streams drop out.
Loki architecture · Configuration docs · HTTP API · Request validation and rate limits

Recent data is visible as soon as you push it

In /root/lk-chunks, start Loki, write date +%s in /root/lk-chunks/anchor.txt, and then load the data with python3 /opt/lab/d5/gen.py chunks "$(cat anchor.txt)". This data lies within 30 minutes of the reference time. Query {app="fresh"} over a one-hour range and count the lines, also count the files under the storage directory /tmp/lokidata/chunks, and write them in /root/lk-chunks/01-boot.txt on two lines as lines=<정수> and chunk_files=<정수> (the placeholders are the integer counts).

path_prefix and storage_config.filesystem.directory in the configuration file decide the storage location. You count the files with find /tmp/lokidata/chunks -type f | wc -l. If it strikes you as odd that the directory is empty even though you just pushed data, that is the starting point of this lab.

Data from five hours ago gets a 204 and is still invisible

Load the data from five hours before the reference time with python3 /opt/lab/d5/gen.py chunks-old "$(cat anchor.txt)". Then query {app="aged"} over a one-hour range around that time and count the lines, and write two lines in /root/lk-chunks/02-aged.txt — push_code=<HTTP 상태 코드> and lines=<정수> (the placeholders are the HTTP status code and the integer count).

The generator exits with a non-zero value on failure. If you want to see the status code directly, push one line by hand with curl -o /dev/null -w '%{http_code}'. The query range must be centered on five hours ago — if you ask for the last hour, of course it is 0.

Measure where the boundary is and confirm it in the configuration

Push one line each into four new streams and check whether they are visible — they are 1, 2, 4, and 5 hours before the reference time, and the labels are {"app":"b1h"}, b2h, b4h, and b5h. Write the result in /root/lk-chunks/boundary.tsv on four lines with no header, each with two tab-separated columns, <시간><탭><yes|no> (the placeholders are the hours, a tab, and yes or no; the hours are 1, 2, 4, and 5). Then find the setting that decides this boundary in the server's /config and write two lines in /root/lk-chunks/03-param.txt — param=<설정 이름> and value=<기본값 그대로> (the placeholders are the setting name and the default value as it is).

curl -s localhost:3100/config prints out the whole configuration that is running now. Look at the querier: block. Write the value as the string the server printed it (a shape such as 1h0m0s). When you query each line, give a generous range around that time.

Force a flush and the same query answers

Record the before and after of a flush in one file. First measure the {app="aged"} line count and the number of files in /tmp/lokidata/chunks, call curl -XPOST http://localhost:3100/flush, and then measure the same two numbers again. Write four lines in /root/lk-chunks/04-flush.txt — aged_before=, chunks_before=, aged_after=, and chunks_after=.

A flush is asynchronous. It takes a few seconds until the files actually appear, so instead of a fixed sleep use a short loop that runs until the number of files increases (do not forget to put an upper bound on it). The query range must be the same as in step 2 so you can compare the two numbers.

When does a chunk close — find the three levers

In the server's /config, find the three settings that decide the conditions under which an open chunk is closed, and write them in /root/lk-chunks/chunkparams.tsv on three lines with no header, each with two tab-separated columns, <설정이름><탭><값> (the placeholders are the setting name, a tab, and the value). The order is chunk_idle_period, chunk_target_size, and max_chunk_age, and the values are exactly as the server printed them.

They are in the ingester: block. If the /config output is long, find the line numbers first with grep -n and then look only near them with sed -n. The same name may appear in other blocks too, so you must check which block the value comes from.

What is left in the index

Check with the series API which streams are in the index now that the flush is done, and write two lines in /root/lk-chunks/06-series.txt — streams=<정수> and apps=<app 라벨 값들을 쉼표로, 사전순> (the placeholders are the integer count and the values of the app label, comma-separated, in alphabetical order). Give a query range of seven hours back from the reference time. Do not count the revive stream you will create in step 8 (it does not exist yet at this step).

Give /loki/api/v1/series the match[]={app=~".+"} and start and end. If you do not give a range, the default is the last few hours, so past streams drop out — that is itself the same kind of trap this lab is talking about.

Applied ① — the order of checks when you get the same report

Write four lines in /root/lk-chunks/runbook.txt. Each line starts with 1. through 4., and each line must contain one check command or value to check that you could actually type. The topic is the order of checks when you receive a report that "logs in a past range look empty." The four lines together must have at least 120 characters excluding spaces.

Just recall in order what you actually used in this lab — whether it was accepted, which path answers, whether it went down to storage, and how the settings are configured. Write the commands as they are so that someone else can read and follow them at dawn.

Applied ② — make one line from six hours ago visible

Push one line whose body contains the word revive-check with the label {"app":"revive"} at six hours before the reference time, make it visible with a query, and then write the body of that line on one line in /root/lk-chunks/08-revive.txt. The rest of the body is up to you.

Just redo what you did in the earlier steps in order — push it in, bring it down to storage, and ask with a range centered on that time. If the query gives an empty result, one step is still missing.