Loki — A Log Store That Does Not Index Logs

I reported zero errors; there were thirty in that hour

Goal

You reproduce for yourself a parser-extracted name colliding with a stream label and being pushed aside, and tidy the result into the shape you want with label_format, line_format, keep, and drop.

Why it matters

In Loki, labels come from two places — the stream labels attached at collection time and indexed, and the labels the parser extracts from the body at query time. If the names overlap, _extracted is appended to the one extracted later and it is pushed aside, with no warning and no error. So level="error" returns 0 lines, and a person reads that as "there are no errors." Name tidying at query time acts only on the result and does not touch the index, so you can use it freely without adding cost — this is why the design of leaving values you worry about for cardinality in the body and extracting them at query time works.

Steps

In /root/lk-format, start Loki, write date +%s in /root/lk-format/anchor.txt, and then load the data with python3 /opt/lab/d5/gen.py format "$(cat anchor.txt)". Then write the one-hour line count of {app="payment"} and the stream label names attached to that stream in /root/lk-format/01-boot.txt on two lines — lines=<정수> and labels=<이름들을 쉼표로, 사전순> (the placeholders are the integer count and the names, comma-separated, in alphabetical order).
Throw {app="payment"} | logfmt | level="error" over a one-hour range and see how many lines the result has, then ask again with the label whose name was pushed aside and count how many lines. Write the answer in /root/lk-format/02-collision.txt on three lines — naive=<정수>, extracted_name=<밀려난 라벨 이름>, and correct=<정수> (the placeholders are the integer counts and the name of the label that was pushed aside).
Use label_format to make the level value in the body come in under the name level, and then filter with level="error". Write the query in /root/lk-format/03-relabel.logql and the answer in /root/lk-format/03-relabel.txt as lines=<정수> (the placeholder is the integer count). The number of result lines must equal correct from step 2.
Write in /root/lk-format/04-line.logql a query that rewrites every line of {app="payment"} into the form <svc>/<본문의 수준>/<dur_ms> (the middle placeholder is the level in the body). Split with only two slashes and put in no other characters. Then write the body of the earliest line in the result in /root/lk-format/04-line.txt on one line.
Write in /root/lk-format/05-bylevel.logql a metric query that counts lines per level in the body. The result must be exactly two series, and the only label of each series must be lvl. Write the values in /root/lk-format/05-bylevel.txt on two lines as info=<정수> and error=<정수> (the placeholders are the integer counts).
Write in /root/lk-format/06-drop.logql a query that uses drop instead of keep in the step 5 query to produce the same result. Then write two lines in /root/lk-format/06-drop.txt — series=<계열 수> and dropped=<버린 라벨 이름들을 쉼표로, 사전순> (the placeholders are the number of series and the names of the dropped labels, comma-separated, in alphabetical order).
In step 3 you created a label with label_format, and in step 5 you grouped the series by the new name. Check whether the number of streams in the index changed after that, and write two lines in /root/lk-format/07-index.txt — streams=<정수> and changed=<yes|no> (the placeholders are the integer count and yes or no). You count the number of streams with the series API.
Write one query in /root/lk-format/08-panel.logql. There are three conditions — (1) return only lines whose body level is error, (2) the returned body must have the form <dur_ms>ms <svc> <region> (split with only two spaces and no other characters), and (3) put the line filter before the body rewrite. Then write the earliest line in the result in /root/lk-format/08-panel.txt on one line.

Notes

The working directory is /root/lk-format. You start Loki yourself in step 1.
The data generator is /opt/lab/d5/gen.py and it uses the format data. The grader does not read this file.
Templates are Go's text/template. You put a dot before the field and wrap it in double curly braces. It is safer to wrap the query in backticks.
Get into the habit of expanding the result's label list once with jq '.data.result[0].stream'. A name that was pushed aside becomes visible only that way.
Common mistake: putting line_format before the line filter. The filters after it see the new body.
Common mistake: measuring with since=1h. Give start and end based on the reference time in anchor.txt.
Log queries and formatting · Metric queries · Labels · HTTP API

Load data in which the labels and the body disagree

In /root/lk-format, start Loki, write date +%s in /root/lk-format/anchor.txt, and then load the data with python3 /opt/lab/d5/gen.py format "$(cat anchor.txt)". Then write the one-hour line count of {app="payment"} and the stream label names attached to that stream in /root/lk-format/01-boot.txt on two lines — lines=<정수> and labels=<이름들을 쉼표로, 사전순> (the placeholders are the integer count and the names, comma-separated, in alphabetical order).

The stream labels are the keys of the result's stream object. Loki 3.x attaches service_name automatically, so that goes into the list too. Sort them alphabetically and join them with commas (with no spaces).

level="error" returns 0 results

Throw {app="payment"} | logfmt | level="error" over a one-hour range and see how many lines the result has, then ask again with the label whose name was pushed aside and count how many lines. Write the answer in /root/lk-format/02-collision.txt on three lines — naive=<정수>, extracted_name=<밀려난 라벨 이름>, and correct=<정수> (the placeholders are the integer counts and the name of the label that was pushed aside).

Expand one result with jq '.data.result[0].stream' and look at the list of keys. When a name extracted by the parser collides with an existing stream label, Loki appends a suffix and pushes it aside. If you ask again with that name, a number comes out.

Restore the name with label_format

Use label_format to make the level value in the body come in under the name level, and then filter with level="error". Write the query in /root/lk-format/03-relabel.logql and the answer in /root/lk-format/03-relabel.txt as lines=<정수> (the placeholder is the integer count). The number of result lines must equal correct from step 2.

You can write label_format in the form 새이름=기존라벨 (the placeholders are the new name and the existing label) or give it a template. Overwriting a name that already exists also works. Order matters — you must filter after renaming.

Rewrite the line with line_format

Write in /root/lk-format/04-line.logql a query that rewrites every line of {app="payment"} into the form <svc>/<본문의 수준>/<dur_ms> (the middle placeholder is the level in the body). Split with only two slashes and put in no other characters. Then write the body of the earliest line in the result in /root/lk-format/04-line.txt on one line.

line_format takes a Go template. You put a dot before the field name. For the level you must use the pushed-aside name you found in step 2. If you receive the results with direction=forward, the earliest line comes first.

Group the series with keep

Write in /root/lk-format/05-bylevel.logql a metric query that counts lines per level in the body. The result must be exactly two series, and the only label of each series must be lvl. Write the values in /root/lk-format/05-bylevel.txt on two lines as info=<정수> and error=<정수> (the placeholders are the integer counts).

Create the name you will use with label_format, keep only that name with keep, and wrap the outside in sum by (lvl) (count_over_time(... [1h])). Also see once what happens if you leave out keep — it shows up right away in the number of series.

What drop throws away and what it keeps

Write in /root/lk-format/06-drop.logql a query that uses drop instead of keep in the step 5 query to produce the same result. Then write two lines in /root/lk-format/06-drop.txt — series=<계열 수> and dropped=<버린 라벨 이름들을 쉼표로, 사전순> (the placeholders are the number of series and the names of the dropped labels, comma-separated, in alphabetical order).

drop throws away names listed one by one. If you combine the stream label list you wrote down in step 1 with the parser label list you saw in step 2, you get what must be dropped. When there is little to keep, keep is shorter, and when there is little to drop, drop is shorter.

Applied ① — changing labels leaves the index as it is

In step 3 you created a label with label_format, and in step 5 you grouped the series by the new name. Check whether the number of streams in the index changed after that, and write two lines in /root/lk-format/07-index.txt — streams=<정수> and changed=<yes|no> (the placeholders are the integer count and yes or no). You count the number of streams with the series API.

If you give match[]={app=~".+"} to /loki/api/v1/series, you get the list of streams currently in the index. See whether the labels you created at query time are there. If they are not, that is exactly the proof that "labels at query time are not stored."

Applied ② — a human-readable error panel query

Write one query in /root/lk-format/08-panel.logql. There are three conditions — (1) return only lines whose body level is error, (2) the returned body must have the form <dur_ms>ms <svc> <region> (split with only two spaces and no other characters), and (3) put the line filter before the body rewrite. Then write the earliest line in the result in /root/lk-format/08-panel.txt on one line.

Order is meaning — a filter after line_format sees the new body. So you must filter first and rewrite afterward. When filtering by level, you can use the name you found in step 2 or the name you created in step 3.