Why level="error" returned zero rows

In one line

When a name extracted by a parser collides with an existing stream label, Loki silently appends _extracted and pushes it aside. label_format, line_format, keep, and drop are the tools that tidy up after that, and all four change only the response and never touch the index.

Why this was needed

The collector was attaching a level label to every Pod. The value was info for all of them — from the collector's point of view that file was the application's standard output, so it saw it as info. But the application also printed its own level in the body as level=error.

The person investigating threw {app="payment"} | logfmt | level="error" and the result was 0 lines. They reported "there are no errors at all," but in reality there were 30 in that period. That was because the level extracted by the parser collided with the stream label level and was renamed to level_extracted. The basis for saying there were no errors was in fact "the names collided."

How it works

When a parser creates a label and the same name already exists, Loki appends the _extracted suffix to the one extracted later. The original label stays as it is. There is no warning and no error. So the first time you query a new stream, you need the habit of expanding one result and looking at the list of keys in metric (or stream) with your own eyes.

There are four tidying tools.

Syntax	What it does
label_format 새이름=기존라벨 or label_format 이름=\{{.field}}``	Renames a label or builds it from a template (the placeholders are the new name, the existing label, and the name)
line_format \{{.a}} {{.b}}``	Rewrites the line body itself
keep a, b	Keeps only the labels listed
drop a, b	Drops the labels listed

Templates are Go's text/template. You insert a value with {{.필드}} (the placeholder is the field name), and you can also use pipes such as {{.a | trim}}. line_format swaps out the whole body, so a line filter after it sees the new body — order is meaning.

All four syntaxes act only on the query result. The streams stored in the index do not change by a single character. So even if you add labels with label_format, the number of streams does not grow and neither does the cost. For values whose cardinality worries you, do not index them as labels; leave them in the body and extract them like this at query time — this is where the "do not index" design actually pays off.

In metric queries these tools matter even more. If the name you want to use in sum by (...) exists only in the body, you have to create it first with label_format, and if labels with scattered values remain, the series split, so you have to tidy them with keep or drop.

What it looks like in the field

Name collisions happen most often with level. This is because the collector, the runtime, and the application each want to say a "level." If you set a team convention such as "labels attached by the collector get a prefix" (for example k8s_), this incident disappears altogether.

The second is hard-to-read log panels on dashboards. If a line is a 200-character JSON, a person cannot scan it. If you use line_format to keep only the four or five fields you need, the same panel suddenly becomes useful. However, if someone wants to search the body in that panel, it is kind to leave a comment saying line_format must go after the line filter.

The third is the question "I added a label, so why is the cost unchanged?" That is normal. Labels created at query time are not stored.

What you will do in the next lab

You put data into the Pod's Loki in which the stream label level and the level= in the body deliberately disagree, and see for yourself level="error" return 0 lines. You find and count the _extracted names, overwrite with label_format to filter by the name you want, rewrite the lines with line_format, and group the series with keep to get the counts per level. Finally, you confirm that the number of streams in the index stays the same even after you change labels.