A Line Chart Cannot Answer a Ranking Question
In one line
"Which one is worst" is a question about ranking, and a line graph cannot answer a ranking. Tables and transformations are the tools that take that question.
Why this was needed
The question that comes up most often in an outage meeting is one of two. "Since when has it been like this" and "which one is worst." The former asks about movement over time, and the latter asks for a list at a single moment. But dashboards are usually built in a shape that can answer only the former.
Suppose the p95 of four handlers is drawn overlaid in one panel. If the values are close to each other, like 0.216, 0.219, 0.229, and 0.233, the four lines stay tangled. You have to find the colors in the legend to work out which line is which handler, and to know the ranking you have to hover the mouse over one time, read four values, and sort them in your head. With eight, nobody does that work. They just say "they all look similar" and move on.
This is not a matter of drawing taste. The shape of the question and the shape of the answer do not match. The answer to a question about ranking is a sorted list, and the shape that shows a sorted list is a table.
How it works
Moving a time series into a table in Grafana splits into three layers. The first is the query. Even with the same PromQL, if you throw it as an instant query, one value per series comes back, and if you throw it as a range query, tens to hundreds of points per series come back. The query editor of the Prometheus data source has separate places to choose between the two and to choose what shape to receive the result in (Prometheus query editor documentation). In the dashboard JSON, this remains per query as the two keys instant and format.
If you pour a range query straight into a table, the number of rows is the number of series times the number of points. With four series viewed for two hours at 60-second intervals, that is 4 times 121, which is 484 rows. Nobody can read that. So the second layer is needed — transformations.
| Transformation | What it does | Effect on the table |
|---|---|---|
reduce |
Reduces a series to a single number | 484 rows become 4 rows |
organize |
Hides columns, renames them, and sets their order | Field becomes 핸들러 (the Korean word for "handler") |
joinByField |
Joins the results of two queries on a common column | Latency and request rate come on one row |
reduce produces "one number per series." If you pick the last value, it is "now," and if you pick the maximum, it is "the worst of this range." The third layer is display. You can attach colors or bars to the cells of a table (table visualization documentation), and there is one rule here. Attach color only to numeric columns that have thresholds. If color is painted even on the handler name column, the color becomes a decoration with no meaning, and at that moment the real red elsewhere loses its power as well. So color is attached not as a panel default but as an override that targets that one column.
Joining two queries into one table is also often needed. "Does a slow handler also receive a lot of traffic" cannot be answered by latency alone or by request rate alone. Only when you join the two by handler name and put them on one row does the eye see the relationship.
What it looks like in the field
On the payment team's health dashboard, the p99 of twelve endpoints was overlaid in a single panel. When an incident happened, people opened that panel and said only "something went up." Each time it took 3 minutes to find out which endpoint it was. When the same query was thrown as an instant query, turned into a table, and sorted by p99 in descending order, at the next incident the name came out 5 seconds after the screen was opened. The query had not changed by a single character.
There is a mistake in the opposite direction, too. If you turn even panels that ask about movement over time, like total request rate, into tables, it gets worse. You cannot tell from a single number whether it is the same as yesterday. The only panels to move into tables are the ones that ask about ranking.
What can and cannot be judged in this environment
Transformations are computed in the browser, not on the server. So in this Pod there is no way to see what a table with the transformations applied actually looks like. There is no image renderer plugin either. Instead, judgment is made in two ways — the transformations, type, and options written in the dashboard JSON model and the instant and format of the queries, and the values that come out when those queries are actually thrown at the data source. How column widths or colors look to the eye you ultimately have to open port 3000 in the web preview and see once.
What you will do in the next lab
You try reading a ranking off a line graph yourself — if you pin down one past time and write the p95 of the four handlers in descending order, it becomes clear in numbers why that cannot be done by eye. Next, you move the same question into a table, measure how many rows a range query becomes in a table, then reduce it with reduce and tidy the columns with organize. You join two queries with joinByField and put them in one table, and attach color by an override to only one column that has thresholds. Finally, you directly turn a ranking panel that is still a line graph into a table and submit it.