TT Lab
Get started
Learn Learning paths Courses

Grafana Dashboards

Move the Ranking Panels into Tables

Continue in TT Lab

Goal

You receive a dashboard with four overlapping line graphs, move the panels that ask about ranking into tables, and build a readable shape with the reduce, organize, and joinByField transformations.

Why it matters

The questions a dashboard has to answer are, broadly, two — "since when has it been like this" and "which one is worst." The former is movement over time, so a line graph fits, but the latter is a sorted list at a single moment, so a line graph cannot answer it. If the values of the four series are close, the lines tangle, and to know the ranking you must pick one time, read four values, and sort them in your head. With eight, nobody does that work. A table is a shape where the screen does that sorting for you, and a transformation is the tool that turns a time series into a shape a table can read. However, if you turn every panel into a table, it gets worse — the only panels to move are the ones that ask about ranking.

Steps

  1. Start Grafana with lab-start-grafana and upload /opt/lab/gfd/gfd-table/ranking.json to Grafana as it is, without editing it (the uid is gfd-table as written in the file, with 4 panels). You can POST it to /api/dashboards/db with curl. After uploading, open it once in the web preview on port 3000 and look by eye at how the four panels appear.
  2. Panel 1 draws the p95 of four handlers overlaid. Pick one past time (at least 1 hour before now, within 6 hours), measure the p95 at that moment for each handler, and write it to /root/gfd-table/02-rank.txt as six lines at= rank1= rank2= rank3= rank4= gap=. at is epoch seconds, rank1 to rank4 are the handler names written in order from the largest p95, and gap is the difference in p95 between 1st and 4th place (seconds).
  3. First, measure how many rows the table would have. Pick a past range (at least 1 hour, ending before now), throw panel 1's query over that range with step 60, and write five lines start= end= series= points= rows= to /root/gfd-table/03-table.txt — series is the number of series returned when the same query is thrown as an instant value at the end time, points is the number of points the range query returned for one series, and rows is the product of the two. Then create panel 5 anew: type is table, and the query is the same as panel 1's with format set to table and thrown as a range query (do not set instant to true).
  4. Attach a reduce transformation to panel 5. Put one entry with id set to reduce in the transformations array, with options.mode as seriesToRows and options.reducers holding only lastNotNull. Leave the query as a range query — the point is to see whether the transformation does the work of reducing. Save the fixed dashboard.
  5. In panel 5's transformations array, attach one more entry with id set to organize after reduce. In options.renameByName, rename the two columns Field and Last * each to a different Korean name, and pin the order of those two columns to 0 and 1 with options.indexByName. Save the fixed dashboard.
  6. Create panel 6 anew. type is table, and there are two queries — the one with refId A is sum by (handler) (rate(http_requests_total{job="shop-api"}[5m])), and the one with refId B is the same p95 query as panel 1. Set instant to true and format to table for both queries. Then put one entry with id set to joinByField in transformations, with options.byField as handler and options.mode as outer.
  7. Leave the default cell display of panel 6 (fieldConfig.defaults.custom.cellOptions.type) as auto, and add to fieldConfig.overrides one override that targets only the p95 value column. The override's matcher.id is byName, and properties holds custom.cellOptions (with type as color-background) and thresholds (two or more steps, the last step being a numeric threshold). Then write to /root/gfd-table/07-cells.txt, as three lines rule1= rule2= rule3=, the rules you must keep when attaching color to table cells, each at least 40 characters and all different from each other.
  8. Panel 2 (핸들러별 5xx 비율, the Korean title means "5xx ratio by handler") is also a panel that asks about ranking. Change type to table, leave the query as it is, set instant to true and format to table, hide the Time column with an organize transformation (set Time to true in excludeByName), and rename the remaining two columns. Do not touch panel 4 (전체 요청률, the Korean title means "total request rate") — it is not a panel that asks about ranking. Then download the dashboard as it stands now and save it to /root/gfd-table/fixed.json (only the .dashboard body), and write to /root/gfd-table/08-review.md, as four lines R1= through R4=, what you changed and why, each at least 30 characters.

Notes

Upload the four-line-graph dashboard as it is

Start Grafana with lab-start-grafana and upload /opt/lab/gfd/gfd-table/ranking.json to Grafana as it is, without editing it (the uid is gfd-table as written in the file, with 4 panels). You can POST it to /api/dashboards/db with curl. After uploading, open it once in the web preview on port 3000 and look by eye at how the four panels appear.

The save API takes the dashboard body inside the dashboard key and sends overwrite along with it. jq -n --slurpfile is convenient for building that shape from the file. Grafana takes a few tens of seconds to come up, so first check whether /api/health responds.

Try reading a ranking off a line graph

Panel 1 draws the p95 of four handlers overlaid. Pick one past time (at least 1 hour before now, within 6 hours), measure the p95 at that moment for each handler, and write it to /root/gfd-table/02-rank.txt as six lines at= rank1= rank2= rank3= rank4= gap=. at is epoch seconds, rank1 to rank4 are the handler names written in order from the largest p95, and gap is the difference in p95 between 1st and 4th place (seconds).

For a single-moment value, send time= along with /api/v1/query. You must pin down the time so that you get the same value when you measure again later. If you pull out metric.handler and value[1] of the response together and line them up with sort -rn, the ranking comes out right away. Look at how close the four values are, and think about whether you could tell them apart by eye on a line graph.

Move the same question into a table

First, measure how many rows the table would have. Pick a past range (at least 1 hour, ending before now), throw panel 1's query over that range with step 60, and write five lines start= end= series= points= rows= to /root/gfd-table/03-table.txt — series is the number of series returned when the same query is thrown as an instant value at the end time, points is the number of points the range query returned for one series, and rows is the product of the two. Then create panel 5 anew: type is table, and the query is the same as panel 1's with format set to table and thrown as a range query (do not set instant to true).

A range query is /api/v1/query_range and you send start, end, and step along with it. The length of data.result[0].values in the returned JSON is the number of points for one series. To create a new panel, download the dashboard, add one more to the panels array, and save it again as it is. Move y down so that gridPos does not overlap.

Keep only one number per series with a transformation

Attach a reduce transformation to panel 5. Put one entry with id set to reduce in the transformations array, with options.mode as seriesToRows and options.reducers holding only lastNotNull. Leave the query as a range query — the point is to see whether the transformation does the work of reducing. Save the fixed dashboard.

reduce reduces a series to a single number. seriesToRows makes one row per series, and that row gets a Field column holding the series name and a column for the chosen calculation. The column name for lastNotNull is Last * — you use this name in the next step. Transformations go, in order, into the transformations array that each panel in the dashboard JSON has.

Make column names and order readable by people

In panel 5's transformations array, attach one more entry with id set to organize after reduce. In options.renameByName, rename the two columns Field and Last * each to a different Korean name, and pin the order of those two columns to 0 and 1 with options.indexByName. Save the fixed dashboard.

organize is a single transformation that hides columns (excludeByName), sets their order (indexByName), and renames them (renameByName). All three options are objects keyed by column name. Transformations are applied one after another in the order written in the array, so the later transformation receives the column names produced by the earlier one. Field is the series-name column and Last * is the last-value column.

Join two queries by label and put them in one table

Create panel 6 anew. type is table, and there are two queries — the one with refId A is sum by (handler) (rate(http_requests_total{job="shop-api"}[5m])), and the one with refId B is the same p95 query as panel 1. Set instant to true and format to table for both queries. Then put one entry with id set to joinByField in transformations, with options.byField as handler and options.mode as outer.

An instant query gives one row per series, so the table is short even without a transformation — that is why reduce is not needed here. The column common to the two results is handler, exactly the label name. After joining, the value columns are distinguished as Value #A and Value #B. outer also keeps rows that exist on only one side, and inner keeps only rows that exist on both sides.

Attach color to only one column that has thresholds

Leave the default cell display of panel 6 (fieldConfig.defaults.custom.cellOptions.type) as auto, and add to fieldConfig.overrides one override that targets only the p95 value column. The override's matcher.id is byName, and properties holds custom.cellOptions (with type as color-background) and thresholds (two or more steps, the last step being a numeric threshold). Then write to /root/gfd-table/07-cells.txt, as three lines rule1= rule2= rule3=, the rules you must keep when attaching color to table cells, each at least 40 characters and all different from each other.

If you hang color on the panel default, even the name column gets painted. For color to have meaning, it must say "it crossed the threshold," and for that it must be attached only to a column that has thresholds. An override is an entry in the fieldConfig.overrides array, where matcher decides which column it applies to and properties decides what to change. First check the name of the p95 value column after the join.

Turn the ranking panel that is still a line into a table

Panel 2 (핸들러별 5xx 비율, the Korean title means "5xx ratio by handler") is also a panel that asks about ranking. Change type to table, leave the query as it is, set instant to true and format to table, hide the Time column with an organize transformation (set Time to true in excludeByName), and rename the remaining two columns. Do not touch panel 4 (전체 요청률, the Korean title means "total request rate") — it is not a panel that asks about ranking. Then download the dashboard as it stands now and save it to /root/gfd-table/fixed.json (only the .dashboard body), and write to /root/gfd-table/08-review.md, as four lines R1= through R4=, what you changed and why, each at least 30 characters.

If you receive an instant query with format: table, the columns are the three Time, handler, and Value. A time column gives no information in a list at a single moment, so you hide it. To keep the file and the screen from drifting apart, you must download again after fixing. The four lines of the record are the place to write, in order, why you turned it into a table, why an instant query, which transformation you attached and why, and which panel you left as a line graph and why.