TT Lab
Get started
Learn Learning paths Courses

Grafana Dashboards

Fold Four Dashboards Into One Dropdown

Continue in TT Lab

Goal

You build four dashboards copied per target yourself and count how many places need fixing, then fold them into a single dashboard with a query variable, multi-value selection, select-all, panel repeat, and a dependent variable. At the end, you count the number of panels that unfold to set an upper limit, merge old copies, and prove that the same answer comes out.

Why it matters

The most common way a dashboard breaks is copying. A copy is not fixed when the original is fixed, and because the screen looks fine, nobody knows. A variable turns that copying into one dropdown. However, adding a variable does not end with one line of declaration. The moment you let it select several values, the values unfold as a regular expression, so the query's matcher must change too, and once you turn on panel repeat, the number of panels drawn on screen is multiplied by the number of options. The convenience and the cost are attached to the same knob, so for a dashboard that uses repeat you must also set an upper limit on the number of panels that unfold. This lab turns that knob once and checks with numbers.

Steps

  1. Start Grafana with lab-start-grafana, and upload four sets, replacing __UID__ and __ROUTE__ in /opt/lab/gfd/gfd-variables/copy-template.json. The uids are gfd-vars-c1 through gfd-vars-c4, and in the __ROUTE__ place you put /api/orders, /api/search, /api/users, and /healthz, one each. Then write three lines to /root/gfd-variables/01-copies.txt — dashboards= is the number of dashboards tagged gfd-vars-copy, panels_per_dashboard= is the number of panels in one of them, and edit_sites= is the number of places you must touch to fix a query once (the product of the two).
  2. Create a new dashboard with the uid gfd-vars. It must have a query-type template variable named route, and that variable reads its values with label_values(http_requests_total, handler). There is one panel, and its query is the 5xx ratio narrowed by handler="$route". Then write the values that variable actually obtains, one per line, to /root/gfd-variables/02-values.txt.
  3. Turn on multi-value selection (multi) and select-all (includeAll) for the route variable, and change the matcher that selects the handler in the panel query from handler="$route" to handler=~"$route". Then write a single line starting with route_all= to /root/gfd-variables/03-interp.txt — it is the string, exactly, that goes into the $route place inside the Prometheus query when all four values are selected (including the parentheses and vertical bars).
  4. Put repeat on the 5xx ratio panel so that one panel is created for each selected value of route. Set repeatDirection to horizontal (h) and maxPerRow to 2. The panel title must include $route so that you can tell which route a picture is for.
  5. Add a second query variable named code. The query is label_values(http_requests_total{handler=~"$route"}, status), and set refresh to the value that also rereads when the time range changes (2). In templating.list, route must come before code. Then add one panel that uses both $route and $code, and one overall request-rate panel that uses no variables, to make three panels. Finally, write to /root/gfd-variables/05-order.txt a line starting with order= (the variable names joined by commas in templating.list order) and a line starting with reason= (why the order must be that way, at least 40 characters).
  6. Turn on multi-value selection (multi) for the code variable too, and put repeat: code on the panel that uses $route and $code together, making two repeated panels. Then write to /root/gfd-variables/06-cost.tsv one line per variable, in templating.list order, with four tab-separated fields <변수이름> <옵션수> <그 변수로 반복되는 패널 수> <둘의 곱> (the placeholders are the variable name, the number of options, the number of panels repeated by that variable, and the product of the two). Next, write three lines to /root/gfd-variables/06-cap.txt — static_panels= is the number of panels without repeat, expanded_total= is the sum of the products plus that number, and max_expanded= is the upper limit you decided to set on this dashboard (an integer at or above the current value).
  7. /opt/lab/gfd/gfd-variables/legacy.json is an old dashboard where the same panel appears four times with only the handler value changed. Merge it into a single dashboard with the uid gfd-vars-new and upload it. There are three conditions — exactly one panel, that panel repeats over a multi-value query variable, and the query is narrowed by that variable. The variable must read its values with label_values(http_requests_total, handler), and multi-value selection and select-all must be on.
  8. Write four lines to /root/gfd-variables/08-proof.tsv. Each line has two tab-separated fields <핸들러 값> <합친 패널의 쿼리에서 변수를 그 값으로 바꾼 PromQL> (the placeholders are the handler value and the PromQL of the merged panel's query with the variable replaced by that value). The first fields of the four lines are the four values of the handler label of http_requests_total, and the second field must not contain the variable symbol ($). The grader throws the query of each line and the query of the panel for the same handler in legacy.json at the same moment and checks whether the values are the same.

Notes

Start by counting how many copies there are

Start Grafana with lab-start-grafana, and upload four sets, replacing __UID__ and __ROUTE__ in /opt/lab/gfd/gfd-variables/copy-template.json. The uids are gfd-vars-c1 through gfd-vars-c4, and in the __ROUTE__ place you put /api/orders, /api/search, /api/users, and /healthz, one each. Then write three lines to /root/gfd-variables/01-copies.txt — dashboards= is the number of dashboards tagged gfd-vars-copy, panels_per_dashboard= is the number of panels in one of them, and edit_sites= is the number of places you must touch to fix a query once (the product of the two).

Starting Grafana takes 20–40 seconds. Wait until curl -s http://127.0.0.1:3000/api/health gives "database": "ok".

Turning the template into one set is just a string replacement. The paths contain slashes, so it is easier to change the delimiter of sed to another character like #.

sed 's#__ROUTE__#/api/orders#g; s#__UID__#gfd-vars-c1#g' /opt/lab/gfd/gfd-variables/copy-template.json > /tmp/c1.json
jq -n --slurpfile d /tmp/c1.json '{dashboard: $d[0], overwrite: true}' \
  | curl -s -X POST -H 'Content-Type: application/json' -d @- http://127.0.0.1:3000/api/dashboards/db
curl -sG http://127.0.0.1:3000/api/search --data-urlencode 'tag=gfd-vars-copy' | jq 'length'

The third number is what this lab wants to get rid of. Right now it is eight places, but with forty targets it would be eighty.

A variable that reads its value list from the data

Create a new dashboard with the uid gfd-vars. It must have a query-type template variable named route, and that variable reads its values with label_values(http_requests_total, handler). There is one panel, and its query is the 5xx ratio narrowed by handler="$route". Then write the values that variable actually obtains, one per line, to /root/gfd-variables/02-values.txt.

A custom type where you list the values by hand goes stale the day a fifth route appears. It must be a query type that asks the data source.

What the variable actually obtains you can find by asking the data source directly. The value list is not stored in the dashboard JSON.

DS=$(curl -s http://127.0.0.1:3000/api/datasources | jq -r 'map(select(.type=="prometheus")) | .[0].uid')
curl -sG "http://127.0.0.1:3000/api/datasources/proxy/uid/$DS/api/v1/label/handler/values" \
  --data-urlencode 'match[]=http_requests_total' | jq -r '.data[]'

You may create the dashboard by clicking in the web preview or upload it with /api/dashboards/db. The grader looks only at the result uploaded to Grafana.

The moment you select several values, the matcher changes

Turn on multi-value selection (multi) and select-all (includeAll) for the route variable, and change the matcher that selects the handler in the panel query from handler="$route" to handler=~"$route". Then write a single line starting with route_all= to /root/gfd-variables/03-interp.txt — it is the string, exactly, that goes into the $route place inside the Prometheus query when all four values are selected (including the parentheses and vertical bars).

How several values are turned into one string is decided by the data source. Prometheus is a data source that uses regular expressions, so the values are joined by vertical bars and wrapped in parentheses.

So the matcher must change too. An equals matcher matches only when the string is exactly the same, so the moment the unfolded regular expression goes in, it matches no series at all and becomes an empty graph. A graph that works when you pick one value and goes empty the moment you pick two is almost always for this reason.

The values in this lab have no regular-expression special characters, so no escaping occurs. The order follows the order of the value list.

Panel repeat — as many panels as targets are created

Put repeat on the 5xx ratio panel so that one panel is created for each selected value of route. Set repeatDirection to horizontal (h) and maxPerRow to 2. The panel title must include $route so that you can tell which route a picture is for.

Repeat applies only to multi-value variables. If you did not turn on multi in the earlier step, there is only one value to repeat, so there is only one panel.

In dashboard JSON, you put "repeat": "<변수이름>" in the panel object (the placeholder is the variable name). For vertical spreading, repeatDirection is v, and in that case maxPerRow is not used.

Inside a repeated panel, the variable resolves to that panel's single value. So if you put the variable in the title, the four titles each differ. To see how many are actually drawn, look by eye in the web preview — the JSON the server returns contains only the single pre-repeat panel.

When a variable depends on a variable — order and refresh

Add a second query variable named code. The query is label_values(http_requests_total{handler=~"$route"}, status), and set refresh to the value that also rereads when the time range changes (2). In templating.list, route must come before code. Then add one panel that uses both $route and $code, and one overall request-rate panel that uses no variables, to make three panels. Finally, write to /root/gfd-variables/05-order.txt a line starting with order= (the variable names joined by commas in templating.list order) and a line starting with reason= (why the order must be that way, at least 40 characters).

If you use the first variable inside the second variable's query, Grafana notices that link and rereads the later value when the earlier value changes. For that, the earlier one must already be resolved — the order of the array is the order of resolution.

refresh is a number. The value that rereads only when the dashboard is opened and the value that also rereads when the time range changes are different. If you leave a variable whose candidates depend on the time range at the earlier value, a person who went to look at yesterday sees today's list.

The overall request-rate panel is counted in the next step as a "panel that does not repeat." Do not narrow it by handler.

Count the panels that unfold and set an upper limit

Turn on multi-value selection (multi) for the code variable too, and put repeat: code on the panel that uses $route and $code together, making two repeated panels. Then write to /root/gfd-variables/06-cost.tsv one line per variable, in templating.list order, with four tab-separated fields <변수이름> <옵션수> <그 변수로 반복되는 패널 수> <둘의 곱> (the placeholders are the variable name, the number of options, the number of panels repeated by that variable, and the product of the two). Next, write three lines to /root/gfd-variables/06-cap.txt — static_panels= is the number of panels without repeat, expanded_total= is the sum of the products plus that number, and max_expanded= is the upper limit you decided to set on this dashboard (an integer at or above the current value).

The number of options is not in the dashboard JSON. The server returns the variable's options empty, so you have to count by asking the data source directly. For route it is the number of values of the handler label, and for code it is the number of values of the status label.

curl -sG "http://127.0.0.1:3000/api/datasources/proxy/uid/$DS/api/v1/label/status/values" \
  --data-urlencode 'match[]=http_requests_total' | jq '.data | length'

The tab must be a real tab character. Use \t in printf or @tsv in jq -r. Deciding the upper limit is the point of this step — repeat is not free; it is a cost multiplied by the number of options.

Merge the four copied panels into one variable

/opt/lab/gfd/gfd-variables/legacy.json is an old dashboard where the same panel appears four times with only the handler value changed. Merge it into a single dashboard with the uid gfd-vars-new and upload it. There are three conditions — exactly one panel, that panel repeats over a multi-value query variable, and the query is narrowed by that variable. The variable must read its values with label_values(http_requests_total, handler), and multi-value selection and select-all must be on.

If you put the queries of the four panels side by side, they differ in only one place. If you change that one place into a variable, it becomes one panel.

jq -r '.panels[] | .targets[0].expr' /opt/lab/gfd/gfd-variables/legacy.json

You may name the variable whatever you like. The grader finds the name that the panel's repeat points to in that dashboard's templating.list and checks it. The matcher must be able to accept several values, so it is the regular-expression kind.

You cannot throw the merged panel's query with promq while it still contains the variable. If you replace the variable with a single value and throw it, you can check whether the same number comes out as the original panel.

Does the merged panel give the same answer as the original four panels

Write four lines to /root/gfd-variables/08-proof.tsv. Each line has two tab-separated fields <핸들러 값> <합친 패널의 쿼리에서 변수를 그 값으로 바꾼 PromQL> (the placeholders are the handler value and the PromQL of the merged panel's query with the variable replaced by that value). The first fields of the four lines are the four values of the handler label of http_requests_total, and the second field must not contain the variable symbol ($). The grader throws the query of each line and the query of the panel for the same handler in legacy.json at the same moment and checks whether the values are the same.

You can pull the merged panel's query out of Grafana.

curl -s http://127.0.0.1:3000/api/dashboards/uid/gfd-vars-new \
  | jq -r '.dashboard.panels[0].targets[0].expr'

Replacing the variable symbol with a value is a string replacement. If you wrote it with braces, like ${target}, you must replace that form too.

What this step proves is "it gives the same answer even after merging." If merging changed the values, that dashboard is not merged but a different dashboard.