The Moment You Copy a Dashboard, It Forks

In one line

Template variables let you stop copying dashboards. Copies inevitably drift apart, and once they have, nobody knows which one is right.

Why this was needed

If there are twelve services, making twelve dashboards seems natural at first. You make one, copy it, and change only the service name in the queries.

The problem comes after that. If you decide to use a 1-minute rate instead of a 5-minute rate on the error rate panel, you have to fix twelve places, and in reality only three or four get fixed. And when the thirteenth service appears, nobody makes a dashboard. That service is invisible to everyone until it has an outage.

Variables remove this problem structurally. There is one dashboard, and only the target being viewed changes from a dropdown at the top of the screen.

How it works

There are several kinds of variables, but the ones that diverge in practice are two.

Kind	Where values come from	Problem
custom	A person writes the list	It goes stale the day a target is added
query	Read from the data	None. Use this one

With a Prometheus data source, you read it with label_values.

label_values(http_requests_total, handler)

The handler values that actually exist in the data right now become the dropdown. If a handler is added, the dropdown grows on its own.

And a variable does nothing just by being declared. The panel query has to be narrowed by that variable.

sum(rate(http_requests_total{handler="$handler"}[5m]))

The notations $handler, ${handler}, and [[handler]] all work. For a variable that can select several values (multi-value), write =~"$handler" instead of = — because Grafana joins the values in the form a|b|c.

Common misconceptions

"Adding a variable makes it slower." Usually the opposite. A query narrowed by a variable reads fewer time series and actually gets faster. What gets slower is when you make a variable out of a label with thousands of values, but that is not a problem with variables; it is a sign that the label cannot be chosen from a dropdown.

"Variable names can be anything." Name it the same as the label. If $svc points to the job label, the person fixing that dashboard six months later will certainly be confused.

What really matters in practice

You cannot use dashboard variables in alert rules. An alert is evaluated outside the dashboard, without a screen. There is no dropdown to resolve $handler, so those characters fly to Prometheus as they are and the query dies with a syntax error.

So when you move a panel query to an alert, you must resolve the variable. To watch a specific handler, write the value directly, and to watch everything, remove the variable condition. This is why "create an alert from a dashboard" does not end with copy and paste.