TT Lab
Get started
Learn Learning paths Courses

Grafana Dashboards

Changing one tag meant editing a file

Continue in TT Lab

Goal

You read a dashboard's versions and save messages, extract the difference between two versions in a form a person can read, pull the data source out into a variable to make it movable, then make the file the original, and even build a tool that checks for itself whether the file and the screen are the same.

Why it matters

If you only put a dashboard in a repository, half a year later the screen and the file are completely different. Nobody has a reason to edit the file, and nothing stops editing on screen. Once you make it provided from a file, saving on screen is blocked from then on, and the file becomes the only way to change the dashboard — that constraint, which looks inconvenient, removes the path by which the screen and the file could drift apart. And the fact that versions are kept and the fact that you know what changed are different. If you do not strip the fields that change on every save, the diff turns entirely red and tells you nothing.

Steps

  1. Start Grafana with lab-start-grafana and upload /opt/lab/gfd/gfd-as-code/start.json (uid gfd-code). Next, change the title of panel 1 to 지금 요청률 (the Korean title means "current request rate") and save it again, attaching a save message of at least 10 characters.
  2. Fetch http://127.0.0.1:3000/api/dashboards/uid/gfd-code/versions and write three lines to /root/gfd-as-code/02-versions.txt. latest= is the latest version number, message= is that version's save message, and first_message= is the save message of version 1. Copy all three values exactly as written in the response.
  3. Create /root/gfd-as-code/normalize.py. It takes the dashboard JSON file path as an argument, keeps only the dashboard body (stripping it if it is wrapped as {"dashboard": ...}), strips the top-level version, id, and iteration, and emits it to standard output with the keys sorted. Keep the panels' id.
  4. Create /root/gfd-as-code/diff.sh. It takes two version numbers as arguments, normalizes the two versions and compares them, and must print nothing and exit with 0 if they are the same, and print the difference and exit with a nonzero value if they differ. Then look at the result of bash /root/gfd-as-code/diff.sh 1 2 and, on the changed= line of /root/gfd-as-code/04-diff.txt, write in at least 20 characters which panel's what changed and how.
  5. Throw a query with a data source uid that does not exist and write the HTTP code that comes back, and this Pod's real prometheus data source uid, to /root/gfd-as-code/05-uid.txt as two lines missing_status= and ds_uid=. Next, create a datasource-type template variable named DS (the target is prometheus), and change both panels to point to that variable (${DS}) and save.
  6. Normalize the current dashboard and save it to /root/gfd-as-code/dash/gfd-code.json, write to /root/gfd-as-code/provisioning/dashboards/lab.yml a provisioning configuration that reads that folder, and then start Grafana again with GF_PATHS_PROVISIONING=/root/gfd-as-code/provisioning. Once the dashboard comes from a file, meta.provisioned in the query response becomes true.
  7. Try saving the current dashboard again as it is, and write the HTTP code that comes back and the message of the response body to /root/gfd-as-code/07-blocked.txt as status= and message=. Then, on the procedure= line, write in at least 40 characters what you must edit and what you must redo from now on to change this dashboard.
  8. Attach a gitops tag to the dashboard. But you must do it by editing the file. Then create /root/gfd-as-code/verify.sh — it takes the dashboard JSON file path as an argument (default /root/gfd-as-code/dash/gfd-code.json), normalizes that file and the current screen and compares them, and must exit with 0 if they are the same and a nonzero value if they differ. Finally, write to /root/gfd-as-code/08-bundle.md two lines: files= (the list of files to put in the repository) and revert= (how to roll back, at least 40 characters).

Notes

If you change it without a save message, nothing remains

Start Grafana with lab-start-grafana and upload /opt/lab/gfd/gfd-as-code/start.json (uid gfd-code). Next, change the title of panel 1 to 지금 요청률 (the Korean title means "current request rate") and save it again, attaching a save message of at least 10 characters.

The save API body is {"dashboard": ..., "overwrite": true, "message": "..."}. If you leave out the message, only the time remains in the version list — which leaves no grounds for judging, half a year later, whether to roll back to this version.

What the version list actually keeps

Fetch http://127.0.0.1:3000/api/dashboards/uid/gfd-code/versions and write three lines to /root/gfd-as-code/02-versions.txt. latest= is the latest version number, message= is that version's save message, and first_message= is the save message of version 1. Copy all three values exactly as written in the response.

You may be surprised by the message of version 1 — it differs from what you wrote and sent. That is the fact to confirm in this step. You can pull out max_by(.version) and select(.version == 1) separately with jq.

Make the two versions comparable

Create /root/gfd-as-code/normalize.py. It takes the dashboard JSON file path as an argument, keeps only the dashboard body (stripping it if it is wrapped as {"dashboard": ...}), strips the top-level version, id, and iteration, and emits it to standard output with the keys sorted. Keep the panels' id.

If fields that change on every save are mixed in, the diff turns entirely red. The same input must always produce the same output, so fix the key order. Python's json.dumps has an option for that.

What changed — a tool that works in both directions

Create /root/gfd-as-code/diff.sh. It takes two version numbers as arguments, normalizes the two versions and compares them, and must print nothing and exit with 0 if they are the same, and print the difference and exit with a nonzero value if they differ. Then look at the result of bash /root/gfd-as-code/diff.sh 1 2 and, on the changed= line of /root/gfd-as-code/04-diff.txt, write in at least 20 characters which panel's what changed and how.

Each version's body is in data of the version list response. The diff command ends with 1 when there is a difference, so you can just use that exit code. Be sure to test that it also works when you give it the same version twice — a tool that works in only one direction cannot distinguish "there is no difference" from "the tool is broken."

Why panels go empty when you move

Throw a query with a data source uid that does not exist and write the HTTP code that comes back, and this Pod's real prometheus data source uid, to /root/gfd-as-code/05-uid.txt as two lines missing_status= and ds_uid=. Next, create a datasource-type template variable named DS (the target is prometheus), and change both panels to point to that variable (${DS}) and save.

The data source proxy path is /api/datasources/proxy/uid/<uid>/api/v1/query. Even if you put in a uid that does not exist, the screen shows an empty graph rather than an error — that is why the cause is hard to find when you have moved. Put the variable in templating.list, and change the panel's datasource.uid to the variable reference string.

Make the file the original

Normalize the current dashboard and save it to /root/gfd-as-code/dash/gfd-code.json, write to /root/gfd-as-code/provisioning/dashboards/lab.yml a provisioning configuration that reads that folder, and then start Grafana again with GF_PATHS_PROVISIONING=/root/gfd-as-code/provisioning. Once the dashboard comes from a file, meta.provisioned in the query response becomes true.

Provisioning configuration is read only once, when Grafana starts — if you only write the file and do not start it again, nothing happens. The configuration contains apiVersion and providers, and options.path points to the folder that holds the dashboard JSON. When starting again, you must leave the data folder (GF_PATHS_DATA) as it is for the version history to remain.

Now you cannot save from the screen

Try saving the current dashboard again as it is, and write the HTTP code that comes back and the message of the response body to /root/gfd-as-code/07-blocked.txt as status= and message=. Then, on the procedure= line, write in at least 40 characters what you must edit and what you must redo from now on to change this dashboard.

It is a request that gets rejected, so nothing changes. Feel free to throw it. You can get the response code with curl -w '%{http_code}'. This constraint, which looks inconvenient, is exactly the mechanism that keeps the screen and the file from drifting apart.

Application — change the screen by editing the file and check that the two are the same

Attach a gitops tag to the dashboard. But you must do it by editing the file. Then create /root/gfd-as-code/verify.sh — it takes the dashboard JSON file path as an argument (default /root/gfd-as-code/dash/gfd-code.json), normalizes that file and the current screen and compares them, and must exit with 0 if they are the same and a nonzero value if they differ. Finally, write to /root/gfd-as-code/08-bundle.md two lines: files= (the list of files to put in the repository) and revert= (how to roll back, at least 40 characters).

After editing the file, you must have Grafana re-read it for it to be reflected on screen (the same as what you did in the earlier step). Test the checker in both directions — if you give it the repository copy, it should pass, and if you give it another dashboard file (for example /opt/lab/gfd/gfd-as-code/start.json), it should fail. A checker that passes whatever you give it is worse than none.