Grafana Admin Password Is Only Read on First Boot: How a Grafana Helm Chart README PR Was Merged in 26 Minutes

Result: Merged (2026-10-17 01:16, grafana-community/helm-charts#896), released with chart grafana 14.0.1. In fact, this was gathered from GitHub's PR, check, and release logs, the chart source (the base commit of the PR 71c86de1), official Grafana documentation, and commit logs from the TT Lab repository. All times are in Korea Standard Time (KST).

Table of Contents

One-line Summary
1. What Was the Problem: The Symptoms We Experienced
2. Cause: Which Lines in the Documentation and Templates
- Direct Measurements
3. What Was Changed: Diff Summary
4. Reaction: 26 Minutes
5. What We Learned: Tips for Open Source Contribution
Links

One-line Summary


PR	grafana-community/helm-charts#896 `[grafana] Document that the admin password is only applied on first start`
Changes	33 lines added for the "Admin user and password" section in the chart README, chart version 14.0.0 → 14.0.1
Reaction	No bot review. Passed 14 checks, maintainer replied with "Thanks!" and approved
Result	PR opened 10-17 00:50 → Merged 01:16 (26 min), grafana-14.0.1 released 28 seconds later

Figure: The moment when the admin password—read only on first boot—desyncs from the Secret

1. What Was the Problem: The Symptoms We Experienced

Grafana in the TT Lab cluster runs via the Grafana chart bundled with the kube-prometheus-stack chart. On 2026-09-24, as we removed the admin password from the values file and updated it to reference a Secret via admin.existingSecret , we needed to document the procedure for changing the password.

This is where we got stuck. If you only update the Secret to a new value, Grafana restarts, but login only works with the old password. Meanwhile, the sidecar responsible for reloading dashboards and data sources calls Grafana using the new value from the Secret, resulting in authentication failures (based on the chart templates). Therefore, we had to add a step to the procedure stating, "First change the password inside Grafana using grafana cli ."

Many others had experienced the same issue. There were multiple reports in the old repository before the chart was migrated (grafana/helm-charts#152, #3568, #3790), but this information was still missing from the new repository's README.

2. Cause: Which Lines in the Documentation and Templates

Chart line numbers are based on the PR's base commit 71c86de1 .

Grafana reads the admin password only once. The configuration documentation entry for admin_password notes that this value is "Set once on first-run." The chart passes this value as the GF_SECURITY_ADMIN_PASSWORD environment variable, and if the DB survives restarts (such as with a persistent volume or external DB), this variable is ignored from the second boot onward.
The sidecar authenticates using the Secret's value. templates/_pod.tpl configured for each sidecar retrieves REQ_USERNAME·REQ_PASSWORD from the same Secret. The moment the Secret and the DB desync, the sidecar's reload call receives 401 .
When left empty, the random value generated by the chart relies on lookup . templates/_helpers.tpl Lines 131–138 of grafana.password reads the Secret existing in the cluster as lookup and uses that value, generating a 40-character random string if it does not exist. In tools that render via helm template (such as Argo CD), lookup returns nothing, resulting in a new value generated on every render.

Direct Measurements

Before writing the PR, instead of blindly trusting the documentation sentences, we measured it. We spun up a temporary pod using the Grafana 13.2.0 image already cached in the TT Lab cluster, and checked the response code of GET /api/user on top of a sqlite data directory that persisted across restarts.

Step	`pw-one`	`pw-two`
First boot, environment variable `pw-one`	200	401
Restart with the same data, environment variable `pw-two`	200	401
To be included in the README, we ran `grafana cli … reset-admin-password --password-from-stdin` with `pw-two`	401	200

helm template(Helm v3.18.2) twice without adminPassword , and the admin-password value came out differently both times, whereas it remained consistent when --set adminPassword=x was provided.
Although the default chart image (13.2.4-distroless) lacks a shell, /usr/share/grafana/bin is located at PATH , allowing us to invoke kubectl exec directly with grafana cli , and the chart supplies GF_PATHS_DATA so that the CLI opens the same DB as the server. We verified this and included it in the PR description.
The sidecar's 401 is a conclusion derived from the templates and the measurements above, and was not directly observed via sidecar logs (unmeasured).

3. What Was Changed: Diff Summary

We inserted an "Admin user and password" section between existing sections in the chart README and, following repository rules, bumped the chart version even though only documentation was modified.

-version: 14.0.0
+version: 14.0.1

+## Admin user and password
+...
+Grafana reads these variables only when it creates the admin user, that is on the first
+start with an empty database. When the database survives pod restarts ...
+- Changing `adminPassword`, or the password in `admin.existingSecret`, and running
+  `helm upgrade` restarts the pod but does not change the password stored in the database.
+- The sidecars ... call the Grafana reload API with the user and password from the same
+  Secret. Once the Secret and the database disagree, these calls fail with `401 Unauthorized`.
+- When neither `adminPassword` nor `admin.existingSecret` is set, ... `lookup` ...
+To rotate the password of a running installation, change it in Grafana first and then
+update the Secret (or `adminPassword`) to the same value:

The command to change the password was written in a form that passes it via standard input (--password-from-stdin) to prevent the value from lingering in shell history or process arguments. In the PR description, we explained what and why we were changing it, the measurement results above, links to issues from the old repository, and a note stating "Only documentation was changed, so helm-unittest was not added," while completing all items in the template checklist (DCO sign-off, chart version bump, title [grafana] prefix). At the end of the description, we added a note stating that AI assistance was used and then reviewed manually.

4. Reaction: 26 Minutes

Time (10-17)	Event
00:35	Commit (DCO sign-off, SSH sign-off)
00:50	PR opened
00:50 ~	Checks: Chart test, generated files, title, DCO, 9 integration tests, Super Linter — 14 passed
01:14	Maintainer comment "Thanks!"
01:16	Approved, merged 5 seconds later
01:16	Release `grafana-14.0.1` 28 seconds after merge

Unlike the SeaweedFS case, there was no review bot, and the maintainer's comment was a single word. This meant there was nothing to fix. There was nothing extra for us to handle.

5. What We Learned: Tips for Open Source Contribution

Old issues remain in the migrated repository. As the chart moved to a new organization, reports from the old repository were not reflected in the new README. Linking old issue numbers in the description lets the maintainer immediately know that this is not "one person's complaint" but "a long-standing known issue."
Pair a documentation sentence with a response code table. "Set once on first-run" is correct but brief. The 200/401 table shows what that sentence actually means in real operations.
Test commands intended for the README firsthand. We verified that grafana cli is in PATH on the distroless image and that it opens the same DB before writing it down. If the commands readers copy and paste are wrong, documentation is worse than nothing.
Align with repository rules first. The rule to bump the chart version even on doc-only changes, the title prefix, and DCO sign-off. A PR with a fully completed checklist gives the maintainer less to review. This time, it automatically led straight to a merge and release.
Document the same trap in our own operational documentation. Upstream documentation is for everyone, but operational procedures must be reproducible as-is within the TT Lab cluster. The order of operations in this PR (Grafana first, Secret later) was already present in our operational procedures.

Titi's Word. The password is a friend who only memorizes things the very first time. Even if you change the note (Secret), the memory stays the same, so you have to tell your friend the new password directly first, and then change the note. When we wrote that down in the docs, a "Thanks!" came back in 26 minutes.