TT Lab
Blog

한국어 English 日本語 中文

Get started
Blog

Blog

Presenting Two Ways to Fix and Letting the Lead Choose: From v2.1.0 to v2.3.0 in the Kubeflow Trainer Installation Guide

Result: Merged (2026-10-28 01:03, kubeflow/trainer#4334), issue #4310 closed, milestone v2.4. Sourced from GitHub issues, PRs, reviews, timelines, inspection records, Trainer source code (the PR's base commit 8e076764, chart values for tags v2.1.0, v2.2.0, and v2.3.0), Kubeflow's AI policy document, and the contribution ledger draft. Times are in KST.

Table of Contents

One-Line Summary

Issue kubeflow/trainer#4310 Installation guide pins chart v2.1.0, so the runtimes and crds flags shown below it are silently ignored
PR kubeflow/trainer#4334 fix(docs): use v2.3.0 in the installation guide
Changes Updated installation and data cache documentation VERSION to v2.3.0 and added a version-pinned runtime installation command. 9 additions, 3 deletions
Response Lead commented on the issue, "Let's bump to v2.3, want to try fixing it yourself?", and on the PR /ok-to-test·/lgtm·/approve all at once
Result Issue opened 10-26 21:43 → Lead reply 10-27 00:28 → PR opened 14:28 → Approved 10-28 00:02 → Merged 01:03. PR took about 10 hours and 35 minutes; from issue to merge took about 27 hours and 20 minutes

1. What Was the Problem: Values Ignored Without Error

TT Lab is in the process of migrating its table tennis scoreboard recognition model learning to Kubeflow, and Kubeflow Trainer is the component that runs those training jobs on Kubernetes. While reading the installation documentation, I noticed a mismatch between the top and bottom sections. Line numbers are based on the PR's base commit 8e076764 .

chart's values.yaml opened across tags reveals the following.

Chart Version runtimes Key crds Key values.schema.json
v2.1.0 None None None
v2.2.0 line 181 None None
v2.3.0 line 190 line 25 None

Helm 은 값 검증 스키마가 없는 차트에 모르는 키를 넘겨도 오류를 내지 않습니다. 그래서 문서를 그대로 따라 하면 2.1.0 제어부가 깔리고, ClusterTrainingRuntime 은 하나도 생기지 않으며, 왜 그런지 알려 주는 메시지도 없습니다.

There were two more side issues. In the kustomize installation section, while the control plane had a version-pinned command (?ref=${VERSION}, line 87), the runtime section only had a ?ref=master command (lines 96–99). The "Install with Release Version" path would mix release CRDs with master runtimes. Before documentation was migrated from kubeflow.org to this repository's doc site (#3308), both commands were present. Also, docs/user-guides/data-cache.md line 51 was also v2.1.0.

I note separately what was and was not verified. Chart values were checked by directly opening tag-specific files (table above, re-verified on 2026-10-29), and I verified that chart 2.3.0 is published in the OCI registry and that manifests/overlays/runtimes·data-cache exists in v2.3.0. However, I did not actually run the Helm and kustomize commands on a live cluster to observe runtimes failing to generate under 2.1.0 (unmeasured). I explicitly stated this in the PR description as well.

2. Why Start with an Issue Instead of a PR

There was a precedent in this repository where a request to simply bump version numbers was closed. On 2026-07-27, issue #3807 "Bump helm install VERSION to v2.2.1" was closed on the same day by the lead, stating that "README versions are merely examples showing users how to select versions."

Submitting the same content directly as a PR could result in it being closed for the same reason. Therefore, three things were decided during the drafting stage (from the ledger's draft notes).

  1. Open an issue first instead of a PR.
  2. Frame it not as "the version is outdated" but as a functional bug where "commands on the same page are silently ignored in that version." Explicitly reference #3807, noting that this case differs because "commands on the same page rely on values that the example version lacks."
  3. Instead of proposing only version pinning, also propose keeping the example version while adding comments such as "runtimes.* is supported since v2.2.0, and crds.enabled since v2.3.0." State that a PR will be submitted either way.

Issue #4310 was opened on 2026-10-26 at 21:43 in this format. It followed the bug report template's field order, and included an AI disclosure at the end of the body stating, "Claude assisted with research and drafting, including reading chart values across tags."

Two hours and 45 minutes later, the lead replied. Stating that the installation guide should be bumped to v2.3, they asked if I would like to fix it myself. In the same comment, they applied the /kind docs and /good-first-issue labels. In effect, this meant choosing the first of the two options.

3. What Was Changed: Diff Summary

The PR was opened on the afternoon of the day the lead's reply was received.

 # docs/operator-guides/installation.md
-export VERSION=v2.1.0
+export VERSION=v2.3.0
 ...
-Run the following command to deploy Kubeflow Trainer built-in runtimes
+Run the following command to deploy Kubeflow Trainer built-in runtimes:
+
+kubectl apply --server-side -k "https://github.com/kubeflow/trainer.git/manifests/overlays/runtimes?ref=${VERSION}"
+
+For the latest changes run:
 
 kubectl apply --server-side -k "https://github.com/kubeflow/trainer.git/manifests/overlays/runtimes?ref=master"

 # docs/user-guides/data-cache.md
-export VERSION=v2.1.0
+export VERSION=v2.3.0

The runtime commands strictly followed the pattern already used by control plane commands (version-pinned commands, with "latest changes" master commands underneath). Chart README, the exact file closed in #3807, was left untouched, and #3807 was linked in the PR description as the reason.

The "How was this tested" section of the PR body stated the following:

AI usage followed Kubeflow's AI Policy. This policy requires building and testing AI-generated contributions locally before opening them, being able to explain them during review, disclosing them in the PR description or commit footers (Assisted-by·Co-authored-by), and ensuring AI does not automatically respond to maintainer feedback. The commit was given Assisted-by: Claude and DCO signatures, and an "AI assistance" paragraph was included in the body stating, "Claude helped find this and draft the changes, and assisted with running the verifications above." The document build mentioned above serves as the policy's "local verification." The commit appears on GitHub as an unsigned commit.

4. Response: Merged Overnight

Time Event
10-26 21:43 Issue #4310 opened. Welcome bot greets with "first issue" message.
10-27 00:28 Lead's reply and /kind docs·/good-first-issue
10-27 14:28 PR opened. Automatically added to project board, two reviewers assigned, size/S label attached.
10-28 00:02 Lead says "Thanks!" along with /ok-to-test·/lgtm·/approve in a single review.
10-28 01:03 Merged, issue #4310 closed, milestone v2.4.

Review comments consisted of just a simple thank you. Since the direction was already decided in the issue, there appeared to be nothing left to question in the PR. All 13 checks, including those run after ok-to-test , passed successfully, with the longest test taking 58 minutes and 42 seconds.

We did not post any separate replies on the issue. The ledger noted: "User replies directly to issues," which effectively translated to replying via the PR the next day. This also aligns with the Kubeflow AI policy guideline that "AI must not automatically respond to maintainer feedback."

5. Lessons Learned: Open Source Contribution Tips

Tt's Brief Note. The first line of the cookbook said, "Take out the version 2.1 pot," and a few lines down it said, "Press the steaming button on the pot." But the version 2.1 pot didn't have a steaming button! Even if you pretend to press it, the pot says nothing, and there are no steamed dumplings for dinner. When I asked, "Should we bump the pot version, or write 'the steaming button is available from version 2.2' instead?", the chef said, "Let's bump the version, would you like to fix it yourself?" The next morning, the fixed cookbook was placed on the bookshelf.

Links

All materials were accessed on 2026-10-29.

Sign in to like

Comments

No comments yet.

Sign in to leave a comment