Rook-Ceph Guide 3: Reading the values of rook-ceph and rook-ceph-cluster charts 1.21.1

Status: Draft (2026-10-29), based on chart originals, unmeasured. All values are from Rook v1.21.1 tag pointing commit 09d2f0d8d883b70b30de796c4390371be725ff3b of deploy/charts/rook-ceph/values.yaml and deploy/charts/rook-ceph-cluster/values.yaml read directly, and the line numbers in the table are the lines of that file. Total resources and capacity are calculated valuesusing defaults from values and Ceph documentation and were not measured. Actual pods and capacity running in the TT Lab cluster (3 VMs, each with a 17GiB empty disk) will be covered in Part 4. Writing convention: Measured data is the default, unmeasured items are marked as "Unmeasured", and items not found in the original text are marked as "Unconfirmed".

Table of Contents

• 1. There are three charts, and there is an order
• 2. Important values in rook-ceph (operator chart)
• 3. Outer values of rook-ceph-cluster
• 4. cephClusterSpec: CephCluster itself
  • Resource requests (resources)
• 5. Default values for block pool, filesystem, and object store
  • cephBlockPools: ceph-blockpool and StorageClass ceph-block
  • cephFileSystems: ceph-filesystem and StorageClass ceph-filesystem
  • cephObjectStores: ceph-objectstore and StorageClass ceph-bucket
• 6. Dangerous places if default values are used as-is
• 7. When reducing memory and CPU in the lab cluster
  • Capacity calculation
• 8. Conclusion
• References

1. There are three charts, and there is an order

The Rook documentation (Helm Charts Overview) introduces three charts for v1.21. It is commonly known that "you just need to install rook-ceph and rook-ceph-cluster two charts," but starting from v1.20, another one has been added in the middle.

If you omit step 2, PVCs will remain stuck in pending. The upgrade documentation notes that without this chart, the CSI drivers will enter a failed state due to missing service accounts, and the installation procedure in the operator chart documentation also follows the order: ‐operator chart → CSI driver chart → cluster‐ (Ceph Operator Helm Chart, Rook Upgrades). ceph-csi-drivers , because installing with default values alone causes the driver to fail, the documentation emphasizes that you must also include the Rook repository's deploy/charts/ceph-csi-drivers/values.yaml. That file adjusts the driver name to match the operator namespace, such as rook-ceph.rbd.csi.ceph.com .

helm repo add rook-release https://charts.rook.io/release
helm install --create-namespace -n rook-ceph rook-ceph rook-release/rook-ceph -f operator-values.yaml

helm repo add ceph-csi-operator https://ceph.github.io/ceph-csi-operator
helm install -n rook-ceph ceph-csi-drivers ceph-csi-operator/ceph-csi-drivers \
  -f https://raw.githubusercontent.com/rook/rook/v1.21.1/deploy/charts/ceph-csi-drivers/values.yaml

helm install -n rook-ceph rook-ceph-cluster rook-release/rook-ceph-cluster \
  --set operatorNamespace=rook-ceph -f cluster-values.yaml

The command is copied directly from the documentation example. The cluster chart documentation recommends not copying the entire values file, but rather creating a file containing only the values you want to change. For reference, the tag Chart.yaml in both charts specifies version: 0.0.1 , suggesting that the chart version numbers are appended during the release process (unconfirmed). Starting from v1.21, OCI chart tags do not include v .

2. Important values in rook-ceph (operator chart)

The entire file is short, around 245 lines (values.yaml).

Most CSI configurations have moved away from this chart. Previously, values like provisioner replica count, kubelet path, and per-driver resources were set under csi in this chart, but upgrade documentation now instructs to find such values in the ceph-csi-drivers chart configuration. What remains in this chart for csi are whether to install subcharts, image overrides, and ServiceMonitor. Configurations for the ceph-csi-operator subchart itself are written in the values ceph-csi-operator section (L86).

3. Outer values of rook-ceph-cluster

From here on is the 844-line cluster chart (values.yaml). cephClusterSpec ). We start by looking at values outside of it.

4. cephClusterSpec: The CephCluster itself

cephClusterSpec goes directly into the spec of the CephCluster CR. The cluster chart documentation notes that this default value is "just an example and unlikely to fit your cluster."

Resource requests (resources)

cephClusterSpec.resourcesis the default value of (L340). Rook documentation states that "if too low, it only logs a warning and starts anyway," and also lists the recommended minimum thresholds corresponding to it(CephCluster CRD). Looking at the source code, this check is based on the limit , and considers it an error if the limit is smaller than the request (spec.go#L486-L520).

The documentation states that if you lower the OSD memory, Rook adjusts osd_memory_target of Ceph to match. As seen in Part 1, the Ceph default for this value is 4Gi, so the chart default (4Gi) is equal to the Ceph default. The resources for MDS and RGW are located in their respective sections rather than here (Section 5).

5. Defaults for Block Pools, Filesystems, and Object Stores

By default, the cluster chart creates one of each of these three types and attaches a StorageClass to each. These StorageClasses are created by the chart template, not the operator (Part 2, Section 9).

cephBlockPools: ceph-blockpool and StorageClass ceph-block

The template (cephblockpool.yaml) appends provisioner: <operatorNamespace>.rbd.csi.ceph.com and pool·clusterID(cluster namespace).

cephFileSystems: ceph-filesystem and StorageClass ceph-filesystem

Since the preserveFilesystemOnDelete of CephFilesystem is not in the chart, the CRD default false is used. This means that deleting the CR also deletes the filesystem (CephFilesystem CRD).

cephObjectStores: ceph-objectstore and StorageClass ceph-bucket

Multi-site (realm, zonegroup, zone), object store users, and EC block pools are only included as comments in values and are not created by default (starting from L743).

6. Areas Dangerous to Use with Default Settings

Warning 1. useAllNodes: true + useAllDevices: true(L418, L419). All available devices on all nodes become OSDs and are formatted. As seen in Part 2, devices that are mounted or have filesystem/OSD signatures are skipped, but empty disks with no signatures are claimed without asking. This targets disks attached for other purposes or those not yet formatted. The CephCluster documentation states that useAllDevices is "not recommended unless in a highly controlled environment with no risk of destroying devices," and recommends specifying nodes using useAllNodes: false in production. At a minimum, specify deviceFilter(e.g., a regex matching only empty disk names in a practice VM) or node-specific devices .

Warning 2. ceph-block of isDefault: true(L531). The moment you install it, ceph-block becomes the default StorageClass for the entire Kubernetes cluster. If another default StorageClass already exists, there will be two defaults, and Kubernetes will use storageClassName the most recently created default StorageClass for PVCs without (Kubernetes: Default StorageClass). New PVCs from other apps will silently go to Ceph. When mounting onto an existing cluster, leave it as false .

Warning 3. Object store data pool EC 2+1 (L674). The Ceph documentation notes that the M=1 profile is "strongly discouraged for production data." Data becomes unavailable during maintenance and can be lost if failures overlap. Also, since K+M=3, host failure domains must include at least three hosts, and if you set min_size to K+1=3 as recommended, a single host going down will cause that pool to stop accepting writes (actual min_size value to be verified in Part 4, unmeasured). For production, consider replication 3 or K=4, M=2.

Warning 4. Things that disappear when deleted. The reclaimPolicy of all three StorageClasses is Delete , so deleting a PVC also deletes the data. For filesystems, preserveFilesystemOnDelete defaults to false , so deleting the CR also deletes the filesystem. Conversely, for object stores, preservePoolsOnDelete: true means the pools are retained. Keep in mind that the defaults differ among the three.

Warning 5. crds.enabled(operator chart L21). Comments warn that changing the value set during initial installation to false later may destroy the cluster. The same comment points to the "Restore CRDs after deletion" section of the disaster recovery documentation in case the CRD is deleted. Determine this value once during installation and do not change it.

Warning 6. requireMsgr2: true and keyType: aes(L231, L176). msgr2 requires kernel 5.11 or higher, and upgrade documentation instructs disabling it with requireMsgr2: false if using an older kernel. keyType: aes is noted in comments as required if the node kernel is below 7.0. Check your node's kernel version first. mon.allowMultiplePerNode(L184) and requireSafeReplicaSize: false(pool CRD, size 1 allowed) should not be enabled outside of test environments.

7. Reducing Memory and CPU in a Practice Cluster

The TT Lab practice cluster consists of three VMs, each with a single empty disk (17GiB). Calculating the resources requestedby Ceph pods with default installation values yields the following via values. Counts assume a set of nodes, a set of disks, and one default pool, filesystem, and object store each.

CSI pods (ctrlplugin, nodeplugin) and ceph-csi-operator requests are excluded as they use chart defaults (unverified). If pods get stuck in Pending , first compare this total with the VM's CPU and memory. Actual scheduled values will be verified in Part 4.

These are the values to tweak when reducing resource usage. The below is a proposaland has not been launched with this combination (unmeasured).

cephClusterSpec:
  mgr:
    count: 1                # mgr 하나와 사이드카 둘이 빠진다
  dashboard:
    enabled: false
  crashCollector:
    disable: true           # cluster-test.yaml 도 끈다
  logCollector:
    enabled: false          # 데몬마다 붙는 로그 사이드카가 빠진다
  resources:
    mon:
      requests: {cpu: 250m, memory: 512Mi}
      limits: {memory: 1Gi}       # 권장 최소 1024MB 와 같다
    mgr:
      requests: {cpu: 250m, memory: 512Mi}
      limits: {memory: 512Mi}     # 권장 최소 512MB
    osd:
      requests: {cpu: 500m, memory: 2Gi}
      limits: {memory: 2Gi}       # 권장 최소 2048MB, osd_memory_target 도 2Gi 가 된다
cephFileSystems:
  - name: ceph-filesystem
    spec:
      metadataServer:
        activeCount: 1
        activeStandby: false
        resources:
          requests: {cpu: 250m, memory: 1Gi}
          limits: {memory: 1Gi}   # 권장 4096MB 보다 낮아 operator 가 경고를 찍는다
cephObjectStores:
  - name: ceph-objectstore
    spec:
      gateway:
        instances: 1
        resources:
          requests: {cpu: 250m, memory: 512Mi}
          limits: {memory: 1Gi}

Since list values (cephFileSystems, cephObjectStores) are completely replaced by Helm, the actual file must include the remaining fields of the defaults (pools, StorageClass) together. The snippet above was shortened to show only modified fields. If RWX volumes or S3 are not needed, removing them entirely with cephFileSystems: [], cephObjectStores: [] yields the greatest reduction. activeStandby: false merely changes the standby MDS to a cold standby while keeping the MDS count at 2 (Part 2, Section 2).

Calculating the request total for the above combination in the same way yields approximately 10.8Gi of memory (osd 6Gi, mds 2Gi, mon 1.5Gi, mgr 512Mi, rgw 512Mi, exporter 150Mi, operator 128Mi) and about 3.6 cores of CPU. In exchange, the OSD memory target drops from 4Gi to 2Gi, shrinking the BlueStore cache (performance impact unmeasured), and the MDS is lower than Rook's recommendation. The performance impact will be covered in a separate performance post after measurements are complete.

An official example in the same direction is Rook storage's deploy/examples/cluster-test.yaml. This configuration omits resource requests entirely, sets mon and mgr to one each, osd_memory_target to 1GiB, and pool size to 1. The header comment introduces it as a "small test cluster where a single node is enough." Since there is only one replica, there is no data protection.

Capacity Calculation

The raw capacity of the disk set is 51GiB. Ceph warns when OSD utilization exceeds mon_osd_nearfull_ratio(0.85), and halts writes when it reaches mon_osd_full_ratio(0.95) (Monitor Config Reference).

Since all pools share the same raw capacity, the rows cannot be simply added. Fixed overhead for BlueStore and RocksDB, and PG unevenness are excluded (unmeasured). Usable capacity can be verified with ceph df in Part 4.

8. Summary

• There are threecharts in the order of rook-ceph → ceph-csi-drivers → rook-ceph-cluster . Omitting the middle one causes CSI to fail.
• The operator chart is short, and defaults are mostly sufficient. Things to verify include crds.enabled, monitoring.enabled, and OBC field allowances.
• Cluster chart defaults are "examples." At a minimum, explicitly configure storage(which disks to use), isDefault, object data pool EC profiles, and resource requests.

TT's Word. values.yaml is a letter saying, "I'll set things up like this by default." However, this letter mixes in quite bold statements like "I'll use all empty disks" and "I'll be the default StorageClass." Please be sure to re-read those few lines before clicking install!

References

Both values and documentation were accessed on 2026-10-29 at the above commits.

• rook-ceph values.yaml: https://github.com/rook/rook/blob/09d2f0d8d883b70b30de796c4390371be725ff3b/deploy/charts/rook-ceph/values.yaml
• rook-ceph-cluster values.yaml: https://github.com/rook/rook/blob/09d2f0d8d883b70b30de796c4390371be725ff3b/deploy/charts/rook-ceph-cluster/values.yaml
• Rook-adjusted ceph-csi-drivers values: https://github.com/rook/rook/blob/09d2f0d8d883b70b30de796c4390371be725ff3b/deploy/charts/ceph-csi-drivers/values.yaml
• Helm Charts Overview, Operator 차트, Cluster 차트, CSI 드라이버 차트 문서: https://github.com/rook/rook/tree/09d2f0d8d883b70b30de796c4390371be725ff3b/Documentation/Helm-Charts
• CephCluster CRD: https://github.com/rook/rook/blob/09d2f0d8d883b70b30de796c4390371be725ff3b/Documentation/CRDs/Cluster/ceph-cluster-crd.md
• CephBlockPool CRD: https://github.com/rook/rook/blob/09d2f0d8d883b70b30de796c4390371be725ff3b/Documentation/CRDs/Block-Storage/ceph-block-pool-crd.md
• CephFilesystem CRD: https://github.com/rook/rook/blob/09d2f0d8d883b70b30de796c4390371be725ff3b/Documentation/CRDs/Shared-Filesystem/ceph-filesystem-crd.md
• CephObjectStore CRD: https://github.com/rook/rook/blob/09d2f0d8d883b70b30de796c4390371be725ff3b/Documentation/CRDs/Object-Storage/ceph-object-store-crd.md
• cluster-test.yaml: https://github.com/rook/rook/blob/09d2f0d8d883b70b30de796c4390371be725ff3b/deploy/examples/cluster-test.yaml
• Ceph Erasure Code: https://docs.ceph.com/en/tentacle/rados/operations/erasure-code/
• Ceph Monitor Config Reference: https://docs.ceph.com/en/tentacle/rados/configuration/mon-config-ref/
• Kubernetes Storage Classes: https://kubernetes.io/docs/concepts/storage/storage-classes/