Rook-Ceph Guide 2: What the Rook Operator Does and Does Not Do

Status: Draft (2026-10-29), based on source code and official documentation, no actual measurements taken. The baseline version is Rook v1.21.1(commit pointed to by the tag 09d2f0d8d883b70b30de796c4390371be725ff3b, released on 2026-10-26) and ceph-csi-operator included as a subchart by the Rook chart v1.1.0 (commit 98eee8980fc69e5c9c02c80958abe06c3faba5eb). All source code links in this article are fixed to this commit. Counting the resources created by the operator directly in the TT Lab practice cluster will be covered in Part 4 (written once the raw data is gathered). Writing rule: Measured data is the default; unmeasured items are marked as "[Unmeasured]", and items not found in the original text are marked as "[Unconfirmed]".

Table of Contents

• 1. The Role of Operators and Rook
• 2. CRD List and What Reconcile Creates
• 3. Mon Placement and Failover
• 4. How the OSD Prepare Job Selects Disks
• 5. CSI Deployment: Shared Among Three in v1.21
• 6. Upgrade Order
• 7. Cluster Cleanup: cleanupPolicy
• 8. Toolbox
• 9. What the Operator Does Not Do
• References

1. The Role of Operators and Rook

An Kubernetes operator is a program that translates human operational knowledge into a controller. When a user writes down "desired state" as a Custom Resource (CR), the operator compares it with the current state, creating what is missing and deleting what is excess. This cycle of comparison and alignment is called a reconcile .

Rook's official documentation (Storage Architecture) divides Rook's work into three layers: the Rook operatorwhich automates Ceph configuration, the Ceph-CSI driverwhich creates and attaches volumes, and the Ceph daemonswhich perform the actual storage. The same document contains the most important single line: "Rook is not in the Ceph data path." As seen in Part 1, application reads and writes travel directly between the application and OSDs, and the operator does not stand in that path. Therefore, even if the operator pod temporarily stops, the I/O of already attached volumes does not pass through the operator. When the operator stops, the things that are delayed are "change operations" handled by the operator, such as creating new OSDs or replacing mons.

2. CRD List and What Reconcile Creates

In v1.21.1, there are deploy/examples/crds.yaml21 CRDs (schemas defining types of CRs).

CephBlockPool, CephBlockPoolRadosNamespace, CephBucketNotification, CephBucketTopic, CephCOSIDriver, CephClient, CephCluster, CephFilesystem, CephFilesystemMirror, CephFilesystemSubVolumeGroup, CephNFS, CephNVMeOFGateway, CephObjectRealm, CephObjectStore, CephObjectStoreAccount, CephObjectStoreUser, CephObjectZone, CephObjectZoneGroup, CephRBDMirror, ObjectBucket, ObjectBucketClaim

Dividing what frequently used ones create during a reconcile into the Ceph side and the Kubernetes side yields the following.

The rest are briefly noted by usage: CephNFS (NFS gateway), CephRBDMirror and CephFilesystemMirror (mirroring to other clusters), CephObjectRealm, ZoneGroup, and Zone (RGW multisite), CephBucketTopic and Notification (bucket event notifications), CephCOSIDriver (COSI driver), CephNVMeOFGateway (NVMe over Fabrics gateway), and CephObjectStoreAccount (RGW account).

The reconcile of CephCluster follows a fixed order. cluster.go Following the reconcileCephDaemonsof , it creates the configuration override ConfigMap, brings up mons (L121), verifies that cluster identity has been established, brings up mgrs (L149), and brings up OSDs (L164). MDS and RGW are not in this function; they are reconciled by separate controllers handling CephFilesystem and CephObjectStore respectively.

3. Mon Placement and Failover

As seen in Part 1, a majority of MONs must be alive for the cluster to operate. Therefore, mon is what Rook takes the most care to protect. Reading the Rook documentation Monitor Healthalong with the source code yields the following.

Identity is a Service IP. In Ceph, a mon's identity is an IP address and must not change. Since pod IPs change every time they restart, Rook creates a separate Kubernetes Service for each mon and uses its ClusterIP as a fixed identity.

Data is bound to nodes. Mon metadata is stored in host directories under the CephCluster's dataDirHostPath(default /var/lib/rook) or volumes created with volumeClaimTemplate . Therefore, mon pods must return to the same node when restarting, which allows them to resume using existing data.

Placement is determined by canaries. When deciding which node to place a new mon on, Rook first spins up a test Deployment with -canary appended to its name to let the Kubernetes scheduler choose a node, and uses that result as the mon's location (scheduleMonitor, mon.go#L731-L740). allowMultiplePerNode: false ). If , anti-affinity preventing mons from residing on the same node is enforced as a mandatory rule, and if , it becomes a preference rule with a weight of 50 (comments in the cluster chart values to be seen in Part 3). true Checks are every 45 seconds, and replacement happens after 10 minutes.

Figure: Sequence of Rook waiting and replacing when a single mon drops out of quorum

health.go: HealthCheckInterval = 45 * time.Second and MonOutTimeout = 10 * time.Minute . The flow is as follows.

1. Every 45 seconds, it queries Ceph for the quorum status.
2. When it encounters a mon missing from the quorum for the first time, it records that timestamp. Meanwhile, Kubernetes tries to restart that pod via its Deployment. The documentation notes that "saving an existing mon is always better than replacing it with a new mon."
3. If it does not return even after 10 minutes, a new mon is provisioned with a new name (the next highest number after the existing ones, e.g., mon-d if mon-a, b, and c existed). The old mon's Deployment is scaled down to 0 (mons using host networking are not scaled down to prevent the new mon from starting on the same host IP), and if the new mon fails, it rolls back (failoverMon, health.go#L742).
4. ). Only one mon is replaced per inspection (health.go#L331).

). There are three exceptions (health.go#L286-L323).

• Timeout 0: Failover is not performed. The documentation notes this is useful for planned maintenance.
• If the node where the mon resided has been deleted , it migrates immediately without waiting 10 minutes (shouldFailoverMonImmediately).
• If the timeout has passed but the mon pod has not even been scheduled to a node(e.g., node drain), it waits for the timeout duration once more. In the documentation's phrasing, "the waiting time is doubled."

Both values can be changed via the CephCluster's healthCheck.daemonHealth.mon.interval·timeout . If the operator environment variable ROOK_MON_OUT_TIMEOUT·ROOK_MON_HEALTHCHECK_INTERVAL is present, it takes precedence (for backward compatibility with older behavior, health.go#L58-L103). The documentation warns that if the interval is too short, mons will be cycled too frequently, and if it is too long, another mon might die and quorum be lost before a new mon is established.

When a mon changes, its address also changes. Every time Rook spins up a mon, it re-notifies the CSI side of the mon address (CephConnection in Section 5, mon.go#L1261-L1269).

4. How the OSD Prepare Job Selects Disks

OSDs are created in two phases: "discovering, preparing, and spinning up the daemon." The first phase is handled by a Job running once per node, and the second phase is handled by the operator. The names are defined in osd.go#L64-L69. The Job is rook-ceph-osd-prepare-<노드>, and the OSD daemon is rook-ceph-osd-<번호> .

Following the events occurring inside the prepare Job in source code order, they proceed as follows.

1. Device list. lsblk Finds the block devices on the node using rbd·nbd·drbd·zram . At this time, devices such as are excluded entirely from the list. This is because the first three are network storages where reads can hang, and zram is volatile memory (disk.go#L37-L39, changes in v1.21.1).
2. Filtering out already used devices(getAvailableDevices, daemon.go#L347-L410). Skips mounted devices, devices with filesystem signatures, and devices that already contain a BlueStore OSD.
3. Querying ceph-volume. For each remaining device, runs ceph-volume inventory --format json <장치> and keeps only those that answer "usable." The rejection reasons are logged as-ically (device.go#L399-L428).
4. Matching user configuration. useAllDevices: true If it is, all remaining devices are used, but LVM logical volumes and loop devices are not selected. deviceFilter (device name regex), devicePathFilter, and per-node devices lists, only matching ones are used.
5. Preparation. The default is ceph-volume raw prepare . If encryption is enabled, multiple OSDs are placed on a single device (osdsPerDevice 2 or greater), or metadata devices are separated, raw mode cannot be used, so it falls back to ceph-volume lvm batch (allowRawMode, volume.go#L497-L525).
6. Reporting results. Writes the list of created OSDs to the ConfigMap rook-ceph-osd-<노드>-status (status.go#L47). The operator reads this and creates a Deployment for each OSD.

The official documentation's requirements (Prerequisites) say the same thing. Things usable as OSDs are raw devices without partitions or filesystems, raw partitions without filesystems, LVM logical volumes without filesystems, and block mode PVs. Therefore, if you "attached a new disk but the OSD was not created," you should first check with lsblk -f to see if any filesystem signatures remain. Conversely, empty disks with no signatures become OSDs without even being queried by useAllDevices: true . This is why Part 3 warns about this default value.

The prepare Job uses a significant amount of memory once depending on the device size. The documentation mentions it is around 1 to 2 Gi on large disks and advises not to set a memory limit for this Job. Once an OSD is prepared, future prepares skip it as "already present," so the heavy usage happens only once.

5. CSI Deployment: Split Among Three in v1.21

CSI(Container Storage Interface) is the standard Kubernetes specification for storage vendor drivers. When a PVC is created, the CSI driver's provisioner(controller side) creates a volume on the storage, and when a pod is scheduled onto a node, that node's nodeplugin attaches (maps and mounts) the volume.

Historically, Rook directly deployed CSI drivers. In v1.21, this responsibility is split among three entities. Since this is the part where first-time installers are most likely to stumble, we will list them in order.

1. ceph-csi-operator. rook-ceph Installed together as a subchart (ceph-csi-operator 1.1.0) by the chart (Chart.yaml, condition csi.installCsiOperator). This operator actually creates the driver pods.
2. ceph-csi-drivers chart. Creates the Driver CR specifying which drivers to enable, OperatorConfig, and the RBAC to be used by the driver pods. This chart is downloaded from the ceph-csi-operator Helm repository rather than the Rook repository and is installed with the values file prepared by Rook (deploy/charts/ceph-csi-drivers/values.yaml). By default, RBD and CephFS are enabled while NFS and NVMe-oF are disabled. Driver names are prefixed with the operator namespace, such as rook-ceph.rbd.csi.ceph.com . The upgrade documentation notes that if this chart is not installed, the CSI driver will fail due to missing service accounts(requirement since v1.20, Rook Upgrades).
3. Rook operator. After mons are up, creates and modifies in the operator namespace the CephConnection CR (list of mon addresses, read affinity, number of RBD mirrors) and the default ClientProfile(Secret name to be used by RBD and CephFS) (ceph_connection.go#L37-L103, config.go#L149-L190). Both are named after the CephCluster namespace. As seen in Section 2, creating separate ClientProfiles for each subvolume group and RADOS namespace remains the same.

When ceph-csi-operator receives a Driver CR, it creates a CSIDriver object, a Deployment ending with -ctrlplugin (provisioner and sidecars), -nodeplugin ending DaemonSet (all nodes), liveness Service, NetworkPolicy, and configuration ConfigMap (driver_controller.go#L254-L264). Thus, you can see rook-ceph.rbd.csi.ceph.com-ctrlplugin-… and rook-ceph.rbd.csi.ceph.com-nodeplugin-… in the Rook quickstart pod list.

The provisioner of the StorageClass must match this driver name exactly. If you want to change the prefix part (CSI_DRIVER_NAME_PREFIX), the documentation warns to do so only during initial installation. It notes that changing it on an already running cluster may cause unexpected behavior (Ceph CSI Drivers).

6. Upgrade Procedure

Rook divides upgrades into two parts: Rook itself (management plane) and Ceph (data plane) .

Rook upgrade. Upgrade the charts in the order of rook-ceph → ceph-csi-drivers → rook-ceph-cluster . The breaking changes in v1.21 specified by the documentation are three:

• Helm OCI 차트 태그에 v is omitted (1.21.0).
• The Ceph msgr2 protocol has become mandatory by default. msgr2 requires Linux kernel 5.11 or higher, and on older kernels, it is disabled via requireMsgr2: false . Already attached volumes continue to use msgr1 until reattached, and mons accept only msgr2 starting from when they are recreated via failover.
• The minimum supported Kubernetes version is v1.32.

Additionally, a new CSI Secret parameter was introduced for block StorageClasses in v1.21. Since the parameters of a StorageClass cannot be changed, it guides you to delete the ceph-block StorageClass before upgrading the chart. It notes that existing PVs are unaffected because a StorageClass with the same name will be recreated.

Ceph upgrade. Initiated by changing the CephCluster image (Ceph Upgrades). Rook checks if each daemon is safe to stop, updates it, waits for stabilization (quorum for mons, clean PGs for OSDs, up for MDS), and then proceeds to the next daemon. Following the reconcile order seen in Section 2, within CephCluster it goes mon, mgr, OSD in order, while MDS and RGW are handled by their respective controllers. The time waited for an OSD to become safe to stop is waitTimeoutForHealthyOSDInMinutes(default 10 minutes). Supported Ceph versions are Squid v19.2.0 or higher and Tentacle v20.2.1 or higher; v20.2.0 is not recommended due to a known data corruption issue when read affinity is enabled. Rook automatically disables read affinity on v20.2.0 (ceph_connection.go#L105-L116).

7. Cluster Cleanup: cleanupPolicy

Deleting a CephCluster does not delete the data. This is the default behavior, designed as a safeguard to prevent accidentally deleted CRs from wiping out data. The chart documentation also notes that even if you delete rook-ceph-cluster , data in /var/lib/rook and OSD disks remains, so you must delete them manually if you want to reuse the disks.

To truly delete them, first add a confirmation phrase to the CephCluster (Cleanup, CephCluster CRD: Cleanup policy).

If you delete the CephCluster while the confirmation phrase is present, the operator remembers the nodes where Ceph daemon pods were located, waits in 5-second intervals until all those pods have disappeared, and then spawns a Job (cluster-cleanup-job-<노드>, label rook-ceph-cleanup) on each of those nodes to delete dataDirHostPath and scrub the OSD disks (cleanup.go#L62-L100). The documentation emphasizes cleaning up application volumes and PVCs first. Failing to do so may leave volumes dangling and require a node restart. The CSI operator's CR (operatorconfigs·drivers·clientprofiles·clientprofilemappings·cephconnections) must also be deleted separately.

If you deleted it without the confirmation phrase, you must manually delete dataDirHostPath on each node and scrub the disks (the documentation's sgdisk --zap-all, dd, blkdiscard, LVM and device mapper cleanup procedure). In particular, chart values comments warn that reinstalling while leaving dataDirHostPath behind will prevent mons from starting in the new cluster.

8. Toolbox

The primary tool for inspecting Ceph is the ceph command. Rook provides a container containing this command as a Deployment ( toolbox ) namedrook-ceph-tools(Toolbox). In the cluster chart, toolbox.enabled is the default false , and when enabled, it runs with the same image as Ceph. There is also a Job approach that runs a command once and logs the results, as well as kubectl plugins that work without the toolbox.

kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph status
kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph osd tree
kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph df

In Part 4, we plan to read these three outputs from the TT Lab practice cluster in alignment with the concepts from Part 1.

9. What the Operator Does Not Do

When adopting an operator, it is easy to expect, "Now it will take care of everything automatically." These are things verified from the original text that Rook does not do, or does not do by default.

There is one more thing to note. useAllNodes: true When a node's taint or affinity is changed so that it is excluded from Rook's targets, the OSD on that node must be removed by "confirming" it through deleting and restarting the operator pod. This is a safeguard to prevent accidental data loss, and the documentation recommends specifying nodes by useAllNodes: false in production environments.

A word from Titi. Rook is a meticulous manager. If a mon is unhealthy, it waits for 10 minutes; if it sees an empty disk, it turns it into an OSD; and when told to delete, it always asks, "Are you sure?" However, it does not transport data itself. That is the job of the Ceph daemons. Therefore, even if the operator rests for a moment, already attached volumes remain perfectly fine!

Reference

Sources and documentation were accessed from the above commit on 2026-10-29.

• Rook v1.21.1 Release: https://github.com/rook/rook/releases/tag/v1.21.1
• Storage Architecture: https://github.com/rook/rook/blob/09d2f0d8d883b70b30de796c4390371be725ff3b/Documentation/Getting-Started/storage-architecture.md
• Monitor Health: https://github.com/rook/rook/blob/09d2f0d8d883b70b30de796c4390371be725ff3b/Documentation/Storage-Configuration/Advanced/ceph-mon-health.md
• CephCluster CRD: https://github.com/rook/rook/blob/09d2f0d8d883b70b30de796c4390371be725ff3b/Documentation/CRDs/Cluster/ceph-cluster-crd.md
• Prerequisites: https://github.com/rook/rook/blob/09d2f0d8d883b70b30de796c4390371be725ff3b/Documentation/Getting-Started/Prerequisites/prerequisites.md
• Ceph CSI Drivers: https://github.com/rook/rook/blob/09d2f0d8d883b70b30de796c4390371be725ff3b/Documentation/Storage-Configuration/Ceph-CSI/ceph-csi-drivers.md
• Ceph-CSI Driver Helm Chart: https://github.com/rook/rook/blob/09d2f0d8d883b70b30de796c4390371be725ff3b/Documentation/Helm-Charts/csi-drivers-chart.md
• Rook Upgrades: https://github.com/rook/rook/blob/09d2f0d8d883b70b30de796c4390371be725ff3b/Documentation/Upgrade/rook-upgrade.md
• Ceph Upgrades: https://github.com/rook/rook/blob/09d2f0d8d883b70b30de796c4390371be725ff3b/Documentation/Upgrade/ceph-upgrade.md
• Cleanup: https://github.com/rook/rook/blob/09d2f0d8d883b70b30de796c4390371be725ff3b/Documentation/Storage-Configuration/ceph-teardown.md
• Toolbox: https://github.com/rook/rook/blob/09d2f0d8d883b70b30de796c4390371be725ff3b/Documentation/Troubleshooting/ceph-toolbox.md
• ceph-csi-operator v1.1.0 Driver Controller: https://github.com/ceph/ceph-csi-operator/blob/98eee8980fc69e5c9c02c80958abe06c3faba5eb/internal/controller/driver_controller.go