Rook-Ceph Explanations Part 1: Where and How Ceph Places Data

Status: Draft (2026-10-29), based on official documentation and paper, no actual measurements. Explanations are written strictly based on what was verified by opening the official Ceph v20.2 (Tentacle) documentation (docs.ceph.com/en/tentacle) and the original Ceph paper. All default values are those stated in that documentation. In the TT Lab practice cluster, ceph status Verifying with the same output is covered in part 4 of this series (written once raw data is gathered). Writing rules: Measured data is the default, unmeasured items are marked as "Unmeasured", and items not found in the documentation are marked as "Unverified".

Table of Contents

• 1. In One Sentence: There Is No Central Catalog Recording Data Locations
• 2. The Daemon Net Composing RADOS
  • MON: Cluster Maps and Paxos Quorum
  • OSD and BlueStore
  • MGR and MDS
• 3. The Two Calculations Where Objects Find OSDs: Pools, PGs, and CRUSH
  • Pools
  • First Calculation: From Object Name to PG
  • Second Calculation: From PG to OSD List
  • How Many PGs Are Good: The PG Autoscale Manager
• 4. Replication and Erasure Coding
• 5. Write Path: Why Clients Talk Directly to OSDs
• 6. The Three Faces On Top: RBD, CephFS, and RGW
• 7. Summary: What to Look At in the Next Part
• References

1. In One Sentence: There Is No Central Catalog Recording Data Locations

When a pod requests a disk in Kubernetes (PVC), someone must provide actual disk space. If you use a single node's disk, the data is bound together with that disk when that node dies. Bundling disks from multiple nodes to make them look like a single storage pool, and keeping multiple copies of data so that reads and writes can continue even if one node dies, is distributed storage. Ceph is its representative open-source implementation, and Rook is the operator that runs Ceph on Kubernetes. This series reads through Ceph's architecture (Part 1), what the Rook operator does (Part 2), and Helm chart options (Part 3) in order.

Many storage systems keep a catalog (metadata server, allocation table) in one place stating "fragments of file X are located on disk 3 of server 3". Because you must query that catalog every time you read and write, the catalog becomes a bottleneck and a single point of failure. The paper first introducing Ceph (Weil et al., OSDI 2006 "ICeph: A Scalable, High-Performance Distributed File System") summarizes replacing this allocation table with a pseudo-random distribution function called CRUSHinstead. Data locations are not written down, but calculated. If the client and the storage daemon share the same map and the same function, anyone obtains the same answer.

The bottom of the figure is RADOS(Reliable Autonomic Distributed Object Store). Whatever data Ceph receives—whether blocks, files, or S3 objects—ultimately becomes a RADOS object(a bundle of name, bytes, and name-value metadata) and is stored. Objects reside in a flat namespace without directories. RBD, CephFS, and RGW on top are translation layers making RADOS appear as block devices, a filesystem, and S3, and the common library through which these three talk to RADOS is librados (Official Documentation: Architecture).

2. The Daemon Net Composing RADOS

A Ceph cluster consists of four types of daemons (background processes) with different roles. In Rook, a single daemon is usually a single pod.

MON: Cluster Maps and Paxos Quorum

To read or write anything in Ceph, a client must first connect to a MON and obtain the latest cluster map. The official documentation explains that the cluster map is actually a bundle of five maps.

Each map has a version number (epoch) that increases whenever it changes. If there is only one MON, the moment that MON dies nobody can receive maps, so multiple units are usually deployed. Because multiple MONs must not output conflicting maps, Ceph uses a consensus algorithm called Paxos to determine map changes, and a majoritymust be alive to make decisions. As documentation examples show, the majority is 1 for 1 MON, 2 for 3 MONs, 3 for 5 MONs, and 4 for 6 MONs. Thus, 3 units survive the failure of 1, 5 survive 2, while 4 units do not increase the survivable count over 3. This is why odd numbers are recommended (Monitor Config Reference). MON records map changes as a single Paxos instance and writes the result to a RocksDB key-value store.

OSD and BlueStore

An OSD is a daemon responsible for a single disk and storing objects on it. The current default storage engine, BlueStore , does not create filesystems like ext4 or XFS on the disk but writes directly (raw) to the device. Object metadata is placed in an embedded RocksDB. Devices can be split into up to three parts (BlueStore Configuration Reference).

• block: The primary device holding data. The simplest deployment uses only this one.
• block.db: A faster device to hold RocksDB metadata (e.g., an SSD alongside an HDD). If it fills up, it spills over to the primary device.
• block.wal: A device exclusively for the write-ahead log (WAL). This is meaningful only when the fast space is less than 1GB.

BlueStore attaches a checksumto all written data and metadata (default crc32c) and verifies it upon reading. Memory cache self-regulates and strives not to exceed osd_memory_target(default 4Gi, minimum 896M). This value meets Rook's resource configurations again in parts 2 and 3.

OSDs exchange heartbeats with neighboring OSDs. By default, they send pings every 6 seconds (osd_heartbeat_interval), and if there is no response for 20 seconds (osd_heartbeat_grace), they notify the MON that the OSD is down. If a down OSD does not return for 10 minutes (mon_osd_down_out_interval), the MON removes that OSD from data placement (out). From that point on, redistribution begins where other OSDs take over that OSD's share. However, if a unit larger than mon_osd_down_out_subtree_limit(default rack) goes down entirely, it is not automatically removed. This is to prevent a power outage in a single rack from triggering a massive data migration (Configuring Monitor/OSD Interaction).

OSDs also perform scrubbing. Light scrubbing, which compares metadata between replicas of the same PG, typically runs daily, while deep scrubbing, which compares data bit-by-bit, runs weekly by default. Deep scrubbing finds bad disk blocks that light scrubbing misses.

MGR and MDS

MGR is an endpoint handling heavy tasks unsuitable for MON (metric collection, dashboards, various modules). MDS is required only when using CephFS. It holds file metadata in memory and processes it separately so that metadata operations like listing or moving directories (ls, cd) do not burden OSDs. File data itself goes directly to RADOS objects without passing through the MDS. We will revisit these two in Section 7.

3. The Two Calculations Where Objects Find OSDs: Pools, PGs, and CRUSH

Pools

A poolis a logical partition containing objects. At least three things are determined per pool: how many copies to keep (size), which CRUSH rule to place them with, and how many PGs to divide into (pg_num). They are created separately for each purpose, such as pools for block storage, filesystem metadata, or S3 data.

First Calculation: From Object Name to PG

A PG(placement group) is a finely divided bundle of a pool. Ceph determines and moves locations in PG units rather than individual objects. Using the documentation's example as-is, when writing an object named liverpool to the pool john (pool number 4), the client calculates as follows.

1. The object name john is hashed.
2. The remainder of dividing the hash value by the pool's PG count is used as the PG number (e.g., 58).
3. The pool number is prepended to form the PG ID: 4.58.

Second Calculation: From PG to OSD List

Next, the CRUSH(Controlled Replication Under Scalable Hashing) algorithm receives the PG ID, CRUSH map, and pool rules, and returns "the list of OSDs to contain this PG". The first OSD in the list is the primary, and the rest are secondaries (Architecture: Calculating PG IDs, Peering and Sets).

The CRUSH mapcontains a list of storage devices (OSDs) and bucketshierarchy is contained. The default bucket types from bottom to top are osd, host, chassis, rack, row, pdu, pod, room, datacenter, zone, region, root . When a new OSD is deployed, it is automatically placed under the host bucket of that node name. The pool rules specify a failure domain, and if you set it to host , CRUSH chooses a different host for each replica. This means that even if an entire host dies, other replicas of the same PG remain on different hosts. In larger environments, you can raise the failure domain to rack (CRUSH Maps).

The reason for dividing the calculation into two steps is the indirection layer. If clients directly knew that "object X is on OSD 3," every time you add or remove an OSD, you would have to notify them of the new locations of all objects. By anchoring objects to PGs and letting only PGs move among OSDs, the unit of movement when the cluster grows or shrinks becomes a few thousand PGs instead of billions of objects. CRUSH itself was first introduced in Weil et al.'s SC 2006 paper "CRUSH: Controlled, Scalable, Decentralized Placement of Replicated Data".

The process by which the OSDs containing a PG reach consensus that "all object states in this PG are identical" is called peering . The set of OSDs currently responsible for that PG is the Acting Set, and the active ones among them form the Up Set. If the primary dies, the next OSD takes over as primary.

How Many PGs Are Good: The PG Autoscaler

If there are too few PGs, data is not distributed evenly across OSDs; if there are too many, peering traffic and memory usage increase. In the past, humans calculated and set pg_num , but now the PG autoscalerdetermines it per pool by observing usage (Placement Groups).

ceph osd pool autoscale-status shows the recommended value (NEW PG_NUM) for each pool. For the object store data pool in the Rook chart we will look at in Part 3, bulk: "true" is left enabled.

4. Replication and Erasure Coding

There are two ways to protect data.

Replicationkeeps multiple complete copies of the same object. The default for osd_pool_default_size is 3. min_size represents "the minimum number of replicas that must be written before success is reported to the client," and a default of 0 means size - (size / 2) (due to integer division, size 3 means 2). Therefore, a size 3 pool continues to read and write even if one replica is lost, but if two are lost, that PG stops I/O (Pool, PG and CRUSH Config Reference).

Erasure coding(EC) splits an object into K data chunks, calculates M coding chunks, and places the K+M chunks across different OSDs. The original data can be recovered as long as K chunks remain. The space overhead is (K+M)/K (Erasure Code).

The documentation notes that EC should not be mistaken for a free lunch. Performance penalties are severe, especially with HDDs and during recovery/backfill. It warns that with M=1, data becomes inaccessible during maintenance, and overlapping failures lead to data loss. Also, because EC pools do not support omap (key-value data attached to objects), RBD and CephFS must store metadata in a replication pool while keeping only data in the EC pool, and must enable allow_ec_overwrites on the EC pool. For EC pools, min_size is recommended to be K+1 or greater. Profiles cannot be changed after a pool is created, so changing them requires creating a new pool and migrating the data.

One more thing: if the number of failure domains is less than the number of replicas, CRUSH cannot satisfy the rule. If the failure domain is host and there are three hosts, a replication size of 3 places one replica per host, and EC 2+1 with K+M=3 fits perfectly. However, if one host dies, there is no fourth host to move a replica to, so the PG remains degraded with missing replicas until that host returns.

5. Write Path: Why Clients Talk Directly to OSDs

The path for writing an object to a replication pool is described in the official documentation as follows.

1. The client connects to the MON, authenticates via CephX, and receives the latest cluster map.
2. The client independently calculates the PG and OSD list using the two calculations from Section 3. No communication with anyone is involved in this calculation.
3. The client sends the object to the primary OSD of the PG.
4. The primary finds the secondary OSDs using its copy of the CRUSH map and replicates the object (the documentation calls this replication a subop).
5. The secondaries send acknowledgments back to the primary once storage is complete.
6. The primary notifies the client of success.

Because replication is handled by the primary rather than the client, the client network transmits data only once. Furthermore, since data does not pass through a central gateway or metadata server, central components do not become bottlenecks or single points of failure. The MON merely distributes maps and does not stand in the data path. This is the practical effect of "computation instead of lists" mentioned in Section 1.

In an EC pool, the primary receives the object, calculates the chunks, and distributes them to each OSD in the Acting Set. Once all chunks are acknowledged, it notifies the client of success.

Maps change. When an OSD goes down or joins, the epoch of the OSD map increases, and the result of the CRUSH calculation changes even for the same PG. The documentation notes that client-side calculation is much faster than querying servers for object locations, and since OSD failures are usually detected and reported to the MON by neighboring OSDs, the MON remains a lightweight process. The exact messages used by clients and OSDs with outdated maps to synchronize are not covered in this article (unverified).

6. The Three Faces Above It: RBD, CephFS, and RGW

A few points are worth noting.

• An RBD image consists of multiple objects. Blocks of a single disk are divided across multiple objects, and each object goes to a different PG and OSD, so I/O for a large disk is distributed across multiple OSDs. The documentation states that thanks to this striping, RBD can be faster than a single server.
• CephFS MDS maintains hot standbys. Extra MDS instances wait as standbys, and if an active MDS dies, the MON decides to fail over to one of them. Because all data, including journals, resides in RADOS, failover is straightforward. Running multiple active MDS instances allows them to partition the directory tree.
• S3 objects and RADOS objects are not 1:1. A single S3 object can be stored as multiple RADOS objects. EC pools natively support only whole-object writes or appends, and the documentation cites RGW as an example. Therefore, large RGW data fits well with EC pools, while data heavily using omap like bucket indexes (which the PG documentation cites as an example for "RGW index") should be placed in replication pools.

7. Conclusion: What to Expect in the Next Part

Up to this point, we have covered Ceph itself. How these daemons are spawned on Kubernetes, how disks are selected, and how MONs are recreated if they fail are the topics of Part 2. And we will examine what values are used to tune these behaviors in Part 3by reading through the chart's values.yaml line by line.

Titi's Note: Ceph doesn't have an information desk to ask, "Where is this file?" Instead, everyone carries the same map (cluster map) and the same calculation method (CRUSH). So there's no information desk to get jammed or go down. As long as your map is up to date, you're good!

References

All documents were accessed on 2026-10-29. Ceph documentation version is Tentacle (v20).

• Ceph Architecture: https://docs.ceph.com/en/tentacle/architecture/
• Monitor Config Reference: https://docs.ceph.com/en/tentacle/rados/configuration/mon-config-ref/
• Configuring Monitor/OSD Interaction: https://docs.ceph.com/en/tentacle/rados/configuration/mon-osd-interaction/
• BlueStore Configuration Reference: https://docs.ceph.com/en/tentacle/rados/configuration/bluestore-config-ref/
• Pool, PG and CRUSH Config Reference: https://docs.ceph.com/en/tentacle/rados/configuration/pool-pg-config-ref/
• Placement Groups: https://docs.ceph.com/en/tentacle/rados/operations/placement-groups/
• CRUSH Maps: https://docs.ceph.com/en/tentacle/rados/operations/crush-map/
• Erasure Code: https://docs.ceph.com/en/tentacle/rados/operations/erasure-code/
• Weil, Brandt, Miller, Long, Maltzahn. "Ceph: A Scalable, High-Performance Distributed File System", OSDI 2006: https://www.usenix.org/legacy/event/osdi06/tech/weil.html
• Weil, Brandt, Miller, Maltzahn.  CRUSH: Controlled, Scalable, Decentralized Placement of Replicated Data , SC 2006: https://ceph.io/assets/pdfs/weil-crush-sc06.pdf