Why 2 Servers Are Not Enough for HA 1: Practical Measurements for PostgreSQL, Redis, OpenSearch, ClickHouse, and SeaweedFS

Status: Measured (2026-10-31). These values were measured in Docker on three TT Lab practice VMs (KubeVirt, Ubuntu 24.04). Each test was run once, and VM power-off tests were not performed (not measured). All numbers are extracted from raw data (write logs left per attempt and cross-checked after recovery). Writing rules: Measured data is the default; unmeasured items are marked as "Unmeasured", and interpretations whose causes could not be verified are marked as "Estimated".

Table of Contents

• 1. Question: Wouldn't 2 servers be enough even if one dies?
• 2. Why a Majority Requires 3
  • 2.1 Impossible to Distinguish Between Dead and Disconnected
  • 2.2 Therefore, "Only the Side with the Majority Becomes the Master"
  • 2.3 The Third Node Can Just Cast Votes Without Holding Data
• 3. How Each of the Five Services Determines Its Master
• 4. Configuration Excerpts
• 5. Test Methodology
• 6. When There Are Only 2 Servers
  • 6.1 How the 2-Server Setup Was Built
  • 6.2 When One Server Is Cut Off for 60 Seconds
• 7. Manual Failover: Humans Declaring "The Other Side Is Dead"
  • 7.1 Cautions for Manual Failover
• 8. Conclusion
• References

1. Question: Wouldn't 2 servers be enough even if one dies?

When asked to set up database high availability in an on-premise environment, the very first mental image that comes to mind is 2 servers. One acts as the primary and receives writes, while the other holds a replica and takes over if the primary dies. Two servers are also the cheapest option in terms of hardware cost and rack space.

This series is a record of directly verifying with five services that this setup does not work automatically. Starting with the conclusion:

• In a 2-server configuration, losing either server completely stopped writes for all five services. A 60-second isolation resulted in 108 seconds for PostgreSQL and 108 seconds for SeaweedFS S3, while the rest ranged from 34 to 62 seconds. Even losing just the replica side caused a halt.
• The only way to resolve the halted state was manual failover by a human. Even with procedures prepared in advance, it took anywhere from 0.9 seconds to 44.2 seconds depending on the service.
• Adding just one small witness VM (2 vCPU, 2 GiB) that stores no data enables automatic failover. In the test where a node was isolated, PostgreSQL promoted itself within 22.6 to 32.7 seconds, and there were 0 lost acknowledged writes. We will look at this result in detail in Part 2.

2. Why a Majority Requires 3

2.1 Impossible to Distinguish Between Dead and Disconnected

Let us assume db1 and db2 monitor each other. From db1's perspective, responses from db2 have stopped. There are two possibilities:

1. db2 is truly dead.
2. db2 is completely fine, but the network between them is cut (network partition).

db1 alone has no way to distinguish between these two because both appear simply as "no response." If db1 believes it is case 1 and becomes the primary when it was actually case 2, db2 will also still consider itself the primary and accept writes. A split brain occurs, and when the network recovers, the data on both sides has already diverged. The writes on one side must be discarded.

2.2 Therefore, "Only the Side with the Majority Becomes the Master"

The standard rule to avoid this problem is majority (quorum) . Only the side that obtains consent from more than half of the N voting members (⌊N/2⌋ + 1) can become the primary. No matter how the network is partitioned, there is at most one side with a majority, so there cannot be two masters.

The problem arises when there are 2 servers. If the number of members is 2, the majority required is 2. If the connection drops, both sides have only 1 vote, so neither side can become primary, and the original primary even steps down voluntarily. This is because otherwise, it would accept writes without knowing what is happening on the other side. As shown in the table below the figure, a 2-server setup cannot tolerate a single failure, just like a 1-server setup. To tolerate a single failure, the number of voting members must be 3.

2.3 The Third Node Can Just Cast Votes Without Holding Data

The third member does not necessarily have to be a server holding data. All that is needed for majority calculation is a vote. Data is kept on the two servers, and a small process that only votes is placed in the third spot. This is called a witness or tie-breaker. The witness VM in this experiment had 2 vCPU and 2 GiB of memory, and the actual memory usage right after installation was 1.5 GiB.

3. How Each of the Five Services Determines Its Master

All five services use a majority, but the entity counting the majority differs for each service. This is why what needs to be placed on the witness also varies.

The reasons why each service was chosen and the trade-offs are as follows.

• PostgreSQL chose synchronous replication. Because commit acknowledgment is sent only after reaching the replica, acknowledged writes are not lost during an automatic failover. strict: false Therefore, if the replica dies, Patroni drops the synchronization requirement and continues accepting writes. The trade-off is that commits made while the replica is missing exist only in a single copy. This is a choice favoring availability.
• OpenSearch does not have a "voting-only" role. We searched for voting_only in the 3.9.0 distribution but could not find it (this role is a feature on the Elasticsearch side). Therefore, the witness was set up as a data-less cluster_manager dedicated node. Since the witness can also be elected as a cluster manager, its heap should not be set too small. We set the heap to 384m with a container limit of 1100m, and it crashed with an OOM error when limited to 700m (measured).
• ClickHouse used Keeper instead of ZooKeeper. It requires no separate JVM and is bundled with the same server binary. The 3 Keepers immediately formed 1 leader and 2 followers.
• SeaweedFS filer stores metadata in PostgreSQL. Since both filers look at the same metadata, if one filer dies, the other seamlessly takes over. However, PostgreSQL failures propagate to the filers.
• SeaweedFS replication 001 means "one additional copy on another server in the same data center and rack" . If there are only two volume servers, all files reside in single copies across the two servers, and if one server goes down, there is no place to satisfy this rule, causing new writes to fail.
• The frontend is HAProxy. PostgreSQL traffic is routed only to nodes where Patroni REST API's GET /primary is 200, while ClickHouse and S3 traffic are load-balanced across live nodes. One instance was placed on the application server spot (the witness VM in this setup), and one on each data node so that filers can locate PostgreSQL.

The total container memory limit for the data nodes was approximately 4.6 GiB (OpenSearch 1.4g, ClickHouse 1.2g, PostgreSQL 768m, Redis 384m, etc.), and right after installation, actual usage was 2.5 GiB for data nodes and 1.5 GiB for the witness.

4. Configuration Excerpts

Passwords and keys are omitted (in reality, they were placed only in the node's .env file with 600 permissions). Addresses were replaced with node names or variables. All services were bound only to the cluster network address and 127.0.0.1.

etcd (Patroni's consensus store). Place the same compose file on all three nodes and change only the node name. The last line is for manual recovery when there are only 2 servers.

services:
  etcd:
    image: gcr.io/etcd-development/etcd:v3.6.16
    network_mode: host
    command:
      - etcd
      - --name=${NODE_NAME}
      - --initial-cluster=${ETCD_INITIAL_CLUSTER}   # db1, db2, witness 세 멤버
      - --client-cert-auth                          # 클라이언트·피어 모두 상호 TLS
      - --peer-client-cert-auth
      - --heartbeat-interval=100
      - --election-timeout=1000
      - --force-new-cluster=${ETCD_FORCE_NEW:-false} # 살아남은 한 대를 단일 멤버로 다시 세울 때만 true
    mem_limit: 256m

Patroni. Leader key TTL ttl determines the upper limit of time from "primary disappearance until promotion".

bootstrap:
  dcs:
    ttl: 30
    loop_wait: 10
    retry_timeout: 10
    synchronous_mode: true          # 커밋이 복제본에 닿은 뒤 확인
    synchronous_mode_strict: false  # 복제본이 죽으면 동기 요구를 내리고 계속
    failsafe_mode: false
    postgresql:
      use_pg_rewind: true
      use_slots: true
watchdog:
  mode: off                         # 컨테이너에 /dev/watchdog 이 없다

Redis and Sentinel. min-replicas-* These two lines are the settings that exacted the highest cost in this series (Part 2).

# redis.conf
appendonly yes
appendfsync everysec
min-replicas-to-write 1     # 10초 안에 응답한 복제본이 1개 이상일 때만 쓰기를 받는다
min-replicas-max-lag 10

# sentinel.conf
sentinel monitor mymaster <db1> 6379 2
sentinel down-after-milliseconds mymaster 5000
sentinel failover-timeout mymaster 60000

OpenSearch. The three nodes gather once at the beginning to form the voting configuration. Roles are provided via environment variables (see the trap in Part 2).

# opensearch.yml
cluster.initial_cluster_manager_nodes: ["db1", "db2", "witness"]
# docker-compose.yml 의 environment
node.roles: cluster_manager,data,ingest   # 증인은 cluster_manager 만

ClickHouse Keeper. One of the three Raft members is the witness. The 30-second session timeout connects to the trap in Part 2.

<coordination_settings>
    <operation_timeout_ms>10000</operation_timeout_ms>
    <session_timeout_ms>30000</session_timeout_ms>
</coordination_settings>
<raft_configuration>
    <server><id>1</id><hostname>db1</hostname><port>9234</port></server>
    <server><id>2</id><hostname>db2</hostname><port>9234</port></server>
    <server><id>3</id><hostname>witness</hostname><port>9234</port></server>
</raft_configuration>

SeaweedFS master. Only the master is run on the witness.

  master:
    profiles: ["data", "witness"]
    command:
      - master
      - -peers=${DB1_IP}:9333,${DB2_IP}:9333,${WITNESS_IP}:9333
      - -defaultReplication=001
    mem_limit: 192m            # 128m 에서 89MiB 까지 오르는 것을 보고 늘렸다(실측)

HAProxy (application side). The application only knows 127.0.0.1:5000. When the primary changes, HAProxy follows using two checks spaced 1 second apart.

listen pg_primary
    bind 127.0.0.1:5000
    option httpchk GET /primary
    http-check expect status 200
    default-server inter 1s fall 2 rise 2 on-marked-down shutdown-sessions
    server db1 <db1>:5432 check port 8008
    server db2 <db2>:5432 check port 8008

5. Test Methodology

• The test clientran on the application server spot (witness VM). Write intervals were 0.05 seconds for PostgreSQL and Redis, 0.1 seconds for OpenSearch, and 0.2 seconds for ClickHouse and S3, with timeouts ranging from 2 to 5 seconds. A log line was recorded for each attempt, and after completion, the storage was read to check whether all acknowledged writes(writes that received a success response) were preserved. Role changes were monitored every 0.25 seconds.
• Isolationis to drop the input and output of the session network interface within that node using iptables. The node is alive, but disconnected from the peer, which is a partition. From the peer's perspective, it is indistinguishable from being powered off.
• Graceful Stopis shutting down all containers on that node with docker stop(30-second grace period).
• Max Write Interruptionis the longest gap between a successful write and the next success. The clock skew between VMs was within ±0.3 seconds (upper bound measured by ssh round-trip).

6. When There Are Exactly 2 Nodes

6.1 How We Reduced It to 2 Nodes

From a setup with a witness, we removed the witness side.

• etcd: member remove Removed the witness with to make it actually 2 members.
• Sentinel: Stopped the witness's Sentinel and sent SENTINEL RESET . The remaining two only know each other, and the quorum is 2.
• OpenSearch: voting_config_exclusions Removed the witness and stopped it with . As a result, the voting configuration was reduced to a single db2 instance. Since 2 is an even number, OpenSearch adjusted it to an odd number (measured).
• ClickHouse Keeper / SeaweedFS master: Stopped the witness instance. In a three-node setup, with only two remaining, the arithmetic that "both must agree to meet the majority of 2" is the same as the 2-member setup, but we did not perform a reconfiguration to actually reduce the member count (reconfiguration to 2 members itself was not measured).

6.2 When Disconnecting One Node for 60 Seconds

At that time, the master for both PostgreSQL and Redis was db2. Therefore, isolating db1 is "losing the replica side," and isolating db2 is "losing the master side."

There are three takeaways.

• The expectation that "it will keep running since one node is left" was most significantly violated in PostgreSQL. Even though only the replica was lost, the primary server shut itself down. This is because it lost the majority in the 2-member etcd. This is not a bug, but normal behavior to prevent split-brain. Patroni's failsafe_mode is a feature that maintains the master role even if the consensus store is lost as long as all members are reachable via REST, but in a partition, the peer cannot be reached, so this case is not rescued (configuration checked only, not measured).
• OpenSearch's results differed depending on "which side was lost." Since the voting configuration was a single db2 instance, losing db1 allowed it to keep running, while losing db2 caused it to stop. The isolated db2 acted as a cluster by itself, accepted 77 local writes, and all 77 writes remained after recovery. Since there was no cluster on the db1 side, data was not split.
• Regardless of which side was lost, the loss of acknowledged writes was 0. The guarantee provided by the majority rule is that it stops rather than loses data.

7. Manual Promotion: A Human Declaring "The Peer Is Dead"

To restart stopped services in a 2-node configuration, human intervention is required. This involves declaring, "The peer is dead, so the remaining single node constitutes a majority." While keeping db2 isolated, we performed this sequentially for each service on db1. The procedure was scripted in advance.

The procedures were performed service by service, starting with PostgreSQL 15.9 seconds after isolation. Therefore, services later in the sequence have larger "from isolation" values. The actual time taken during an outage is closer to "time to detect and decide + second column from the right." There were zero confirmed write losses across all five services.

The reason PostgreSQL took the longest is that even after re-establishing etcd, it has to wait for the old leader key to expire. This 30 seconds is precisely ttl .

7.1 Cautions for Manual Promotion

• Humans must guarantee that "the peer is truly dead." If the peer was still alive and accepting writes, this moment would result in split-brain (two masters). In operations, ensuring this requires powering off the peer or disconnecting its network fencing first and then promote.
• For about 20 seconds immediately after the ClickHouse Keeper forced recovery, insert_quorum=2 inserts responded with success. Between +100s and +120s from the isolation baseline, there are 98 cases. Since db2 was still isolated, confirmation from 2 replicas is impossible. We presume that db2's old "alive" flag remained in the recovered Keeper state until the session expired, counting it as 2 live replicas (presumption. The basis is that successes were concentrated within 30 seconds of the session timeout, followed by TOO_FEW_LIVE_REPLICAS resulting in 1,346 failures). After a forced recovery, you must not trust successful responses for quorum inserts until the session time has passed.
• Reverting (bringing back the dead node) was not tested (unmeasured). We only note the sequence. Add db2 via etcd member add and start with an empty data directory → Patroni reinit db2 → Re-register Redis REPLICAOF and Sentinel → OpenSearch clears db2 data and joins (the UUID differs from theunsafe-bootstrap cluster) → Keeper expands back to a 3-server configuration → Restore SeaweedFS master peers → Restore fs.configure with 001 → volume.fix.replication -apply.

8. Conclusion

How a configuration with an added witness behaves under each failure type, and the pitfalls left by the condition that "there are still 2 data nodes," will be covered in Part 2.

Titi's Word. When two people try to majority-vote, it's always a tie of 1 to 1. That's why both stop, wondering, "Can I decide this myself?" You just need one more friend to cast a vote. That friend doesn't even need to carry any luggage (data)!

References

• ClickHouse Keeper Guide (Configuration examples and single-node forced recovery procedure): https://clickhouse.com/docs/guides/sre/keeper/clickhouse-keeper (Accessed October 31, 2026)