Why Two Servers Cannot Achieve HA 2: What a Single Witness Changes and Remaining Pitfalls

Status: Measured (2026-10-31). Values measured in Docker on three TT Lab practice VMs (KubeVirt, Ubuntu 24.04). Each test was run once, and VM power-offs were not performed (unmeasured). Configurations and test methods are in Part 1. Writing rule: Measured data is the default, unmeasured items are marked as "Unmeasured", and interpretations whose causes could not be verified are marked as "Estimated".

Table of Contents

• 1. Overview at a Glance
• 2. Maximum Write Outage by Failure Type
  • 2.1 Until the Master Changes
  • 2.2 Reading by Service
  • 2.3 Even When Shutting Down the Witness
• 3. Partition: Does the Isolated Former Master Accept Writes?
• 4. Pitfalls Operators Must Know
  • 4.1 Redis min-replicas-max-lag Is Directly the Loss Window During Partitions
  • 4.2 Writes Saved Despite Error Responses, and Idempotency Keys
  • 4.3 Quorum Window Immediately After ClickHouse Keeper Force Recovery
  • 4.4 OpenSearch Took 65 Seconds to Detect an Isolated Node
  • 4.5 The Official redis:8 Image and Sentinel Restart Loop
  • 4.6 Place the Witness in a Third Failure Domain
  • 4.7 SeaweedFS: Cannot Write with a Single Volume Server, But Does Not Fail Immediately either
  • 4.8 Collection of Short Pitfalls
• 5. Limitations of These Measurements
• 6. Conclusion: Checklist When Moving to 2 Nodes + Witness

1. Overview at a Glance

In Part 1, we saw that with only two data nodes, all five services stop when a single node is lost. This time, we injected three types of failures into a configuration containing a voting-only witness VM (data nodes db1 and db2 + witness).

• Isolation: Cut off one node's network session using iptables. The node is alive but in a partitioned state cut off from its peer.
• Graceful Shutdown: Shut down one node's container entirely using docker stop(30-second grace period).
• Witness Shutdown: Stopped all of the witness's etcd, Sentinel, OpenSearch, Keeper, and SeaweedFS master.

First, the results.

• Automatic failover succeeded. During isolation, PostgreSQL promoted a new master in 22.6 to 32.7 seconds, and Redis Sentinel in 6.5 to 7.0 seconds. During graceful shutdown, PostgreSQL transitioned in 2.8 to 3.1 seconds.
• Loss of confirmed writes was 0 across all tests and all services.
• Shutting down the witness for 90 seconds resulted in 0 write failures for PostgreSQL, Redis, and OpenSearch. This is because when two data nodes remain alive, a majority (2/3) is preserved.
• However, the condition "two data nodes" remains unchanged. Redis's min-replicas-to-write 1, ClickHouse's insert_quorum=2, and SeaweedFS's replication 001 are all settings requiring that "the peer data node must be present to write," which blocked writes until the peer returned. Most of the long values in the table below came from here.

2. Maximum Write Outage by Failure Type

Maximum write outage is the longest gap (in seconds) between a successful write and the next successful write. Values in parentheses are failure count / trial count. ClickHouse is basic insertion, quorum=2 is insertion via insert_quorum=2 , and S3 represents two SeaweedFS S3 gateways behind HAProxy.

※ The 268.8-second Redis value for db2 graceful shutdown occurred because the restarted Redis fell into a restart loop due to an official image bug and never recovered (Section 4.5). The second column from the right shows the same test run on db1 after the fix. Write intervals are 0.05 seconds for PostgreSQL and Redis, so 0.09 seconds means there was no interruption.

Read (query) failures were almost nonexistent. OpenSearch _count was 0 to 1 times per test, and ClickHouse and S3 were 0 to 1 times across all tests in the witness configuration.

2.1 Until the Master Changes

Seconds measured from the moment the failure was injected.

2.2 Reading by Service

• PostgreSQL: During isolation, promotion took 22 to 34 seconds because it had to wait for the old master's leader key TTL (ttl: 30) to expire. During graceful shutdown, Patroni voluntarily released the key and transitioned in 3 seconds. The old primary server was rewound with pg_rewind after recovery and reattached as a replica. If you want to shorten the isolation time, reducing ttl is a direct knob, but the reduced value was not measured (Unmeasured).
• Redis: The failover itself finished in 6.5 to 7.9 seconds. However, since the new master was also min-replicas-to-write 1 , it rejected writes via NOREPLICAS until the replica (old master) returned. The write outage was "failure duration + 10 to 15 seconds". With only two data nodes, this setting is equivalent to "stop writes if one node dies." The trade-offs and alternatives are covered in Section 4.1.
• OpenSearch: Graceful shutdown caused less than 1 second of outage as the node gracefully withdrew. Isolation halted some writes for about 65 seconds (Section 4.4).
• ClickHouse: Since both replicas accept writes, basic inserts only paused for 2 to 5 seconds while HAProxy removed the dead node. insert_quorum=2 failed with TOO_FEW_LIVE_REPLICAS until the peer returned. Catch-up was fast: after a 150-second outage, 896 replication queue items were cleared in 16 seconds.
• SeaweedFS: When only one volume server remains under replication 001 , new write allocations fail. This was expected. What differed from expectations was that failures did not arrive immediately, but each request hung for about 10 seconds before timing out(Section 4.7). Reads did not fail in any test.

2.3 Even When Shutting Down the Witness

The brief hiccups in ClickHouse and S3 occurred because at that moment, the Keeper leader and SeaweedFS master leader were on the witness. Electing new leaders took about 5 seconds and 12 seconds respectively, and S3 experienced additional interruption until volume servers reattached to the new leader. It is worth remembering that although a witness is just there to "cast a vote," in Raft it can also become a leader.

3. Partition: Does the Isolated Former Master Accept Writes?

Automatic failover working is not the end. If the old master on the severed side accepts writes in the meantime and discards them later, the app loses data despite receiving a successful response. We isolated db1 (the master at the time) for 40 seconds, and inside db1 , wrote directly to its local service every 0.5 seconds. After recovery, we cross-checked whether those writes survived.

PostgreSQL, ClickHouse, and SeaweedFS, which strictly follow majority rules, had the isolated side reject writes. Data loss occurred only in Redis.

4. Pitfalls Operators Must Know

From here on are issues actually experienced during these measurements.

4.1 Redis min-replicas-max-lag Is Directly the Loss Window During Partitions

Redis replication is asynchronous. When the master receives a write, it immediately returns success and sends it to replicas later. min-replicas-to-write 1 and min-replicas-max-lag 10 mean "accept writes only if there is at least 1 replica that responded within the last 10 seconds." The isolated master continues to accept writes until the replica's last response exceeds 10 seconds.

In our measurements, that window opened precisely like that. The isolated db1 successfully accepted 22 writes during the first 10.7 seconds, while Sentinel promoted db2 as the new master at +7.0 seconds. After recovery, db1 became a replica of db2 and was overwritten by db2's data, and not a single one of the 22 writes survived. From the app's perspective, these are writes that received a "success" response.

There are three choices, and none of them are free.

• min-replicas-max-lag Lower . This narrows the window, but rejects writes even if replicas are slightly delayed.
• min-replicas-to-write is set to 0. Even if one data node is lost, writes continue (eliminating the 105-second outage from section 2.2), but the writes accepted by the old master during a split increase without limit.
• To eliminate this window entirely, a storage system with synchronous consensus is needed rather than Redis. Using Redis only for disposable data like caches is also a viable solution.

This configuration corresponded to the first setup (1, 10), and its cost was measured as both "up to about 10 seconds of lost writes during a split" and "a write freeze when a single data node is lost."

4.2 Writes Saved Despite Failure Responses, and Idempotency Keys

While there were zero instances of lost acknowledged writes, there was a discrepancy in the opposite direction: writes that received a failure response but were actually saved.

(Values in the table are "Number of saved writes with a failure response / Number of failure responses")

• For OpenSearch, these were writes that succeeded after a client timeout.
• ClickHouse insert_quorum=2 reported that "quorum was not met," but the data was written to one replica and successfully replicated once the peer recovered. Nearly all failure responses were like this. You must not treat "failure = not written."
• The behavior was the same when a majority of Keepers were lost (test stopping 2 out of 3 Keepers). Inserts hung with a 5-second timeout without errors (13 occurrences over 60 seconds), and once the Keepers were revived, all 13 inserts were eventually saved.

If the client retries upon failure, the same row gets inserted twice. Therefore, retried writes require an idempotency key. OpenSearch lets the app specify and include a document _id , while ClickHouse attaches insert_deduplication_token . First, you need to check whether your app's retry logic assumes that "it failed, so it does not exist."

4.3 Quorum Window Immediately Following ClickHouse Keeper Force Recovery

This was covered in Part 1, Section 7.1. In a two-node setup, for about 20 seconds immediately after reviving the Keeper as a single server using --force-recovery , even while the peer replica remained isolated, insert_quorum=2 98 insert operations reported success. Following that, TOO_FEW_LIVE_REPLICAS 1,346 operations failed with . It is presumed that the peer's old "alive" status remained on the recovered Keeper until the session timeout (30 seconds) expired (estimated). After a force recovery, do not trust the success of quorum inserts until the session timeout has passed.

4.4 OpenSearch Took 65 Seconds to Detect an Isolated Node

During a graceful stop, the node leaves the cluster voluntarily, resulting in less than 1 second of write interruption. Isolation was different. It took 64.8 and 65.2 seconds for the node to be dropped (yellow), during which writes to indices with primary shards on that node were halted. Searches (_count) failed only 0 to 1 time per isolation test.

OpenSearch monitors nodes using default failure detection settings (a 10-second timeout per check, requiring 3 consecutive failures to judge a node dead). Since isolation goes through a two-step process for this determination, it is believed to take around 65 seconds. cluster.fault_detection.* Lowering this value can reduce the time, but it was not measured with a lower value (unmeasured). Because temporary network flickers could cause nodes to be evicted, re-measurement would be necessary if the value is reduced.

4.5 Official redis:8 Image and Sentinel Restart Loop

This accounts for the 268.8 seconds in the table from Section 2. Causes were intertwined as follows:

1. The entrypoint of the official redis:8 image appends all .so in the modules folder with --loadmodule to launch Redis.
2. During a failover, Sentinel sends CONFIG REWRITE to Redis. This rewrites the running configuration to the file, embedding those loadmodule lines directly into redis.conf .
3. On the next startup, the modules appended by the entrypoint overlap with the modules embedded in the file, attempting to load the same module twice and causing it to crash with Can't load module ... server aborting . The container then restarts and crashes again.

Because this only manifests when using Sentinel, experiencing a failover once, and then restarting that Redis instance, it is not caught by post-installation tests. If you do not use modules, the fix is simple: overwrite the modules folder with an empty tmpfs so they are never loaded. Remove the already embedded lines.

  redis:
    image: redis:8.8.4
    command: ["redis-server", "/conf/redis.conf"]
    tmpfs:
      - /usr/local/lib/redis/modules   # 진입점이 붙일 모듈이 없게 한다

sed -i '/^loadmodule /d' /path/to/redis.conf   # 이미 박힌 줄 지우기

After the fix, in the same test (db1 graceful stop), Redis returned to normal, and the remaining 134.8-second outage was the cost of min-replicas from Section 4.1. This issue has already been reported upstream (redis/docker-library-redis#468, redis/redis#14178), and the fix on the Redis core side (redis/redis#16009) was merged on 2026-10-29 and is scheduled to be released in the next version.

There is one more small pitfall along the same lines. Sentinel rewrites requirepass into the ACL hash line. Because health checks that read passwords from the configuration file break, we made pre-authentication responses (NOAUTH) count as "alive" this time.

4.6 Place Witnesses in a Third Failure Domain

In this test, the witness VM was on the same physical server as db2. Since we only introduced VM-level failures, this did not affect the results, but if that physical server dies, db2 and the witness are lost together, wiping out 2 out of 3 votes. The system then stops just like a two-node configuration.

The value of a witness comes from the fact that it "never dies together with either data node." In production, place the witness in a third failure domain, such as a different rack, a different building, a different power supply and switch, or a small VM in the cloud. Since a witness only casts votes, a small VM is sufficient. Our witness used 2 vCPU and 2 GiB of memory, consuming 1.5 GiB right after installation. However, as seen in Section 2.3, a witness can also become a Raft leader, so if the round-trip latency to the data nodes is too high, the downtime during leader re-election could be prolonged (unmeasured).

4.7 SeaweedFS: It cannot be used with a single volume server, but it does not fail immediately either

In replication 001 , when only one volume server remains, there is nowhere to place new writes. Up to this point, it behaves as configured. The problem was that failures only appeared after hanging for about 10 seconds per request. Two factors overlapped.

• The surviving volume server continued sending heartbeats to the dead former master leader and only attached to the new leader after 54 seconds . Meanwhile, the new leader reported "no volume server capacity registered yet" because its registered capacity was 0. This phrasing differs from the actual cause.
• When the isolated master returned, it disrupted the perfectly functioning leader, causing a leaderless period of 6 to 19 seconds.

Both issues were submitted upstream. For the problem of volume servers being slow to find the new leader, we submitted a fix PR (seaweedfs#12259, reducing the reproduction time from 25 seconds to 4 seconds), and for the problem of the returning master demoting the leader, we filed an issue (seaweedfs#12260). In the same experiment, running with -raftHashicorp(Raft with PreVote enabled) prevented the leader from being disrupted. If you use SeaweedFS with two data nodes in production, keep in mind that uploads will hang rather than fail immediately when one node is down, and reflect this in your application's timeouts. When an entire volume server disk was lost, volume.fix.replication -apply matched all 24 volumes in 30.7 seconds, during which there was 1 S3 write failure and 0 read failures. In a test where a single filer was paused for 60 seconds, there were 0 failures out of 748 S3 writes.

4.8 Collection of Short Pitfalls

5. Limitations of This Measurement

Please check the conditions before using these numbers.

• It is Docker on a VM. It was run with Docker Compose inside a KubeVirt VM. The values may differ on physical servers, other hypervisors, or Kubernetes operator configurations. Container memory limits were also set smaller than in production (e.g., OpenSearch at 1.4g).
• Each test was run only once. Isolation was tested three times by varying the node and length, graceful shutdown twice (of which Redis values from section 4.5 must be viewed separately due to a defect), and witness shutdown and isolation/manual promotion of a two-node setup were each tested once. These are neither distributions nor medians of repeated identical conditions.
• VM power-off was not performed (unmeasured). "Isolation" was treated as equivalent to power-off as seen from the peer side and used as a substitute. Phenomena specific to power-off, such as disk cache loss, are not included in this measurement.
• The witness was on the same physical server as one of the data nodes. Since only VM-level failures were injected, the results are unaffected, but physical server failures were not measured.
• Unmeasured items: Rollback (bringing dead nodes back in), finding the primary server via libpq multi-host connection (target_session_attrs=read-write), Patroni failsafe_mode, reducing OpenSearch failure detection values, actual reconfiguration of 2-member Keeper and SeaweedFS master, and watchdog fencing.
• Gaps in security configuration: etcd, PostgreSQL, and OpenSearch used TLS with their own CA, but Redis and Sentinel used only password authentication, and Patroni REST, SeaweedFS, and ClickHouse HTTP were unencrypted plaintext within the session network. In production, TLS must be added to these three as well.
• Clock drift between VMs was within ±0.3 seconds, so differences of less than 1 second were not interpreted.

6. Conclusion: Checklist for 2 Nodes + Witness

A word from TT. Having a witness friend makes majority voting work well. However, since there are still only two friends carrying the load, some services say, "I'll put the load down only when my partner arrives" if one takes a break. And when you retry after hearing "It failed," make sure to attach a name tag (idempotency key). It gets troublesome if the same load comes in twice!