Three Hosts Were Logging by Three Different Clocks

Goal

From logs stamped by three hosts with different clocks, you pick out notations that lack an offset or have a different meaning, use a reference event to split and estimate each host's local offset and clock error, and rewrite every line in trustworthy UTC to restore the reversed causal order.

Why it matters

The first thing you decide in an outage investigation is "what happened first." The basis for that order is the time in the log, and that time is a claim, not a fact. Without an offset, you cannot tell from the outside which region's clock it is, -00:00 means something different from Z, and even if the offset is accurate, if the host's clock is a few seconds ahead, the cause is recorded after the effect. Making times trustworthy is not preparation for the analysis but the analysis itself.

Steps

1. Create and run /root/clock/gen_clock.py to create app-seoul.log, db-frankfurt.log, and edge-newyork.log under /root/clock/raw/.
2. /root/clock/stamps.json — write how the time notation differs for each file and how many lines can be converted to UTC right now.
3. /root/clock/dst_probe.json — measure with zoneinfo the local times in regions with daylight saving time that disappear or occur twice.
4. /root/clock/anchors.json — find the check signals that left marks in all three files.
5. /root/clock/skew.json — use the reference events to split and estimate each host's local offset and clock error.
6. /root/clock/fixed.ndjson — rewrite every line with the corrected UTC time and line them up in time order.
7. /root/clock/causality.json — count the requests where the cause was stamped after the effect before correction, and compare with after correction.
8. /root/clock/clock_report.md — leave a report of what you corrected and on what grounds.

Notes

• Write every corrected ts in the form 2026-03-08T06:05:00.200Z — an uppercase T between date and time, three millisecond digits, and an uppercase Z at the end.
• Unify the rule for reading a line's time into one. If an offset is attached, read it as it is; if not, read it as UTC for now. Whether that assumption is right is decided by the reference events.
• Python's strptime reads both -05:00 and -00:00 with %z. Use zoneinfo.ZoneInfo and datetime's fold to handle the daylight saving time interval.
• A local offset is a multiple of 15 minutes. If you fold the difference from the reference to the nearest 15 minutes, you get the local offset, and the remaining seconds are that host's clock error.
• Common mistakes: reading -00:00 as meaning the same as Z, sorting with offset-less lines left as UTC, discarding the leftover few seconds as noise, and fixing the offset to a single value on the day of a daylight saving time transition.
• Collect all outputs of this lab under /root/clock/. They disappear when the session ends, so keep the important ones on screen.

Reproduce the logs of three hosts

Create and run /root/clock/gen_clock.py to create app-seoul.log (65 lines), db-frankfurt.log (41 lines), and edge-newyork.log (61 lines) under /root/clock/raw/.

The three files were written by different hosts using their own clocks, so their time notations differ. One has no offset at all, one attaches the offset as -00:00, and one attaches the local offset properly. First create /root/clock/raw and write the three files in it.

Count which lines can be converted to UTC right now

In /root/clock/stamps.json, for each host name (app-seoul · db-frankfurt · edge-newyork), write lines, offset_style, utc_resolvable, and distinct_offsets. offset_style is none if there is no offset at all, -00:00 if every offset is -00:00, and explicit otherwise. utc_resolvable is the number of lines you can convert to UTC by looking at that line alone.

The originals are the three files /root/clock/raw/app-seoul.log, db-frankfurt.log, and edge-newyork.log. Only for lines with an offset attached is UTC settled by that line alone. distinct_offsets is the offset strings that actually appear in that file, collected without duplicates, and if two values appear in one file, think about what that window straddled.

Measure the times that disappear and the times that occur twice

In /root/clock/dst_probe.json, write the six below as an array in exactly this order. Each item has four keys, zone, local, count, and utc, where count is the number of UTC instants that correspond to that local time, and utc is an array of those instants in ascending order in the form 2026-03-08T08:30:00.000Z. (1) America/New_York 2026-03-08 02:30:00 (2) America/New_York 2026-03-08 04:30:00 (3) America/New_York 2026-11-01 01:30:00 (4) Asia/Seoul 2026-03-08 02:30:00 (5) Europe/Berlin 2026-03-29 02:30:00 (6) Europe/Berlin 2026-10-25 02:30:00

Attach the region with zoneinfo.ZoneInfo and switch datetime's fold between 0 and 1 to make two candidates. Convert each candidate to UTC and back to that region, and only those that match the original local time are real. If none comes back, that local time does not exist, and if two different ones come back, it is a time that occurs twice.

Find the reference events stamped in all three files

In /root/clock/anchors.json, write db-frankfurt as reference_host, and in anchors, for each probe= value that appears in all three files, put four keys, corr, app-seoul, db-frankfurt, and edge-newyork, sorted by corr in ascending order. The three host values are the time strings exactly as written in that file.

The originals are the three files /root/clock/raw/app-seoul.log, /root/clock/raw/db-frankfurt.log, and /root/clock/raw/edge-newyork.log. The check signals sprayed by the operations scheduler are stamped as probe=SYNC-xxxx. A signal that is in only one host cannot be used as a yardstick, so keep only the intersection of the three files. The time strings must be put in as the original text, including the offset, so you can trace back later. Choose the reference from among the hosts that attach an offset and tell you UTC, the one whose clock you can trust.

Split and estimate the local offset and the clock error

In /root/clock/skew.json, write reference_host and hosts. hosts has zone_offset_minutes, skew_seconds, and anchors_used for each host. For each reference event, take the difference of the time that line states (as is if it has an offset, read as UTC if not) minus the reference host's time, fold that difference to the nearest 15 minutes to get zone_offset_minutes, and the leftover seconds are skew_seconds.

The material is the /root/clock/anchors.json you made in step 4. One difference contains a mix of two things — how far the clock face is shifted from UTC (the local offset) and how wrong that clock is (the clock error). IANA local offsets are multiples of 15 minutes, so you can split the two. If there are several reference events, the values may wobble, so gather them into one value such as the median, and leave how many you used in anchors_used.

Rewrite every line with a trustworthy time

In /root/clock/fixed.ndjson, write every line of the three files, one per line, as ts, host, raw_ts, corr, and msg, and sort in ascending ts order. ts is the UTC obtained by subtracting zone_offset_minutes and skew_seconds from the time that line states, in the form 2026-03-08T06:05:00.200Z. raw_ts is the original time string, corr is the req= or probe= value (null if none), and msg is the body remaining after the time.

The material is the three files under /root/clock/raw/ and the /root/clock/skew.json you made in step 5. Correction is two subtractions — subtract the amount the clock face is shifted, and subtract the amount the clock is wrong. Do not overwrite the original string; leave it alongside as raw_ts. If you change the reference later, you must recompute from the beginning, and you cannot if you do not have the original text.

Count the requests where the cause was stamped after the effect

In /root/clock/causality.json, write pairs, inverted_before, inverted_after, and examples. pairs is the number of req=RQ- values that appear in both edge-newyork and db-frankfurt. inverted_before is the number of cases, before correction (the time each line states as is), where the front end's time is later than the database's time, and inverted_after is the number recounted with the corrected times. examples are the first three, in ascending order, of the values that were reversed before correction.

The corrected times are already in the /root/clock/fixed.ndjson you made in step 6, and the pre-correction times can be recovered from the raw_ts of the same file. The front end receiving the request is the cause and the database processing that request is the effect, so if the front end's time is later, judging by the records alone the effect happened before the cause. The key of this step is that both hosts attach their offsets correctly.

Leave what you corrected and on what grounds

In /root/clock/clock_report.md, write four sections: ## 시계가 어떻게 어긋나 있었나 ## 무엇을 기준으로 삼았나 ## 보정한 뒤 무엇이 달라졌나 ## 다음에 받을 때의 요구사항 (in order, these mean: how the clocks were off, what was used as the reference, what changed after correction, and the requirements for the next delivery). It must include, as numbers, the total number of corrected records, app-seoul's zone_offset_minutes, and the number of inversions before correction.

The materials are /root/clock/skew.json, /root/clock/fixed.ndjson, and /root/clock/causality.json. A correction value is not a measurement but an assumption you built from the reference events, so you must write what that assumption rests on so the next person can trace back. In the last section, write what to demand of the customer so that this estimation disappears entirely.