Overall Architecture: From the Phone to the Judgment Server, and What Leaves the Device

Status: Draft (2026-10-26), reflects actual measurements. The architecture is based on the repository code as of 2026-10-25, and numbers are measured directly by us, with dates and conditions attached to each table. Actual user path latency over the internet and field accuracy on mobile phones have not yet been measured. Writing rules: Measured data is the default, and unmeasured items are marked as "Unmeasured".

Table of Contents

• 1. How Many Machines Are Attached to a Single Scoreboard?
• 2. A Map of Four Columns
  • Why the Engine Was Split into Two Pieces
  • Phone Side: Buildless Modules and Service Workers
• 3. What Happens When Auto-Scoring Is Turned On
  • Time Share and Limits
  • Measurement: When Concurrent Sessions Increase
  • Measurement: From the End of a Rally to the Appearance of the Score
• 4. Deployment: Reading Code and Weights from the Model Repository
  • Web as Images
  • Workers as Shared Volumes Instead of Images
  • Features as Environment Variable Switches
  • Incidents Caused by This Approach
  • Fences Between Columns
• 5. What Leaves the Device and What Does Not
  • Long-Term Plan: GPU on the Phone
• 6. Conclusion
• References
• Further Reading

1. How Many Machines Are Attached to a Single Scoreboard?

Using TT Lab at a table tennis club is simple. You set up a phone next to the table tennis table, open /tt/play/cam , and tap "Turn on Camera and Start." A large scoreboard appears on the screen over the camera view, and tapping the half of the screen belonging to the scoring player increases their score. The scoreboard informs you of the server turn, deuces, and court changes at the end of a game according to the rules.

Up to this point, there is only a single machine, the phone. Scoring rules, history, score reading, and recording all run inside the phone browser. The server only comes into play when the user chooses one of three options:

1. Turn on auto-scoring. Camera frames are sent to the server's GPU worker to receive suggestions for rally winners.
2. Share live. Only scoring events are sent to the server so that others can watch in real time.
3. Tap "Record this match". Game scores of finished matches are saved on the server and reflected in ratings after opponent confirmation.

There is also a path to upload and analyze recordings separately (/tt/videos), but that is a separate feature where users upload video files directly.

This article draws this architecture as a single map, detailing what travels between columns, and specifically what leaves the device and what does not. The inner workings of judgment rules are unveiled in parts 2 and 3, model principles in part 4, training in part 5, and records in part 6.

2. A Map of Four Columns

Why the Engine Was Split into Two Pieces

Looking closely inside (3), there are two layers: the part requiring GPU and OpenCV (live/), and the pure Python engine(ttscore/) which only receives ball coordinates and events. Since the engine is a collection of functions that require neither GPU nor OpenCV, tests run in CI using only synthetic trajectories without video. Even if the ball model is swapped out, the referee rules must continue to pass tests unchanged. This separation was detailed in the previous series, Part 1, across four layers: "perception, events, refereeing, and slow confirmation."

Match rules are duplicated once more. Because the phone screen must count scores without calling the server, it usesttscore/rules.py, which ports the Python rule engine ( backend/static/tt/play/lib/scorekeeper.mjs ) to JavaScript. The live viewing server (backend/app/tt_watch.py) also has a third copy of the same rules. Since having three copies inevitably leads to divergence, the Python tests and Node tests were configured to read the same case files(backend/static/tt/play/data/rules-cases.json). Modifying only one side causes the tests on the other side to fail.

Phone Side: Buildless Modules and Service Workers

The screen code consists of ES modules read directly by the browser without a bundler. Module addresses use import maps with content hashes (?v=), ensuring that code changes alter the address and prevent mixing with old caches. The table tennis zone has a service worker (/tt/sw.js, scope /tt/) that caches screen code but does not cache APIs and record screens. This is because the manual scoreboard must keep running even if the table tennis club Wi-Fi drops, and score records must always reflect the latest server values.

/tt/play/cam The first screen of

does not initially load heavy chunks (auto-scoring, recording, live) until they are used, as most people only use the manual scoreboard.

3. What Happens When Auto-Scoring Is Turned On

1. Auto-scoring is the longest path in this architecture. Let's trace step-by-step from when a frame is downscaled and sent until a score suggestion is returned. The camera video is downscaled to 512×288, compressed to JPEG quality 0.7, and sent at up to 30 frames per second (backend/static/tt/play/cam-auto.js, lib/live.mjs). Even if displayed flipped like a mirror on the screen, the sent version is the unflipped original. An 8-byte header (sequence number, timestamp in ms) is prepended to each frame. If 3 frames are sent without receiving a response, no new frames are sent. This is a mechanism to prevent the queue from growing indefinitely on slow connections.
2. The web server guards the gate. WebSocket relay (backend/app/tt_live.py) only accepts users logged in via Google and tracks their slot and time quotas. Frames simply pass through.
3. The worker converts frames into ball coordinates. The order of operations is: increase brightness → ball model (BlurBall) → select candidates near the previous ball → maintain ball trajectory for the last 2.5 seconds → fill short gaps → extract events → filter using learned bounds (veto) (live/core.py).
4. The engine makes the judgment. The referee runs only after receiving the "Start Match" signal, and when a rally ends, it outputs the winner, reason, confidence, and whether "Confirmation Required" is true (Part 3).
5. The browser asks with a chip. The judgment does not go directly to the scoreboard but appears as a chip. If a human does not respond, definitive judgments are applied as-is after 4 seconds, and confirmation-required judgments after 9 seconds (Part 3).

The ping-pong table is detected separately outside this flow. "Table detection" is run once before the match starts (using a segmentation model much heavier than the ball model), and thereafter tracked every frame using optical flow (Part 4).

Time Quotas and Limits

GPUs are expensive and shared resources. Therefore, there are multiple layers of limits. Below are the default values in the code (we could not verify if separate values exist in production config files).

If the connection drops, the browser attempts to reconnect up to five times at intervals of 1, 2, 4, 8, and 15 seconds. Reconnection is not attempted for closures where reconnecting is useless, such as when login is required or quotas are exhausted (close codes 4401, 4429, 4430, 4431, etc.).

Measurements: When concurrent sessions increase

A single worker handles up to three sessions. When three sessions arrive simultaneously, they share a single GPU time slice among the three. Below are the measured values under a condition where the same GPU was shared with a text-to-speech service on 2026-10-11 (docs/tt/PERFORMANCE.md 4.2).

With three sessions, 21–30% of sent frames were overwritten by new frames and discarded before being processed. This is because the worker has a single GPU loop, and every time the loop runs, only the most recent windowof each session is grouped into a single batch for processing, allowing newer windows to overwrite delayed ones (live/server.py Header, "Latency First"). For scoring judgments, being on timeis more important than not missing a single frame. The trade-off is that in fast rallies, 30fps effectively drops to 16–20fps, and as shown in the previous series Part 5, lower frame rates in this engine increase "incorrect judgments without confirmation" instances.

Measurements: From rally end to score display

The target is "End of rally → Judgment within 1.5 seconds" (real-time budget set by the research director). The engine considers a rally ended if there is no new bounce for 1.5 seconds after the last bounce or if the ball is not visible for 1 second. In other words, a significant portion of the budget is spent waiting to see if it has truly ended.

Attempts to reduce latency by shortening the end threshold (such as ending sooner when the ball disappears) were made multiple times but all rejected (director E03·E10·E24·E28). Ending early results in scoring too prematurely during long defensive rallies.

4. Deployment: Reading Code and Weights from the Model Repository

The deployment methods for the web and workers differ. Since this difference once caused an incident, it is described in detail.

Web via images

The web server follows a standard approach. When pushed to the repository, CI runs tests, builds a container image, and updating the image tag in the GitOps repository allows ArgoCD to automatically synchronize it to development and production environments. Frontend code (backend/static/tt/) is also included inside this image.

Workers use shared volumes instead of images

GPU workers do not build new images. They reuse the exact AI service image (PyTorch 2.9, Python 3.11) that was already running on the same GPU server, and load the engine code, BlurBall model code, weights, and OpenCV wheel from the model repository (shared volume) . Since engine code changes frequently while heavy dependencies like PyTorch rarely do, you don't need to rebuild multi-gigabyte CUDA images every time you modify the code. The trade-off is that the Python version of the image becomes locked, which is revisited in the SAM 3 story in Part 4.

The script that populates the code and weights (live/seed-pvc.sh) works as follows:

1. It verifies the weight's sha256 against the value specified in the code. The ball model default weights 3545206c…, and the table tennis model (SAM 2.1·EfficientTAM) fetches the official distribution bundle, verifies its sha256, and extracts it.
2. It downloads new files to the .new folder first, and replaces them all at once after the download completes. This prevents workers from reading partially downloaded code.
3. It restarts the worker. This constitutes deployment.

When reading weights, the worker re-verifies the sha256 and reads them strictly using PyTorch's weights_only=True (live/blurball.py). Since weight files are pickles and can execute arbitrary code, this option prevents the creation of any objects other than tensors.

Features controlled by environment variable switches

Almost all new features follow the code default off, enabled via environment variable pattern. As of 2026-10-25, the switches enabled in the live worker configuration are as follows:

Conversely, the fine-tuned ball weights (TTSCORE_BALL_WEIGHTS) and trajectory filling (TTSCORE_BALLISTIC_FILL) have finished experimentation but failed to meet the conditions for activation and are absent from the configuration (disabled). The advantage of the switch approach is that disabling is easy. By simply deleting a single line of configuration while leaving the code and weights intact, it reverts to the previous behavior.

An incident caused by this approach

We also experienced the downside firsthand. Unlike the web application, worker configurations exist outside of automated deployment, meaning a human must apply them while code is populated separately. Consequently, configuration and code can drift apart. On October 18, 2026, an actual incident occurred where TTSCORE_EDGE_REVIEW=1 was enabled in the configuration, but the engine code in the model repository did not yet contain that feature (Log E41). The switch was turned on, but nothing happened, and no errors were raised. We resolved this by re-populating the code, and since then, we strictly follow the sequence when enabling features: "Apply configuration → Populate code → Restart → Verify that the feature exists in the code inside the worker."

For the same reason, when enabling the vertex model (TT_TABLE_KP) on October 25, 2026, we did not rely solely on the configuration file; we actually sent test images to the production worker and checked the responses for traces of the vertex model. All 212 responses recorded evidence that the vertex model had executed (Log E44, Part 4).

Fences between compartments

Because workers handle user video feeds, we erected multiple layers of fences.

• Incoming connections to the zone where AI workers reside are blocked by default; only pods whose names are whitelisted, like web servers, can access a single port on the worker.
• Recorded video analysis workers do not receive DB and storage keys. They only use the web's internal API, and even then, both a service token and a per-task lease token are required. Outbound connections are restricted to DNS and web servers only.
• Workers operate without Kubernetes service account tokens attached and with all Linux capabilities dropped.
• Worker logs do not retain video, ball coordinates, or personal information (live/server.py).

5. What leaves the device and what does not

The guiding principle of the TT Lab table tennis screen is stated in backend/static/tt/play/cam.js a single line in the preamble: Camera footage never leaves the device. Breaking it down by feature:

Auto-scoring is an exceptionto this principle because finding the ball requires a GPU to inspect the video, and those GPUs currently reside on the server. Therefore, we made the exception narrow.

• Data is sent only when the user turns it on. No evaluation occurs before it is turned on and before "Start Match" is pressed.
• Resolution is reduced to 512×288 for transmission. Since that matches the input size of the ball model, there is no reason to send more.
• Web streaming only passes frames through, and workers keep only the 3-frame window viewed by the ball model in memory. They are not written to disk (backend/app/tt_live.py preamble: "Videos are not saved").
• Records corrected by human review are retained without video, and only lines consented to for training can be exported (Part 6).

Recorded video analysis is a feature where users manually select and upload files, so its nature differs. Originals are deleted after 30 days, managed by a daily early-morning cleanup job.

Long-term plan: Moving GPUs to mobile phones

The path to bringing auto-scoring back within the principle is performing ball detection directly on mobile devices. Although the ball model is small at 1.49 million parameters, whether it can achieve 60fps in mobile browsers has not yet been measured. For now, "sending to the server without storing" is a pragmatic compromise.

6. Conclusion

• The manual scoreboard is a self-contained product relying on a single phone. The server only appears when you select auto-scoring, live features, or history.
• Auto-scoring is the sole path for sending frames to the server, and the server does not store frames.
• The engine was isolated and tested in pure Python, and rules were unified across three copies—Python, JavaScript, and the live server—using identical test case files.
• Workers read code and weights from the model repository by verifying sha256, and features are toggled via switches. Because an incident actually occurred where configuration and code drifted, we now verify the code inside the worker itself when enabling features.

A word from TT: A scoreboard originally just needs a single phone. The server is only a friend you call when you want it to "help watch the ball." And that friend has promised not to write down what it sees in a diary.

References

1. T. Gossard, F. Radovic, A. Ziegler, A. Zell. "BlurBall: Joint Ball and Motion Blur Estimation for Table Tennis Ball Tracking." CVPR Workshops (CVsports), 2026. arXiv:2509.18387.
2. N. Ravi et al. "SAM 2: Segment Anything in Images and Videos." ICLR, 2025. arXiv:2408.00714.
3. Y. Xiong et al. "Efficient Track Anything." ICCV, 2025. arXiv:2411.18933.
4. Our measurements: Repository docs/tt/PERFORMANCE.md 2.13·2.16·4.2, Research logs E00·E07·E41·E44.

Further Reading

• This series Part 2: Manual Scoreboard and Match Rule Engine, Part 3: Automatic Decision, Part 6: Match Records and Privacy.
• Previous series TT Score Model Anatomy Part 1: Four Layers and Time Budget: How ball detection and the engine divide the 16.7ms frame budget.
• Reading Vision Models 3: Running in Real-Time: Latency budget, batching, on-device inference.