Listening to Bounces via Sound: Onset Detection, Surface Classification, and the Record of Failure in Real Recordings

Status: Draft (2026-10-13), incorporating actual measurements (including negative results). Classification experiments and degradation experiments are values measured on temporary CPU pods and laptops on 2026-10-11–12. The endpoint (score) effect was measured only with mock onsets created from ground-truth timestamps, not actual sounds. Smartphone recording conditions are unmeasured. Paper figures are marked as "Paper". Writing rule: Actual measurement data is the default, and unmeasured items are marked as "Unmeasured".

Table of Contents

• 1. Why Sound?
• 2. Principle 1: Onset Detection
• 3. Principle 2: Surface Classification
• 4. Actual Measurement: Nearly Perfect in Clean Recordings
• 5. Actual Measurement: When Conditions Worsen
• 6. Negative Result: Effectiveness Could Not Be Confirmed in Real Recordings
  • Simulation: What Would the Score Be If There Was Sound?
• 7. How to Attach It
• References

1. Why Sound?

You can follow a rally in a table tennis hall even with your eyes closed. The "clack" of the table and the "thwack" of the racket alternate. There are three reasons why sound is appealing.

1. It perseveres where video is weak. Sound can be heard even when it is dark, the ball is blurred, or it is blocked by the player's body. Conditions where ball tracking largely collapsed in Episode 2 (darkness 0.685, occlusion 0.842) do not affect sound.
2. Time is fine-grained. 60fps video has one frame every 16.7ms, but 44.1kHz sound has 44 samples within 1ms. The exact moment of a bounce can be pinpointed more accurately than with video frames.
3. It directly distinguishes hits and bounces. To the most common mistake in Episode 5—"reading a hit as a bounce"—sound can directly reply, "What was heard at that moment was a racket sound."

2. Principle 1: Onset Detection

An onset is the moment a sound suddenly starts. The sound of the ball hitting the table is a short impulse sound that rises sharply within a few ms and disappears, so finding the onset gives the bounce timestamp.

The intuition is "a place where the sound intensity suddenly became louder than just before." Computation typically involves three steps.

1. Create an onset intensity curve. Measure energy for each short window (several ms), or add up only the increase per frequency component compared to the previous window (spectral flux). The reason for looking only at the increase is to capture the starting momentrather than the duration of the continuous sound.
2. Follow the baseline. Since ambient noise levels change constantly, instead of a fixed threshold, a multiple of a recent average (moving average or exponential moving average) is used as the threshold.
3. Pick peaks. Local maxima exceeding the threshold are set as onsets, and peaks that are too close together are merged into one.

The classic literature in this field is Bello et al.'s tutorial on onset detection (IEEE Trans. Speech and Audio Processing, 2005), and Python's librosa provides onset.onset_detect for "picking peaks from the onset intensity curve."

The table tennis sound paper (Gossard et al., arXiv 2409.11760, IEEE STAR 2025) chose a simpler path. A 10kHz high-pass filter (5th-order Butterworth) cuts out low-frequency noise, and the energy of a 1ms window is compared with an exponential moving average. Ball impact sounds have strong high-frequency components, so most human speech and footsteps are filtered out. In clean recordings, precision was 0.98 and recall 1.00; when mixed with speech, they were 0.98 and 0.95 (paper, recordings with a directional microphone placed 50cm–2m from the player).

3. Principle 2: Surface Classification

Once an onset is found, we determine "what kind of surface was hit." A short segment immediately following the onset (15ms for TT Sounds) is sliced, features are extracted, and classification is performed.

Features are numerical summaries of the sound's "timbre." Our experiment used 77 dimensions. These include the mean and standard deviation of 20 MFCCs tailored to human hearing frequency sensitivity, shape indicators like spectral centroid and spread, and log energy per frequency band. Tables are large panels producing low, sustained resonances, while rackets are made of rubber and wood, producing short, dull sounds. This difference is reflected in the features.

A simple classifier was sufficient. Logistic Regression, Random Forest, and histogram-based gradient boosting (HistGradientBoosting) were compared. The paper reported a surface classification F1 score of 0.97 using a small CNN, taking 4.2ms on a laptop CPU (paper).

4. Actual Measurement: Nearly Perfect in Clean Recordings

This was done using 5,702 15ms samples from TT Sounds (University of Tübingen, CC BY-NC 4.0). It consists of 3,396 rackets, 777 tables, 290 floors, and 1,239 others, sliced from 115 original recordings.

There was one crucial decision. The 80/20 split provided by the dataset means neighboring samples sliced from the same recording are included in both training and testing. Neighboring samples sound almost identical, so testing essentially asks to recall memorized items. Therefore, we performed 5-fold cross-validation by original recording unit, ensuring that only previously unheard recordings entered the test set. Results measured with dataset-based splitting were 0.1–0.8%p higher.

Feature extraction took 4.7ms per sample, and prediction took 0.001–0.04ms (CPU). This is well within real-time requirements.

5. Actual Measurement: When Conditions Worsen

Degradation was applied only to the test folds of the same samples. Speech mixed actual coach voice audio, while music and reverberation were synthesized. These are the recall / precision for "Is it a table sound?"

Two things are clear.

• Classifiers trained only on clean sound collapse under noise. At a white noise SNR of 10dB, only 0.9% of table sounds were found. Most performance is recovered if noise, speech, and distance are mixed in during training (0.85–0.96).
• Adjacent table sounds cannot be distinguished even when mixed in during training. Precision lingered around 0.61–0.77. This is because adjacent table bounces are also identical table sounds. A 15ms segment alone cannot tell if it is "our table." This is why you must not establish bounces based on sound alone in a table tennis hall with multiple tables.

6. Negative Result: Effectiveness Could Not Be Confirmed in Real Recordings

Up to this point, we used cleanly recorded separate audio. The real question is: "Does it work in audio captured alongside match video?"

The first wall we hit was data. OpenTTGames videos do not have audio tracks(ffprobe confirmed that all 12 episodes have only a single video stream). We could not find publicly available data containing video, audio, and bounce ground truth together.

Therefore, we used T3Set (amateur multi-ball training, CC BY-NC-SA 4.0), which includes audio alongside the video. The video file (mkv) and audio (AAC 44.1kHz) are bundled in the same container, guaranteeing visual alignment. We attached the above classifier to 8 rounds (8 players).

No evidence emerged showing that bounces could be distinguished in real training center recordings. The evaluation metric itself was weak. The misalignment between the racket sensor clock and the video filename (in seconds) varied from −1.5 seconds to +3 seconds per round, shaking up even the hit time alignment. Therefore, audio was not added to the engine.

Simulation: What Would the Score Be If There Was Sound?

Since there is no data containing video, audio, and ground truth together, we created fake onsets from ground-truth bounce and hit times of 12 real matches to measure "what it would be like if sound of this quality existed." We re-emphasize that this is not real audio. Because the design is structured so that audio does not change the judgment but only changes it to "Needs Verification," the winner accuracy remains identical at 93.1% across all rows.

Here is how to read this. Clean and synchronized audio brings 2 incorrect judgments forward to human reviewers. Unsynchronized audio causes racket sounds to overlap with bounce locations, doubling the "needs verification" cases. Since a human has to re-examine almost every rally, it is useless. To use sound, estimating audio-video misalignment and judging "Is this sound reliable?" must come first.

7. How to Attach It

The design for attaching it to the engine has been decided.

• If a "table" onset exists within ±20ms (one 60fps frame) of a bounce candidate, increase the reliability of that bounce.
• If it is a V within the table band but the sound at that moment is a "racket," treat it as a hit (targeting Episode 5's "reading hits as bounces").
• Table sounds heard while the ball is not visible are posted only on the verification screen as "sound-only bounces" and are not passed to the referee (because the side is unknown).
• If there are no video candidates, sound alone does not establish a bounce (adjacent table).
• Output a health score (0–1) for each sound, and if it is low, do not use that sound. Do not use absence of evidence as evidence of absence.

One thing is needed before that: A test set consisting of smartphone videos shot according to shooting guidelines, with bounce timestamps labeled by humans. How the above figures turn out with smartphone microphones, table tennis hall reverberations, and adjacent table sounds can only be measured once that test set exists (unmeasured). We also investigated methods of creating noise-robust classifiers using pre-trained general-purpose audio embeddings, but experiments are not yet finished (in progress).

TT Out. It understood 99% of the "tok" sound recorded in a quiet room, but it still couldn't recognize the real table tennis hall noise. Recording what it failed to do is the way to wander less next time.

References

• Gossard et al., Sound-Based Spin Estimation in Table Tennis: https://arxiv.org/abs/2409.11760
• TT Sounds Dataset: https://github.com/cogsys-tuebingen/tt_sounds (CC BY-NC 4.0 in README)
• Bello et al., A Tutorial on Onset Detection in Music Signals, IEEE Trans. Speech and Audio Processing 13(5), 2005
• librosa onset.onset_detect: https://librosa.org/doc/0.10.2/generated/librosa.onset.onset_detect.html
• T3Set: https://zenodo.org/records/15516144 (CC BY-NC-SA 4.0)
• Our Measurements: Repository docs/tt/PERFORMANCE.md 2.7 (Sound Experiment), 2.9.6, Code experiments/sound/·experiments/fusion/sound_robust.py