Real-time voice conversation, measured latency by component: ASR 0.2s, LLM first token 0.07s, TTS half the length

Status: Draft (2026-10-06), empirical measurement post. The numbers below are measured directly by the author. The time up to the first sound of a turn is an estimated calculationbased on component-wise measurements and was not measured continuously from start to finish. Most measurements were conducted using synthesized speech and a single conversational prompt, so they may differ from actual user environments. Section 4-3 provides re-measured ASR values using human-read speech (public speech corpora), including non-native English. Writing rule: Empirical data is the baseline, and unmeasured items are marked as "Unmeasured".

Table of Contents

• 1. What Was Measured and Why
• 2. Environment and Methodology
• 3. Text-to-Speech (TTS) Results
  • 3-1. VoxCPM2
  • 3-2. Magpie TTS Multilingual 357M
• 4. Automatic Speech Recognition (ASR) Results
  • 4-1. Qwen3-ASR-1.7B
  • 4-2. Nemotron 3.5 ASR Streaming 0.6B
  • 4-3. Re-measured Values with Human Speech: Native Reading and Non-Native English
• 5. Conversational LLM Results: Gemma 4 12B 4-bit
• 6. Issues Discovered During Measurement
  • 6-1. Japanese: Scores Suffer Due to Discrepancies Between Sound and Notation
  • 6-2. Deployed Servers May Be Older Than the Latest Code
  • 6-3. Container Fails to Start if Driver Versions Differ Across GPU Servers
  • 6-4. Pitfalls Encountered in Measurement Tools
• 7. What Is the Latency of a Single Turn?
  • What Should the End-of-Speech Detection Timeout Be? (Empirical Measurement with Read Speech)
• 8. Potential Interview Questions
• Unverified Items
• References
• Verified Versions and Dates

1. What Was Measured and Why

Part 19investigated open-source candidates, but all values were based on documentation. To integrate a "verbal back-and-forth" feature into foreign language speaking practice, we need to directly measure the latency of the following three components:

1. Automatic Speech Recognition (ASR): Converts the learner's speech into text.
2. Conversational LLM: Generates a response to the transcribed text.
3. Text-to-Speech (TTS): Reads the response aloud.

All three components were measured in Korean, Japanese, Chinese, and English, comparing two pairs of open models (Qwen3-ASR and Nemotron 3.5 ASR, VoxCPM2 and Magpie TTS) using the same inputs.

2. Environment and Methodology

Because the two model pairs were measured on different GPUs , direct speed comparisons should be approached with caution (memory usage and accuracy can be compared regardless of the GPU). For TTS, the real-time factor (synthesis time ÷ audio duration) was calculated using the synthesis time and audio duration reported by the server; a value less than 1 indicates generation faster than playback.

3. Text-to-Speech (TTS) Results

3-1. VoxCPM2

Median of 3 runs in a warm state (model loaded, voice profile already configured).

• Regardless of language, the synthesis time is approximately half the audio duration. The real-time factor across all twelve combinations ranged from 0.494 to 0.582 (median 0.521), with a variance across 3 runs of the same combination within 0.02.
• Shorter sentences exhibit slightly higher factors. Fitting a linear equation to the three points per language yields a formula nearly identical across all four languages: synthesis time equals audio duration × approx. 0.47 + approx. 0.23 seconds(0.475 × duration + 0.23 seconds based on all twelve points). This fixed overhead of about 0.23 seconds per sentence can be interpreted as inflating the ratio for shorter sentences, though the exact nature of this overhead could not be verified.
• The server synthesizes the entire sentence and returns it all at once. The time to the first sound is simply the sentence synthesis time, which is 1.2 to 2.1 seconds for short sentences of 2 to 4 seconds. In a conversation, this value is the primary factor delaying the first sound.
• Initial voice cloning/creation calls take an additional 3 to 3.5 seconds. This only applies to the very first time, as the server designs and saves the voice profile for subsequent reuse.
• It consumes approximately 6,600 to 6,900 MiB of GPU memory. Initial model loading took 385 seconds, including weight downloads on the first startup.

3-2. Magpie TTS Multilingual 357M

Tested with the same sentences (8GB-class laptop GPU, median of 3 runs).

• It consumes only about 750 MiB of additional GPU memory. This is calculated by subtracting 5,078 MiB (with only the ASR model loaded) from 5,832 MiB (with both models loaded). This is about one-ninth of VoxCPM2's memory usage.
• The median real-time factor for the three properly functioning languages is 0.466 (0.447 to 0.541). This is comparable to or slightly lower than VoxCPM2, though measured on a different GPU.
• Korean could not be used out-of-the-box in this configuration (causes and workarounds below). Sentences of 20 to 67 characters resulted in audio clips lasting 0.4 to 4.3 seconds (compared to 4 to 13 seconds normally), which yielded empty results when passed to ASR. The model card states that Korean is supported, and the same invocation method worked normally for English, Japanese, and Chinese. Checkpoint versions were not the root cause. We tested all three checkpoints in the same environment—main, v2607 (which the model card notes added Korean), and v2602—and all three Korean sentences resulted in audio clips lasting 0.4 to 2.1 seconds (0.7 to 1.3 seconds for v2602). Transcribing them back yielded a character error rate of 1.00, which persisted even when text normalization was enabled. For English, all three versions were normal with the same invocation (character error rate 0.00 to 0.07). Initializing with normalization enabled took 4.5 to 4.7 seconds. The root cause was the code selecting the tokenizer based on the language code. The v2607 checkpoint has 15 tokenizers including korean_chartokenizer (tokenizing a Korean sentence with this yields 40 tokens), but NeMo 3.0.0's get_tokenizer_for_language looks up the language code in an internal table (LANGUAGE_TOKENIZER_MAP) and falls back to english_phoneme if not found. Because Korean was missing from this table, ko, ko-KR, korean all routed to the English phoneme tokenizer. When a Korean sentence passed through the English phoneme tokenizer, only 5 tokens remained (mostly unknown characters), causing the model to finish in 0.4 to 0.8 seconds (English, Japanese, and Chinese mapped to their respective tokenizers and worked normally). Modifying just this function to use korean_chartokenizer for Korean (a workaround) resulted in audio outputs of 3.5, 5.8, and 3.8 seconds, with back-transcribed character error rates of 0.00, 0.00, and 0.05 via ASR. Synthesis times were 1.6, 2.7, and 1.7 seconds (real-time factors around 0.46 to 0.47), matching other languages. In short, the model and checkpoints support Korean, but this NeMo code version failed to wire it up. Whether newer NeMo versions have added this mapping was not verified (unverified). This test used 3 sentences, and voice quality was not evaluated.
• The back-transcription error rate is a rough indicator of synthesis quality, obtained by feeding synthesized speech into another ASR model (Nemotron 3.5 ASR) and comparing it against the original text.

4. Automatic Speech Recognition (ASR) Results

The 12 audio files (2 to 13 seconds) generated by the previous TTS step were transcribed with specified language targets.

4-1. Qwen3-ASR-1.7B

• Transcription took 0.15 to 0.6 seconds, with a real-time factor of 0.04 to 0.08. Being about 10 times faster than synthesis (approx. 0.5), it is not a bottleneck.
• Even when running with automatic language detection instead of specifying the language, all 12 were correct.
• Loading the model increased GPU memory by approximately 5,756 MiB. The first request took 174 seconds (including weight downloading and loading), while pure inference took 6 seconds.

4-2. Nemotron 3.5 ASR Streaming 0.6B

Tested on the same 12 audio files (8GB-class laptop GPU, language specified via locale, transcribing entire files at once).

• Examples of transcription errors includetranscribing How was as How is, 三明治 , and 三名制, 집 근처에 as 집근처에 . Across these 12 samples, Qwen3-ASR was more accurate, whereas Nemotron is smaller and designed for streaming.
• Accuracy by chunk size. The model allows choosing processing chunk sizes from 80, 160, 320, 560, and 1,120 ms. Measuring average error rates across the same 12 samples yielded nearly identical results for 80ms (4.3%), 160ms (4.0%), 320ms (3.9%), and 560ms (3.9%), with a spike only at 1,120ms (11.3% ). This was caused by a single audio sample returning empty results (persisting across re-runs), but re-measuring with human speech showed 0 empty results for Korean regardless of chunk size (Section 4-3). This appears to be an artifact specific to these 12 synthetic samples. These values are approximations applying the setting to the entire file, and actual streaming latency was not measured.

4-3. Re-measured Values with Human Speech: Native Reading and Non-Native English

The 12 items above are synthesized voices and thus do not contain human pronunciation and intonation. Nemotron 3.5 ASR was re-evaluated using public speech corpora: 30 sentences per language from FLEURS (human-read sentences) and 100 sentences from speechocean762 (a corpus of English read by Chinese speakers with pronunciation evaluation scores). The setup used full-file transcription at once, a default chunk size of 1,120ms, and an 8GB laptop GPU.

When non-native English was broken down by total pronunciation score (0-10), the error rates varied significantly.

• The Korean character error rate of 5.8% is comparable to the model card's FLEURS value (7.12%, 1.12s chunk). Although the sample size is small at 30 sentences, it was consistent with the card. The average of 3.9% across the 12 synthetic voices above (average of four languages) is lower than the values for human voices (Korean 5.8%, Japanese 14.7%, Chinese 18.2%), confirming that it was optimistic.
• **In non-native English, the error rate was about 4 times (38.8%) that of native reading (10.0%)**, and worsened sharply as pronunciation scores dropped. Below a total score of 6, the median was 100%, meaning most utterances could not be transcribed. This indicates that in language learning apps, beginner learners' speech is the hardest to recognize and thus requires the most feedback. The human-evaluated scores were provided by the corpus.
• Smaller chunk sizes reduce accuracy, but for Korean, the difference was almost negligible.

For English and non-native English, reducing the chunk size from 1,120ms to 80ms increased the error rate by 3-6 percentage points. For Korean, it remained flat at 5.6-6.2% with virtually no difference, and the model card's Korean values (7.12-7.70%) are also flat. Empty results did not occur only at 1,120ms (for non-native English, they were actually most frequent at 1,120ms with 12 occurrences), and the 1,120ms-and-above phenomenon seen in the synthetic sample above was not reproduced.

• Limitations. These are read speech recordings consisting of 30 sentences each for Korean, Japanese, Chinese, and English, and 100 sentences for non-native English. Conversational styles, background noise, Korean spoken by speakers of native languages other than Korean learners' English were not measured (unmeasured). This is an approximation using chunk settings applied to the entire file and does not represent actual streaming fixed latency.

5. Conversational LLM Results: Gemma 4 12B 4-bit

Tailored to the learner's level (CEFR A2), we sent the instruction "Answer in one or two sentences and end with a simple question" along with the user utterance "I read a book at a cafe yesterday" in four languages, receiving 5 streaming responses per language. Time to first tokenis measured from sending the request until the first character arrives, and first sentenceuntil the first punctuation mark appears (the point where speech synthesis can begin).

• The first token takes around 0.07 seconds, and generation is about 85 tok/s. This was also consistent with server-side internal measurements (processing 67 input tokens took 95.7ms, i.e., 700 tok/s, with 11.6ms per generated token).
• The first sentence finishes in 0.1 to 0.2 seconds. English took the longest because its first sentence is longer.
• The instructions were followed. All 20 instances answered in one or two sentences and ended with a question. Korean example: 와, 정말 좋았겠네요! 카페에서 책 읽는 건 정말 행복한 일이에요. 어떤 책을 읽었어요? Quality (correction, level tailoring) must be evaluated by humans, and this measurement is objective regarding speed only.
• Increased concurrent requests delay the first token. When Japanese prompts were sent concurrently, 2 requests yielded 0.206 seconds, while 4 requests yielded a median of 0.352 seconds and a maximum of 0.552 seconds (since the server was configured to handle only 2 concurrent requests, the rest waited).
• GPU memory usage was about 8,450 MiB. Model loading took 145 seconds, including weight downloading.

The thinking mode resulted in completely empty responses. In the initial measurements, all 20 responses were empty. Looking at the raw responses, content was an empty string, and reasoning_content accumulated English analysis such as Role: Friendly conversation partner for a Japanese learner... Constraint 1: ... , meaning the 80-token budget was entirely consumed by thinking. This happens because the model defaults to thinking first in llama.cpp's chat template, and turning it off via server options (--reasoning-budget 0) and request options (chat_template_kwargs: {enable_thinking: false}) resolved the issue. Thinking mode must be disabled in voice conversations. Leaving it enabled delays the first sound by several seconds, and if the budget is short, no response is produced at all.

6. Issues Discovered During Measurement

6-1. Japanese: Correct Pronunciation But Different Orthography Results in Point Deductions

The error rate for middle Japanese sentences was 15%, but the pronunciation was not wrong.

たいてい vs 大抵, 七 vs 7, and the presence or absence of commas represent different spellings of the same expression. If a pronunciation practice service compares recognition results with the target sentence using character similarity, learners may lose points due to such spelling differences even if they speak perfectly. Chinese numbers (七 vs 7) is expected to have the same issue. The way to fix it is to normalize numbers and notations before comparison, or to compare based on pronunciation (reading) rather than characters.

6-2. The deployed server may be older than the latest code

Chinese synthesis in one environment failed with 404 Not Found . This was because the Chinese voice preset in the server code existed in the new code but not in the deployed image. When a new feature does not work, you must check the version of the deployed imagerather than the code first.

6-3. If GPU servers have different driver versions, images will not spin up

A vLLM image built for CUDA 13 failed to start on the driver 570 series (up to CUDA 12.8) with Error 804: forward compatibility was attempted on non supported HW . The same image runs fine on the driver 595 series. CUDA 13.x requires driver 580 or higher. When using multiple servers together, you must check the driver version on each server and match it with the CUDA version of the image . The LLM measurements in this article were configured with an image for CUDA 12 (llama.cpp).

6-4. Pitfalls encountered in the measurement tools

The biggest lesson here is the case of the word error rate. A bug in the measurement code can make it look like a flaw in the model. If you get a value where the error rate exceeds 1, you should suspect the inputs (sampling rate, length, language specification) first rather than the model.

7. What is the turn latency?

The time from when the user finishes speaking until the first sound plays is the sum of the following:

(말 끝 판단) + ASR + LLM 첫 문장 + TTS 첫 문장 + 전송

A calculated estimatestitched together from component measurements by sentence length (we did not measure continuously from start to finish). The first sentence of the LLM is an actual sentence from the measurements above, and the TTS time was calculated by plugging the reading length of that sentence into the formula from section 3-1 (synthesis ≈ 0.475 × audio length + 0.23 seconds).

This table does not include Voice Activity Detection (VAD) and network transmission. Assuming end-of-speech detection takes 0.3 to 0.5 seconds, that much is added accordingly. How much this assumed value incurs in the cost of cutting off mid-speech was measured right below using human-read sentences. The range of publicly available values is 200–500ms recommended by NVIDIA Blueprint, 0.2s default for Pipecat, 0.5s for LiveKit (0.3s for streaming turn detector), and 500ms for OpenAI examples, so the assumption falls within this range. The target latency in the same document (from the end of speech to the start of response) is 600–1,500ms, and the server-end latency of the blueprint using large GPUs and streaming TTS is 0.86–0.93s (vendor figures). The fact that the Korean total in this article plus VAD, which is about 1.9–2.3s, is larger is interpreted as being because the TTS in this article is not streaming (time to the first sound of TTS in public cases is 0.07–0.37s). Learners often hesitate, so short judgments might cut off their speech; research addressing this was not found in accessible materials (unconfirmed).

What Should the End-of-Speech Detection Time Be? (Read-Speech Measurements)

End-of-speech detection assumes that speech has ended if the user is silent for a certain period of time. If that duration is shorter than natural pauses within a sentence, speech is cut off mid-sentence, and if it is too long, the response is delayed by that amount. In FLEURS sentences read by humans (60 per language, average 10–14 seconds), speech segments were found using Silero VAD (6.2.3, threshold 0.5, single CPU thread), and the intervals between adjacent speech segments within a single sentence were counted as "in-sentence pauses." The script and raw data are at docs/experiments/2026-10-07-vad-turn-gaps/ , and were run on a single CPU thread without a GPU.

• If end-of-speech is set to 0.3 seconds, 40 to 52% of reading sentences are judged to have ended mid-speech. The lower end of the above assumption (0.3 seconds) is too short for this data. It decreases to 13–28% for 0.5 seconds, 7–10% for 0.7 seconds, and 0–1.7% for 1 second.

• Calculating this, when adding 0.5 seconds for end-of-speech judgment to the above turn total, it is about 2.1 to 2.3 seconds for Korean, and about 2.3 to 2.5 seconds when using 0.7 seconds (this is calculated by adding to the total in the table above and was not measured from start to finish). Increasing the judgment time to reduce interruptions is a trade-off where that value is directly added to the response latency.

• The cost of VAD itself was small. It had a median of 3.2 ms (maximum 55.1 ms) to process 1 second of audio on a single CPU thread, and 0.10 ms per 32 ms frame. The latency of end-of-speech judgment consists almost entirely of waiting time settings rather than calculation.

• Limitations. Because this is reading speech and sentences are longer than a conversational turn (the longer the sentence, the more pauses there are), the values in conversational speech may differ, and pauses for stuttering or thinking are not included. Only a single VAD (Silero 6.2.3) and a single threshold were used; turn-detection models that judge the end of speech based on meaning were not measured. We counted whether there were pauses in each sentence or not, but did not judge whether each pause was actually where the speech ended.

• The bottleneck is Text-to-Speech (TTS). About 60 to 85% of a turn's time is TTS, while ASR (0.2 seconds) and LLM (0.1 to 0.2 seconds) combined are around 0.5 seconds.

• Cutting the first sentence short makes the first sound faster. If Japanese and Chinese responses start with short backchanneling, the total drops to just over 1 second. English was the slowest because the first sentence was long. We think that instructing conversational prompts to start with short backchannels would achieve the same effect, but we did not test it.

• Approaching ChatGPT levels (around 1 second after speech ends). Japanese and Chinese are already in that range, and English and Korean can be reduced by further splitting the first sentence or using streaming synthesis (which this server does not use). This is a speculation and was not tested.

8. Potential Interview Questions

• How do you break down the latency of real-time voice conversations? Divide it into end-of-speech judgment, ASR, LLM first sentence, TTS first sound, and transmission, and measure each stage. Here, ASR was 0.2 seconds, the LLM first sentence was 0.1 to 0.2 seconds, and TTS was 0.7 to 1.8 seconds, making TTS the bottleneck.
• Is a real-time factor of 0.5 sufficient? It means it is faster than playback, which is sufficient when chaining long answers, but the time to the first sound is a separate matter. Non-streaming servers must synthesize the entire sentence before producing the first sound.
• What should you watch out for when scoring learners using speech recognition results? Even if the sound is correct, the transcription may appear differently (たいてい Dae 大抵). Normalize or compare based on reading before comparing.
• What do you look at when two models share a single GPU? VRAM headroom (adding about 6,900 MiB for TTS and about 5,800 MiB for ASR), host RAM, and server driver version (whether it matches the CUDA version in the image).
• What do you do when measured values defy common sense (error rate > 1)? Suspect the measurement code before the model. Errors in input conversion (sampling rate) are a common cause.

Things That Were Not Verified

• Conversation quality. Speed was measured, but qualities such as adaptation to the learner's level, error correction, and naturalness require human evaluation and were not evaluated. Only a single prompt ("I read a book at a café yesterday") was tested.
• English recognition accuracy for Korean learners and Korean speech recognition accuracy for native speakers of other languages. Non-native English was measured only using an English corpus read by Chinese speakers (Section 4-3).
• Actual end-to-end latency connecting components. Section 7 is based on calculated estimations.
• Direct comparison of values measured on different GPUs. Two model pairs were measured on different GPUs.
• Quality of Magpie TTS Korean sounds(The cause was identified as a tokenizer mapping omission in the NeMo code, Section 4. Bypassing it yields a text-to-reply character error rate of 0-5% for 3 sentences). The 1.12-second chunk empty result of Nemotron ASR was not reproduced in human speech.
• Concurrent conversation counts (TTS, ASR). Only one was measured at a time (LLM measured 2 to 4 concurrent ones). The TTS server holds a lock processing requests one by one, so receiving two conversations simultaneously appears to queue them up, but this was not measured. Because this TTS server is currently in active service, load testing was not run on it, and the direction was verified using public figures. Magpie TTS values in NVIDIA blogs (vendor figures, NIM serving) state that when increasing concurrent streams from 1 to 64, the time to first sound increased from 32 to 239 ms on B200, 47 to 275 ms on H100, 79 to 395 ms on A100, and 53 to 962 ms on small systems (DGX Spark) (approx. 7.5x, 5.9x, 5.0x, and 18x respectively). This serves as evidence that under the same load, smaller hardware increases concurrency latency more severely, and these figures cannot be directly transferred to the environment of this article.
• First-sound latency in streaming synthesis and actual finalization latency in streaming speech recognition. This measurement did not use streaming paths.
• Barge-in (situations where the user interrupts while a response is being given) and end-of-speech judgment in conversational audio. Intra-sentence pauses in reading speech were measured above, but barge-ins and conversational speech were not measured, and only the public scope of Section 7 was verified.

References

• Part 19 (Open Source Investigation), Part 21 (Principles), Part 17 (DCGM Metrics, Driver Version)

Verified Versions and Dates

As of 2026-10-06. voxcpm 2.0.3, Qwen3-ASR-1.7B, Qwen3-ForcedAligner-0.6B, Nemotron 3.5 ASR Streaming 0.6B, Magpie TTS Multilingual 357M, NeMo 3.0.0, Gemma 4 12B QAT q4_0, llama.cpp b11429.