Checking the Principles of Voice Conversations with Numbers: Decoding is Memory Bandwidth, Synthesis is Number of Frames, and Streaming is Cache

Status: Draft (2026-10-06), Principles article. The principle explanations are based on official materials for each model (linked at the end of the article) and general inference theory, while the numbers used for cross-checking are values measured directly by the author in Episode 20. The formulas are simplified to understand the phenomenon, and each section clarifies which parts were verified and which parts were estimated. Writing rule: Measured data is the default, and unmeasured items are marked as 「Unmeasured」.

Table of Contents

• 1. At a Glance: What Determines Time Varies by Component
• 2. Conversational LLM: Reading the Entire Model Every Time a Token is Generated
  • Principle
  • Cross-check
  • Direct Verification: Confirming the Upper Bound with Models of Different Sizes and Precisions
  • Why Speed Doesn't Drop Even When Concurrent Requests Increase, and Its Limitations
  • Input Processing is Compute-Bound, and Short Inputs Come Out Slower
• 3. Text-to-Speech: Repeating as Many Times as the Number of Patches to Create
  • Principle
  • VoxCPM2: 76ms per patch
  • Magpie TTS: Two frames at a time
  • Principle by Which Streaming Synthesis Reduces Time to First Sound
• 4. Streaming Speech Recognition: What Is Computed Once Is Not Recomputed
  • Principle
  • What Cache Does
  • Does Increasing Chunk Size Improve Accuracy?
• 5. Putting It Together: Bottleneck Types Differ by Component
• 6. Likely Interview Questions
• External Evidence: Confirmed via Papers and Official Documentation
• Things Not Confirmed
• References
• Verified Versions and Dates

1. At a Glance: What Determines Time Varies by Component

The values in Episode 20 were about 「how long it took」, whereas this article is about 「why it takes that long」.

2. Conversational LLM: Reading the Entire Model Every Time a Token is Generated

Principle

For a Transformer to compute a single next token, it must use the weights of all layers once. With a single request, each weight undergoes only one or two multiplications and finishes, leaving the GPU compute units idle while the speed at which weights are fetched from memory (memory bandwidth) determines the time. Therefore, the theoretical upper bound of token generation speed is as follows.

초당 토큰 수 ≤ 메모리 대역폭 ÷ 가중치 바이트 수

Conversely, the input (prompt) processing stage computes dozens of tokens all at once, so multiple tokens reuse the same weights, making compute capacity the bottleneck. This is why input processing is much faster than generation on the same model.

Cross-check

• It is utilizing 66% of the upper bound. The remainder likely breaks down into KV cache read by attention, computations reverting the quantization ranges, and intervals for launching kernels per layer, though the exact breakdown for each has not been isolated (estimated).
• Input processing is 8.2x faster than generation(700 ÷ 85.6). This aligns with the direction predicted by the principles, which is why the first token is not significantly delayed even when the input is long. Conversely, long answers are delayed in direct proportion to the token count. However, since 67 tokens is short, it is a value mixed with fixed costs per request. The measured values by input length are found below in 「Input Processing is Compute-Bound, and Short Inputs Come Out Slower」.
• Quantization manifests directly in speed. If the same model is kept at 16-bit, the weights would be about 25 GB (13 billion parameters × 2 bytes, calculated), lowering the upper bound of the same formula below one-third to 905 ÷ 25 ≈ 36 tok/s. Since it does not even fit into this GPU's 24 GB of memory, 4-bit is a prerequisite rather than a choice. The 16-bit speed of this model was not measured (unmeasured, calculated). When precision was halved with a smaller model, the gain was slightly smaller than calculated (1.54x instead of 1.7x) (direct verification below), so the difference between 4-bit and 16-bit is also likely to be smaller than the calculated value.
• Thought tokens are generated at the same speed. In the reasoning of Episode 20, a response where all 80 tokens were spent on thought took about 1.0 second (matching 80 ÷ 85.6 ≈ 0.93 seconds). In voice conversations, if the model uses N thought tokens, the user hears silence for N ÷ 85 seconds. The reason why thoughts must be turned off emerges from calculation rather than intuition.

Direct Verification: Confirming the Upper Bound with Models of Different Sizes and Precisions

Since the cross-check above sourced its bandwidth value from the specifications, we could not verify whether the upper bound itself was correct. Therefore, we measured the bandwidth directly firston a single 8GB-class GPU and measured the generation speeds of four models of differing sizes and precisions using the same method (vLLM 0.31.0, 1 concurrent request, median of 5 runs for 128 generated tokens).

• GPU read-only bandwidth: 250 GB/s(sum of reading large tensors via PyTorch, median), copy 227 GB/s
• fp16 matrix multiplication 4096³: 31.8 TFLOPS(median)

Weight size is the loaded memory from server logs, and 「Weight × Speed」 is a calculation assuming weights are read once per generated token.

• The principle holds true. Regardless of model size, 「Weight × Speed」 clusters near the bandwidth (86–98%). This means speed is determined not by the parameter count, but by the bytes that must be read.
• The effect of halving precision was smaller than calculated. When changing 1.7B from bf16 to fp8, the weight becomes 1.7x smaller (3.46 GB vs. 2.04 GB, calculated), but the speed only increased by 1.54x (68.8 vs. 105.7). Viewed in terms of time per token, an additional 0.4 to 1.3ms is spent per token beyond reading time (calculation: 1 ÷ speed − weight ÷ 250 GB/s). Attention, kernel launching per layer, and calculations to revert 8-bit values are believed to be involved, though the exact breakdown for each has not been isolated (estimated).
• The bandwidth of the 24GB-class GPU was also measured using the same method. It was 905 GB/s read-only (approx. 97% of the approx. 936 GB/s known specification value), and the generation speed × weight of the 12B 4-bit model (85.6 × 7.0) was about 599 GB/s, which is 66% of the measured bandwidth. In other words, the 64% calculated earlier using specification values is nearly identical to 66% using measured values. This means the discrepancy was not caused by assuming bandwidth, and whether the lower percentage compared to the 8GB-class GPU's vLLM results (86–98%) stems from differences in the engine (llama.cpp vs. vLLM), the 4-bit format (per-section dequantization calculations), or model architecture could not be determined (unverified). Testing the same model on the same GPU across both engines remains future work.

Why Speed Doesn't Drop Even When Concurrent Requests Increase, and Its Limitations

Since the cost of reading weights occurs only once regardless of the number of requests, batching multiple requests together for processing (continuous batching) amortizes the read cost. In Episode 20, when 2 requests were sent simultaneously, the first token took 0.206 seconds, and for 4 requests, the median was 0.352 seconds. Because this server only had 2 slots open, the 4-request case involved queuing in pairs of two, so it was not an experiment that cleanly demonstrated the benefits of batching.

Therefore, we opened the four models above with 16 slots and increased the number of concurrent requests to measure total throughput (generating 128 tokens per request, median of 3 runs).

The unit is total throughput (tok/s), and the last two rows were calculated from the table above. Although requests were increased 16-fold, individual requests slowed down by only 8–14%, while total throughput reached nearly 14–15x. The principle of sharing the weight-reading cost manifests directly.

The point where this benefit ends was only estimated through calculation. Since each token requires 2 operations per parameter, based on the measured compute throughput (31.8 TFLOPS) and measured bandwidth (250 GB/s), compute becomes the bottleneck past approximately 130 concurrent requests for bf16 and approximately 64 for fp8 (calculated, simplifying by ignoring attention and KV cache reads). Since we only opened up to 16 requests this time, that threshold was not verified (unmeasured), and even the 4B model at the 16-request mark used only about 18% of its measured compute limit (2 × 3.6B × 775 ≈ 5.6 TFLOPS), keeping it firmly in the bandwidth-bound region.

Input Processing is Compute-Bound, and Short Inputs Come Out Slower

Input processing was measured using the same server, with prefix caching disabled and using distinct random inputs per request.

• The speed becomes constant at 1,024 tokens or more. This is about 127 times and about 65 times faster, respectively, than generation with the same model (69 and 53 tok/s). Calculating this out, for the 1.7B model, excluding embeddings, the parameter count is 1.4B, so 8,730 tok/s corresponds to about 24 TFLOPS (2 x 1.4B x 8,730); for the 4B model, the count is 3.6B, so 3,438 tok/s corresponds to about 25 TFLOPS. For both models, this is about 75-80% of the measured matrix multiplication throughput of 31.8 TFLOPS, which aligns well with the principle and figures that input processing is compute-bound.
• Short inputs have a high cost per token. At 256 tokens, the speed was lower by about 35% for the 1.7B model and about 9% for the 4B model. This is because the fixed cost per request is spread across fewer tokens in short inputs, making it appear more significant. Since voice conversation inputs are typically short, the 700 tok/s (67 tokens) measured for the 12B model above should also be interpreted as being affected by this.
• The initial measurement was wrong, which I realized by comparing it against compute limits. Initially, I repeatedly sent the same sentence, and the 4B model came out to about 91,000 tok/s. This corresponds to about 660 TFLOPS (2 x 3.6B x 91,240), which is a value a GPU cannot achieve. This was because the prefix cache hit and the input was not computed; I disabled the cache, re-measured with a different input for each request, and obtained the table above. This is an error that can be filtered out simply by checking measured values against hardware limits.

3. Text-to-Speech: Repeat as many times as the number of patches to create

Principle

Neither of the two synthesis models outputs sound all at once, but rather produce short units (160ms worth of patches for VoxCPM2, codec frames for Magpie) one by one from the beginning (autoregression). Therefore, time is directly proportional to the audio length, and a fixed cost independent of length is attached to the beginning of the sentence. The formula fitted to twelve points in Part 20 has this shape.

합성 시간 ≈ 0.475 × 오디오 길이(초) + 0.23초

VoxCPM2: 76ms per patch

According to the VoxCPM2 technical report, this model does not use discrete tokens, but rather has a structure that generates continuous latent representations through diffusion and autoregression . It reduces speech to 25 latent frames (40ms) per second, groups 4 frames into a patch, and creates patches one at a time. Therefore, one step of autoregression corresponds to 160ms of audio, which is 6.25 stepsper second. Connecting this number with the measurements yields the following.

This means that the cost of running one language model step and 10 diffusion stepsfor each step is 76ms, and in this structure, it can be expected that reducing the number of diffusion repetition steps will speed things up almost proportionally. The speed and quality when reducing the number of repetitions were not tested (unmeasured). The fixed cost of 0.23 seconds could not be found in the paper either. The real-time factor in the paper is a wall-clock value including the fixed cost, so the longer the length, the closer it gets to the slope. No material indicates whether this 0.23 seconds is sentence encoding, reference voice processing, decoder initialization, or the server wrapper, and it was not broken down (unconfirmed).

Official descriptions state that this model supports streaming generation (generate_streaming), and the technical report (Table 11) reports a real-time factor of 0.30 for a PyTorch implementation on a single high-end consumer GPU (24GB class) (peak VRAM about 8GB), and 0.13 when using a serving engine (Nano-vLLM). Our measured slope of 0.475 is about 1.6 times slower than the 0.30 in the paper, but it could not be determined whether this was due to the GPU type, the server method of synthesizing and returning entire sentences, or wrapper overhead (unconfirmed). VRAM (6,600 to 6,900 MiB) is in the same range as the paper's roughly 8GB.

Magpie TTS: Two frames at a time

Magpie TTS Multilingual is an encoder-decoder transformer with 364 million parameters. The official description has two core points: Frame stackingis a method where the decoder predicts two audio frames at once in a single step, cutting the number of repetitions in half, and local transformersare small modules that refine the dependencies between the frames created this way to recover the reduced quality. This essentially halves the number of decoder steps required to produce the same length of sound.

• Even though it has over 5 times fewer parameters and 9 times less memory, the speed is similar. This aligns with the interpretation that what determines time is not the number of parameters, but 「number of steps to make x cost per step」. However, the two models were measured on different GPUs, and because Magpie's frame ratio could not be verified, the time per frame could not be calculated like VoxCPM2 (unconfirmed).
• Official Time-To-First-Token figures (such as 47ms on H100) are based on streaming. The method used in the measurements synthesized and received the entire sentence, so they cannot be compared with the same numbers.
• Korean was not generated normally in this environment(Part 20, 3-2). It was not confirmed whether this was a structural issue or a configuration issue.

Why streaming synthesis reduces the time to first sound

Because patches are generated from the beginning, there is no need to wait until the end of the sentence; playback can begin as soon as the first patch comes out (the VoxCPM2 report explains that it is structured to convert generated patches into sound immediately). The time to the first sound is reduced from the 「total sentence synthesis time」 (Part 20 measurement: 1.2 to 2.1 seconds) to the 「fixed cost per sentence plus the time of the first single patch」 (0.23 seconds + 0.08 seconds, about 0.3 seconds). This is a calculation and was not tested on the server used for measurements. Since playback is slower than synthesis (real-time factor 0.5), the condition that subsequent audio does not stutter is also required.

4. Streaming Speech Recognition: Do not recompute what has already been computed

Principle

The speech recognition model Nemotron 3.5 ASR consists of 24 FastConformer encoder layers and an RNNT decoder. Two things create latency.

1. Frame length. The encoder reduces the sound by a factor of 8, making a single encoder frame 80ms (based on the general setting of extracting features at 10ms intervals, and the chunk value of 80ms in the model card matches this interpretation).
2. Right context (amount of look-ahead). To determine the meaning of the current frame, we want to look a little bit into future frames. Since we must wait until those frames arrive, the amount of look-ahead is precisely the latency.

The model determines these two with att_context_size = [왼쪽, 오른쪽] . The 56 left frames are the amount remembering 4.48 seconds of the past (56 x 80ms), and the right side determines the chunk. The length of a chunk processed at once is (right + 1) x 80ms.

Since all five values match the table in the card, it can be considered correct to interpret that 「chunk = (right context + 1) x 80ms」.

What the cache does

Older streaming methods (buffer methods) cut recent audio into overlapping windows and re-encoded from the beginning every time. This essentially computes overlapping sections multiple times. The cache-aware method stores the encoder representations of all self-attention layers and convolution layers in the cache, and when new audio arrives, it only computes the new frames to update the cache. As official explanations state, each frame is processed exactly once. As a result, the amount of computation is proportional to the newly incoming length, not the audio length. Throughput figures are vendor specs: the multilingual 3.5 model card lists about 240 concurrent streams for an 80ms chunk setting on a single H100 (baseline Parakeet 1.1B using the buffer method is about 14), and about 2,400 streams for the 1.12-second setting (baseline about 400), while the NVIDIA blog for the English-only model lists 560 streams at a 320ms chunk (these are values for two different models and should not be mixed up). 「Median 24ms until final results」 is a partner pipeline value from the English model blog. It was not reproduced in the author's experimental environment.

Because the RNNT decoder is structured to output a character (or a blank token meaning 「none yet」) every time an encoder frame is produced, characters are outputted sequentially as frames come in. Therefore, results stream out before speech ends, and once speech ends, only the final chunk needs to be finalized.

Does increasing chunk size improve accuracy?

Generally, the longer the right context, the more future the model sees to make decisions, which typically increases accuracy while also increasing latency. Applying chunk-size settings to the same 12 voices and measuring the average character error rate yields the following (this is an approximation applying settings to the entire file, not actual streaming).

• Going from 80ms to 320ms, it improves slightly from 4.3% to 3.9%, and stays the same afterward. This means that for these 12 synthetic audio samples, a context of 320ms or more was no longer helpful. Even when changing to 80-160ms, which minimizes latency, the accuracy loss was 0.4 to 0.3 percentage points. The fact that they are clean synthetic voices may have made these results overly optimistic.
• The 11.3% at 1,120ms could not be explained. It was due to a single audio sample resulting in an empty output, and it remained the same when measured twice. The Korean FLEURS character error rate in the model card is actually lowest at the 1.12-second setting (7.12%), so our values go in the opposite direction of the card. When re-measured with 30 human-read Korean FLEURS sentences, there were 0 empty results regardless of chunk size, and the character error rate was flat at 5.6 to 6.2% (Part 20, Section 4-3), so this phenomenon appears to be specific to the 12 synthetic audio samples rather than a general property of the model. The cause was not broken down.

5. Putting Them Together: The Type of Bottleneck Varies by Component

This is why synthesis accounted for 60-85% of the calculation time in a single turn of part 20. Speech recognition is lightweight computation, and LLMs are already utilizing 66% of the bandwidth upper limit with little room to squeeze further, leaving only synthesis 「repeating as much as the audio length」 structure, causing time to increase linearly as sentences get longer.

6. Potential Interview Questions

• How is the upper limit of LLM token generation speed estimated? Divide the memory bandwidth by the weight size in bytes. For a 4-bit 12B model, the measured bandwidth was 905 GB/s ÷ 7.0 GB ≈ 129 tok/s, and the actual measured speed was 85.6 tok/s (66%).
• Why is input processing much faster than generation in the same model? Input processing reads weights once and reuses them across multiple tokens, making it compute-bound. Generation reads weights again for every token, making it bandwidth-bound (8,730 vs. 69 tok/s in the 1.7B model).
• How much slower does a single request become when concurrency is increased to 16 requests? Across four small models, it slowed down by only 8-14% while total throughput increased by about 14-15x. This is because the cost of reading weights is amortized.
• How do you check if measured values are suspicious? Compare them against hardware limits. An input processing speed of 91,000 tok/s exceeds the compute capacity GPUs can achieve, which revealed that prefix caching had hit.
• Why does quantization also help with speed? When weight bytes decrease, the bandwidth upper limit increases proportionally. 4-bit is about 3.5x faster than 16-bit (calculated). When switching a small model from 16-bit to 8-bit, the theoretical computation speedup was 1.7x, but the actual measured speedup was 1.54x, with gains diminishing due to per-token fixed costs.
• How does cache-aware streaming speech recognition differ from the buffer-based approach? Instead of re-encoding overlapping windows, layer-wise representations are kept in a cache so each frame is computed only once. The chunk size is (right context + 1) × 80ms.
• How can the initial audio latency (time-to-first-audio) be reduced in autoregressive text-to-speech? Play back the first few frames immediately as they are streamed. Playback must be slower than synthesis (real-time factor < 1) to avoid stuttering.

External Evidence: Verification via Papers and Official Materials

The numerical values in this article are directly measured, and the underlying principles were cross-checked against the materials below. External figures were not directly compared due to differing models, GPUs, and workloads; only structural and directional alignmentwas evaluated.

Vendor figures (such as H100 concurrent streams in model cards) were cited only for directional trends because conditions were only partially specified on the cards. PNAS and certain model cards had restricted access, so we read alternative copies or papers instead, and omitted figures from materials we could not open.

Unverified Items

• The cause of the discrepancy between 66% and 86-98%. Bandwidth was measured directly on both GPUs, but the 24GB class used llama.cpp with a 4-bit model, while the 8GB class used vLLM with bf16/fp8 models, resulting in different engines and formats. Measurements using the same model and both engines on the same GPU are needed to isolate this.
• Where the remaining 34% of 66% goes. KV cache reading, attention, and kernel gaps were not broken down.
• The breakdown of VoxCPM2's 76ms per frame, the effect of reducing diffusion iteration steps, and the exact nature of the 0.23-second fixed cost.
• Time-to-first-audio in streaming synthesis and actual confirmed latency in streaming speech recognition. The streaming path was not used during measurements.
• The 16-bit speed of the 12B model (37 tok/s is calculated). Precision effects were smaller than calculated in smaller models.
• Regimes exceeding 16 concurrent requests. The point where bottlenecks shift to computation (approx. 130 requests for bf16, approx. 64 for fp8) is theoretical and was not measured.
• The nature of the short-input fixed cost in input processing. The reason why speed drops at 256 tokens (request processing, kernel startup overhead, etc.) was not decomposed.
• Comparison with other inference engines. Direct verification was performed using only vLLM, and the engine was not matched with the 12B measurements above (which used llama.cpp).
• Descriptions of model architectures followed official documentation without reading the original research papers in full.

References

• Nemotron Speech ASR Cache-Aware Streaming Explanation: https://huggingface.co/blog/nvidia/nemotron-speech-asr-scaling-voice-agents
• Nemotron 3.5 ASR Model Card: https://huggingface.co/nvidia/nemotron-3.5-asr-streaming-0.6b
• VoxCPM: https://github.com/OpenBMB/VoxCPM
• Magpie TTS Explanation: https://huggingface.co/blog/nvidia/magpie-tts-multilingual-voice-agents
• Parts 20 (measured), 8 (KV cache size calculation), and 4 (inside vLLM) of this series

Verified Versions and Dates

As of 2026-10-06. llama.cpp b11429, Gemma 4 12B QAT q4_0, vLLM 0.31.0 (bandwidth, concurrency, and input processing verified using Qwen3 0.6B, 1.7B, and 4B-Instruct-2507 FP8), VoxCPM2 (voxcpm 2.0.3), Nemotron 3.5 ASR Streaming 0.6B, NeMo 3.0.0.