NVIDIA Open Source for Building Real-Time Voice Chat Apps: Nemotron Voice Agent, PersonaPlex, NeMo

Status: Draft (Updated 2026-10-07), research article. The contents of this article are based on documentation; the results of direct execution are only from Part 20(Magpie Korean and NeMo code tokenizer mapping). The facts below were compiled by reading the GitHub repositories and Hugging Face model cards of each project. Since they were read through page summarization tools, please re-verify the original text for numbers and license phrases before writingwith them. Please read "Unverified Items" at the end of the post first. Writing rule: Measured data is the default, and unmeasured items are marked as "Unmeasured".

Table of Contents

• 1. Conclusion First
• 2. Two Structures
  • 2.1 Cascade: Separating Listening, Thinking, and Speaking Stages
  • 2.2 Full-Duplex: A Single Model Listens and Speaks Simultaneously
• 3. Summary by Candidate
  • 3.1 Nemotron Voice Agent Blueprint
  • 3.2 Nemotron 3.5 ASR Streaming 0.6B (Speech Recognition)
  • 3.3 Magpie TTS Multilingual (Text-to-Speech)
  • 3.4 PersonaPlex-7B
  • 3.5 NeMo voice_agent Example
• 4. Feasibility on Consumer GPUs
• 5. App Design Draft (Untested)
• 6. Measurement Plan
• 7. Potential Interview Questions
• 8. Common Misconceptions
• Unverified Items
• References
• Verified Versions and Dates

1. Conclusion First

If you are building an app for real-time conversation in Korean, within the currently verified scope, the approach of chaining Automatic Speech Recognition (ASR), LLM, and Text-to-Speech (TTS)is correct. This is because NVIDIA's open models supporting Korean exist in two components of this approach.

2. Two Structures

2.1 Cascade: Separating Listening, Thinking, and Speaking Stages

The sequence is: Microphone → Voice Activity Detection (VAD) and speech end judgment → ASR(speech to text) → LLM(generate response text) → TTS(text to speech) → Speaker.

• Pros: Components can be swapped independently. You can replace only the components weak in Korean with other ones, and you can continue using your existing deployed LLM. Intermediate results (recognized text, response text) can be logged for easy debugging.
• Cons: Latency accumulates at each step. You must directly handle the logic for determining whether speech has ended and the logic for stopping playback when the user interrupts (barge-in).

2.2 Full-Duplex: A Single Model Listens and Speaks Simultaneously

PersonaPlex uses this approach. Because a single model receives continuous audio as input while simultaneously outputting voice, interruptions, backchannels, and rapid turn-taking feel natural. The latency reported by the PersonaPlex paper (arXiv 2602.06053, Table 6) as Full-Duplex-Bench is about 0.170 seconds for turn-taking and about 0.240 seconds for response when a user interrupts (these are paper figures for the English model; the model card could not be opened because access approval is required).

• Cons: English-only, components cannot be swapped individually, and it is difficult to connect response content to the model's own LLM or knowledge base.

3. Summary by Candidate

3.1 Nemotron Voice Agent Blueprint

NVIDIA-AI-Blueprints/nemotron-voice-agentis a reference implementation released by NVIDIA.

• Configuration: ASR is chosen from Nemotron ASR Streaming, Parakeet CTC, or Parakeet RNNT Multilingual; LLM from Nemotron 3.5 Lightning 30B, Nemotron 3 Super 120B, or Nemotron 3 Nano Omni; TTS from Magpie TTS Multilingual or Chatterbox TTS Multilingual. The backbone is the open-source Pipecat, accelerated by NVIDIA NIM microservices.
• Deployment Profiles: Cloud (CPU only without GPU, using NVIDIA API endpoints), Server (NVIDIA GPU workstation, default approx. 80GB VRAM), Single GPU (one workstation GPU, DGX Spark, or Jetson Thor, available VRAM 28GiB or more). Launched via Docker Compose v2.20 or later.
• Recent Releases: v2.2.0 (2026-09-23) added self-hosted Nemotron 3.5 Lightning, single-GPU recipes, and OpenAI Realtime-compatible WebSocket gateway. v2.0.0 (2026-07-09) integrated the React client.
• License: Source code is Apache-2.0, documentation and assets are CC-BY-4.0.
• Latency: Advertises "sub-second end-to-end latency" and multiple concurrent streams. These figures are claims from the documentation, and measurement conditions could not be verified.
• Korean: The blueprint's docs/how-to/configure-tts.md lists Korean (ko-KR) among the supported languages of Magpie TTS Multilingual. Conversely, the LLM embedded in the blueprint (Nemotron 3 Super) does not list Korean among its supported languages (English, German, Spanish, French, Italian, Japanese, Chinese). Therefore, a separate LLM must be chosen for Korean conversations. Since the two components below support Korean, a configuration is possible if you select Korean-supported models for those parts (this article confirms documentation; actual operation was tested in Part 20).

3.2 Nemotron 3.5 ASR Streaming 0.6B (Speech Recognition)

Model Carddetails are as follows.

• It has 600 million parameters and supports 40 language locales. Among them, 19 "transcription-ready" locales include Korean (ko-KR) , grouped alongside Japanese, English, Spanish, German, etc.
• Streaming chunk sizes can be chosen from 80, 160, 320, 560, and 1120ms (att_context_size). Smaller chunks mean faster transcription and lower accuracy—a trade-off balance (general principle, not verified with this model).
• Word error rates listed on the card are Spanish 4.11%, English 7.91%, German 8.31%, and French 9.03% (based on 1.12s chunks). Korean is listed in Character Error Rate (CER) , showing 7.59, 7.70, 7.27, 7.18, and 7.12% across 80ms to 1.12s chunks with language specified in FLEURS evaluation, and 8.31 down to 7.30% with automatic language detection.
• License is denoted as openmdw-1.1 .
• Pipecat-related posts (Daily) note that this model's "time to final segment" is under about 250ms under real usage conditions, and mention that a C++ server for benchmarking and a Python server that automatically switches between multilingual and English models have been released.

3.3 Magpie TTS Multilingual (Text-to-Speech)

Model Carddetails are as follows.

• It supports 12 languages and includes Korean (ko) (Arabic, Chinese, English, French, German, Hindi, Italian, Japanese, Korean, Portuguese, Spanish, Vietnamese). Korean was added in checkpoint v2607 (released 2026-07-21) , whereas previous checkpoints (v2602, v2512) did not have Korean. The card lists Korean CER as 2.69% and speaker similarity as 0.807, with evaluation data noted by the card as internal data. Korean functionality was not split solely by checkpoint version. When testing all three checkpoints in Part 20, NeMo 3.0.0 did not connect Korean to the model's korean_chartokenizer (lacking a Korean entry inLANGUAGE_TOKENIZER_MAP and falling back to the English phoneme tokenizer), so even in v2607, audio output terminated in 0.4 to 0.8 seconds. Explicitly specifying that connection resulted in normal synthesis. In other words, the model card's Korean support matches the checkpoints, but the NeMo code version being used must wire it up.
• Parameters are listed as 364M on the card (named 357m), with a 569MB GGUF quantized version available. The license is the NVIDIA Open Model License.
• The card's open weights provide 5 voices (Aria, Jason, Leo, Sofia, John Van Stan), and NIM's Korean voice list (verified via search results only) differs from this. Depending on which one you use, the Korean voice may vary.
• Streaming latency figures were not found on the card. Standard mode supports up to 20 seconds of content, while long-text mode supports longer text.

3.4 PersonaPlex-7B

GitHuband Model Carddetails are as follows.

• It is a 7B full-duplex voice conversation model based on the Moshi architecture and weights. Speaking style and persona are determined through text role prompts and voice conditioning.
• It responds to English voice input exclusively with English voice.
• The code is under the MIT license, and model weights are under the NVIDIA Open Model License, requiring agreement to the license on Hugging Face to download.
• Execution is done via libopus-dev installation and pip install moshi/. followed by the web interface (localhost:8998). If VRAM is insufficient, you can use --cpu-offload , though the documentation does not specify the required VRAM.

3.5 NeMo voice_agent Example

NVIDIA-NeMo/Speech's exampleis a conversational agent combining NeMo's speech recognition and text-to-speech with a Hugging Face LLM. For speech recognition, you can choose from the Parakeet real-time model (default) and Nemotron streaming variants; for text-to-speech, from Kokoro-82M up to FastPitch-HiFiGAN. It requires a single GPU and recommends 21GB of VRAM for a 9B LLM or 13GB for a 4B LLM. It states that it currently supports English input and output only.

4. Feasibility on Consumer GPUs

The author's experimental environment uses an 8GB-class laptop GPU and a 24GB-class consumer GPU. The blueprint's single-GPU profile requires 28GiB or more of VRAM , so neither GPU alone can run it as-is. However, since this requirement corresponds to the default setup that also includes a 30B-class LLM, splitting the components may change the situation.

The two rows for ASR and TTS in the table have been updated with values measured by directly loading them onto an 8GB-class GPU in part 20. The VRAM usage when including the LLM and the total latency linking all three components have still not been measured (unmeasured). Since the blueprint is said to provide an OpenAI Realtime-compatible WebSocket gateway, it is also worth checking the method of connecting an existing LLM server (untested).

5. Draft App Design (Untested)

브라우저(마이크, 스피커)
   │  WebSocket 또는 WebRTC
   ▼
Pipecat 파이프라인
   ├─ VAD + 발화 종료 판단
   ├─ ASR : Nemotron 3.5 ASR Streaming (ko-KR)
   ├─ LLM : vLLM 으로 서빙하는 소형 모델
   └─ TTS : Magpie TTS (ko)
   │  끼어들기가 감지되면 TTS 재생과 LLM 생성을 중단
   ▼
스피커

• Latency breakdown. The time from when the user finishes speaking until the first sound plays is the sum of "speech end detection + final recognition + LLM first token + TTS first audio." Each stage must be measured separately to identify the bottleneck.
• Barge-in. When the user starts speaking, both the currently playing audio and the ongoing LLM generation must be canceled together.
• Privacy. Since voice is personal data, policies on whether to store it and how long to retain it must be determined first. It is safer to keep only the recognized text in the logs and not the raw audio.
• Concurrency. As users increase, ASR and TTS will share the GPU, so latency per concurrent user count must be measured separately (unmeasured).

6. Measurement Plan

The sequence for turning the content summarized in writing into numbers. Conducted using temporary pods on an 8GB-class GPU.

1. ASR standalone: Real-time factor (processing time vs. audio length) with Korean voice samples, latency by chunk size, and VRAM usage.
2. TTS standalone: Time to first audio and VRAM usage with Korean sentences.
3. End-to-end latency linking the three components and barge-in response time.
4. Korean recognition accuracy: Although the card provides the Korean CER (FLEURS), it needs to be re-verified using voice data from our environment. This will be checked using a custom small-scale test sentence set, noting the limitation of a small sample size.

7. Potential Interview Questions

• What is the difference between cascade and full duplex? Section 2 above. The balance between component swapping/debugging versus natural barge-in.
• How do you break down latency in real-time voice conversations? It is divided into speech end detection, final recognition, LLM first token, and TTS first audio, measuring each separately.
• How is barge-in handled? When user speech is detected, playback and generation are canceled simultaneously.
• What does the chunk size in streaming ASR affect? It trades off latency against accuracy.

8. Common Misconceptions

• "If it's an NVIDIA conversational model, Korean is supported": PersonaPlex and NeMo examples only support English.
• "Spawning the blueprint immediately makes it a Korean app": You must manually select and configure components that support Korean, and this configuration has not been tested yet.
• "It is open source, so it is commercially free": The licenses for the code (Apache-2.0, MIT) and the model weights (NVIDIA Open Model License, openmdw-1.1) differ. You must read each original text before use.

Unverified Items

• Because it was read through a tool that summarizes page content, the numbers and license phrases may differ from the originals. The full text of each license has not been opened.
• Korean TTS quality and the Korean recognition error rate on our voice data rather than the card's (the card has CER based on FLEURS).
• Whether the blueprint officially supports a Korean configuration out of the box (the TTS documentation mentions Korean, but the built-in LLM does not support Korean).
• All latency and VRAM figures. None of the models were run in the author's experimental environment.
• The difference between Magpie TTS open weights and NIM Korean voices (verified only via search results).
• NVIDIA speech projects other than the blueprint (NVIDIA/voice-agent-examples, NVIDIA-NeMo/labs-Voice-Agent) were only identified by name without reading their contents.

References

• https://github.com/NVIDIA-AI-Blueprints/nemotron-voice-agent
• https://github.com/NVIDIA-AI-Blueprints/nemotron-voice-agent/releases
• https://github.com/NVIDIA/personaplex
• https://huggingface.co/nvidia/personaplex-7b-v1
• https://huggingface.co/nvidia/nemotron-3.5-asr-streaming-0.6b
• https://huggingface.co/nvidia/magpie_tts_multilingual_357m
• https://github.com/NVIDIA-NeMo/Speech/tree/main/examples/voice_agent
• https://www.daily.co/blog/open-source-multilingual-transcription-from-nvidia-nemotron-3-5-asr/

Checked Version and Date

As of 2026-10-05. Nemotron Voice Agent Blueprint v2.2.0 (2026-09-23), PersonaPlex-7B-v1, Nemotron 3.5 ASR Streaming 0.6B, Magpie TTS Multilingual 357m.