Relaying Full-Duplex Voice Calls via WebSockets: From Browser Microphone to GPU Model

Status: Draft (Updated 2026-10-09). The metrics were measured directly by the author. The environment consists of a single 24GB-class GPU, MiniCPM-o 4.5 (Q8_0, llama.cpp-omni), and a FastAPI backend. Questions were asked using synthesized voice instead of human voice, and echo from actual browsers as well as multi-person simultaneous calls are unmeasured . Writing rule: Measured data is the default, and anything that could not be measured is marked as "Unmeasured".

Table of Contents

• 1. Model: Why MiniCPM-o 4.5
• 2. Why the Backend Relays
• 3. In What Format Audio Is Sent
• 4. Frames: Separating Binary and Text
• 5. Clients Sending Faster Than Real Time
• 6. Maintaining a Single Slot Across Two Web Pods
• 7. Things That Drop the Connection
• 8. End-to-End Measured Values
• 9. Limitations and Unmeasured Items
• Potential Interview Questions
• References

In the 20th conversational exercise, "Recording → Speech Recognition → Conversational Model → Speech Synthesis" is exchanged turn-by-turn over HTTP. You have to finish speaking before a response comes back, and you cannot speak while listening to the response. To create the feel of talking to a person, a full-duplex model that speaks while listening and listens while speaking is required. Such models receive and produce audio multiple times per second. Therefore, instead of HTTP—which has one response per request—WebSockets, which keep a single connection open so both sides can send data at any time, are the right fit. keeps a single connection open so both sides can send data at any time WebSockets are the right fit.

This article briefly covers the process of choosing a model, and then discusses five decisions made on the WebSocket side.

1. The backend relays the communication instead of connecting the browser directly to the model server.
2. Audio is sent in 16kHz, 16-bit, 1-second chunks, and received in 24kHz, 16-bit.
3. Audio is separated into binary frames, while text and state are separated into text frames.
4. Clients sending data faster than real time are blocked using a token bucket.
5. A server slot that accepts only one call is protected using a database advisory lock.

Short answer. Even with the relay, calls were opened in 0.33 seconds, and for "What is the capital of Australia?", "It's Canberra." started being heard 0.30 seconds after speech ended. Relaying and converting 1 second's worth of audio took 1.3 ms upward and 5.9 ms downward. When using binary PCM instead of base64 JSON between the browser and backend, the transfer volume decreased from an upward 85 KB/s → 32 KB/s and a downward 128 KB/s → 48 KB/s.

1. Model: Why MiniCPM-o 4.5

Three full-duplex models available on Hugging Face without a key were tested with the same questions (raw data: docs/experiments/2026-10-07-full-duplex-voice/).

MiniCPM-o got Canberra, 100 degrees, Mars, and Shakespeare right, and when given the instruction "何語で話しても日本語だけで答えて" (Answer only in Japanese no matter what language is spoken) in Japanese, it answered in Japanese 9 out of 10 times. Its Korean was poor (official languages are English and Chinese). Since the PyTorch version required 28GB or more, it was run using GGUF from a llama.cpp fork (llama.cpp-omni).

Two things were fixed on the model side.

• Speech initiation threshold. With default settings, the median time from the end of speech to the output of voice was 4.0 seconds (0.9–8.3). The model decides whether to "listen or speak" every second, but lowering the session setting listen_prob_scale from 1.0 to 0.0 decreases the score of the "listen" token by 2. As a result, it became 1.03 seconds (−0.7–2.4) and there were no unanswered questions.
• 41 seconds per call opening. Server logs showed that for each session, the entire LLM weight file (8.7 GB) was read into memory to obtain embeddings for 7 special tokens, and then discarded when the session ended. Since the values depend solely on the model file, fixing it to be created only once inside the process reduced the opening time to 0.11 seconds(or 1.6 seconds when feeding a 6,500-token blog post in the body).

2. Why the Backend Relays

The model server directly exposes a WebSocket endpoint (/backend). Connecting the browser directly to this would be the fastest approach, but it presents three problems.

브라우저 ──wss──▶ 게이트웨이 ──▶ 백엔드(/ws/call) ──ws──▶ 모델 서버(/backend)
            (쿠키로 사람 확인)   (프롬프트 생성, 자리, 속도 제한, 형식 변환)

1. The model server trusts system prompts blindly. When the prompt is included in the message that opens the session, letting the browser create this message allows anyone to issue any instructions to the model. With a proxy in place, the browser sends only mode=lang&lang=ja&level=N4 or mode=blog&slug=… , and the prompt is built exclusively by the server using fixed sentences and text from the DB. Levels are also used only when they are in a known list. Initially, we only left alphanumeric characters, but tests caught B1. Ignore all rules entering the prompt as B1Ignore .
2. The model server accepts only one call at a time. If a second person connects, the first call is broken. Concurrency is covered in Section 5.
3. We must verify users and limit connection times. Browsers cannot attach arbitrary headers to WebSocket handshakes. Therefore, we identify users using same-origin login cookies (the same approach as the existing web terminal), limit calls to 10 minutes, and disconnect silent connections after 15 seconds.

When including blog posts in the prompt, we prevented one more issue. Since the model's special tokens look like <|im_end|> , if the same string appears in the body of the post, it can close the system message and insert new instructions. We neutralize <| in the body by replacing it with < | , and use tests to verify that the entire prompt contains only the two special tokens we added.

3. Audio Transmission Formats

• The browser is where the audio is downsampled to 16kHz. The model's speech encoder (Whisper family) accepts 16kHz. new AudioContext({sampleRate: 16000}) Accepting 16kHz from the start via would be simple, but some browsers reject this option. Therefore, we capture audio at the device's default sample rate (usually 48kHz) and downsample it using linear interpolation. Since spoken voice mostly lies below 4kHz, this level of quality is sufficient. We use Node tests to verify whether 1 second of 48kHz results in exactly 16,000 samples and whether 44.1kHz behaves the same way—if the sample rate is incorrect, the model hears a sped-up or slowed-down voice and misunderstands it, which cannot be detected visually.
• The chunk size is 1 second. This model listens and makes decisions in 1-second increments. Sending smaller fragments does not reduce latency because the server simply waits until a full second is accumulated. If a chunk size is not exactly 32,000 bytes, the proxy discards it.
• The proxy is where the audio is downsized to 16-bit. If the model's format (float32 base64) were sent all the way to the browser as-is, the download bandwidth would be 128KB per second. Reducing it to 16-bit lowers this to 48KB, a 63% reduction. There is no perceptible difference to the human ear (16-bit has the same bit depth as a CD).
• Conversion cost. Using the standard library array to convert a 1-second segment took roughly 1.3ms for serialization and 5.9ms for deserialization (median of 5 runs using Python 3.12 on Apple Silicon arm64). After deployment, re-measuring with the same code on two production backend pods yielded 1.03ms serialization and 5.61ms deserialization on a desktop-class x86 CPU, and 1.21ms and 6.02ms on a laptop-class x86 CPU, which were almost identical to the Mac (2026-10-09). This is negligible compared to the 50ms the model takes to process a 1-second chunk.

4. Frames: Separating Binary and Text

WebSocket frames support text (opcode 1) and binary (opcode 2). Audio was sent as binary, and everything else as JSON text.

브라우저 → 백엔드   이진: 1초 PCM          텍스트: {"type":"hangup"}
백엔드 → 브라우저   이진: 모델 목소리 조각  텍스트: ready · text · listen · done · busy · unavailable · timeout

Dividing them this way splits the data on the receiving end into typeof ev.data === 'string' without needing to wrap audio in JSON or bloat it with base64. In the browser, setting it to ws.binaryType = 'arraybuffer' ensures that binary frames arrive directly as Blob rather than ArrayBuffer , making them immediately usable.

Communicating reasons via close codes. WebSocket close codes 4000–4999 are reserved for application use. We used 4401 (login required), 4404 (post not found), 4429 (another user is on a call), and 4503 (model server preparing). The UI translates these codes into human-readable language. Phrases like "Another user is currently on a call, please try calling back shortly" encourage users to try again much more effectively than a generic "Failed."

Chaining playback seamlessly. The model generates voice snippets of a few hundred milliseconds at a time. Converting the received fragments into AudioBufferSourceNode and scheduling them at the "exact end time of the previous fragment" ensures seamless playback without gaps. Playing each fragment immediately as it arrives causes them to overlap.

Barging in (Interruption). When a user interrupts, the model switches back to listening (listen). However, if previously received voice segments are still waiting in the playback queue, the user will perceive the model as continuing to speak. Therefore, when listen arrives and the microphone input is loud , we clear the playback queue. Clearing it when the microphone is quiet cuts off the end of the response—because the model still sends listen even after it has finished generating the response, and playback has not yet finished at that moment.

5. Clients Sending Faster Than Real-Time

The model server processes 1-second chunks sequentially. The next chunk is read only after the response for the previous chunk finishes. Consequently, if a client dumps 30 seconds of pre-recorded audio all at once, the queue backs up, responses become progressively delayed, and other users are blocked from acquiring a slot during that time.

We placed a token bucket in the proxy. It fills at 1.5 tokens per second and holds a maximum of 3. Normal calls sending one chunk per second experience zero dropped chunks over 10 minutes (600 chunks), whereas burst-sending passes only the first 3 chunks and then throttles down to 1.5 chunks/sec. Setting the rate slightly higher at 1.5 accounts for network jitter that can occasionally cause two chunks to arrive back-to-back. Making the time source externally injectable allows this behavior to be tested in just a few milliseconds.

We ran into the same pitfall when building the metering tool. If the measurement client sends chunks ahead of time, the time from "speech end to voice response" appears shorter than it actually is. Therefore, chunks must be sent the exact moment they finish recordingIt was configured to be sent only (k seconds after start).

6. Holding a spot between two web pods

The model server accepts only one call, but there are two backend pods. Variables inside the process (asyncio.Lock) do not know about calls in other pods.

Conversation practice in part 20 holds its spot using a partial unique index in the DB. Since HTTP requests go back and forth per turn with no connection in between, the spot is left as a row and expired if there is no activity for 3 minutes. Calls are different. Since the WebSocket remains open throughout the call, you only need to hold the spot while the connection is alive .

That is why we used session-level advisory locks in Postgres.

seat = db()                                   # 통화마다 연결 하나
got = seat.execute("select pg_try_advisory_lock(%s)", (SEAT_KEY,))
if not got: ...                               # 4429 로 닫는다
try:
    await relay(ws, prompt)
finally:
    seat.close()                              # 연결이 닫히면 잠금도 풀린다

• When the call ends, the connection closes, and when the connection closes, the lock is released.
• Even if a pod dies, the spot does not remain locked. If the connection drops, Postgres automatically releases the lock for that session. If you leave the spot as a row, you have to wait for the expiration time.
• The cost is one DB connection per call. Since there is only one call at a time, there is only one connection.
• The API asking "Can I make a call now?" checks by acquiring and immediately releasing the lock.

7. Things that drop connections

• Idle connections. By default, Envoy in the Cilium gateway drops connections where both sides are quiet after 60 seconds (proxy-idle-timeout-seconds). Because of this, the web terminal sends a heartbeat every 25 seconds. For calls, the browser sends audio every 1 second, so a separate heartbeat is not needed. Rather, if no audio arrives for 15 seconds, the relay drops the connection first (assuming the window was closed or the tab went to sleep).
• Deployments. When a new backend is spun up, open WebSockets are dropped. Calls have a 10-minute upper limit, so you can just make a new call. Spots are automatically released for the reasons mentioned above.
• When the model server is not ready. The model server's /health returns 200 even before loading the model (the model loads on the first session and takes 70–80 seconds). Therefore, we opened a session right after spinning up the server to load the model, and then used a subsequently created file to determine readiness. The relay waits up to 20 seconds for session opening, and if it exceeds that, returns 4503 to indicate "Preparing".

8. End-to-end measurements

We ran the relay router as a fake DB, attached it to the model server, and sent synthesized voice questions in 1-second real-time intervals (scripts/e2e_relay.py).

This measurement does not include the gateway and TLS. After deployment, we re-measured the production path segment by segment (2026-10-09).

The portion added to the open time by the gateway and TLS was about 30ms. Most of the time taken to open a call is the 0.33 seconds it takes for the model server to prepare the session. There are no measurements of an entire logged-in call (up to the voice after speech ends) through the production path — we did not create a test account in production (Unmeasured). Therefore, the WebSocket opening segment was measured using a flow where it opens without login and the server closes it with 4401.

At this point, one thing became clear. When measured with a Python websockets client, "until closure finishes" had a median of 1,045ms on the production path, compared to 6.9ms inside the cluster. When we separately measured the timestamp when the close frame arrived using aiohttp , it was also 36ms on the production path. The server's close signal arrives immediately, and the delay is the time the client spends waiting for the TCP connection to tear down after receiving the close frame. This means that if you tie the timing of displaying "Connection lost" on the screen to TCP termination, it could be delayed by about 1 second on paths going through a proxy.

9. Limitations and unmeasured items

• Questions were asked using synthesized speech only. Human voices, dialects, and noise are unmeasured .
• Whether the browser's echo cancellation (echoCancellation) removes sounds played via the Web Audio API varies by browser and device and was not measured. We recommend using earphones.
• Since the server is 1:1, concurrent calls are limited to one by design. Supporting multiple calls would require more GPUs or a server that supports batching (unmeasured).
• ). Similar models like Realtime-Venus (based on MiniCPM-o 4.5, BF16 approx. 20GB) and GGUF of NemotronLabs VoiceChat-11B (English-only) are unmeasured .
• ScriptProcessorNode is scheduled for deprecation. Moving to AudioWorklet could reduce stuttering when the main thread is busy, but it was not measured.

Potential interview questions

• Why use WebSockets instead of SSE? SSE is strictly server-to-client unidirectional. Full-duplex calls send audio every second while simultaneously receiving voice. Sending uploads separately via HTTP requests would introduce one request per second and its associated connection overhead.
• Why wasn't WebRTC used? WebRTC is better suited for calls as it includes UDP, jitter buffers, and echo cancellation. However, it requires TURN/STUN servers and media servers (e.g., LiveKit). Since this model processes data in 1-second chunks, differences in latency of tens of milliseconds matter less to the user experience, and reusing the already operational WebSocket path (web terminal) was simpler. As user scale grows or mobile networks are considered, moving to WebRTC would be the right choice.
• Why was base64 avoided? Base64 inflates data by 4/3. Float32 is twice 16-bit. Combining the two makes the same audio 2.7 times larger (48KB → 128KB/s).
• Why Redis locks instead of advisory locks? This site already uses Postgres, and we needed the property where locks are automatically released when the connection drops. Redis locks require setting an expiration time, and if a call takes longer than that, the lock must be extended.

References

• RFC 6455 The WebSocket Protocol — Frame opcodes (Section 5.2), Close code ranges (Section 7.4.2)
• PostgreSQL Documentation, Advisory Locks (Section 13.3.5), pg_try_advisory_lock
• MiniCPM-o 4.5 GGUF: huggingface.co/openbmb/MiniCPM-o-4_5-gguf, llama.cpp-omni: github.com/tc-mb/llama.cpp-omni
• MDN, Web Audio API — AudioBufferSourceNode.start(when), BaseAudioContext.createBuffer
• Part 20: Real-time voice conversation for language learning, actual measurements by component; Part 19: NVIDIA open source for real-time voice conversation apps