The Microphone the Server Did Not Hear: Bringing the Gemini Live API WebSocket Example Up to Date

Result: Merged (2026-10-27 18:04, google-gemini/cookbook#1433). ) Actually, this was taken from GitHub PRs, reviews, line comments, comments, timelines, inspection logs, Cookbook source (the PR's base commit d9df60de), Live API references, drafts of contribution logs and reply drafts, and the voice relay code in the TT Lab repository. Times are in Korea Standard Time (KST).

Table of Contents

• One-Line Summary
• 1. What Was the Problem: Where the Example and Reference Diverged
• 2. Cause: Which Line of the Example?
• 3. What Was Changed: Two Commits
• 4. Reaction: "A Larger Modification Than It Looks"
• 5. What Was Learned: Tips for Open Source Contributions
• Links

One-Line Summary

1. What Was the Problem: Where the Example and Reference Diverged

TT Lab's voice conversation feature is a relay where the server connects the browser and the Gemini Live API via WebSocket (TT Lab repository backend/app/lang_call.py). Without using the SDK and handling the WebSocket protocol directly, microphone audio is sent as realtimeInput.audio to audio/pcm;rate=16000 , and when the server sends serverContent.interrupted , the playback queued in the browser is cleared, but it is not cleared for turnComplete . The comments in that code record that on 2026-10-09, four questions were sent one by one, answered correctly in four languages, and the first audio after speech ended was 0.88–1.48 seconds. A note is also included that speed comparisons are unmeasured because they were measured individually.

We separate what we measured from what we did not. That our relay works well with the current fields was measured as noted above. However, running this example as is media_chunks whether it is truly ignored was not measured (unmeasured). Because the mic, camera, and API key were needed, I wrote in the PR description, "I only checked the syntax with Python compile() and could not run it all the way through." Therefore, the initial PR had the weight of "cleaning up deprecated fields into current fields." The reviewer let me know in Section 4 just how big of a deal this issue was. What went wrong is detailed line by line in Section 2.

2. Root Cause: Which Line of the Example

The Gemini Cookbook has two sets of examples that do the same thing: one using the SDK, quickstarts/Get_started_LiveAPI.py and the other using WebSockets directly quickstarts/websockets/Get_started_LiveAPI.py . While reading the second one, which uses the same approach as our relay, I noticed four places that differed from the reference. The line numbers are based on the PR's base commit d9df60de .

The comment in the receiving loop stated, "If you interrupt the model, it sends an end of turn, so we must clear the queue to allow interruptions." However, the end of turn arrives on every response even without an interruption. This effectively discards unplayed audio at the end of each response. Lines 187–199 of the SDK version example lines 187–199 were already interrupted fixed to .

In short, the WebSocket version example was not updated when the protocol changed, and only the SDK version example in the same repository was fixed. The reference is mediaChunks was marked as deprecated, the WebSocket version still sent data to that field, and the interruption handling remained in the old way (clearingturnComplete after the SDK version changed). It is common for one side to be left unchanged when two sets of examples do the same thing. The draft notes in the contribution log state that the last substantive change to the WebSocket version was in 2025-11, and only the model name has changed since then.

3. What Was Changed: Two Commits

The first commit (10-17) fixed the four items mentioned in Section 2 (+24/−30).

-            msg = {"realtime_input": {"media_chunks": [frame]}}
+            msg = {"realtime_input": {"video": frame}}
 ...
-                    "media_chunks": [ { "data": ..., "mime_type": "audio/pcm" } ]
+                    "audio": {
+                        "data": base64.b64encode(data).decode(),
+                        "mime_type": f"audio/pcm;rate={SEND_SAMPLE_RATE}",
+                    }
 ...
+            # If you interrupt the model, the server sets `interrupted`.
+            if server_content.get("interrupted"):
+                while not self.audio_in_queue.empty():
+                    self.audio_in_queue.get_nowait()
+
+            # A single message can carry several parts, so play all of them.
+            for part in server_content.get("modelTurn", {}).get("parts", []):
+                if b64data := part.get("inlineData", {}).get("data"):
+                    self.audio_in_queue.put_nowait(base64.b64decode(b64data))
+
+            if server_content.get("turnComplete"):
+                print("\nEnd of turn")

Because the files used CRLF line endings, the patch preserved CRLF. One change was intentionally omitted: while this example sends the API key as ?key= in the WebSocket address, the SDK sends the key in the x-goog-api-key header over the same WebSocket. Addresses are easily left in logs and exception messages. However, since we did not re-verify whether header authentication works when using WebSockets directly and it is not documented in the reference, we did not include it in the PR and only mentioned it as a "point for separate discussion" at the end of the main text. There is no record in the log of an issue being opened for this suggestion.

The second commit (10-19) incorporated the review feedback (Section 4). We changed the two places where WebSocket frames were decoded using raw_response.decode("ascii") (the configuration response and the receiving loop) to json.loads(raw_response) . json.loads receives bytes and decodes them as UTF-8. As suggested by the bot, we changed .get("modelTurn", {}) to (server_content.get("modelTurn") or {}) so that even when the value is explicitly null , it is treated as an empty dictionary (the same applies toinlineData ). To prevent crashes in close() when the microphone stream is not opened, we also added if self.audio_stream: . The final diff shows 29 insertions and 34 deletions.

The use of AI was disclosed in a single sentence at the end of the PR body: "Prepared with the help of an AI assistant (Claude) and reviewed and verified by me." The draft notes in the contribution log mention that there are no AI policy statements in the repository's contribution guidelines. The first commit contains only the DCO sign-off and SSH signature without any AI footer, while the second commit includes Assisted-by: Claude and a DCO sign-off, but no SSH signature. This repository requires the Google CLA, and that check passed.

4. Reaction: "A Bigger Fix Than It Looks"

The request for changes was a welcome one. The reviewer noted that it was "a bigger fix than it looks" and mentioned that they had sent the same audio clip directly via WebSocket to the gemini-3.8-live model. media_chunks produced no response at all (regardless of whetherrate= was present or not), whereas realtime_input.audio returned the correct answer. In effect, the microphone was essentially dead in the main branch example. They also confirmed that realtime_input.video and interrupted handling were working. Furthermore, they requested one more fix: the server sends the answer's text (outputTranscription) along with it by default, and decoding the frames as ASCII causes non-English responses to crash. They noted this can be reproduced on main by typing "Say hello in Japanese."

We fixed it with the second commit on the same day, and replied that since the configuration response also contained the same decode("ascii") , it was fixed together. At that time, we noted that we had only verified locally that a single frame containing Japanese characters now decoded correctly, and had not tested it again against the actual server. A few hours later, we added the results of running the receiving loop itself against a mock WebSocket (no network, no audio, Python 3.12). When feeding four items: a Japanese character frame, a modelTurn: null frame, an inlineData: null chunk, and turnComplete , before the fix it stopped at the first frame with UnicodeDecodeError: 'ascii' codec can't decode byte 0xe3 , whereas after the fix it processed all four and audio chunks reached the playback queue. We reiterated here as well that we "did not run it against the actual server." These two replies remain as drafts in the log.

Another reviewer ran eight checks that could be performed without actual hardware and stated there were "no blocking issues." They reported extracting the example's class into ast , hooking up mock cameras, screens, and microphones, and verifying multiple chunks, binary and text frames, playback persisting after turnComplete , playback clearing on interrupted , and Unicode. They also noted that running the same checks on main reproduced the old issue, while adding the caveat that they did not test with actual microphones, speakers, or servers.

The final verification was done by the first reviewer. They tested thoroughly with the second commit, confirming 5-minute voice conversations (on main, the model could not hear the microphone at all), recognition when sending screens, and successful playback and "End of turn" output even when prompted with a Japanese greeting on the first message. They also pointed out that we had jointly discovered the decoding issue in the configuration response. Because this reviewer's review state was still "Changes requested," which blocked merging, we left a note of thanks and asked them to change their status. Two hours and 22 minutes later, it was changed to approved. The merge was performed by a maintainer five days later. The merge commit db6fb41f is a squash combining the two commits, and the author is our account.

5. Lessons Learned: Tips for Open Source Contributions

• 「Two sets of examples serve as references for each other.Since the SDK version of the example in the same repository was already handling things correctly, "let's make it match the neighboring example" served as a shorter justification than "let's match the reference."
• 「If you state what you couldn't test, reviewers will test it for you." We couldn't run this example to the end and stated as much. After testing it directly, the reviewer discovered a larger truth that we hadn't claimed: "the microphone is dead on main." Conversely, had we exaggerated by claiming we "tested it," this conversation likely wouldn't have happened.
• 「Accept minor bot suggestions immediately.」 (… or {}) might seem trivial, but the reviewer also pointed out the exact same thing, noting that "it's optional, but the bot's point..." It keeps the review cycle short to simply incorporate correct suggestions without arguing.
• 「The "Changes requested" status is not automatically cleared.Even if a reviewer leaves an LGTM comment, merging will remain blocked if the review status stays unchanged. The next step is to politely ask them to update the status along with a word of thanks.
• 「Check both the notice and the signature for every commit.The first commit is missing the AI footer, and the second commit is missing the SSH signature. Although it was not a problem because the notice was in the body, it violates the rule.

A word from TT. The walkie-talkie manual said, "Speak using the old button." But that button was no longer connected to anything. Moreover, every time the other person finished speaking, it would erase the words we hadn't heard yet. When I handed over a note suggesting we switch to the new button, the teacher tried out the walkie-talkie personally and said, "This is a bigger deal than I thought; we can't hear anything at all with the old button!" They also discovered that the walkie-talkie would freeze if we replied in Japanese. Once we fixed that too, we chatted for five minutes and got a "thumbs up".

Links

All materials were accessed on 2026-10-29.

• PR: google-gemini/cookbook#1433 (Merge commit db6fb41f)
• Fixed example (base commit): quickstarts/websockets/Get_started_LiveAPI.py
• SDK version example (base commit): quickstarts/Get_started_LiveAPI.py lines 187–199
• Live API Reference: https://ai.google.dev/api/live#BidiGenerateContentRealtimeInput