Voice
Stream captured audio to an NPC and play back its spoken reply.
Voice runs over the same connection as text. The SDK gives you the wire methods — streaming audio chunks in, receiving synthesized speech back — but capturing the player's microphone and deciding when they've stopped talking is up to you, using the browser's own audio APIs.
VoiceNPC component, the Web SDK does not capture your microphone or run voice-activity detection for you — the browser sandbox makes that a much more involved, permission-gated integration than in a compiled game engine. This is a real gap today, not a hidden limitation: you own capture (e.g. the MediaRecorder API or an AudioWorklet) and pass the SDK raw PCM bytes.Sending audio
Stream chunks as you capture them, then send one final chunk with end: true — or let the server auto-finalize once it has buffered about 5MB, whichever comes first:
// pcmChunk: ArrayBuffer | Uint8Array of raw audio you've already captured
connection.sendVoiceChunk(pcmChunk, false);
// ...more chunks as they're captured...
connection.sendVoiceChunk(finalChunk, true); // end: truedataendsenderIdOnce the server finishes transcribing, a transcript event fires with what it heard, followed by the normal chat events — voice replies arrive as one complete chatChunk rather than streamed token by token, since grounding runs before anything is sent back.
Playing the reply
Synthesized speech arrives as raw binary WebSocket frames — no JSON envelope, one frame per sentence, in playback order:
connection.on("audio", (bytes) => {
// bytes: ArrayBuffer of raw synthesized speech for one sentence.
// Decode/queue it with the Web Audio API in your own playback pipeline.
playAudioChunk(bytes);
});