The Realtime Voice Mountain — Step 1 Embedding an Open vLLM, Step 2 Akka Streams, and Why ElevenLabs Stays Plugged In

TL;DR — On real-time streaming voice, ElevenLabs is the bar. Sub-second latency, near-natural prosody, actual real-time. AgentZero Lite today doesn't chase that — we VAD-batch utterances before sending to STT and accept the latency hit, which lets us treat voice behind a clean byte[] PCM → string interface. But to feel like the assistant we want — and to do it without a Realtime API in the loop — we need to climb. Adopting ElevenLabs Realtime or OpenAI Realtime is genuinely cheap on the implementation side: they already ship the streaming protocol, sentence-boundary handling, and interrupt semantics. The mountain only exists when you stitch a local open STT model and a local open TTS model into something Realtime-API-like yourself, because none of the open pieces ship that orchestration layer for you. Step 1: prove that an open voice-LLM (vLLM) embeds end-to-end on consumer hardware. Step 2: an Akka.NET Streams Graph that handles mid-stream long-sentence speech, request-unit switching, and mid-stream LLM function calls for sentence-boundary detection. The goal isn't to beat ElevenLabs Realtime or OpenAI Realtime — it's to reach a minimum usable approximation of that experience on a local-only stack. Step 2 may end up harder than the multi-CLI core itself. So Realtime APIs stay plugged in throughout — I'm a paying user; this isn't a cost argument, it's an ownership-and-architecture one. The road for fully open local realtime voice is long.

Where realtime streaming voice actually is in 2026

Why even attempt the open path

1. Local control of the audio modality. The same reason we ship an on-device Gemma 4 for the function-call agent applies here — voice data is sensitive, and a captain's bridge (per the previous chapter) shouldn't have to leave the machine for every utterance.

2. Architectural ownership of the streaming protocol. Bek's critique on X named it precisely: behind a unified OpenAI-compatible HTTP endpoint, the streaming chunk semantics still differ per backend — chars vs ms boundaries, finalization shape, interim vs final signaling. If we own at least one path end-to-end, we get to shape that protocol seam intentionally instead of papering over it.

3. The captain-can-interrupt problem. Real conversational voice means the user can cut in mid-output. That requires the TTS layer to expose cancellation primitives that match how our Akka actor mesh talks. ElevenLabs' API has good interrupt support; doing it ourselves teaches us exactly how that protocol should look.

Today's design — VAD utterance batching, not streaming

flowchart LR
  Mic[🎙 NAudio<br/>16 kHz mono PCM]
  Mic --> VAD[Frame VAD + Utterance VAD<br/>40 silent frames = 2 s threshold]
  VAD -->|UtteranceEnded| Drain["ConsumePcmBuffer to byte array"]
  Drain --> STT{ISpeechToText<br/>factory}
  STT --> Whisper[Whisper.net]
  STT --> OAI[OpenAI Whisper-1]
  STT --> Webnori[Webnori-Gemma audio]
  STT --> LocalG[Local Gemma audio]
  Whisper --> Txt[full transcript]
  OAI --> Txt
  Webnori --> Txt
  LocalG --> Txt
  Txt --> Bot[AgentBotActor.StartReactor]

  classDef now fill:#1e3a3a,stroke:#4EC9B0,color:#fff
  class Drain,Txt,Bot now

Step 1 — Embedding an open vLLM end-to-end

• Microsoft MAI-Voice-1 via Foundry Local — interesting because Foundry Local already ships in Windows; Windows ML integration path looks clean. (Hat tip to @nkanauzu for surfacing this on X.)

• Kokoro — small footprint, surprisingly good prosody, ONNX path available.

• XTTS-v2 — multi-lingual, but heavier and license is restrictive.

• Parakeet TDT — primarily STT side but worth probing for the streaming protocol design.

1. The model loads and synthesizes inside Project/AgentZeroWpf without an external server process.

2. First-audio-out latency is bounded — say <1.5 s on a non-GPU developer machine.

3. The streaming output exposes a chunk interface we can plug behind an ITtsStream abstraction without leaking backend-specific chunk semantics into consumer code.

interface ITtsStream {
  IAsyncEnumerable<AudioChunk> Synthesize(string text, CancellationToken ct);
  Task SignalSentenceBoundary();        // optional, for mid-stream coordination
  ValueTask DisposeAsync();              // explicit cancellation
}

record AudioChunk(byte[] Pcm16, bool IsFinal, TimeSpan Position);

Step 2 — Akka.NET Streams Graph for mid-stream long-sentence speech

flowchart LR
  S[Source<br/>produces items] --> B((Broadcast))
  B --> FA[Flow A<br/>transform]
  B --> FB[Flow B<br/>transform]
  FA --> M((Merge))
  FB --> M
  M --> Snk[Sink<br/>consumes]

  classDef source fill:#1e3a5f,stroke:#3794FF,color:#fff
  class S source
  classDef flow fill:#3a1e5f,stroke:#C586C0,color:#fff
  class FA,FB flow
  classDef junction fill:#5f3a1e,stroke:#FFA500,color:#fff
  class B,M junction
  classDef sink fill:#1e3a3a,stroke:#4EC9B0,color:#fff
  class Snk sink

1. Source / Flow / Sink are the primitives. Every Akka Streams pipeline ends with one Sink and starts with one Source; Flows are the in-between transforms. The linear DSL stops here.

2. Broadcast and Merge are junctions. Broadcast duplicates each upstream item across N downstream branches; Merge interleaves N upstream branches into one downstream. Junctions are how you bend a pipeline into a non-linear shape.

3. Back-pressure flows backwards through everything. If the Sink is slow, Merge slows down, Flow A / Flow B slow, Broadcast slows, Source slows. You don't write any of that — the materializer wires it for you.

flowchart LR
  Src[Source<br/>LLM token stream]
  Src --> Acc[Stage: accumulate<br/>tokens into buffer]
  Acc --> Boundary{Stage:<br/>sentence-boundary<br/>detection}
  Boundary -->|complete sentence| Sentence[Source: completed sentence]
  Boundary -->|partial / continuing| Acc
  Sentence --> Switch[Stage: request-unit<br/>switching / noise filter]
  Switch --> TTS[Sink: ITtsStream<br/>synthesize chunk]
  TTS --> Audio[AudioChunk stream]

  classDef stage fill:#3a1e5f,stroke:#C586C0,color:#fff
  class Acc,Boundary,Switch stage
  classDef io fill:#1e3a5f,stroke:#3794FF,color:#fff
  class Src,TTS io
  classDef out fill:#1e3a3a,stroke:#4EC9B0,color:#fff
  class Sentence,Audio out

2.1 Sentence-boundary detection mid-stream — via LLM function call

2.2 Request-unit switching (noise filter)

2.3 Cancellation and back-pressure

1. Cancel the LLM token source.

2. Drain or discard buffered sentences depending on policy.

3. Stop the in-flight TTS synthesize call cleanly.

4. Release the audio device without click artifacts.

Honest assessment — Step 2 may be harder than the core

1. Real-time audio constraints — clicks, gaps, glitches are immediately perceptible to humans.

2. LLM unpredictability — token timing is bursty; sentence boundaries arrive irregularly.

3. User interrupt semantics — the human can change intent mid-response, and the system must respond gracefully.

Why Realtime APIs stay the default — and what the open work is actually for

• Realtime API path (ElevenLabs Realtime / OpenAI Realtime) — the streaming fabric is built into the API. Implementation = client integration, mostly. Recommended unless local-only is a hard constraint.

• Local open path (open STT + open TTS + our own orchestration) — the streaming fabric is the work. Step 1 + Step 2 above. Activated only when audio cannot leave the machine, or the user wants total ownership of the modality.

flowchart LR
  User[User intent]
  User --> Dispatch{TTS dispatch}
  Dispatch -->|default for realtime,<br/>cheap to adopt| EL[Realtime API<br/>ElevenLabs / OpenAI<br/>built-in streaming]
  Dispatch -->|local-only constraint| Open[Local open stack<br/>Step-1 + Step-2<br/>approximation only]
  Dispatch -->|Step 1 not yet ready,<br/>fallback| EL
  EL --> Audio[audio out]
  Open --> Audio

  classDef premium fill:#3a1e5f,stroke:#C586C0,color:#fff
  class EL premium
  classDef open fill:#1e3a3a,stroke:#4EC9B0,color:#fff
  class Open open

What this means for users right now

1. Voice input today — Whisper.net / OpenAI Whisper / Webnori-Gemma / Local Gemma audio, all behind VAD utterance batching. Works, ~600 ms latency penalty, no streaming. Stable.

2. Voice output today — Windows SAPI / OpenAI tts-1, both also batch. Works, no streaming chunk semantics leaking out, no real-time interruption.

3. Voice output Step 1 (planned) — ITtsStream interface + at least one open backend (probably Kokoro or MAI-Voice-1 via Foundry Local) embedded end-to-end. Adds streaming as an option without changing existing batched paths.

4. Voice output Step 2 (planned, may take a while) — Akka Streams Graph for sentence-boundary-aware long-form streaming with cancellation and request-unit switching.

5. ElevenLabs path — kept as a first-class provider throughout. If real-time matters now, that's the answer.

The long road for fully open local realtime voice

• Ship Step 1 (embed an open vLLM, prove the seam).

• Decide based on Step 1's actual feel whether Step 2's complexity is worth it for our UX.

• Keep ElevenLabs plugged in regardless — they earned the bar.

TECH LINKS

• 𝕏 @webnori

• 📘 Akka Labs (Facebook group)

• 📝 More writings — webnori wiki