Adding Providers and a Voice — AgentZero Lite Expands Beyond a Single Local Model

TL;DR — Two surfaces of AgentZero Lite expanded at once. The LLM layer went from a single local Gemma-4 to five backends (local Gemma, local Nemotron Nano 8B, Webnori, OpenAI, LM Studio, Ollama) behind one OpenAI-compatible interface. The modality layer grew a voice — a STT/TTS subsystem with four STT engines, two TTS engines, dual-VAD mic capture, and a live pipeline that reuses the same LLM gateway. Neither expansion needed a new architecture; both fell out of the same design principle that already defined the agent: one abstract interface, many implementations.

The two surfaces that just expanded

flowchart LR
  subgraph Before
    A1[Local Gemma 4<br/>+ Nemotron]:::dim
    T1[Text only]:::dim
  end
  subgraph After
    A2[Local Gemma 4<br/>Local Nemotron<br/>Webnori<br/>OpenAI<br/>LM Studio<br/>Ollama]
    T2[Text + Voice<br/>STT × 4 / TTS × 2]
  end
  A1 -.expanded.-> A2
  T1 -.expanded.-> T2
  classDef dim fill:#f5f5f5,stroke:#999,color:#666

Surface 1 — From one local LLM to five providers

public interface ILlmProvider
{
    string ProviderName { get; }
    Task<List<LlmModelInfo>> ListModelsAsync(CancellationToken ct = default);
    Task<LlmResponse> CompleteAsync(LlmRequest request, CancellationToken ct = default);
    IAsyncEnumerable<LlmStreamChunk> StreamAsync(LlmRequest request, CancellationToken ct = default);
}

flowchart TD
  S[LLM tab<br/>Local · External] --> F[LlmProviderFactory]
  F -->|Local| LH[LLamaSharp host<br/>Gemma 4 / Nemotron / …]
  F -->|External| OAC[OpenAiCompatibleProvider]
  OAC -->|baseUrl + apiKey| W[Webnori]
  OAC -->|baseUrl + apiKey| O[OpenAI]
  OAC -->|baseUrl| LMS[LM Studio]
  OAC -->|baseUrl| OLL[Ollama]

Where this connects back to AIMODE

Surface 2 — From text to voice

Pipeline

flowchart LR
  M[Mic<br/>NAudio<br/>16 kHz mono] --> V[VAD<br/>frame + utterance]
  V -->|UtteranceEnded| STT[ISpeechToText]
  STT -->|text| GW[LlmGateway.OpenSession]
  GW -->|reply| CL[TtsTextCleaner<br/>strip markup]
  CL --> TTS[ITextToSpeech]
  TTS -->|audio bytes| PB[VoicePlaybackService<br/>NAudio]
  PB -->|while playing| MUTE[Muted=true<br/>mic suppressed]
  MUTE -.-> M

STT — four engines, one interface

public interface ISpeechToText
{
    string ProviderName { get; }
    Task<string> TranscribeAsync(byte[] pcm16k, string? language, CancellationToken ct);
}

TTS — three options

Mic capture: dual-VAD + pre-roll + mute-while-playing

public sealed class VoiceCaptureService : IDisposable
{
    public event EventHandler<float>? AmplitudeChanged;          // ~50 ms ticks → UI level meter
    public event EventHandler<bool>? SpeakingStateChanged;        // VAD frame state
    public event EventHandler? UtteranceStarted;
    public event EventHandler<byte[]>? UtteranceEnded;            // PCM 16k mono → STT
    public bool Muted { get; set; }                               // suppress while TTS plays
}

• Frame-level VAD (~50 ms) — for the live amplitude meter the user sees in the Voice Test panel.

• Utterance-level VAD (~2 s of trailing silence) — the actual segment boundary that triggers STT. Without this, every breath becomes a transcription request.

• Pre-roll buffer (~1 s) — captured before VAD trips, so the first consonant of a sentence isn't clipped off.

• Muted flag — set true while VoicePlaybackService is playing TTS audio. Without this the speaker output gets re-captured by the mic and the loop talks to itself.

What this expansion didn't change

• No new actor was introduced. Voice still talks to the same AgentReactorActor topology described in the foundation report. STT/TTS are services, not actors.

• No new gateway. Voice reuses LlmGateway.OpenSession(). Whatever the LLM tab decides answers the voice — local Gemma, OpenAI, anything.

• No chunked streaming yet. All STT/TTS calls take a complete utterance. Streaming is a known follow-up; the absence is intentional, not an oversight, because chunked Whisper + chunked TTS on five providers is its own multi-week problem.

• No CUDA in the installer. Whisper.net's CUDA runtime is real and works, but bundling it doubled installer size for a feature most users will run on small models. CPU-only first; GPU later if the data says it matters.

The shared design principle, restated

Why the free-OS-voice angle is load-bearing

Situational use cases this stack unlocks

What's coming from the open-source side

STT (open / efficient, 2026 picks)

TTS (open / efficient, 2026 picks)

Microsoft's own move (April 2026)

• MAI-Voice-1 generates 60 seconds of audio in 1 second (60× real-time).

• Microsoft's embedded-speech path lets newer Win11 boxes (24H2 + DX12 + 8 GB RAM) run TTS locally — exactly the same OS-integration story SAPI already gives us, but neural-HD-grade.

• 2026-03 Foundry release notes mention Neural HD TTS with MAI-voice-1 integration.

What plugs in where

flowchart LR
  subgraph existing[Existing slots]
    S[ISpeechToText]
    T[ITextToSpeech]
  end
  subgraph stt_open[Open STT models]
    Pk[Parakeet TDT 1.1B]
    DW[Distil-Whisper]
    WT[Whisper Turbo V3]
    FW[Faster-Whisper]
    WX[WhisperX<br/>+ diarization]
  end
  subgraph tts_open[Open / future TTS]
    Ko[Kokoro 82M]
    XT[XTTS-v2]
    Mai[MAI-Voice-1<br/>via Foundry Local]
  end
  Pk --> S
  DW --> S
  WT --> S
  FW --> S
  WX --> S
  Ko --> T
  XT --> T
  Mai --> T

Closing — what the next slot is for

• Local Gemma audio model when the catalog ships one — fills the LocalGemmaStt placeholder, gives a fully-offline voice loop.

• A streaming provider (e.g., Anthropic's streaming API) — adds nothing structural, just a new ILlmProvider impl.

• A new voice provider — same.

TECH LINKS

• 𝕏 @webnori

• 📘 Akka Labs (Facebook group)

• 📝 More writings — webnori wiki