Voice-Driving Claude CLI on a Free Stack — A Working End-to-End Round-Trip in AgentZero Lite

Series: AgentZero Lite — Part 11. The build report for this work is Part 10 — Shipping Voice Output to AIMODE (the engineering story). This part is a different angle on the same commit: the user-facing demo. It works. Here's what that looks like and why.

TL;DR

• I spoke to AgentZero Lite's bot. The bot wrote my instruction into a Claude CLI terminal. Claude did the work, the bot read the result back from the terminal, summarised it, and the summary came through my speakers — all in a single voice turn.

• Zero API keys in the audio path. STT is Whisper.net running a local GGML model. TTS is Windows SAPI (the OS-bundled Heami voice). Claude CLI is the only LLM in the loop, and it's the user's existing CLI — not a hosted API call.

• The same wiring works for Codex CLI, Gemini CLI, or any other peer agent that lives in a terminal — the bot doesn't know which CLI it's typing into. It just owns the keyboard.

• The voice turn auto-mutes the mic during TTS playback so the bot doesn't transcribe itself. That detail sounds trivial; it isn't (build report Part 10 for the post-mortem).

The show, before we open the box

What you see on screen

┌─────────────────────────────────────────────────────────────────────────────────┐
│ AgentZero Lite                                                       [_][□][X]  │
├──────────────────────────────────────────┬──────────────────────────────────────┤
│  [CMD] [EDIT] [Claude•] [Codex] [+]      │  ◂ AgentBot                [ AI ▼ ]  │
│ ─────────────────────────────────────    │ ──────────────────────────────────── │
│                                          │  ▸ You                               │
│  D:\Code\AI\AgentZeroLite>               │    "터미널 작업 요약해 줘"            │
│  $ git log --oneline -5                  │                                      │
│  988b4cd feat(voice): AI-mode TTS …      │  ◂ Gemma                             │
│  c17717e chore: bump version to 0.2.1    │    "활성 Claude 터미널의 최근 5개    │
│  2ae8b75 fix(ui/dock): break floating-…  │     커밋은 voice / dock / version 세 │
│  5622080 fix(ui/dock): don't auto-grab…  │     영역에 집중되어 있고, 가장 최근  │
│  95c6227 DOCUPDATE                       │     변경은 AI 모드 TTS 출력입니다."  │
│                                          │                                      │
│  Claude>                                 │  ✓ done · 3127ms · 1 turn            │
│                                          │                                      │
│                                          │  ┌─────────────────────────────────┐ │
│                                          │  │ 🎙 Listening · Whisper small    │ │
│                                          │  │  ▆▆▅▂▁▁▁▂▃▅▆▆▇  [🔇] [🔊 75%]  │ │
│                                          │  └─────────────────────────────────┘ │
└──────────────────────────────────────────┴──────────────────────────────────────┘
                                                                ↓
                                     🔊 (speakers play the Gemma reply,
                                          mic auto-mutes for the duration)

What you say, and what happens

1. You speak. "터미널 작업 요약해 줘." (Or in English: "Summarise what the terminal is doing.")

2. The mic indicator pulses, then settles. The bot transcribes locally — no spinner waiting on a cloud call.

3. Your phrase appears as a chat bubble, immediately followed by the bot's reply: a couple of sentences summarising what the active terminal is showing.

4. Your speakers play the bot's reply. The mic indicator dims to muted while the bot is speaking — no echo, no double-take.

5. The mic comes back on automatically the moment playback drains. You can speak the next thing without touching the keyboard or the mute button.

Why this is a different feeling than "voice assistant"

Why a "free stack" is the headline, not a footnote

• An OpenAI or ElevenLabs key for TTS,

• A Deepgram or Google Speech key for streaming STT,

• A custom server pipeline that fans the audio out and back,

• Or, in many "free" demos, an open-source TTS model whose voice sounds like a 1996 GPS unit.

What's inside

Component map

flowchart LR
    subgraph User["User"]
        Mic[("🎙 Microphone")]
        Speaker[("🔊 Speakers")]
    end

    subgraph WPF["AgentZero Lite (WPF host)"]
        VCS[VoiceCaptureService<br/>NAudio WaveInEvent<br/>16 kHz mono 50 ms frames]
        VSA[VoiceStreamActor<br/>under /user/stage/voice]
        Whisper[Whisper.net<br/>local GGML]
        SAPI[Windows TTS<br/>SAPI / Heami]
        Naudio[NAudioPlaybackQueue]
        AgentBot[AgentBotActor<br/>+ AgentReactorActor<br/>tool loop]
        TermActors[TerminalActor pool<br/>per ConPTY tab]
    end

    subgraph Agents["Worker agents (in ConPTY tabs)"]
        Claude[Claude CLI]
        Codex[Codex CLI]
        Other[Gemini / shells / REPLs]
    end

    Mic -->|PCM frames| VCS
    VCS -->|MicFrame| VSA
    VSA -->|TranscribeRequest| Whisper
    Whisper -->|transcript| VSA
    VSA -->|VoiceTranscriptReady| AgentBot
    AgentBot -->|StartReactor| TermActors
    TermActors -->|send_to_terminal<br/>read_terminal| Agents
    Agents -.->|tool result text| TermActors
    TermActors -->|ReactorResult<br/>final summary| AgentBot
    AgentBot -->|SpeakText| VSA
    VSA -->|SynthesizeRequest| SAPI
    SAPI -->|wav bytes| Naudio
    Naudio -->|WaveOut| Speaker
    Naudio -.->|PlaybackStarted/Stopped| VSA
    VSA -.->|OnTtsPlaybackChanged| VCS

One full voice round-trip, beat by beat

sequenceDiagram
    autonumber
    actor U as User
    participant Mic as VoiceCaptureService
    participant VSA as VoiceStreamActor
    participant W as Whisper.net
    participant Bot as AgentBotActor
    participant Reactor as AgentReactorActor
    participant Term as TerminalActor (Claude tab)
    participant CLI as Claude CLI
    participant TTS as Windows TTS
    participant PB as NAudioPlaybackQueue
    participant Spk as Speakers

    U->>Mic: speak: "터미널 작업 요약해 줘"
    Mic->>VSA: MicFrames (50 ms PCM, 16 kHz)
    Note over VSA: VAD segmenter buffers<br/>until ~2 s silence
    VSA->>W: PCM segment (~3.5 s)
    W-->>VSA: transcript (chars=15)
    Note over VSA: WhisperHallucinationFilter ✓<br/>VoiceCommandInterceptor → SummarizeTerminal<br/>Snapshot GetConsoleText() once
    VSA->>Bot: StartReactor(prompt + terminal snapshot)
    Bot->>Reactor: run tool loop
    Reactor->>Term: tool: read_terminal (group=0, tab=2)
    Term-->>Reactor: terminal screen text
    Reactor->>Reactor: model decides "done"<br/>with summary in args.message
    Reactor-->>Bot: ReactorResult(success, finalMessage)
    Bot->>Bot: AddBotMessage(finalMessage)
    Bot->>VSA: SpeakText(finalMessage, voice)
    VSA->>TTS: synthesize chunk(s)
    TTS-->>VSA: wav bytes per sentence
    VSA->>PB: Enqueue(audio, format=wav)
    PB->>Spk: WaveOut.Play()
    PB-->>VSA: PlaybackStarted
    VSA-->>Mic: OnTtsPlaybackChanged(true) → Muted=true
    Note over Mic,Spk: Mic captures audio for the level meter<br/>but VAD/forwarder are frozen
    PB->>Spk: …chunks drain in arrival order…
    PB-->>VSA: PlaybackStopped (queue idle)
    VSA-->>Mic: OnTtsPlaybackChanged(false) → Muted=false
    Note over Mic: User can speak again

Why an actor sits between the mic and the speaker

graph TB
    subgraph Stage["StageActor — supervisor"]
        AgentBot[AgentBotActor<br/>chat orchestration]
        Reactor[AgentReactorActor<br/>tool loop]
        Voice[VoiceStreamActor<br/>singleton<br/>under /user/stage/voice]
        WS[WorkspaceActor<br/>per workspace]
        TA1[TerminalActor<br/>Claude tab]
        TA2[TerminalActor<br/>Codex tab]
        TA3[TerminalActor<br/>shell tab]
    end
    AgentBot --- Reactor
    Reactor -.->|read/send| TA1
    Reactor -.->|read/send| TA2
    Reactor -.->|read/send| TA3
    AgentBot -.->|SpeakText| Voice
    Voice -.->|VoiceTranscriptReady| AgentBot
    WS --- TA1
    WS --- TA2
    WS --- TA3

1. One backpressure boundary for the whole audio path. Source.Queue<MicFrame>(64, OverflowStrategy.DropHead) at the front of the input graph means the soundcard can never block the actor. If STT lags, old frames drop instead of stalling capture — the user hears no clicks, sees no jitter.

2. Atomic teardown of the output side via KillSwitch. When the user says "그만" (stop), one Tell — BargeIn — collapses the OUTPUT graph: token queue closed, kill switch shut, playback queue stopped, mic auto-unmuted. Mid-sentence interruption with no leftover audio in the pipeline.

3. Settings reload happens at the seam, not the leaves. VoiceRuntimeFactory.BuildStt(VoiceSettingsStore.Load()) is invoked at the moment a worker actor spawns, so changing the TTS provider in the settings panel takes effect on the next worker without restarting the actor or the app.

The five pillars

Pillar 1 — Whisper hallucination filter

public static bool IsLikelyHallucination(string? transcript)
{
    if (string.IsNullOrWhiteSpace(transcript)) return false;
    var trimmed = transcript.Trim();
    if (Patterns.Symbolic.Contains(trimmed)) return true;   // ♪, [Music]
    var n = Normalize(trimmed);                             // letters/digits, lowercase
    if (n.Length == 0) return false;
    return Patterns.Normalised.Contains(n);
}

Pillar 2 — Voice command interceptor

flowchart LR
    T[Transcript] --> C{Classify}
    C -->|"그만 / stop"| Stop[VoiceCommandIntent.StopSpeaking]
    C -->|contains 터미널 + 요약| Sum[VoiceCommandIntent.SummarizeTerminal]
    C -->|else| Pass[VoiceCommandIntent.PassThrough]
    Stop --> BI[Tell BargeIn → kill OUTPUT graph]
    Sum --> SS[Snapshot GetConsoleText → embed → SendThroughAiToolLoopAsync]
    Pass --> SCI[fill txtInput → SendCurrentInput]

Pillar 3 — Auto-mute envelope (NAudio contract + self-tracked flag)

stateDiagram-v2
    direction LR
    [*] --> Idle
    Idle --> Busy: Enqueue + Started ✓
    Busy --> Busy: Enqueue (no event)
    Busy --> Busy: chunk drains, next clip starts<br/><i>Started fires again — BUG</i>
    Busy --> Idle: queue drains + Stopped ✓

stateDiagram-v2
    direction LR
    [*] --> Idle
    Idle --> Busy: Enqueue + Started ✓<br/>_started = true
    Busy --> Busy: Enqueue (no event)
    Busy --> Busy: chunk drains, next clip starts<br/><i>silent — _started already true</i>
    Busy --> Idle: queue drains + Stopped ✓<br/>_started = false

private bool _autoMutedByTts;

private void OnTtsPlaybackChanged(bool isPlaying)
{
    if (isPlaying)
    {
        if (!_voiceCapture.Muted)
        {
            SetVoiceMicMuted(true, source: "tts-auto");
            _autoMutedByTts = true;
        }
    }
    else
    {
        if (_autoMutedByTts)
        {
            SetVoiceMicMuted(false, source: "tts-auto");
            _autoMutedByTts = false;
        }
    }
}

Pillar 4 — AIMODE tool loop

flowchart TD
    Start([User instruction +<br/>optional terminal snapshot])
    Start --> Sys[Append system prompt + tool catalog<br/>on first send only]
    Sys --> Loop{For iter in 0..MaxIterations}
    Loop --> Gen[Stream LLM response]
    Gen --> Parse{Extract first JSON object<br/>+ ParseToolCall}
    Parse -->|done| End([Final summary returned<br/>via ReactorResult])
    Parse -->|tool name| Exec[ExecuteToolAsync]
    Exec -->|list_terminals| LT[Host returns groups + tabs]
    Exec -->|read_terminal| RT[Host calls GetConsoleText / ReadOutput]
    Exec -->|send_to_terminal| ST[Host writes via WriteAndEnter]
    Exec -->|send_key| SK[Host sends control sequence]
    Exec -->|wait| W[Task.Delay clamped 1..30 s]
    LT --> Result[Append --- TOOL RESULT --- to history]
    RT --> Result
    ST --> Result
    SK --> Result
    W --> Result
    Result --> Loop
    Loop -->|cap reached| Fail([Failure surfaced via FailureReason])

Pillar 5 — Static snapshot for the summary prompt

flowchart LR
    Voice[Voice intent: SummarizeTerminal] --> Snap["GetConsoleText()<br/>called ONCE at request time"]
    Snap --> Build["BuildTerminalSummaryPrompt<br/>fenced code block"]
    Build --> Bot["SendThroughAiToolLoopAsync<br/>aiInput = enriched<br/>displayText = user phrase"]
    Bot --> LLM[LLM sees a static window]
    LLM --> Sum[One-shot summary]

private static string BuildTerminalSummaryPrompt(string userPhrase, string terminalText)
{
    return
        $"{userPhrase}\n\n" +
        "아래는 사용자가 보고 있는 활성 터미널의 현재 화면 출력입니다. " +
        "이 내용을 한국어로 간결하게 요약해 주세요. " +
        "어떤 작업이 실행되었고 어떤 결과 / 에러가 있었는지 핵심만 짚어주세요.\n\n" +
        "```\n" + terminalText + "\n```";
}

What one round-trip looks like in the diagnostic log

[t0] [BOT-Voice-pipe] [t0] utterance-start
[+2.6s] [BOT-Voice-pipe] [t1] utterance-end | t1-t0=2600ms · pcm=115200 bytes (~3.60s)
[+2.6s] [BOT-Voice-pipe] [t2] pipeline-start | provider=WhisperLocal · lang=ko ·
        peak=-10.2dBFS · rms=-32.5dBFS · VAR=25.0%
[+5.9s] [BOT-Voice-pipe] [stage] STT transcribe | 3319ms · chars=15
[+5.9s] [BOT-Voice]  (batch) summarize-terminal | terminalChars=4821
[+5.9s] [AIMODE]    StartReactor sent (backend=External, model=google/gemma-4-e4b)
[+9.0s] [AIMODE]    result success=True turns=1 elapsed=3127ms
                    final="현재 활성 Claude 터미널의 최근 5개 커밋은 …"
[+9.5s] [BOT-Voice] Mic MUTED (tts-auto)
[+9.5s] [BOT-Voice] TTS started — autoMuted=True micMuted=True
[+15.4s] [BOT-Voice] Mic UNMUTED (tts-auto)
[+15.4s] [BOT-Voice] TTS stopped — autoMutedNow=False micMuted=False

Codex compatibility (and beyond)

• A tab running claude becomes a Claude peer.

• A tab running codex becomes a Codex peer.

• A tab running gemini becomes a Gemini peer.

• A tab running pwsh becomes a shell.

• A tab running python becomes a REPL.

• A tab running psql becomes a database session.

• "Tell Claude to refactor file X" — bot writes the instruction into the Claude tab via send_to_terminal, hits Enter, then summarises the response when Claude finishes.

• "List the terminals" — bot calls list_terminals, reads aloud what's in each tab.

• "Cancel that" — 그만 tears down the OUTPUT graph if the bot is mid-sentence; you get audio silence within the next chunk boundary.

What's still on the table

• Latency is honest, not great. ~3.3 s STT on CPU is the dominant cost. Vulkan halves it. large-v3 improves accuracy but adds another second. There's no streaming partials yet — the segmenter waits for ~2 s of silence before declaring an utterance, which adds another second to the perceived delay. Streaming partials are a stretch goal for the next iteration.

• MaxTokens=128 on voice replies is a guardrail, not a strategy. It keeps replies speakable but truncates legitimate longer answers. A better answer is sentence-chunked TTS that starts speaking before the LLM finishes, but that requires token-level streaming through the OUTPUT graph; today the OUTPUT graph receives the whole final message at once.

• Mute envelope still has a race. When two SpeakText arrivals straddle a sub-200 ms gap (rare in practice), the playback queue's natural drain and the next start can overlap such that the mic auto-unmutes for one or two frames. A small "hold mute for N ms after stop" timer would close it; deferred until we see whether real users hit it.

• Settings hot-reload is partial. Factory closures load settings at call time, so new TTS workers see the latest provider — but the existing pool keeps the original. Switching providers mid-session technically requires a mic toggle. An explicit "settings changed → reset pool" Tell would close it.

Closing

• Built with Claude Code Opus 4.7 (1M context)

Screenshot of it working

• TIP : a basic command-line text editor on Windows (like MS-DOS Editor) - TestCLI for VoiceInput

• https://psmon.github.io/AgentZeroLite/

TECH LINKS

• 𝕏 @webnori

• 📘 Akka Labs (Facebook group)

• 📝 More writings — webnori wiki