Voice-Driven Multi-Terminal Control — How a Local Gemma 4 Function-Caller Pilots Claude, Codex and Anything Else with a Prompt

TL;DR — In AgentZero Lite, the voice channel is not a chat input. It is the captain's bridge for a fleet of terminal AIs. You speak; an on-device Gemma 4 with a GBNF-locked tool catalog decides which terminal to read, which terminal to write, which key to send, when to wait; and the Akka.NET actor mesh carries those decisions to N concurrent CLI agents — Claude Code, Codex, Cursor CLI, or a plain shell. Same captain, swappable crew. Same bridge, swappable language model behind it. The point of this chapter is to show the seam: how voice becomes a function call, and how a function call becomes a keystroke into another agent's terminal.

The shape of the feature — UI view

"Tell Claude to refactor the auth module and tell Codex to write the tests for it. When Claude finishes, paste its diff into Codex."

┌──────────────────────────────────── AgentBot ─────────────────────────────────┐
│  Mode: [AI ▾]   Backend: Gemma 4 E4B (local)   Reactor: ▶ running  R3/12     │
├───────────────────────────────────────────────────────────────────────────────┤
│                                                                               │
│  🎙  YOU — "Tell Claude to refactor the auth module and tell Codex…"          │
│                                                                               │
│  🤖  REACTOR — Thinking ░░░░░░░░░ ▓                                           │
│        ↳ Tool: list_terminals                  → 3 terminals found            │
│        ↳ Tool: send_to_terminal #1 (Claude)   "refactor the auth module…"     │
│        ↳ Tool: send_to_terminal #2 (Codex)    "write tests for auth…"         │
│        ↳ Tool: wait 8s                                                        │
│        ↳ Tool: read_terminal  #1               → captured 142 lines           │
│        ↳ Tool: send_to_terminal #2             "<paste diff from Claude>"     │
│        ↳ Tool: done                                                           │
│                                                                               │
│  🤖  REACTOR — done in 4 turns · 11.2s                                        │
│                                                                               │
├───────────── Workspace tabs (each is an independent ConPTY) ──────────────────┤
│  ┌────[1] Claude Code──┬──[2] Codex──┬──[3] PS──┬──[4] +]──────────────────┐  │
│  │  > /refactor auth   │ > write tests for…    │ PS> _    │                │  │
│  │  ✓ refactored 3 files│ writing test_auth.py │          │                │  │
│  │  diff: 142 lines    │ pytest collected 17  │          │                │  │
│  └─────────────────────┴───────────────────────┴──────────┴────────────────┘  │
│                                                                               │
│  [🎙 listening · Whisper.net small]   ▮▮▮▮▮▯▯▯▯  amplitude · VAD: speaking   │
│                                                                               │
│  > _                                                                          │
└───────────────────────────────────────────────────────────────────────────────┘

The mental model — captain, comms, crew

Pipeline layer 1 — voice becomes text

flowchart LR
  Mic[🎙 NAudio<br/>16 kHz mono PCM]
  Mic --> RMS[Frame RMS<br/>every ~50 ms]
  RMS --> Meter[UI amplitude meter<br/>9-bar gradient]
  RMS --> VAD[Utterance VAD<br/>~40 silent frames<br/>= 2 s threshold]

  VAD -->|UtteranceStarted| Buf[ring buffer<br/>collect PCM]
  VAD -->|UtteranceEnded| Drain[ConsumePcmBuffer]

  Drain --> STT{ISpeechToText<br/>factory}
  STT -->|local CPU| W[Whisper.net small<br/>~487 MB]
  STT -->|cloud REST| O[OpenAI Whisper-1]
  STT -->|local server| WG[Webnori-Gemma audio]
  STT -->|on-device LLM| LG[Local Gemma audio]

  W --> Txt[transcript text]
  O --> Txt
  WG --> Txt
  LG --> Txt

  Txt --> Disp[Dispatcher.Invoke<br/>txtInput on UI thread]
  Disp --> Send[SendCurrentInput<br/>mode-aware]

  Send -->|Chat/Key mode| Term[active terminal write]
  Send -->|AI mode| Bot[AgentBotActor.StartReactor]

  classDef pri fill:#1e3a5f,stroke:#3794FF,color:#fff
  class Bot,Send pri

• Project/AgentZeroWpf/Services/Voice/VoiceCaptureService.cs — dual-VAD service, exposes AmplitudeChanged, UtteranceStarted, UtteranceEnded events. The 40-frame silence threshold (line 41) is empirically tuned — shorter clips bleed into the next utterance, longer feels laggy on the bridge.

• Project/AgentZeroWpf/Services/Voice/VoiceRuntimeFactory.cs — BuildStt(VoiceSettings) switch returning the right ISpeechToText implementation. Adding a new engine is one new branch and one new class.

• Project/AgentZeroWpf/UI/APP/AgentBotWindow.Voice.cs — wires capture events to the input box and the AUTO toggle. AUTO ON means each UtteranceEnded automatically calls SendCurrentInput; AUTO OFF means the user still has to press Enter. STT engines are warmed up off-thread to hide cold-start latency (Whisper.net is ~487 MB CPU model load).

• The mode awareness is critical — the same voice transcript routes to a terminal write in Chat/Key mode but to the AIMODE reactor in AI mode. Same input, different captain.

Pipeline layer 2 — text becomes a tool plan (the GBNF function-call loop)

flowchart TB
  Req[user request<br/>natural language]
  Req --> Reactor[AgentReactorActor]
  Reactor --> Choose{IAgentToolLoop<br/>factory}

  Choose -->|primary, on-device| Local[AgentToolLoop<br/>Gemma 4 E4B + GBNF]
  Choose -->|fallback, REST| Ext[ExternalAgentToolLoop<br/>Nemotron Nano 8B-v1]

  subgraph LOOP [tool-call loop · same shape on both backends]
    Plan[generate next step]
    Plan --> JSON{JSON<br/>tool call}
    JSON -->|GBNF-locked| Strict["{tool, args}<br/>schema-valid by construction"]
    JSON -->|native FC| Native["OpenAI-style<br/>function call"]
    Strict --> Dispatch[dispatch to tool]
    Native --> Dispatch

    Dispatch --> T1[list_terminals]
    Dispatch --> T2[read_terminal]
    Dispatch --> T3[send_to_terminal]
    Dispatch --> T4[send_key]
    Dispatch --> T5[wait]
    Dispatch --> T6[done]

    T1 --> Obs[observation]
    T2 --> Obs
    T3 --> Obs
    T4 --> Obs
    T5 --> Obs

    Obs --> Plan
    T6 --> Stop((finalize))
  end

  Local --> LOOP
  Ext --> LOOP

  Stop --> Result[ReactorResult<br/>Success, FinalMessage,<br/>TurnCount, ElapsedMs]
  Result --> UI[stream to AgentBot UI]

  classDef gbnf fill:#3a1e5f,stroke:#C586C0,color:#fff
  class Local,Strict gbnf
  classDef ext fill:#1e3a5f,stroke:#3794FF,color:#fff
  class Ext,Native ext

• Project/ZeroCommon/Llm/Tools/AgentToolLoop.cs — local Gemma path. Constructor (line 54–58) wires ChatTemplates.Gemma and constructs the Grammar from AgentToolGrammar.Gbnf with GrammarRootRule. KV cache reuse means follow-up prompts only re-process the new turn.

• Project/ZeroCommon/Llm/Tools/ExternalAgentToolLoop.cs — REST path. First user send seeds LlmMessage.System(AgentToolGrammar.SystemPrompt); subsequent sends append only the new user request to a stateless history list (line 66).

• Project/ZeroCommon/Actors/AgentReactorActor.cs — the FSM around the loop. Lines 55–89 lazy-construct the loop with OnTurnCompleted and OnGenerationProgress callbacks that re-enter the actor mailbox as TurnCompletedInternal / GenerationProgressInternal. This is what keeps streaming UI updates flowing without violating actor encapsulation.

• Project/ZeroCommon/Actors/Messages.cs lines 159–208 — the public reactor protocol: StartReactor, ReactorProgress(phase, text, round), ReactorResult(success, finalMessage, turnCount, elapsedMs, failureReason), ReactorBindings(hostFactory, optionsFactory, toolLoopFactory).

Pipeline layer 3 — tool plan becomes keystrokes in N terminals

flowchart TB
  Reactor[AgentReactorActor<br/>/user/stage/bot/reactor]
  Reactor -->|emits tool call| Bot[AgentBotActor<br/>/user/stage/bot]
  Bot -->|SendToTerminal<br/>WorkspaceName, TerminalId, Text| Stage[StageActor<br/>/user/stage]

  Stage -->|forwards by ws-name| WS1[WorkspaceActor<br/>/user/stage/ws-default]
  Stage -->|forwards by ws-name| WS2[WorkspaceActor<br/>/user/stage/ws-secondary]

  WS1 -->|terminalId lookup| TA1[TerminalActor<br/>term-claude]
  WS1 -->|terminalId lookup| TA2[TerminalActor<br/>term-codex]
  WS1 -->|terminalId lookup| TA3[TerminalActor<br/>term-pwsh]
  WS2 -->|terminalId lookup| TA4[TerminalActor<br/>term-other]

  TA1 -->|ITerminalSession.Write| C1[ConPtyTerminalSession<br/>child: claude.exe]
  TA2 -->|ITerminalSession.Write| C2[ConPtyTerminalSession<br/>child: codex]
  TA3 -->|ITerminalSession.Write| C3[ConPtyTerminalSession<br/>child: pwsh.exe]
  TA4 -->|ITerminalSession.Write| C4[ConPtyTerminalSession<br/>child: any CLI]

  C1 -.->|stdout bytes| TA1
  C2 -.->|stdout bytes| TA2
  C3 -.->|stdout bytes| TA3
  C4 -.->|stdout bytes| TA4

  TA1 -.->|TerminalOutput| Stage
  TA2 -.->|TerminalOutput| Stage
  TA3 -.->|TerminalOutput| Stage
  TA4 -.->|TerminalOutput| Stage

  Stage -.->|broadcast / read_terminal| Reactor

  classDef agent fill:#3a1e5f,stroke:#C586C0,color:#fff
  class Reactor,Bot agent
  classDef supervisor fill:#1e3a5f,stroke:#3794FF,color:#fff
  class Stage,WS1,WS2 supervisor
  classDef worker fill:#1e3a3a,stroke:#4EC9B0,color:#fff
  class TA1,TA2,TA3,TA4 worker

1. Names from user input must not poison the addressing scheme. Workspace and terminal names go through ActorNameSanitizer before becoming actor paths — Akka rejects /, :, spaces.

2. Supervision isolates failures. A Claude CLI that hangs or crashes is one TerminalActor's problem, not the bridge's. The supervising WorkspaceActor restarts or escalates.

3. The ITerminalSession seam is what keeps the actor layer testable headlessly. Real ConPTY sessions live in AgentZeroWpf/Services/; the actors only know the interface in ZeroCommon/Services/ITerminalSession.cs. Tests can swap a fake session and assert on the message protocol.

4. Active-terminal tracking (SetActiveTerminal at StageActor.cs lines 87–91) gives the reactor a sane default for send_to_terminal when the user doesn't name one — "send this to whichever terminal is in front."

• Project/ZeroCommon/Actors/StageActor.cs — top supervisor. Children: one AgentBotActor (singleton) and N WorkspaceActors. Holds (activeWorkspace, activeTerminalId) so the reactor doesn't need to ask each turn.

• Project/ZeroCommon/Actors/WorkspaceActor.cs — owns _terminals dict (line 25), routes SendToTerminal by terminal ID (lines 62–79).

• Project/ZeroCommon/Actors/TerminalActor.cs — wraps ITerminalSession. Bidirectional: outbound WriteToTerminal becomes pipe writes; inbound stdout becomes TerminalOutput events broadcast to Stage.

• Project/ZeroCommon/Data/Entities/CliDefinition.cs — the database row that says "Claude" maps to claude.exe with these args. Built-in seeds (IsBuiltIn = true) cover CMD / PowerShell 5/7 / Claude. Adding Codex or Cursor CLI is one row.

End-to-end sequence — voice to keystrokes

sequenceDiagram
  autonumber
  actor U as User
  participant V as VoiceCaptureService
  participant S as STT (Whisper.net)
  participant W as AgentBotWindow
  participant B as AgentBotActor
  participant R as AgentReactorActor
  participant L as AgentToolLoop<br/>(Gemma 4 + GBNF)
  participant Stg as StageActor
  participant TA1 as TerminalActor<br/>(Claude)
  participant TA2 as TerminalActor<br/>(Codex)

  U->>V: speaks utterance
  V->>V: dual-VAD: speaking → silent ≥ 40 frames
  V->>W: UtteranceEnded
  W->>S: ConsumePcmBuffer → transcribe
  S-->>W: text transcript
  W->>B: StartReactor(userRequest=text)
  B->>R: forward StartReactor

  R->>L: build loop (lazy)
  R->>L: send userRequest

  loop tool-call loop
    L->>L: GBNF-constrained sample
    L-->>R: tool call JSON {tool, args}
    R-->>W: ReactorProgress(Acting, "send_to_terminal #1")

    alt send_to_terminal
      R->>Stg: SendToTerminal(ws, termId, text)
      Stg->>TA1: forward
      TA1->>TA1: ITerminalSession.Write(bytes)
      TA1-->>Stg: TerminalOutput (echo, prompt)
      Stg-->>R: observation
    else read_terminal
      R->>Stg: ReadTerminal(ws, termId, lines)
      Stg-->>R: lines
    else done
      L-->>R: tool=done, message=...
    end
  end

  R-->>B: ReactorResult(success, msg, turns, ms)
  B-->>W: stream to chat UI
  W-->>U: render final message + per-turn log

Where Claude, Codex and friends fit — the model-agnostic terminal layer

• It receives WriteToTerminal(text) and writes those bytes to a pipe.

• It reads bytes from the pipe and wraps them in TerminalOutput.

• That is the entire contract.

Why this design holds together

1. One abstract interface, many implementations. ISpeechToText for voice. IAgentToolLoop for the function-call loop. ILlmProvider for the underlying chat backend. ITerminalSession for the terminal seam. Adding a fifth STT engine, a third tool-loop backend, a sixth provider, or a new terminal type each costs a single class — never a structural change.

2. Schema-valid by construction at the LLM boundary. GBNF gives us function calling on Gemma 4 without trusting the model to behave. Without this, a 4 B-parameter local model would be too unreliable to pilot real terminals.

3. Actors all the way down for routing. The captain → comms → crew chain crosses async boundaries (mic threads, LLM inference threads, ConPTY pipe threads, UI thread). Akka.NET makes those boundaries the addressable shape of the system instead of the bug surface.

Status (2026-04-28)

• Voice capture & STT — shipping. Four engines available; AUTO toggle live.

• AgentBotActor + AgentReactorActor + tool-loop FSM — shipping. Both backends exercised.

• Six-tool GBNF catalog — shipping. AgentToolGrammar.Gbnf enforced at sampler level.

• Multi-terminal actor mesh — shipping. ConPTY hosts in WPF tabs, sanitized actor paths, peer-signal protocol live.

• Built-in CLI definitions — CMD / PowerShell 5 / PowerShell 7 / Claude seeded; Codex / Cursor / others are user-defined rows.

TECH LINKS

• 𝕏 @webnori

• 📘 Akka Labs (Facebook group)

• 📝 More writings — webnori wiki