AgentZero Lite v0.6.1 — Voice in, OS out: when your CLI grows hands

A minimalist Windows shell for the AI era just learned two new tricks: it can drive your desktop, and you can drive it with your voice. Mission M0014 wired in a symmetric OS-Control surface — every read-only capability is exposed both as a CLI verb (AgentZeroLite.exe -cli os …) and as an LLM tool (os_*). Mission M0008 had already brought a fully on-device voice pipeline (Whisper + Vulkan, Akka Streams, command interception). Put them together and the on-device assistant can self-operate the computer hands-free — within a deliberate approval gate.

What shipped

Architecture at a glance

flowchart LR
    subgraph IN[Inputs]
      MIC[🎙 Mic]
      KBD[⌨ Keyboard]
      RCLI[📡 Remote CLI<br/>WM_COPYDATA]
    end

    subgraph VP[Voice pipeline · M0008]
      VAD[VAD + Akka Streams<br/>VoiceSegmenterFlow]
      STT[Whisper.net<br/>CPU + Vulkan]
      VCI[VoiceCommand<br/>Interceptor]
    end

    subgraph BOT[AgentBot · UI gateway]
      MODE{AI Mode?}
      LOOP[AgentLoopActor<br/>IAgentLoop]
    end

    subgraph TOOLS[Toolbelt · M0013/M0014]
      TR[Terminal-relay<br/>tools]
      OS[OS-Control facade<br/>OsControlService]
    end

    subgraph WIN[Windows surface]
      UIA[UI Automation tree]
      GDI[GDI BitBlt → PNG]
      INP[SendInput<br/>gated]
    end

    MIC --> VAD --> STT --> VCI
    VCI -- "stop / summarize" --> BOT
    VCI -- pass-through --> BOT
    KBD --> BOT
    RCLI --> BOT

    MODE -- off --> TR
    MODE -- on --> LOOP
    LOOP --> TR
    LOOP --> OS

    OS --> UIA
    OS --> GDI
    OS -.gate.-> INP

    OS --> AUDIT[(tmp/os-cli/audit/<br/>YYYY-MM-DD.jsonl)]

• One facade, many callers. OsControlService is reached by the CLI dispatcher, the in-process LLM toolbelt, and the E2E smoke probe. They all stamp a caller field on the audit line, so a forensic reader can tell which channel originated a click.

• The LLM never sees pixels. Screenshots return a path, not bytes. The agent loop's context never inflates with image data, and the prompt-injection attack surface stays small.

• Voice is just another input. It funnels into the same bot dispatch the keyboard uses. Once the transcript arrives, AI Mode + tool calls decide what happens — including whether os_* tools fire.

The symmetric surface

1. --allow-input on the CLI verb (per-call), or

2. Process env var AGENTZERO_OS_INPUT_ALLOWED=1 (any of 1 / true / yes).

{
  "ok": false,
  "error": "OS input simulation is gated. Set environment variable AGENTZERO_OS_INPUT_ALLOWED=1 (LLM/GUI) or pass --allow-input on the CLI verb to enable.",
  "verb": "<tool name>"
}

How the voice path classifies a transcript

flowchart TD
    T[Whisper transcript] --> N{IsNullOrWhiteSpace?}
    N -->|yes| P[PassThrough no-op]
    N -->|no| TR2[Trim + strip trailing punctuation]
    TR2 --> S{exact match<br/>StopPhrases?}
    S -->|yes| STOP[StopSpeaking<br/>cancel TTS, skip LLM]
    S -->|no| K{contains 터미널 + 요약<br/>or terminal + summar?}
    K -->|yes| SUM[SummarizeTerminal<br/>snapshot + ask LLM]
    K -->|no| PT[PassThrough<br/>forward to bot]

Where the artefacts go

tmp/os-cli/
├── audit/2026-05-08.jsonl        one line per call (cli, llm, e2e)
├── screenshots/2026-05-08/       PNG outputs (HH-mm-ss-fff prefix)
└── e2e/2026-05-08.log            E2E smoke summary

{
  "ts":     "2026-05-08T11:42:09.123+09:00",
  "caller": "cli | llm | e2e",
  "verb":   "list_windows | screenshot | mouse_click | …",
  "args":   { },
  "ok":     true,
  "error":  null
}

Use cases — what this combination unlocks

1. Voice-driven screen review while you keyboard the next tab

"터미널 작업 요약해 줘."

2. Hands-free TTS interruption

"그만."

3. Remote desktop check-up via CLI

# From a Claude tab (or any peer terminal) on the same box:
AgentZeroLite.exe -cli os list-windows --filter "Visual Studio"
AgentZeroLite.exe -cli os screenshot --hwnd 0x000A0234
AgentZeroLite.exe -cli os element-tree 0x000A0234 --depth 5

4. LLM-driven UI inspection (without typing on your behalf)

"내 노트패드 열려 있어? 열려 있으면 내용 캡처해."

1. os_list_windows({ "title_filter": "Notepad" }) → returns hwnd list.

2. os_screenshot({ "hwnd": <hwnd>, "grayscale": true }) → returns path.

3. os_text-capture (CLI, future LLM exposure) for the textual content.

4. done with the operator-readable summary.

5. Self-piloting micro-flows on operator-approved sessions

$env:AGENTZERO_OS_INPUT_ALLOWED = "1"

Anti-patterns we deliberately ruled out

• No screenshot bytes in LLM context. The toolbelt returns a path string. Inlining bytes wastes tokens and exposes the prompt-injection attack surface.

• No os_launch LLM tool. os_activate requires a pre-existing hwnd. The CLI has open-win, but exposing it to the LLM creates a recursion risk (LLM driving an instance of itself). If a future mission needs this, it lands behind its own approval flag.

• No reused screenshot paths. Filenames embed millisecond precision (HH-mm-ss-fff) so two near-simultaneous captures don't collide.

• No "trivial" call left unaudited. Even os_list_windows logs, because enumeration itself is a fingerprinting signal worth tracing.

Origin parity (AgentWin)

• Imported: list-windows, get-window-info, screenshot, element-tree, text-capture, dpi, activate, mouse-click, mouse-move, mouse-wheel, keypress.

• Skipped: scroll-capture (folded into element-tree), copy <text> (clashes with existing copy verb), virtual-desktop service (a future, dedicated mission).

Try it

pwsh Docs/scripts/launch-self-smoke.ps1 -Configuration Debug

• Releases page: https://github.com/psmon/AgentZeroLite/releases/tag/v0.6.1

• Source: https://github.com/psmon/AgentZeroLite

Closing note

TECH LINKS

• 𝕏 @webnori

• 📘 Akka Labs (Facebook group)

• 📝 More writings — webnori wiki