Hybrid Isn't a Choice — Plugging into All Three Giants Is Inevitable; Mixing Open Models In Is the Strategy

Series: AgentZero Lite — Part 12. The previous part, Voice-Driving Claude CLI on a Free Stack, showed what one specific hybrid configuration looks like end-to-end. This part zooms out: why "pick a giant" is not a thing anymore, why open models in your composition is what you actually choose, and what the architecture looks like that lets a single app run all of those at once.

TL;DR

• You don't pick OpenAI or Anthropic or Google. You plug into all three. Each one wins different turns — coding, voice, multimodal, reasoning, on-device — and your users will route around you the moment your app refuses to follow. Picking one giant is a posture for a research demo, not a product.

• Once you're already running a multi-giant surface, mixing in open models is the only knob you actually control. Provider pricing is unpredictable in both directions (Anthropic briefly pulled Claude Code from the $20 Pro plan in April 2026; OpenAI cut Realtime API rates the same month). What finance and ops teams can plan against is fixed cost — and only the open path gives you that.

• AgentZero Lite is built for the composition, not for any one provider. The on-device LLM (Gemma 4 today, Nemotron Nano in flight) is the in-shell coordinator; the giants live in their own terminal tabs. The user's recipe — Paid+Free, Paid+Free+Free, Free+Free — is a configuration choice, not an architecture rewrite.

• What turns a plain LLM into an agent is function calling, not the model. Gemma 4 is a token generator until you constrain its decoding to a JSON tool-call grammar. That grammar is half a page. Once it exists, the same actor stage that drives Claude Code drives Gemma 4 drives Nemotron Nano.

• Gemma 4 is the inflection point, not the destination. From April 2026 onwards, purpose-specific small models — fine-tuned for terminal log triage, voice command parsing, code-review summarisation — are expected to keep accelerating. Three weeks after Gemma 4, NVIDIA shipped Nemotron 3 Nano Omni. The cadence isn't slowing.

What hybrid looks like in this app

flowchart LR
  U["User"] -->|"ask in KO/EN"| BOT["AgentBot AIMODE"]
  BOT -->|"GBNF JSON tool call"| LL["Local LLM<br>Gemma 4 / Nemotron Nano"]
  LL -->|"send_to_terminal"| RE["AgentReactorActor<br>FSM"]
  RE -->|"ConPTY write + Enter"| TT["Top-tier CLI in tab<br>claude / codex / gemini / aider"]
  TT -->|"reply via -cli bot-chat"| RE
  RE -->|"continuation cycle"| LL
  LL -->|"summary"| BOT
  BOT -->|"text or voice"| U

The economics — a snapshot, not a trend

"We pay $X for the GPU, $Y for the model download, $Z for electricity. The marginal voice minute is free. Top-tier kicks in only on turns flagged for it."

What top-tier still wins at — and what open is rapidly closing in on

• Voice prosody and latency — OpenAI Voice and ElevenLabs are still in another class. For production (audiobook narration, branded TTS, customer-facing IVR), top-tier is the right call.

• Frontier reasoning depth — anything that benefits from a 200K+ context window, very long chain-of-thought, or recent web grounding still favours Claude / GPT / Gemini.

• Best-in-class coding — Claude Code's tool-use and codebase agency are genuinely ahead of any open model running on a single user GPU.

• Multimodal at the edge. Gemma 4 (Apr 2, 2026) shipped four variants from 2B to 31B with on-device deployment as the headline use case (HuggingFace blog; InfoQ). The smaller variants run on a phone or a Raspberry Pi. The licence is Apache 2.0.

• Single-GPU omni-modal. NVIDIA Nemotron 3 Nano Omni (Apr 28, 2026) is a 30B Mixture-of-Experts model that activates only 3B parameters per token, fits in 25 GB of RAM at 4-bit, and ships as open weights with commercial licensing (DeepInfra editorial). It handles audio + video + image + text in a single backbone — the exact shape AgentZero Lite's voice subsystem currently glues out of separate single-modality models.

• Function-calling reliability. Both Gemma 4 and the Nemotron line ship with robust tool-use templates. The reliability gap to top-tier on structured output (JSON-only, schema-constrained) is now small enough that constrained-decoding (GBNF) closes it for most production flows.

What turns a plain LLM into an agent — function calling, walked through

Route A — constrained decoding via GBNF grammar (what AgentZero Lite uses)

root         ::= "{" ws "\"tool\":" ws tool-name ws ","
                 ws "\"args\":" ws args-object ws "}"

tool-name    ::= "\"list_terminals\""
              |  "\"read_terminal\""
              |  "\"send_to_terminal\""
              |  "\"send_key\""
              |  "\"wait\""
              |  "\"done\""

args-object  ::= "{" ws (kv (ws "," ws kv)*)? ws "}"
kv           ::= string ws ":" ws value
value        ::= string | number | boolean
string       ::= "\"" char* "\""
ws           ::= [ \t\n]*

{ "tool": "send_to_terminal",
  "args": { "tab": "Claude", "text": "Summarise the last failure trace." } }

Route B — native tool-use template (what Nemotron Nano uses)

How AgentZero Lite plays both sides — the architecture

1. The actor stage runs the loop, the LLM does not

/user/stage                       StageActor       (supervisor + broker)
    /bot                          AgentBotActor    (chat-input router)
    /ws-<name>                    WorkspaceActor   (one per folder)
        /term-<id>                TerminalActor    (one per ConPTY tab)
    /reactor                      AgentReactorActor (FSM around the LLM)

Idle  ──StartReactor──▶  Thinking  ──prompt sent──▶  Generating
                                                       │
                                                       ▼
                                                     Acting  ──tool result──▶  Thinking  (continuation)
                                                       │
                                                       └──────done──────▶  Done

2. One contract, two backends — ILocalLlm

public interface ILocalLlm : IDisposable
{
    string ModelId { get; }
    string Backend { get; }     // "gemma4-gbnf" | "nemotron-native" | …
    bool   Ready   { get; }

    // ONE inference call. Caller supplies the tool catalog; receiver returns
    // the structured tool-call payload (or a "done" signal). KV cache lives
    // inside the implementation; the actor does not see it.
    Task<ToolCall> NextTurnAsync(
        ChatHistory history,
        IReadOnlyList<ToolDef> tools,
        CancellationToken ct);
}

// AgentReactorActor — Generating state
When<TurnReady>(t =>
{
    var call = await _llm.NextTurnAsync(_history, _tools, ct);
    if (call.Done) return GoTo(Done);
    var result = await _toolHost.ExecuteAsync(call, ct);
    _history.AppendToolResult(call, result);
    return GoTo(Thinking);   // continuation cycle
});

3. Voice as the synthesis — a free stack mimicking a paid one

mic ──▶ Whisper.net (local GGML) ──▶ AgentReactorActor ──▶ chunked text out ──▶ Windows SAPI / OpenAI TTS ──▶ speakers
                  STT                       FSM                                       TTS

Strategic compositions — what hybrid actually buys you

What's next — Gemma 4 is the inflection point

1. Apache 2.0 licence with commercial use — the legal blocker that kept earlier open-weights models out of consumer products is gone for this family.

2. Four variants spanning phone-class to 31B — the same model identity (and therefore the same community fine-tuning ecosystem) can target a Pixel and a developer GPU. That is new.

3. A real fine-tuning ecosystem from week one — HuggingFace community fine-tunes started landing within days of the April 2 release. Specialised variants (code-only, multilingual-strong, instruction-tight) compound on a base model that's now actually base — not hobbled by licence.

4. Vendors are racing to fill the niche — NVIDIA's Nemotron 3 Nano Omni shipped 26 days later. That is not a coincidence; that is a market signal that "small, open, on-device, multimodal" is now the contested ground.

• Workflow specialists — small models fine-tuned for terminal log triage, code-review summary, voice command parsing, error-message rewriting. A 4B model that wins at its specific job against a generic 32B.

• Domain specialists — small models tuned on industry corpora (legal review, medical notes summarisation, financial compliance) where the data sovereignty argument forces on-device anyway.

• Multimodal collapse — Nemotron 3 Nano Omni's pattern (one backbone, all modalities) reaches more vendors. The voice subsystem in Project/ZeroCommon/Voice/ exists because we had to glue multiple single-modality models together; an omni-modal small model could collapse half of it.

Closing — three questions for the reader

1. Do you really think you can ship a serious AI product on top of just one of OpenAI / Anthropic / Google? If not — and most can't — then you're already running hybrid. The question is whether you've named it as a strategic capability or are still treating it as integration debt.

2. How do you prevent the on-device path from being a Potemkin demo? A small model that fails 30% of the time is worse than no small model — it trains the user to distrust the surface. AgentZero Lite uses GBNF + native templates + a deterministic FSM to keep the failure mode loud. What's your equivalent guardrail?

3. What does the playground look like a year out? This project's stated value today is being a place where top-tier and open can be benchmarked head-to-head on the same UI. If Gemma 4-class models keep landing every six weeks, the playground becomes a competitive surface — every dev with the app is, in effect, running an A/B between a free model and the model their employer pays for. That is a research artifact, and it is also a buying signal. Who wants that data?

TECH LINKS

• 𝕏 @webnori

• 📘 Akka Labs (Facebook group)

• 📝 More writings — webnori wiki