An LLM Is Not an Agent: Five Layers Between a Local Model and a Real Conversation

TL;DR — A local on-device LLM is, by itself, a token generator. It is not an agent. AgentZero Lite turned a Gemma-4 running on a user's machine into something that can hold a real conversation with another AI (Anthropic's Claude CLI in a neighbouring terminal tab) by stacking five distinct layers on top of inference. This is the field report.

The premise: an LLM alone is not an agent

flowchart BT
  L[Local LLM<br/>tokens out]
  S[+ Structure<br/>GBNF grammar OR native tool template]
  T[+ Tools<br/>5-tool catalog over real ConPTY]
  C[+ Concurrency<br/>off-UI-thread reactor actor]
  P[+ Protocol<br/>identity-first handshake, peer-name routing]
  D[Discipline harness<br/>research before code]
  A((Agent))
  L --> S --> T --> C --> P --> A
  D -.-> A

1. A harness before a feature

2. Prompts ask. Grammars constrain.

root      ::= ws "{" ws "\"tool\"" ws ":" ws toolname ws ","
                 ws "\"args\"" ws ":" ws args ws "}" ws

toolname  ::= "\"list_terminals\"" | "\"read_terminal\""
            | "\"send_to_terminal\"" | "\"send_key\""
            | "\"wait\"" | "\"done\""

args      ::= "{" ws "}" | "{" ws kv (ws "," ws kv)* ws "}"
kv        ::= string ws ":" ws value
value     ::= string | integer | boolean

3. The terminal is just five tools

flowchart TD
  U[User prompt] --> L[AgentToolLoop]
  L -->|system + grammar/template| G{LLM backend}
  G -->|Gemma 4 path| GBNF[GBNF sampler<br/>5-tool catalog]
  G -->|Nemotron path| TPL[Llama-3.1 chat template<br/>native tool channel]
  GBNF --> P[Parse JSON]
  TPL --> P
  P --> H[IAgentToolHost<br/>WorkspaceTerminalToolHost]
  H -->|tool result| L
  L -->|loop until 'done'<br/>or MaxIterations cap| L

4. Inference belongs in an actor, not a window

5. Identity is the protocol

private static string BuildFirstContactHeader(string peerName = "Claude") =>
    "[AgentBot Handshake — first contact, please read carefully]\n\n"
  + "You are " + peerName + " and I am AgentBot, "
  + "an on-device AI agent running inside the AgentZero Lite shell.\n"
  + "\"" + peerName + "\" is the name of YOUR terminal tab — "
  + "both of us will use that name to refer to you, and you will use it "
  + "as your --from identity when replying.\n\n"
  + "Step 1 — Verify the CLI exists.\n"
  + "Step 2 — Acknowledge using the same CLI.\n"
  + "    AgentZeroLite.exe -cli bot-chat \"DONE(handshake-ok)\" --from "
  + peerName + "\n";

public enum HandshakeState
{
    NotConnected,    // never sent intro
    HandshakeSent,   // intro dispatched, awaiting bot-chat reply
    Connected,       // peer has called back at least once
}

public sealed record TerminalSentToBot(string PeerName, string Text);
public sealed record MarkHandshakeSent(string PeerName);
public sealed record QueryHandshakeState(string PeerName);
public sealed record HandshakeStateReply(string PeerName, HandshakeState State);

sequenceDiagram
  participant U as User
  participant B as AgentBot<br/>(local Gemma)
  participant R as AgentReactorActor<br/>(Akka, off-UI-thread)
  participant H as WorkspaceTerminalToolHost
  participant P as Peer terminal<br/>(Claude CLI)

  U->>B: "Talk to the Claude tab"
  B->>R: StartReactor (Mode 2)
  R->>H: send_to_terminal(<handshake intro>)
  H->>P: "You are <peerName>, I am AgentBot.<br/>Verify CLI: -cli help.<br/>Ack via bot-chat DONE(handshake-ok)"
  Note over R,H: MarkHandshakeSent
  P->>P: runs -cli help, confirms
  P->>B: AgentZeroLite.exe -cli bot-chat<br/>"DONE(handshake-ok)" --from <peer>
  B->>R: TerminalSentToBot (peer signal)
  Note over R: HandshakeState → Connected<br/>Phase H done → Phase C
  R->>H: substantive turn
  H->>P: real exchange
  P->>B: real reply via bot-chat
  R->>U: result

The tech stack at a glance

Closing — what made the tokens become a conversation

• Structure removes "the model produced unparseable output" failures.

• Tools remove "the model has no way to act on the world" failures.

• Concurrency removes "the act of thinking blocks the act of using" failures.

• Protocol removes "the model lies about a counterparty it has not actually reached" failures.

• Discipline (the harness) removes "we built the wrong thing" failures.

TECH LINKS

• 𝕏 @webnori

• 📘 Akka Labs (Facebook group)

• 📝 More writings — webnori wiki