docs/PHASE2: analyze — lmcp v0.5.4 probed, transport simplified

Live-probed against lmcp v0.5.4 (boltzmann) + hossenfelder broker proxy:

Transport simpler than spec:
- lmcp only implements POST-per-RPC with Connection: close; no held-open
  SSE channel. Combined with capabilities.tools.listChanged=false, no
  client-side listener is needed in v1. Drops the planned M.get_sse
  addition to ffi/curl.lua — Phase 1's M.post covers MCP.

Bearer auth is universal across the fleet — config schema grew
auth_token (literal) and auth_env (env-var indirection) fields per
server, mirroring PHASE0 §10's key_env convention.

Streaming tool_calls delta shape verified — accumulator by `index`,
function.arguments arrives as chunked JSON-string. Matches the
formulate-phase assumption in §5.

Resolutions:
  Q17 transport abstraction — POST-only, no SSE channel for lmcp.
  Q21 error mapping       — result.isError (model-recoverable, feed
                             back as tool turn) vs JSON-RPC error
                             (unknown method/tool, transport-level).
  Q18 role:"tool" turn    — accepted at protocol level (live-probed).
                             Mistral-nemo template verification
                             blocked by the hossenfelder model-field
                             routing bug; full closure carried to
                             Phase 7 verify.

Open-end recorded in §11: the hossenfelder proxy routes every request
to the loaded fast model regardless of model field, blocking Phase 2
testing against mistral-nemo specifically. Parallel to the SSE
buffering issue at marfrit/aish#15; same root (boltzmann proxy code).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

docs/PHASE2.md

@@ -2,7 +2,7 @@
 
 **Project:** aish — AI-augmented conversational shell
 **Document:** Phase 2 Requirements, Architecture & Design Decisions
-**Status:** Formulate (pre-analysis)
+**Status:** Analyze (formulate complete; live-probed against lmcp v0.5.4 + hossenfelder proxy)
 **Date:** 2026-05-12
 
 PHASE0.md is the locked substrate; PHASE1.md is layered on top. This
@@ -49,14 +49,15 @@ Three pillars per PHASE0.md §11 row 2:
 
 | Decision | Choice | Rationale |
 |---|---|---|
-| MCP transport | HTTP+SSE per the MCP spec's HTTP transport flavor | Matches `lmcp`'s native transport. Reuses the libcurl easy interface + the Phase 1 SSE parser in `ffi/curl.lua`. Avoids spawning child processes (stdio transport requires Phase 1-style PTY plumbing per server, more moving parts). Stdio is left for a possible Phase 2.1 if a stdio-only MCP server becomes necessary. |
+| MCP transport | HTTP POST per RPC, `Connection: close` per response, **no long-lived SSE GET channel** in v1 | Analyze finding (2026-05-12): lmcp v0.5.4 only implements the trivial POST-and-respond flavor of the spec's streamable-HTTP transport. Its GET /mcp endpoint announces the POST endpoint then closes — there's no server→client notification channel to listen on. Combined with lmcp's `capabilities.tools.listChanged = false`, aish doesn't need an SSE GET listener at all for lmcp. Stdio transport is left for a possible Phase 2.1 if a stdio-only MCP server becomes necessary. |
-| MCP protocol version | `2025-03-26` (or whatever `lmcp` announces in its `initialize` response) | Track lmcp's spec target. Negotiated at connect time; aish caps at the spec version it knows. |
+| MCP protocol version | `2025-03-26` (confirmed by live probe of boltzmann:8080/mcp) | lmcp pins this in `MCP_VERSION`. aish sends the same in `initialize`; future model bumps are negotiated at connect time. |
+| MCP auth | Bearer token via `Authorization: Bearer <token>` header, per-server | Analyze finding: every lmcp deployment in mfritsche's fleet (boltzmann/hertz/pve*/nc/etc.) requires Bearer auth. Phase 2 config supports `auth_token` literal and `auth_env` env-var indirection per server (mirrors `key_env` in the models registry). lmcp servers without auth (broglie/higgs LAN-only) just leave the field nil. |
 | Tool-call wire format | OpenAI `tools` field on `/v1/chat/completions` body; `tool_calls` on assistant deltas; `role:"tool"` turn with `tool_call_id` for results | Standard, supported by llama.cpp and OpenRouter. Aligns with the existing `/v1/chat/completions` substrate invariant. |
 | Tool namespacing | `<server-alias>.<tool-name>` for both the wire-level tool name and `:mcp tools` listing | Avoids name collisions across servers. The alias comes from the config key or the connect URL hash. |
 | `CMD:` coexistence with tool-calls | Both stay live, no policy preference. Substrate invariant §3 unchanged. | Resolves Q6 (see §10). `CMD:` is the local-shell route; MCP tools are structured-API routes; they serve different purposes. Future phases (Norris, Phase 3) may prefer tools when both are available, but Phase 2 doesn't enforce. |
 | Authorization default | Per-call confirm (mirrors PHASE0.md §10 `confirm_cmd` for shell) | Conservative default; user can opt into auto-approval per tool or per server via config. Resolves Q8. |
 | System prompt augmentation | Hybrid: static frame in `broker.lua` system prompt + dynamic `tools` array in the request body | Tool list goes in the API field where it belongs; the system prompt only mentions that tools exist and how to use them. Per-request body cost is bounded (tools change rarely; small schemas). Resolves Q9. |
-| Tool-call streaming | Streaming-from-day-one — `broker.chat_stream`'s on_delta callback widens to handle `tool_calls` deltas in addition to text deltas | Resolves Q10. Phase 1 SSE landed first, so we're not retrofitting; we just extend the parser. |
+| Tool-call streaming | Streaming-from-day-one — `broker.chat_stream`'s on_delta callback widens to handle `tool_calls` deltas in addition to text deltas | Resolves Q10. Phase 1 SSE landed first, so we're not retrofitting; we just extend the parser. **Wire shape confirmed at analyze** (2026-05-12 probe vs hossenfelder): `delta.tool_calls[]` arrives indexed; id+type+function.name appear on the opening delta; `function.arguments` is a JSON-string that arrives in character-fragment chunks; finish_reason "tool_calls" closes the call. Accumulator strategy matches §5. |
 | Tool-call concurrency | Sequential dispatch in Phase 2 v1 — process `tool_calls[0]` to completion, then `[1]`, etc. | Simpler error handling; tool effects often order-dependent (e.g. write-then-read). Parallel dispatch deferred (see Q20). |
 | MCP server lifecycle | aish does not manage MCP server processes (parallel to PHASE0.md §12 llama.cpp rule) | Declared in config or connected by URL; aish is a client only. |
 
@@ -66,14 +67,14 @@ Three pillars per PHASE0.md §11 row 2:
 
 | File | State after Phase 1 | Phase 2 changes |
 |---|---|---|
-| `mcp.lua` | **New file** (not in PHASE0 §4 layout; this Phase amends the layout to add it) | Implement: `M.connect(url, alias) -> session`, `session:initialize()`, `session:list_tools() -> [{name, description, schema}]`, `session:call_tool(name, args) -> result`, `session:close()`. JSON-RPC 2.0 over HTTP POST for client→server, SSE GET for server→client notifications. Per-session state: connected, tools-cache, pending request ID counter. |
+| `mcp.lua` | **New file** (not in PHASE0 §4 layout; this Phase amends the layout to add it) | Implement: `M.connect(url, opts) -> session` (opts: `alias`, `auth_token`, `auth_env`), `session:initialize()`, `session:list_tools() -> [{name, description, schema}]`, `session:call_tool(name, args) -> result | tool_error`, `session:close()`. JSON-RPC 2.0 over HTTP POST (`Content-Type: application/json`, `Accept: application/json`, `Authorization: Bearer <token>`). Per-session state: alias, base-url, auth, tools-cache, request-ID counter. No persistent SSE channel — POST is one-shot per RPC. |
 | `safety.lua` | Stub | Implement Phase 2 surface only: `M.confirm_tool_call(tool_name, args, policy) -> bool`. Reads `config.mcp.auto_approve` (per-tool and per-server) before prompting. Norris destructive-op heuristic and HALT gate stay Phase 3. |
 | `broker.lua` | Streaming `chat_stream(cfg, msgs, on_delta)` | Request body grows `tools = mcp.tools_schema()` (assembled from all connected sessions). on_delta callback widens to `on_delta(kind, payload)` where `kind ∈ {"text", "tool_call"}`; tool_call payload includes id+name+arguments-delta. `M.chat` wrapper updates to buffer both. |
 | `context.lua` | turns = {{role, content}, ...} + `pending_exec_output` | New role: `"tool"`. Assistant turns may carry `tool_calls = [{id, name, arguments}]`. `to_messages()` flattens these into OpenAI-shape messages. Alternation rules: assistant-with-tool_calls is followed by N tool turns (one per call), then assistant text. |
 | `repl.lua` | meta cmds + ask_ai stream loop | After ask_ai sees `tool_calls`, enter a tool-execution sub-loop: confirm-gate each call via `safety.confirm_tool_call`, dispatch via `mcp.session:call_tool`, append tool turn to context, re-issue the broker request. Loop until assistant emits text without tool_calls. New meta: `:mcp connect <url> [alias]`, `:mcp list`, `:mcp tools`, `:mcp disconnect <alias>`. |
 | `renderer.lua` | streaming text + exec frame | Add `tool_call_begin(name, args)`, `tool_call_end(result, ok)`. Visual style: indented, dim, parallel to the exec frame. |
 | `config.lua` | example with models/shell/context/history | Schema additions: `mcp = { servers = { alias = { url = "..." } }, auto_approve = { ["alias.tool"] = true } }`. Documented in §10 below. |
-| `ffi/curl.lua` | post + post_sse | One probable addition: GET request helper (for the server→client SSE channel of MCP transport). Or wrap the SSE-GET inside `mcp.lua` directly if it's tight enough. Decided at analyze. |
+| `ffi/curl.lua` | post + post_sse | **No additions in v1** — analyze finding ruled out the long-lived SSE GET channel for lmcp. Phase 1 `M.post` (already does sync POST with response capture) is sufficient for MCP JSON-RPC. |
 | `history.lua` | JSONL session log | Tool turns are logged like any other turn — `{role:"tool", tool_call_id:"...", content:"..."}`. Resume reconstructs them via `ctx:append` like user/assistant turns. |
 
 §4 module-layout amendment: `mcp.lua` slots between `broker.lua` and
@@ -81,57 +82,66 @@ Three pillars per PHASE0.md §11 row 2:
 
 ---
 
-## 4. MCP Transport
+## 4. MCP Transport (analyze findings — lmcp v0.5.4)
 
-`lmcp` exposes an HTTP+SSE transport. From a client perspective there are
+lmcp implements only the **synchronous POST** flavor of the MCP
-two channels per session:
+streamable-HTTP spec. Each RPC is one HTTP transaction:
 
 ```
-client → server:   POST <base-url>     Content-Type: application/json
+client → server:   POST /mcp           Content-Type: application/json
+                                       Accept: application/json
+                                       Authorization: Bearer <token>
                                        Body: { jsonrpc:"2.0", id, method, params }
                                        Returns: { jsonrpc, id, result | error }
-
+                                       Connection: close
-server → client:   GET  <base-url>     Accept: text/event-stream
-                                       (held open; receives notifications/...)
 ```
 
-### Handshake (per MCP spec)
+lmcp's `GET /mcp` exists but only sends a one-shot `event: endpoint`
+announcing the POST URL, then closes — there is no held-open
+server→client channel. Combined with the `listChanged: false`
+capability lmcp announces in `initialize`, **aish does not open a
+persistent SSE channel** to lmcp servers in v1. Notifications-from-server
+are out of scope here; track for v2 if a richer server appears.
 
-1. `initialize` request: `{ protocolVersion, capabilities, clientInfo }`.
+### Handshake
-2. Server response: `{ protocolVersion, capabilities, serverInfo }`. Cache
+
-   the announced server capabilities — only invoke RPCs the server says
+1. `initialize` request: `{ protocolVersion: "2025-03-26", capabilities: {}, clientInfo: { name: "aish", version: "..." } }`.
-   it supports (e.g. `tools` capability).
+2. Server response (lmcp): `{ protocolVersion: "2025-03-26", capabilities: { tools: { listChanged: false } }, serverInfo: { name, version } }`.
-3. `notifications/initialized` (one-way notification) signals end of
+3. `notifications/initialized` POST (one-way; lmcp returns HTTP 202 with no body).
-   handshake.
 
 ### Tool discovery
 
 1. `tools/list` RPC → `{ tools: [{ name, description, inputSchema }] }`.
-2. Cache per-session. Re-fetch on `notifications/tools/list_changed`
+2. Cache per-session **for the session lifetime** — lmcp announces
-   from the SSE channel.
+   `listChanged: false`, so there's no need to refetch or listen for
+   change notifications.
 
 ### Tool invocation
 
-1. `tools/call` with `{ name, arguments }` → `{ content: [{type, text}], isError }`.
+`tools/call` with `{ name, arguments }`. lmcp distinguishes two failure
-2. The full text payload flows back to the model as the next `role:"tool"`
+modes:
-   turn's content.
 
-### SSE channel
+- **Tool-handler exception** → JSON-RPC `result` with `isError: true`
+  and `content: [{ type:"text", text: "Error: ..." }]`. **Model-recoverable**:
+  feed it back to the model as the next `role:"tool"` turn content and
+  let it react.
+- **Unknown method or unknown tool** → JSON-RPC `error` with
+  `code: -32601` ("Method not found" / "Tool not found"). **Transport
+  level**: surface as an aish status, do not feed to model.
 
-Held open via `ffi/curl.post_sse` style — but using a GET (Phase 2 likely
+This split resolves Q21.
-needs `M.get_sse` mirroring the POST variant). Each event is a JSON-RPC
-notification. Phase 2 v1 only listens for `notifications/tools/list_changed`;
-other notification types (progress, log) are ignored and tracked for
-later phases.
 
 ### Lifecycle
 
 - Connect on startup (from `config.mcp.servers`) — best effort; failures
-  are status-logged, don't abort aish.
+  are status-logged, don't abort aish. "Connect" here means: do the
-- `:mcp connect <url>` adds a session at runtime; alias auto-derived from
+  `initialize` round-trip + cache `tools/list` results.
-  hostname or supplied as second arg.
+- `:mcp connect <url>` adds a session at runtime; alias auto-derived
-- `:mcp disconnect <alias>` closes both channels.
+  from hostname or supplied as second arg.
-- On aish quit, all sessions are closed cleanly (best effort).
+- `:mcp disconnect <alias>` drops cached state. There's no long-lived
+  HTTP connection to close (every RPC was already `Connection: close`).
+- On aish quit, sessions are just forgotten — nothing to clean up
+  server-side.
 
 ---
 
@@ -228,22 +238,36 @@ function M.confirm_tool_call(name, args, cfg)
 end
 ```
 
-Config schema:
+Config schema (analyze-revised — Bearer auth fields added):
 
 ```lua
 mcp = {
     servers = {
-        local_fs = { url = "http://localhost:9700/" },
+        boltzmann = {
-        gitea    = { url = "http://localhost:9701/" },
+            url       = "http://boltzmann.fritz.box:8080/mcp",
+            auth_env  = "BOLTZMANN_MCP_TOKEN",  -- read from env at startup
+        },
+        broglie = {
+            url = "http://broglie.fritz.box:8080/mcp",
+            -- no auth (LAN-only deployment)
+        },
+        nc = {
+            url        = "https://nc.reauktion.de:8080/mcp",
+            auth_token = "literal-token-if-not-using-env",  -- alternative
+        },
     },
     auto_approve = {
-        ["local_fs.read_file"] = true,    -- specific tool
+        ["boltzmann.read_file"] = true,    -- specific tool
-        ["gitea.*"]            = true,    -- whole server
+        ["broglie.*"]           = true,    -- whole server
     },
     max_tool_depth = 8,
 }
 ```
 
+Auth precedence per server: `auth_token` literal > `auth_env` indirection
+> nil (no Authorization header sent). Mirrors PHASE0 §10's `key_env`
+convention for cloud model API keys.
+
 Norris mode (Phase 3) will extend this: when autonomous, the destructive-op
 heuristic decides; for non-destructive tools, auto_approve. Outside scope here.
 
@@ -328,12 +352,15 @@ Specifically out of Phase 2 scope despite proximity:
 
 | # | Question | Impact | Resolve by |
 |---|---|---|---|
-| Q17 | MCP transport abstraction: design `mcp.lua` from day one for both HTTP+SSE and stdio transports (transport_t interface), or hard-code HTTP+SSE and refit if a stdio-only server appears? | mcp.lua API shape | Phase 2 (plan) |
+| Q17 | ~~MCP transport abstraction: stdio vs HTTP+SSE~~ | mcp.lua API shape | **Resolved at analyze.** Hard-code POST-only HTTP for v1. lmcp doesn't use the long-lived SSE channel and `listChanged: false` removes any v1 need for it. Stdio transport tracked as Phase 2.1 / out-of-scope here. |
-| Q18 | Tool result re-injection: standard OpenAI `role:"tool"` turn, or `[tool: X]` prefix to next user turn (matching the §6 exec-output pattern)? Strict chat templates may reject `tool` role — needs verification against mistral-nemo specifically. | context.lua + broker.lua | Phase 2 (analyze) |
+| Q18 | Tool-result re-injection: standard OpenAI `role:"tool"` turn, or `[tool: X]` prefix to next user turn (matching the §6 exec-output pattern)? | context.lua + broker.lua | **Partly resolved.** Live probe (2026-05-12, hossenfelder) shows `role:"tool"` accepted by the proxy + the loaded model (qwen2.5-coder-1.5b). Mistral-nemo-specific template testing is **blocked** by the hossenfelder proxy routing all `model` field values to the loaded fast model — see open-end below. Default v1 path: `role:"tool"` (standard); fallback to `[tool: X]` prefix is plumbed but unused unless a strict template rejects it during Phase 7 verify. |
-| Q19 | Large tool-result payloads (file reads, query dumps): pass-through, truncate at N chars, or summarize via fast model? Token-budget pressure scales with tool use. | context.lua + executor of tool-result | Phase 2 (plan); Phase 4 may refine with memory.jsonl |
+| Q19 | Large tool-result payloads: pass-through, truncate at N chars, or summarize via fast model? | context.lua + executor of tool-result | Phase 2 (plan); Phase 4 may refine with memory.jsonl |
-| Q20 | Parallel `tool_calls`: sequential v1 is safe; spec allows parallel. Move to parallel when both calls are read-only (declared in tool metadata)? | mcp.lua dispatch | Phase 2 (verify) — track for v2 |
+| Q20 | Parallel `tool_calls`: sequential v1 is safe; spec allows parallel. Move to parallel when both calls are read-only? | mcp.lua dispatch | Phase 2 (verify) — track for v2 |
-| Q21 | MCP server error mapping: JSON-RPC `error` response → tool-result content with `isError=true` (model can react), or aish-level transport error (broker aborts)? | mcp.lua + broker.lua | Phase 2 (plan) |
+| Q21 | ~~MCP error mapping~~ | mcp.lua + broker.lua | **Resolved at analyze.** lmcp distinguishes: `result.isError=true` (handler exception, model-recoverable, feed back as tool turn content) vs JSON-RPC `error` (unknown method/tool, transport-level, surface as aish status). See §4. |
-| Q22 | aish's own command surface as an MCP server (eat your own dog food: expose `aish.exec`, `aish.read_file`, etc. via MCP so other clients can drive aish)? | scope expansion / new module | **Out of Phase 2.** Tracked for "maybe Phase 4 or later"; flagging here so it's not silently lost. |
+| Q22 | aish's own command surface as an MCP server | scope expansion | **Out of Phase 2.** Parked for Phase 4+ if interest stays. |
+
+Open-end carried forward to Phase 7 (verify):
+- **Hossenfelder proxy `model`-field bug** (separate from aish): the proxy at `:8082` routes all requests to the loaded fast model regardless of the request's `model` field — chunks return `"model":"qwen2.5-coder-1.5b-q4_k_m.gguf"` even when `mistral-nemo-12b-instruct` was asked for. This **blocks live-verification of mistral-nemo's chat-template tool-role behavior**. Fix lives in boltzmann (parallel to the SSE-buffering bug tracked at [aish#15](https://git.reauktion.de/marfrit/aish/issues/15)). Phase 7 needs the proxy fix to fully close Q18.
 
 Resolved at formulate (above in §2 table):
 - Q6 (CMD: vs tools coexistence) — both, no policy preference, substrate unchanged.
@@ -342,6 +369,10 @@ Resolved at formulate (above in §2 table):
 - Q9 (system-prompt augmentation) — hybrid: static frame + dynamic `tools` body field.
 - Q10 (tool-call streaming) — streaming-from-day-one on top of Phase 1 SSE.
 
+Resolved at analyze (2026-05-12, live probes vs lmcp v0.5.4 + hossenfelder):
+- Q17 (transport abstraction) — POST-only, no SSE channel needed for lmcp.
+- Q21 (error mapping) — isError vs JSON-RPC error split per §4.
+
 ---
 
 *End of Phase 2 Manifest — aish*