Foundation for Phase 7. broker.chat_stream now emits a third
on_delta kind ("usage") after the stream completes successfully;
broker.chat returns (text, usage). Backward-compatible — existing
callers that ignore the new kind / second value continue working
via Lua's drop-extra-returns semantics.
Changes:
- build_request widens (A3 + R3) — `(model_cfg, msgs, stream, opts)`.
opts.tools / opts.max_tokens / opts.include_usage / opts.category
all live inside opts now. Both internal call sites updated.
- opts.include_usage defaults to true for streaming requests; sets
`stream_options: { include_usage: true }` in the request body.
B1: required for local llama.cpp to emit usage; cloud honors as
a no-op (emits anyway).
- on_event captures `doc.usage` into a closure-local `final_usage`.
N1: the check is INDEPENDENT of the choice/delta branches — local
emits usage on choices=[] chunks (choice nil) while cloud emits
with non-empty choices + finish_reason. Both shapes funnel here.
- After curl.post_sse returns successfully (NOT on transport/api
errors), if final_usage is set, emit on_delta("usage", {prompt_tokens,
completion_tokens, total_tokens, cost, model, category}). cost is
nil for local (R6 preserves the nil vs 0 distinction the
accumulator needs). model is model_cfg.model — caller-stable per
B4 + R2 so call_broker's fallback retry attributes usage to the
fallback's model name without wrapper-side tracking.
- M.chat (R1 — BLOCKER fix): on_delta now also captures kind=="usage"
alongside "text"; M.chat returns (text, usage). Without this fix
4 of 5 non-streaming categories (summarize / delegate /
memory_summarize / probe) would silently report zero usage.
Smoke verified against live hossenfelder:8082:
- CLOUD chat -> (text, usage); cost=2.9e-05, model=anthropic/...
- LOCAL chat -> (text, usage); cost=NIL (correct per R6),
model=qwen-coder-7b-snappy-8k
- CLOUD stream -> on_delta("usage", {...}) with category="test"
echoed; model name caller-stable.
Regression: test_safety 87/87, test_router_model 31/31, repl loads.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Phase 3 commit #2 per docs/PHASE3.md §12. Adds the LLM-probe gate on
top of commit #1's static patterns. Together they form is_destructive.
broker.lua extension:
- opts.max_tokens (A2) — passed through to the request body. Phase 3
probes cap at 4 tokens for YES/NO replies.
- opts.timeout_ms — overrides model_cfg.timeout_ms per-call. Probe
uses 15000ms cap regardless of the model's normal timeout
(the user's deep model has 1800000ms for long generations; the
probe must stay snappy).
- M.chat now accepts an opts table (same shape as chat_stream's).
Backwards compatible — existing callers passing (cfg, msgs)
unaffected.
safety.lua additions:
- llm_probe(cfg, system, cmd): single broker.chat call returning
"YES"/"NO"/"YES_FAILSAFE"/"YES_UNPARSEABLE" — fail-safe defaults.
- llm_second_opinion(cmd, cfg): two-probe protocol per R-B2.
Probe 1: "Is this destructive?" — YES → flag.
Probe 2 (only if probe 1 said NO): "Is this safe?" inverted
question — NO → flag (disagreement = HALT).
Both NO → safe.
- Session-scoped cache _llm_cache keyed by normalized command
(lowercased + whitespace-collapsed). Mitigates Q23 latency for
repeated commands within a Norris run.
- Model-selection precedence: cfg.safety.llm_model (explicit)
→ cfg.models.deep (independent local class) → cfg.models[default].
Fail-safe YES if none configured.
- is_destructive(cmd, cfg): runs static patterns first (always),
then LLM if cfg present + not explicitly opted-out. cfg=nil
yields static-only mode (handy for tests).
End-to-end verified against hossenfelder using qwen-coder-7b-32k as
the deep probe (qwen3-30b-a3b-instruct in repo's config.lua isn't
currently loaded on the local backend):
cat /etc/hostname → hit=false (LLM: NO, NO inverted = safe)
rm /tmp/x.log → hit=true (LLM flagged; static missed
because no -r/-f flags)
cp /etc/passwd /tmp/passwd.bak → hit=false (safe copy)
cache: second probe on same cmd → 0s wall time
static-only (cfg=nil): rm -rf /tmp/x → static hit, no LLM call
opt-out (llm_second_opinion=false): cp x y → hit=false, no probe
Test corpus (test_safety.lua, 87 cases) still all pass — cfg=nil
preserves the static-only behavior.
Note: production config.lua currently has `deep = qwen3-30b-a3b-instruct`
which isn't loaded on the proxy backend right now; Norris users will
hit the fail-safe (everything flagged destructive) until either the
deep model is brought up OR cfg.safety.llm_model = "cloud" is set
to route the probe through anthropic/claude-haiku-4.5. Update the
config or model deployment for production use — covered by Phase 3
verify test case.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Phase 2 commit #5 per docs/PHASE2.md §12. Streaming broker grows
tool-call support without taking a dependency on mcp.lua (caller
supplies the tools array — B5 from review).
chat_stream signature widens to (cfg, msgs, on_delta, opts):
opts.tools - optional array, passed to the request body as the
OpenAI-shape tools field. OMITTED entirely when nil or
empty (#tools == 0) — some servers reject "tools": [].
on_delta callback shape widens to (kind, payload):
kind = "text", payload = string (Phase 1 path; unchanged
semantics, signature
changes from (delta) to
("text", delta))
kind = "tool_call", payload = {id, name, arguments}
emitted ONCE per call on
finish_reason "tool_calls"
after the streaming
accumulator pulls
fragmented JSON-string
arguments together.
Accumulator behavior:
- Keyed by delta.tool_calls[i].index.
- If index is absent on a delta (some llama.cpp builds omit it on
single-call streams; C2 in review), default to 0 with a one-shot
stderr debug status per stream.
- id and name captured from the opening delta of each slot.
- function.arguments concatenated across all deltas as the raw
JSON-string; caller (repl.lua / future Phase 2 commit #6) does
dkjson.decode.
- On finish_reason "tool_calls" the accumulator emits all collected
calls in index order and resets.
M.chat external contract unchanged (C1): wrapper now uses the new
(kind, payload) shape internally but exposes the same text-string
return. No caller of M.chat passes opts.tools so tool_call kinds are
silently dropped.
repl.lua minimal companion edit: ask_ai's chat_stream callback updated
to the new shape. Text path unchanged; tool_call kinds are no-op
placeholders until commit #6 lands the sub-loop. Keeps Phase 1 streaming
functional between #5 and #6.
Smoke-tested against hossenfelder/8082 (post-#23 fix):
- text-only: ok=true, kind="text" deltas received
- with opts.tools: model emitted one tool_call,
accumulator collected id + name=get_weather + args={"city":"Paris"}
correctly across fragmented deltas
- opts.tools={}: server accepted (field omitted as required)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Phase 1 streaming consumer per PHASE1.md §3.
broker.chat_stream(model_cfg, messages, on_delta) -> true | (nil, err)
broker.chat(model_cfg, messages) -> content | (nil, err)
(now a thin buffer over
chat_stream)
The HTTP shape unifies on stream:true. on_event from ffi/curl.post_sse
decodes each event's JSON, extracts choices[1].delta.content, and calls
on_delta(content) for non-empty string deltas. The `[DONE]` sentinel is
filtered. SSE-framed error envelopes ({"error":{"message":...}} arriving
as data:) surface as "api: ..." errors.
build_request is factored out so chat_stream and (future) any
non-streaming consumer share URL/body/header construction.
Live verification against hossenfelder fast preset:
- chat_stream("Count one to five..."): 9 incremental deltas streamed
token-by-token, assembled to "1 2 3 4 5"
- chat("Reply with exactly: pong"): "pong" returned via buffer
Error envelope path is correct by inspection but not exercised live —
hossenfelder passes through bogus model names rather than rejecting.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Phase 0 implementation per PHASE0.md §6.
M.chat(model_cfg, messages) -> content_string | (nil, errmsg)
Builds the OpenAI-compat JSON body:
{ model, messages, stream: false, temperature: model_cfg.temperature ?? 0.2 }
Sends Content-Type and (optionally) Authorization Bearer pulled from
model_cfg.key_env's process environment. Default timeout 60s; overridable
per-model via model_cfg.timeout_ms.
Error surfaces split:
"transport: ..." curl-side (TCP/TLS/timeout)
"decode: ..." non-JSON response body
"api: ..." OpenAI-style { error: { message } } envelope
"broker.chat: no choices[1].message.content..." shape miss
Tested against four canned mock responses (nc -lN listener feeding
HTTP/1.0 + Connection: close so EOF terminates the body): happy path,
api error envelope, raw-text non-JSON, empty choices[]. The on-wire
request body verified as well: POST path, headers, model/messages/
temperature/stream JSON.
Live test against a real llama.cpp/hossenfelder endpoint deferred per
issue #12 (broker endpoint configuration).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- README, .gitignore, CLAUDE.md (project conventions)
- docs/PHASE0.md — full Phase 0 manifest (locked substrate)
- 10 root .lua modules + 4 ffi/ bindings, all stubs raising NotImplemented
with module-scoped responsibilities matching the manifest
- config.lua wired to current dirac/hossenfelder endpoints (qwen-coder-7b
snappy/32k + cloud via OpenRouter through hossenfelder)
File names match docs/PHASE0.md §4 exactly. Module bodies fill in across
later phases; the tree shape is locked.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
marfrit
claude-noether