broker + repl + safety: GBNF grammar-sampling passthrough (closes #88)

llama.cpp constrains the sampler to ONLY emit tokens matching a
GBNF grammar. For small models this kills format drift at the
token level — `CMD: <cmd>` is enforced by the sampler rather than
hoped for via prompt discipline.

Probe finding (this commit's pre-implementation): cloud (Anthropic
via Bedrock) silently IGNORES the `grammar` field — returns normally
via standard sampling. Default passthrough is safe for all routes;
no per-model opt-in/opt-out needed in v1.

Changes:

- broker.lua build_request: `if opts.grammar then req.grammar =
  opts.grammar end`. Misformed grammar surfaces at request time
  via the existing transport-error path.

- repl.lua ask_ai: `grammar_override = config.routing.grammars
  [req_class]` (same gating shape as #86's system_prompts override).
  Passed via opts.grammar in the call_broker invocation.

- safety.lua is_destructive threads cfg.safety.probe_grammar through
  opts.grammar so llm_probe constrains the YES/NO output. Skips
  the regex-match dance entirely when the model can't drift.
  Caller-provided opts.grammar takes precedence over cfg.

- config.lua gains two commented examples:
  * routing.grammars per class
  * safety.probe_grammar for the destructive probe

6 unit cases verified (stubbed curl.post_sse / broker.chat):
  - default: no grammar in body
  - opts.grammar -> body contains grammar JSON-encoded
  - safety probe_grammar reaches llm_probe via opts
  - no probe_grammar configured -> opts.grammar nil
  - caller opts.grammar takes precedence over cfg.safety.probe_grammar

E2E against live local broker:
  - `routing.grammars.default = "root ::= \\"ACK\\""` configured;
    prompted "tell me a long story about a fox" -> model output
    EXACTLY "ACK" (sampler forced; would normally produce paragraphs).
    Grammar passthrough end-to-end confirmed.

Regression: test_safety 87/87, test_router_model 31/31, repl loads.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
2026-05-17 07:00:36 +00:00
parent 047d629a66
commit 74e4bffb37
4 changed files with 64 additions and 3 deletions
+10 -1
View File
@@ -999,6 +999,14 @@ function M.run(config)
and config.routing.system_prompts
and req_class
and config.routing.system_prompts[req_class]
-- #88: per-class GBNF grammar passthrough. llama.cpp constrains
-- the sampler to only emit tokens matching the grammar — kills
-- format drift on small models. Cloud silently ignores the
-- field (probed Anthropic/Bedrock returns normally).
local grammar_override = config.routing
and config.routing.grammars
and req_class
and config.routing.grammars[req_class]
local depth = 0
local final_resp = ""
@@ -1030,7 +1038,8 @@ function M.run(config)
tool_calls_seen[#tool_calls_seen + 1] = payload
end
end,
{ tools = tools_schema(), category = "main" })
{ tools = tools_schema(), category = "main",
grammar = grammar_override })
if rehydrator then
local tail = rehydrator:flush()
if tail ~= "" then