broker + repl + safety: GBNF grammar-sampling passthrough (closes #88)
llama.cpp constrains the sampler to ONLY emit tokens matching a GBNF grammar. For small models this kills format drift at the token level — `CMD: <cmd>` is enforced by the sampler rather than hoped for via prompt discipline. Probe finding (this commit's pre-implementation): cloud (Anthropic via Bedrock) silently IGNORES the `grammar` field — returns normally via standard sampling. Default passthrough is safe for all routes; no per-model opt-in/opt-out needed in v1. Changes: - broker.lua build_request: `if opts.grammar then req.grammar = opts.grammar end`. Misformed grammar surfaces at request time via the existing transport-error path. - repl.lua ask_ai: `grammar_override = config.routing.grammars [req_class]` (same gating shape as #86's system_prompts override). Passed via opts.grammar in the call_broker invocation. - safety.lua is_destructive threads cfg.safety.probe_grammar through opts.grammar so llm_probe constrains the YES/NO output. Skips the regex-match dance entirely when the model can't drift. Caller-provided opts.grammar takes precedence over cfg. - config.lua gains two commented examples: * routing.grammars per class * safety.probe_grammar for the destructive probe 6 unit cases verified (stubbed curl.post_sse / broker.chat): - default: no grammar in body - opts.grammar -> body contains grammar JSON-encoded - safety probe_grammar reaches llm_probe via opts - no probe_grammar configured -> opts.grammar nil - caller opts.grammar takes precedence over cfg.safety.probe_grammar E2E against live local broker: - `routing.grammars.default = "root ::= \\"ACK\\""` configured; prompted "tell me a long story about a fox" -> model output EXACTLY "ACK" (sampler forced; would normally produce paragraphs). Grammar passthrough end-to-end confirmed. Regression: test_safety 87/87, test_router_model 31/31, repl loads. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
+23
@@ -260,6 +260,29 @@ return {
|
||||
-- Do not ask clarifying questions.]],
|
||||
-- -- reasoning routes to cloud; no override usually needed
|
||||
-- },
|
||||
--
|
||||
-- -- Issue #88: per-class GBNF grammar passthrough. llama.cpp
|
||||
-- -- constrains the sampler to ONLY emit tokens matching the
|
||||
-- -- grammar — eliminates format drift on small models. Cloud
|
||||
-- -- (Anthropic/Bedrock) silently ignores the field, so default
|
||||
-- -- passthrough is safe; no per-model opt-out needed. Misformed
|
||||
-- -- grammar surfaces as a broker error at request time.
|
||||
-- grammars = {
|
||||
-- code = [[root ::= "CMD: " [^\n]+ "\n"]],
|
||||
-- default = [[root ::= ("CMD: " [^\n]+ "\n") | [^\n]+ "\n"]],
|
||||
-- },
|
||||
-- },
|
||||
--
|
||||
-- Issue #88 (continued): for the safety LLM probe (YES/NO
|
||||
-- destructive classification), set safety.probe_grammar to force
|
||||
-- the probe model to emit exactly YES or NO. Eliminates the
|
||||
-- regex-match fallback for unparseable verdicts; small models
|
||||
-- become reliable enough to use as the probe.
|
||||
--
|
||||
-- safety = {
|
||||
-- llm_second_opinion = true,
|
||||
-- llm_model = "fast",
|
||||
-- probe_grammar = [[root ::= ("YES" | "NO")]],
|
||||
-- },
|
||||
|
||||
-- ── Phase 5 context summarization on sliding-window eviction.
|
||||
|
||||
Reference in New Issue
Block a user