test-case: :route check classifies prompts correctly #46

New Issue

2026-05-13T11:36:28Z

claude-noether commented

2026-05-13 11:36:28 +00:00

Steps

Boot aish (any config; routing block doesn't need to be enabled).
Run :route check Explain how MMAP works on Linux
Run :route check Traceback (most recent call last):
Run :route check ls /tmp
Run :route check how does ASLR work?
Run :route on
Run :route check ls /tmp again

Expected

Step 2: class=reasoning model=nil (routing currently disabled) — heuristic fires but config.routing.classes is empty, so no override.
Step 3: class=code model=nil (routing currently disabled) — same.
Step 4: class=default model=nil (routing currently disabled).
Step 5: class=reasoning model=nil ... — 19 chars + "how does" hits the threshold.
Step 6: auto-routing on.
Step 7: class=default model=nil (no "routing disabled" suffix anymore).

What this exercises

router.classify_model heuristics (code-fence, traceback, length+keywords, default)
The 15-char minimum length on reasoning keywords (catches "how does X work?" without false-positive on bare "explain")
N1: :route check ALWAYS runs heuristic regardless of cfg.routing.auto
The (routing currently disabled) suffix appears when auto is off

## Steps 1. Boot aish (any config; routing block doesn't need to be enabled). 2. Run `:route check Explain how MMAP works on Linux` 3. Run `:route check Traceback (most recent call last):` 4. Run `:route check ls /tmp` 5. Run `:route check how does ASLR work?` 6. Run `:route on` 7. Run `:route check ls /tmp` again ## Expected - Step 2: `class=reasoning model=nil (routing currently disabled)` — heuristic fires but config.routing.classes is empty, so no override. - Step 3: `class=code model=nil (routing currently disabled)` — same. - Step 4: `class=default model=nil (routing currently disabled)`. - Step 5: `class=reasoning model=nil ...` — 19 chars + "how does" hits the threshold. - Step 6: `auto-routing on`. - Step 7: `class=default model=nil` (no "routing disabled" suffix anymore). ## What this exercises - router.classify_model heuristics (code-fence, traceback, length+keywords, default) - The 15-char minimum length on reasoning keywords (catches "how does X work?" without false-positive on bare "explain") - N1: :route check ALWAYS runs heuristic regardless of cfg.routing.auto - The (routing currently disabled) suffix appears when auto is off

claude-noether added the test-case label 2026-05-13 11:36:28 +00:00

claude-noether commented

2026-05-13 11:50:33 +00:00

PASS (autonomous run, 2026-05-13). 4-case classification corpus: Explain how MMAP works on Linux → class=reasoning; Traceback (most recent call last): → class=code; ls /tmp → class=default; how does ASLR work? → class=reasoning. (routing currently disabled) suffix appears on all 4 (auto off). After :route on, the suffix correctly disappears from subsequent :route check output (verified via direct probe). N1 honored. Closing.

**PASS** (autonomous run, 2026-05-13). 4-case classification corpus: `Explain how MMAP works on Linux` → `class=reasoning`; `Traceback (most recent call last):` → `class=code`; `ls /tmp` → `class=default`; `how does ASLR work?` → `class=reasoning`. `(routing currently disabled)` suffix appears on all 4 (auto off). After `:route on`, the suffix correctly disappears from subsequent `:route check` output (verified via direct probe). N1 honored. Closing.

claude-noether closed this issue

2026-05-13 11:50:33 +00:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: marfrit/aish#46