lurker: say in the ROOM when a model swap happened, not only in the journal (#118, lurker half)
The lurker falls back from claude to opencode on any dead end — right, since posting a failure verbatim helps nobody. But the reply then carried only the SECOND model's name, so a thin answer read as "the configured model had little to say" when the configured model had in fact never finished. Live on 2026-08-02: a Fable architecture run for @reviewer died on error_max_turns, the lurker fell back to opencode/deepseek, deepseek burned its own budget on the same search and answered nothing useful. That Fable had run at all appeared solely in `journalctl --user -u bullpen-lurker@reviewer` — exactly where nobody looks while reading a disappointing reply. Same defect idea #118 fixed for the grinder: report the model that actually served, and say so when it was not the one asked for. The banner only appears on an actual swap; the common case stays as quiet as before. tests/test_lurker_fallback_visible.py pins all three: the banner names both models on a swap, no banner when the configured model answers, and a double dead-end still returns a real diagnosis rather than a banner wrapped around nothing. Mutation-checked — cutting the announcement fails exactly the first test and leaves the other two green. (The first attempt at that mutation silently did not apply, which made three green tests meaningless; redone via a file with an explicit marker assertion.)
This commit is contained in:
@@ -325,7 +325,14 @@ def run_agent(prompt):
|
||||
# If opencode ALSO dead-ends, hand back the more informative of the two so the room at
|
||||
# least sees a real diagnosis instead of two layers of "(no result)".
|
||||
if oc and not oc.lstrip().startswith(f"(@{NICK}"):
|
||||
return oc
|
||||
# SAY IT IN THE ROOM, not only on stderr. The reply carries the second model's
|
||||
# name, so a thin answer reads as "the configured model had little to say" when
|
||||
# what actually happened is that the configured model never finished. On
|
||||
# 2026-08-02 a Fable architecture run died on max_turns, deepseek answered
|
||||
# instead, and the only trace was journalctl. Same defect bullpen idea #118 fixed
|
||||
# for the grinder: report what SERVED you, and say so when it was not what you asked.
|
||||
return (f"⚠ answered by the fallback: {MODEL or 'claude'} {why}, "
|
||||
f"served by opencode {_ocmodel()}\n\n{oc}")
|
||||
return ans if (ans and not ok) else oc
|
||||
return ans
|
||||
|
||||
|
||||
@@ -0,0 +1,77 @@
|
||||
"""A model swap must be visible in the ROOM, not only in the journal (#118, lurker half).
|
||||
|
||||
The lurker falls back from claude to opencode on any dead end — a good behaviour, since
|
||||
posting a failure verbatim helps nobody. But the reply then carried only the SECOND
|
||||
model's name, so a thin answer read as "the configured model had little to say" when what
|
||||
really happened was that the configured model never finished.
|
||||
|
||||
Live on 2026-08-02: a Fable architecture run for @reviewer died on `error_max_turns`, the
|
||||
lurker silently fell back to opencode/deepseek, and deepseek burned its own budget on the
|
||||
same search and answered nothing. The only record that Fable had run at all was
|
||||
`journalctl --user -u bullpen-lurker@reviewer` — exactly where nobody looks when reading a
|
||||
disappointing reply. Same defect bullpen idea #118 fixed for the grinder: report the model
|
||||
that actually served you, and say so when it was not the one you asked for.
|
||||
"""
|
||||
import importlib.machinery
|
||||
import importlib.util
|
||||
import os
|
||||
import sys
|
||||
|
||||
import pytest
|
||||
|
||||
REPO = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def lurk(monkeypatch, tmp_path):
|
||||
monkeypatch.setenv("HOME", str(tmp_path))
|
||||
(tmp_path / "spec_lurker").mkdir()
|
||||
sys.modules.pop("lurk", None)
|
||||
argv, sys.argv = sys.argv, ["bullpen-lurker", "spec"]
|
||||
try:
|
||||
ld = importlib.machinery.SourceFileLoader(
|
||||
"lurk", os.path.join(REPO, "lurker", "bullpen-lurker"))
|
||||
mod = importlib.util.module_from_spec(importlib.util.spec_from_loader("lurk", ld))
|
||||
ld.exec_module(mod)
|
||||
finally:
|
||||
sys.argv = argv
|
||||
return mod
|
||||
|
||||
|
||||
def test_fallback_announces_itself_in_the_reply(lurk, monkeypatch):
|
||||
monkeypatch.setattr(lurk, "RUNTIME", "claude", raising=False)
|
||||
monkeypatch.setattr(lurk, "MODEL", "fable", raising=False)
|
||||
monkeypatch.setattr(lurk, "run_claude", lambda p: ("(no result, rc 1)", False))
|
||||
monkeypatch.setattr(lurk, "run_opencode", lambda p: "here is the architecture verdict")
|
||||
monkeypatch.setattr(lurk, "_ocmodel", lambda: "bosch/deepseek")
|
||||
|
||||
out = lurk.run_agent("some prompt")
|
||||
assert "architecture verdict" in out, "the fallback's answer must still come through"
|
||||
assert "fable" in out, "the reply must name the model that was ASKED for and failed"
|
||||
assert "deepseek" in out, "…and the model that actually served"
|
||||
|
||||
|
||||
def test_no_banner_when_the_configured_model_answers(lurk, monkeypatch):
|
||||
"""The common case must not get noisier — same posture as the grinder's model tag."""
|
||||
monkeypatch.setattr(lurk, "RUNTIME", "claude", raising=False)
|
||||
monkeypatch.setattr(lurk, "MODEL", "fable", raising=False)
|
||||
monkeypatch.setattr(lurk, "run_claude", lambda p: ("a fine answer", True))
|
||||
monkeypatch.setattr(lurk, "run_opencode", lambda p: pytest.fail("must not fall back"))
|
||||
|
||||
out = lurk.run_agent("some prompt")
|
||||
assert out == "a fine answer"
|
||||
assert "⚠" not in out
|
||||
|
||||
|
||||
def test_double_dead_end_still_returns_a_diagnosis(lurk, monkeypatch):
|
||||
"""When BOTH runtimes fail there is nothing to announce — the room needs the error,
|
||||
not a banner wrapped around an empty string."""
|
||||
monkeypatch.setattr(lurk, "RUNTIME", "claude", raising=False)
|
||||
monkeypatch.setattr(lurk, "MODEL", "fable", raising=False)
|
||||
monkeypatch.setattr(lurk, "NICK", "spec", raising=False)
|
||||
monkeypatch.setattr(lurk, "run_claude", lambda p: ("(@spec: claude API error)", False))
|
||||
monkeypatch.setattr(lurk, "run_opencode", lambda p: "(@spec: opencode gave nothing)")
|
||||
monkeypatch.setattr(lurk, "_ocmodel", lambda: "bosch/deepseek")
|
||||
|
||||
out = lurk.run_agent("some prompt")
|
||||
assert "spec" in out and "error" in out.lower()
|
||||
Reference in New Issue
Block a user