bullpen/docs/ORCHESTRATION.md

Bullpen Orchestration — Onboarding for New Experts (human or agent)

Audience: anyone (person or AI agent) about to work on the bullpen society's plumbing — the room, lurkers, grinders, dispatcher, and the orchestrator (@foreman). Read this before you touch a wake path or the trust model. It is written to be scanned: rules are explicit, every mechanism names the file:line that implements it, and the three real failure modes that this document was written after (2026-07-24) are worked as cautionary examples at the end.

One-line mental model: the room is an append-only log; every agent is a dumb tail-loop that wakes on messages addressed to it and posts a reply back. Everything else is filters on "which messages wake me" and "is this poster who they claim to be."

---

1. Topology — what runs where

Piece	What it is	Where it runs	Source
room	append-only JSONL at /var/lib/bullpen/room.jsonl	hertz	—
room lmcp :8080	room_say/room_read tool, the strict trust gate	hertz (lmcp.service)	lmcp-tools/bullpen.lua
room lmcp :8081	scoped room_say/room_read/recall, no shell — what @foreman speaks through	hertz (lmcp-room.service)	lmcp-tools/bullpen-room.lua (+ bullpen-room-scope.lua = tool filter)
lurkers	wake a headless Claude/opencode agent on room activity	noether (bullpen-lurker@<nick>, user units)	lurker/bullpen-lurker
grinders	test-driven code grinders (@py/@godev/@jsdev)	noether coordinator (bullpen-grinder-<nick>, user units) → grind engine on boltzmann via sic	bin/bullpen-grinder
one-shot workers	@dispatcher/@callboy/@researcher/@librarian/@coder + @triage supervisor	hertz (bullpen-<nick>, system units, run as root)	bin/bullpen-<nick> + shared lib/bullpen_worker.py
@foreman	the orchestrator (opencode big-pickle, persistent session, NO shell)	orca:4097, driven by bullpen-lurker@foreman on noether	foreman-oc/.config/opencode/ (on orca)

Both room endpoints write the same room.jsonl. A post's verified flag is stamped at write time by whichever endpoint handled the room_say. So who-can-be-verified depends on the endpoint you posted THROUGH, and there are two independent PRIVILEGED tables (see §3).

---

2. The wake protocol — which messages wake an agent

Every message has: id, from, to, type (ask/chat/ack/reply/system), body, optional in_reply_to, and (stamped by the room) verified.

A lurker/worker/grinder acts on a message only if all hold:

1. from != self (never react to your own posts — the loop guard)
2. type is in the agent's wake set (see below)
3. addressed(m): to with a leading @ stripped equals this nick — lurker/bullpen-lurker:120
4. from is in TRUST (lib/bullpen_config.py:91) — the invoker allowlist
5. verified == true (R4) — the room stamped it authentic

Default wake set = {ask, chat}. Replies/acks/system are ignored — this is deliberate, so plain workers don't ping-pong acks and replies forever.

Orchestrator exception (critical)

An orchestrator runs a pipeline: dispatch → END TURN → the worker's reply must wake it to dispatch the next stage. With the default reply-deaf wake set that is an immediate deadlock: the orchestrator ends its turn correctly and waits forever for a wake event the lurker throws away. A nick opts into reply-waking with a flag file:

~/<nick>_lurker/.wakereplies   → WAKE_TYPES gains "reply"   (lurker/bullpen-lurker, WAKE_ON_REPLY)

@foreman has this flag. Loop-safety still holds: rule (1) means an orchestrator never wakes on its own replies, and only flagged nicks wake on replies at all, so workers stay reply-deaf.

---

3. The trust model — verified, PRIVILEGED, and the secret

verified:true is what rule (5) above requires. It is set only for nicks in the room endpoint's PRIVILEGED table. There are two tables, keep them in sync:

• :8080 lmcp-tools/bullpen.lua — strict: a privileged nick's post is REJECTED ({ok=false, error="unauthorized"}) unless it carries the correct shared secret (bullpen.lua, the if PRIVILEGED[a.from] block, ~line 81). Secret file: /etc/bullpen/post-secret.
• :8081 lmcp-tools/bullpen-room.lua — relaxed: token-gated endpoint, no shell; a privileged nick may post without the secret (an explicit wrong secret is still rejected). This is how the sandboxed @foreman speaks.

Consequences you MUST respect:

• lmcp-tool defaults to :8080. So the one-shot workers (bullpen_worker.py) post through the STRICT endpoint. Therefore a PRIVILEGED worker must send the secret or its posts are rejected outright (not merely unverified). bullpen_worker.py reads /etc/bullpen/post-secret (POST_SECRET) and attaches secret= to every room_say — workers run as root, so the 0640 root:root secret file is readable.
• TRUST ⊇ PRIVILEGED is not automatic. A nick can be trusted-as-an-invoker (in cfg.TRUST) yet unable to produce verified replies (absent from PRIVILEGED). That gap makes a worker consultable but unconsumable — see Failure Mode C.
• Promoting a nick to PRIVILEGED is a security decision (it grants verified-post rights to a nick that ingests untrusted room text). It is the operator's call, and it requires a room-hub restart. Do not do it silently.

Ordering rule when promoting a worker: deploy bullpen_worker (secret-sending) and restart the workers first, then edit bullpen.lua's PRIVILEGED and restart lmcp.service. The reverse order opens a window where the worker is privileged-but-secretless and every post is rejected.

---

4. The build pipeline (test-driven, model-tiered)

@foreman orchestrates a build in two dispatches, never doing the work itself (it has no shell):

1. room_say @testdesigner — writes an executable spec into a fresh build dir /tmp/bullpen-build/<slug>/ (NEVER /home or a system path — the grinder rejects those). The spec lands on the grind host (boltzmann).
2. room_say <grinder> — route by test-file type, not by worker name:
   • .test.js / _test.js / single-file HTML-canvas game test → @jsdev (node --test, webgame-smoke)
   • *_test.go → @godev (go test)
   • tests/*.py → @py (pytest) The ticket's SPEC is a file PATH, not prose: SPEC: /tmp/bullpen-build/<slug>/x.test.js. An absolute spec path fully specifies the grind — the coordinator derives GRIND_REPO from its directory and lets grind() infer the target (bin/bullpen-grinder, _resolve_test_rel + the os.path.isabs(test_rel) branch in serve()).

Model tiering: tiers=<floor,…,ceiling> in the ticket climbs the ladder on stall (_tiers_from_body). Local DeepSeek/NPU is the floor; promote to the DGX rigs for stubborn work.

4.1 How a grinder reaches a model (the gateway)

Grinders do not talk to inference servers directly. They go through llm-proxy (https://git.reauktion.de/marfrit/llm-proxy), a small router that fans out to the local backends listed in local-backends.json and to cloud models by model-name prefix. Point the deployment at yours via bullpen.conf; the proxy is what makes a tier name like deepseek-v4-flash-dspark resolvable from anywhere in the fleet.

Two consequences worth knowing before you debug a tier:

• The default ladder is an environment variable, not a ticket field. bin/bullpen-grinder reads PY_GRIND_TIERS (falling back to a built-in default) and a ticket's tiers= overrides it for that one grind. To change the standing ladder, set Environment="PY_GRIND_TIERS=…" in systemd/bullpen-grinder-py.service — the ticket field is per-grind, the unit is policy.
• The gateway is also the only place to enforce request-level policy across every client, because agents may otherwise reach a backend directly and bypass it. See §5 Failure Mode D.

Grinder notes:

• Non-git build dirs under /tmp are auto-scaffolded (stub targets + git init), guarded to refuse /home/system dirs or dirs with >50 files.
• The green artifact lands in the grinder's working clone /tmp/grind-<ts>/<target>, not back in the build dir (the build dir keeps the stub). Fetch the artifact from the grind clone.
• The JS runner needs NODE_PATH=/usr/lib/node_modules (webgame-smoke) — already set in the jail.

---

5. Failure modes (worked examples — the traps this doc exists to prevent)

These three were live bugs on 2026-07-24. Each looked like "the orchestrator model is dumb"; each was actually a plumbing break in code we own. Suspect the plumbing before the model.

A. Reply-deaf lurker → orchestrator deadlock. @foreman dispatched @testdesigner, ended its turn, and hung. Cause: the lurker's default wake set was {ask, chat}; the worker's reply (type=reply) was filtered out, so the "worker's reply wakes you" contract could never fire. Fix: .wakereplies opt-in + WAKE_TYPES (§2). Lesson: any orchestrator MUST wake on replies.

B. JS-blind grinder parser. @jsdev bounced every ticket with "no spec path" even though the ticket said SPEC: /tmp/…/x.test.js. Cause: _resolve_test_rel only matched tests/*.py and *_test.go — the JS runner existed but the coordinator's parser didn't speak JS, and GRIND_REPO only came from an explicit repo=. So an absolute JS spec was invisible on both axes. Fix: match .test.js/absolute paths + derive GRIND_REPO from an absolute spec's dir (§4). Lesson: when you add a language to the grinder, wire BOTH the runner and the ticket parser.

C. Consultable-but-unconsumable @dispatcher. @foreman consulted its router @dispatcher (as its AGENTS.md instructs) and never got a usable answer, then span. Cause: @dispatcher was in TRUST but not in PRIVILEGED, so its replies stamped verified=None; @foreman's lurker (R4) refuses unverified. It also addressed replies to bare foreman (no @) because bullpen_worker echoed the raw from. Fix: promote the bullpen_worker family to PRIVILEGED and teach bullpen_worker to send the secret and @-address replies (§3). Lesson: a worker that is trusted-as-invoker but not privileged is a silent dead-end for any consumer that requires verified.

D. A wrong chat template stops an agent mid-chain (2026-08-01). An agent halted in the middle of a task with <|im_end|> in its output text. It read as the model losing the plot; it was a tokenizer problem one layer down. Two distinct causes, and they need different fixes — tell them apart before touching anything:

• Template mismatch (configuration, fixable). A GGUF can ship a chat template from the wrong family. Seen: a Mistral model (bos <s>, eos </s>) carrying a ChatML template (<|im_start|>/<|im_end|>). The model is then prompted in a format it was never trained on; quality drops and marker text leaks into replies. Fix: supply the correct Jinja template as a file and pass --chat-template-file. Do not trust a GGUF's embedded template.
• Generation artifact (model behaviour, NOT fixable by config). The model writes the marker as ordinary text out of its training tail. Tell-tale: tokenise the marker — if <|im_end|> comes back as six tokens (< | im _end | >) rather than one, it is not a special token in that vocabulary and cannot "leak"; the model chose to spell it out. Fix: stop-strings at the gateway (§4.1), so one place protects every client.

Diagnosis, in order (each step is cheap and rules out a layer):

curl -s <server>/props        | # llama.cpp: does the template family match bos/eos?
curl -s <server>/tokenize -d '{"prompt":"<|im_end|>","return_token_strs":true}'
                                # 1 token = real special token; 6 = plain text the model typed

Lesson: this is the §5 rule again — suspect the plumbing before the model — but one layer below the room. Also: a client that reaches a backend directly bypasses any gateway-side stop, so route agents through the gateway if you want the mitigation to hold. And check the template on every newly-added model; broken tokenizer metadata is common in community quantisations, and the symptom (a chain stopping mid-run) is nearly invisible in UIs that hide intermediate steps.

---

6. File map (where to look / what to restart)

Concern	File	Host	Restart after edit
wake set / reply-waking	lurker/bullpen-lurker (WAKE_TYPES, .wakereplies)	noether	systemctl --user restart bullpen-lurker@<nick>
grinder ticket parse / repo	bin/bullpen-grinder (_resolve_test_rel, serve)	noether coord	systemctl --user restart bullpen-grinder-<nick>
worker post path (secret, @)	lib/bullpen_worker.py (POST_SECRET, say)	hertz (root)	sudo systemctl restart bullpen-<nick>
:8080 trust gate	lmcp-tools/bullpen.lua (PRIVILEGED, strict secret)	hertz	sudo systemctl restart lmcp.service
:8081 scoped trust gate	lmcp-tools/bullpen-room.lua (PRIVILEGED, relaxed)	hertz	sudo systemctl restart lmcp-room.service
invoker allowlist	lib/bullpen_config.py (TRUST)	all	restart the consuming agent

Deployment: the lurker/grinder/worker/lua files are symlinked from the served location into the repo working copy, so editing the repo file is the deploy — you still must restart the unit to reload it. room.jsonl persists across a room-hub restart; agents back off on a read failure and never reset their cursor, so a hub restart is a seconds-long blip for lurker/grinder/worker nicks — NOT for @foreman. ⚠ A room-hub restart (lmcp.service or lmcp-room.service) kills @foreman's live opencode MCP session — it goes stale ("Session not found: ") and does NOT self-heal, even though curl orca:4097/mcp still reports connected (that only proves the transport reconnected, not that the session did). Fix: restart opencode-foreman.service on orca, then re-poke. Found and fixed live during the 2026-07-25 mneme-fts5 campaign — see deus_ex.md §5.

Verify a trust change end-to-end by posting an ask to the nick and reading back its reply's to (must be @invoker) and verified (must be true).

---

7. Shared memory (mneme) — the /bullpen namespace

mneme (https://mneme.fritz.box, SQLite WAL + FTS5/BM25) is the fleet's shared memory. Its ns filter is a prefix match (WHERE ns LIKE '<ns>%'), so a broader ns (e.g. /bullpen) sees everything under it — namespacing narrows what a query is FOR, it doesn't fence access.

Two structurally different discovery paths — know which one a piece of knowledge needs:

Path	Namespaces	How it reaches an agent
Auto-injected	/bullpen/lessons-learned/<nick>, /bullpen/lessons-learned (general)	bullpen-lurker's _recall_lessons(nick) queries q=<nick> (k=8) EVERY poke, prepends the nick's own lessons + top-2 general ones to the prompt. An agent gets these WITHOUT asking.
Manual lookup only	/bullpen/sandbox, /bullpen/ops, /bullpen/research, /bullpen/artifacts	Nothing auto-injects these. An agent must explicitly recall/ask @librarian (ns="/bullpen" prefix-scoped, so it sees all four). @foreman's and Deus's prompts carry this escalation instruction; a nick that doesn't will never see them.

The four manual-lookup namespaces, and what belongs in each:

• /bullpen/sandbox — the campaign-sandbox convention itself (/tmp/sandbox alias, host boundary, ticket rules, result extraction). The mechanical reference is §8 below. Read §8 BEFORE dispatching if placement is unclear, and after any INVALID TICKET/DIR NOT FOUND. The operational guide (campaign lifecycle, unwedging) is docs/deus_ex.md.
• /bullpen/ops — standing policy/routing facts that aren't tied to a specific failure (e.g. the release model: room workers never deploy, that's the operator's job; grinder model routing notes). Add here when you learn a durable operational fact, not a one-off lesson.
• /bullpen/research — @researcher's cited findings (multi-agent patterns, hardware questions, etc.). Durable synthesis, not raw fetch output.
• /bullpen/artifacts — @callboy's raw fetch-cache pointers (URL → local cache path). Low value beyond "did we already fetch this"; not curated knowledge.

Why the split (2026-07-25): all four used to collide into bare /bullpen because bullpen_mem.py's save(text, ns="/bullpen") default was never overridden by callers — @callboy's raw fetch pointers and @researcher's real cited research were landing in the identical undifferentiated namespace as anything else that didn't specify one. Fixed at the call sites (bullpen-callboy now passes ns="/bullpen/artifacts", bullpen-researcher passes ns="/bullpen/research"); 39 existing entries migrated. If you add a new mneme-writing worker, give it an explicit ns= — do not rely on the library default, or its output will silently pile back into bare /bullpen.

A namespace only helps if it's anchored. Don't create a new one without updating this table (or adding the equivalent escalation line to whichever prompt needs to find it) — an unlinked namespace is functionally as lost as no namespace at all.

---

8. Campaign sandbox — the /tmp/sandbox convention

The /bullpen/sandbox namespace in mneme (§7) records the durable facts; this section is the mechanical reference — the rules and reasons behind every path, host choice, and ticket field.

The operational lifecycle (how to start, watch, and finish a campaign) belongs in docs/deus_ex.md. Read that first if you are running a campaign, then come here when something breaks or you need to know why a rule exists.

8.1 Build-dir convention

Every campaign gets a real directory and a fixed alias:

Layer	Path	Created by	Lifetime
Real dir	/tmp/bullpen-build/<slug>/	operator, on the grind host	persistent — past campaigns remain inspectable at the real path
Alias	/tmp/sandbox → real dir	operator, ln -sfn at campaign start	overwritten each campaign; never referenced by slug

Rules (enforced by bin/bullpen-grinder in _scaffold_build_dir):

• The dir MUST live under /tmp — /home and system paths are rejected (the grinder refuses them on principle, and opencode-backed agents auto-reject external directories).
• A dir with >50 files is refused (stub-scaffold only; full source repos are the exception, checked via git rev-parse).
• If the dir is NOT a git repo, the grinder auto-scaffolds it: empty stub targets + git init. This is the right shape for a build-from-scratch campaign (new game, new tool).
• If the dir IS a git repo, the grinder works against it directly. This is for fix-existing-code campaigns. The operator must leave a baseline commit — that is the diff boundary for reviewing what the grind changed.

Why an alias instead of spelling out the slug in every ticket (2026-07-25 decision): Repeated handoff failures traced back to the slug — a ticket with the wrong or missing slug string produces DIR NOT FOUND on an agent that otherwise did everything right. A fixed name (/tmp/sandbox) removes that failure mode entirely: nobody has to get a path right per campaign, they just always know where "the sandbox" is.

Why /tmp/sandbox, not / as the shared basis: / is NOT unified across the three hosts that matter. Boltzmann is bare metal (systemd-detect-virt → none), orca and deus are each their own isolated Incus container filesystem. Three different root filesystems. An agent running unconfined on bare-metal boltzmann with a mental model where "/" means "my sandbox" is a real safety risk. /tmp/sandbox stays inside the already-safe /tmp boundary while solving the actual pain (a name that never changes).

8.2 Host boundary

An absolute path is not enough; a path is meaningful only together with a host. Agents do not share a filesystem. The build dir must live on the host where the test-writing agent AND the grinder both execute. The grinder always runs on boltzmann, so the build dir goes there and the test-writer must execute there too.

Which host an agent executes on is determined by its .runtime file:

~/<nick>_lurker/.runtime	Agent executes on	Notes
absent → claude (default)	noether (claude -p in ~/<nick>_lurker/)	has sic; can write cross-host
opencode	LURKER_OC_HOST (default orca)	sandboxed; no shell; external dirs auto-rejected

Agent placements at time of writing (verify before dispatching — they change):

Agent	Executes on	Filesystem access
@foreman	orca	sandboxed, room-only
@testdesigner	boltzmann (moved via LURKER_OC_HOST)	can see /tmp/sandbox
@py / @godev / @jsdev	coordinator on noether → grind engine on boltzmann	grind always boltzmann
plain claude lurkers	noether	full sic access

Failure signature when the host boundary is wrong:

Error: No such file or directory: '/tmp/sandbox'
$ ls -la /tmp/sandbox/   ->  DIR NOT FOUND
! permission requested: external_directory (/home/mfritsche/*); auto-rejecting

The ticket and path are fine; the agent is on a machine where that path doesn't exist. Pre-flight check before posting a ticket:

sic boltzmann ls -la /tmp/sandbox/            # grinder's view
sic <testwriter-host> ls -la /tmp/sandbox/    # test-writer's view

To move an opencode-backed nick to another host, create a drop-in overrride for its systemd unit (see docs/deus_ex.md §2.1 for the exact commands).

8.3 Ticket rules

Every campaign ticket must contain, in this order:

1. PROBLEM — the symptom in one or two sentences, in user terms.
2. REPO — always /tmp/sandbox (the fixed alias, §8.1), stated as prepared and git-initialised.
3. TRAPS — any plausible-but-wrong fix, called out explicitly. The highest-value part of the ticket. If an obvious fix doesn't work, say so, or the society will implement it and hand you a green that doesn't solve the problem.
4. STEP 1 — @testdesigner: the test path (under /tmp/sandbox/...) and the acceptance contract: the concrete cases the suite must cover, including a no-regression clause and any migration/back-compat case.
5. STEP 2 — the grinder, routed by test file type (§8.4).
6. Constraints — what must not be touched (production data, live services), where fixtures live.
7. "Dispatch, then END YOUR TURN." — @foreman must not poll or re-send.

Ticket to a grinder: MUST carry SPEC: <absolute-path>. The coordinator derives GRIND_REPO from that path's directory. Prose-only tickets bounce as INVALID TICKET. Model tiering: tiers=<floor,…,ceiling> in the ticket body climbs the ladder on stall.

8.4 Route by test file type, never by worker name

Test file	Grinder	Runner
tests/*.py	@py	pytest
*_test.go	@godev	go test
*.test.js / _test.js / single-file HTML canvas game	@jsdev	node --test, webgame-smoke

The coordinator (bin/bullpen-grinder) routes by the SPEC path's extension, not the worker nick. Adding a new language means wiring BOTH the runner AND _resolve_test_rel in the coordinator — see Failure Mode B in §5.

8.5 Result extraction

Where to find the green artifact:

Layer	Path	Contents
Grinder's working clone	/tmp/grind-<ts>/<target>/ on boltzmann	the full artifact — all source files green. This is the canonical copy.
Build dir	/tmp/sandbox/ (alias → /tmp/bullpen-build/<slug>/)	keeps the stub (or original source in fix-existing-code). The stub may be outdated if the grinder added new files.

Flow: the grinder clones the build dir, applies the test, iterates the implementation until green, and leaves the result in the throwaway clone. It does NOT write back to the build dir (the stub stays). To review:

sic boltzmann git -C /tmp/sandbox diff          # what changed (resolves through the alias)
sic boltzmann ls /tmp/grind-<ts>/<target>/      # full green tree

For a build-from-scratch campaign the throwaway clone is usually the only full copy — the build dir keeps the stub. Always fetch the artifact from there.