aish/docs/PHASE6.md

aish — Phase 6 Manifest

Project: aish — AI-augmented conversational shell Document: Phase 6 Requirements, Architecture & Design Decisions Status: Plan + review fold-in (tree at 4407029) Date: 2026-05-16

Review findings (independent agent, 2026-05-16) — 2 BLOCKERs resolved in-place, 7 CONCERNs folded, 6 NITs applied:

R1 (BLOCKER, RESOLVED). §4 fence detector's outside-state branch drops the leading '``' chunk of a split-fence. The §4 pseudocode as written ("look for ```<lang>\n in chunk; if found [...] else: emit chunk as-is") emits the partial-fence chunk immediately, so the next chunk no longer sees the full marker. Contradicts B2's split-fence requirement. Fix folded into §4: outside-state also holds a small tail (up to 10 chars) when the chunk's tail could be a fence prefix; flushes on next push. Same pattern as the secrets.lua streaming rehydrator (secrets.lua ~213). Pseudocode + algorithm updated.

R2 (BLOCKER, RESOLVED). highlighted() file placement was ambiguous in §3 vs §12. highlighted() needs _shq (currently a repl.lua M.run-local closure) and require("executor"). Resolution: highlighted() stays in repl.lua; renderer.lua exposes renderer.set_highlight(enabled, detected, highlight_fn). The filter state machine in renderer.lua calls back through highlight_fn(body, lang) at fence-close. No executor dependency in renderer.lua; no _shq lift. §3 + §12 commit 5 updated to state this explicitly.

R3 (CONCERN, FOLDED). PTY raw-mode toggle per code block. Each executor.exec call calls libc.set_raw(0) briefly. For an assistant turn with N fenced blocks that's N raw-mode toggles on the streaming hot path. Smoke-test for cursor/flicker before locking in. Added to §12 commit 5 risk row.

R4 (CONCERN, FOLDED — risk noted, needs verify at implement-time). tree-sitter highlight --lang X invocation grammar is unverified. The upstream tree-sitter CLI's highlight subcommand canonically takes a path argument and infers language from the file extension via ~/.config/tree-sitter/config.json. A --lang flag may not exist. Since B4 confirmed zero fleet hosts have tree-sitter installed, this can't be probed locally. Resolution: §4 amended — at commit 5 implement time, VERIFY against a real install. If --lang is wrong, switch to writing the tmpfile with the matching extension (/tmp/lua_XXX.py) and pass the path. Path-based discovery is the CLI's documented primary mode.

R5 (CONCERN, FOLDED). :tree off semantics ambiguous. §6 listed it as "clear ctx.project" but didn't clarify whether subsequent :tree (no arg) re-uses cached opts or falls back to config defaults. Clarified in §6: :tree off is a one-shot clear of ctx.project; subsequent :tree re-scans with config defaults or the explicit arg if given.

R6 (CONCERN, FOLDED). cwd-coupling differs between :diff and :tree. :diff reads libc.getcwd() at meta invocation time; :tree's captured ctx.project is fixed at scan time (per A8). After cd /other-project, :diff shows the new project's diff but ctx.project still holds the old project's tree. Documented in §5 (the diff section now cross-refs §6 / A8) so the user-facing expectation is clear.

R7 (CONCERN, FOLDED). :tree refresh opts caching unspecified. Should :tree refresh re-use the last explicit :tree <N> depth override, or fall back to cfg.project.tree_depth? Resolution: cache the last opts on ctx._project_opts; :tree refresh reuses them; falls back to config defaults if no prior call. §6 updated.

R8 (CONCERN, FOLDED). :reset interaction with ctx.project. Phase 4 established that :reset does NOT clear ctx.memory_items (parity is desirable — startup-injected facts persist across a user-driven context reset). ctx.project should follow the same rule: :reset clears ctx.turns and pending_exec_output and ctx.summary (per Context:reset at context.lua ~343), but NOT memory_items and NOT project. Documented in §3 + §12 commit 1.

R9 (CONCERN, FOLDED). Status-bump duplication between §12 commits 5 and 6. Commit 5 sub-step (e) said "PHASE6 status → Implement"; commit 6 also said the same. Resolved: commit 5e does NOT bump the status (only HELP update); commit 6 owns the status bump (along with the config example). One owner per change.

R-N1..N6 (NITs, APPLIED): N1. §4 algorithm pseudocode now includes the SOL/post-newline anchor requirement (mid-line backticks in prose don't open a fence). The plan §12 risk row already promised this; now §4 matches. N2. §4 detection block gained a comment explaining the read("*l") and pipe:close() pattern — close return-value is ignored per B3; presence of an output line is the signal. N3. §5 :diff staged → git diff --cached table row dropped (the meta is a thin pass-through; user types the right git flags). :diff --cached works directly. Surface is honest. N4. §6 _scan_project_tree switched from os.execute("cd " .. shq .. " && git rev-parse ...") to git -C <dir> rev-parse --git-dir — no subshell, more idiomatic. N5. §12 "Open at plan-time" first bullet (dir-arg vs hardcoded getcwd) dropped — already decided in §6's signature; not open. N6. §11 wording on Phase 7+ left as-is (reviewer marked purely cosmetic).

Analyze findings (2026-05-16):

A1. renderer.lua surface clean — assistant_delta(chunk) already concatenates into a stream_buf then emit()s the chunk; assistant_flush() finalizes with a trailing newline if missing. The fence-aware highlight filter slots in between chunk receipt and emit without restructuring; no callers besides repl.lua touch stream_buf so the filter state can live alongside it.

A2. executor surface clean — executor.exec(cmd) already forkpty-spawns, captures + live-streams output, returns (out, code). Phase 6's :diff and _scan_project_tree reuse this path verbatim; no new IO model. git-rooted commands inherit cwd from the parent (which libc.chdir already mutates), so a :diff after cd reads the right repo.

A3. context composition order locked — current to_messages builds sys_content = base + [background] + [earlier summary] + NORRIS suffix. Phase 6 inserts [project] between [background] and [earlier summary]. Same Norris-suppression guard already in place (if not self.norris_active).

A4. Q-H1 RESOLVED: tmpfile roundtrip for tree-sitter highlight write+read. Avoids ARGMAX risk on large code blocks (vs printf BODY | tree-sitter ...) and shell-escape complexity. Two file handles, deterministic cleanup via os.remove. Sketch:

```lua
local tmp = os.tmpname()
local w = io.popen(("tree-sitter highlight --lang %s > %s")
                   :format(lang, tmp), "w")
w:write(body); local _, _, code = w:close()
local f = io.open(tmp, "rb"); local out = f:read("*a"); f:close()
os.remove(tmp)
if code ~= 0 then return body end  -- pass-through on failure
return out
```

A5. Q-D1 RESOLVED: no confirm gate on :diff. git diff is read-only; matches :history, :sessions, :safety check — none of which gate. Permission DSL (#9) only applies to AI-suggested CMD: lines, not user-issued metas.

A6. Q-D2 RESOLVED: tiered resolution for @<token>. The mention parser tries <token> as a file path first; if it doesn't resolve AND the token contains .., retry as a diff range. This keeps @../sibling.txt (path) working AND allows @origin/main..feature (ref range — resolves via second attempt since no such file exists). No grammar prefix needed.

A7. Q-H2 RESOLVED: highlighting is assistant-output only in v1. expand_mentions content lands in the user-turn payload — visible on the terminal via readline echo, not via assistant_delta. Filing "highlight @path-expanded code in echo" as v2 polish. Reason: intercepting readline echo for ANSI injection is non-trivial and orthogonal to the stream filter.

A8. Q-T1 RESOLVED: project tree captured at scan time, not auto- refreshed on cd. cd /other-project leaves the existing ctx.project stale; :tree refresh is the manual verb to update. Auto-refresh on cd intercept is a v2 polish (the cd interceptor in executor.maybe_chdir is a clean hook for it).

A9. Q-T2 RESOLVED: rely on .gitignore via git ls-files in repos; fall back to find with simple excludes outside. Custom include/exclude glob lists deferred to v2. Reason: most users live inside git repos; .gitignore already encodes their notion of "noise". Out-of-repo users get the simple fallback and can scope via :tree <depth>.

A10. expand_mentions punct-peel does NOT strip / — so @HEAD~1..HEAD, peels the , and the underlying token HEAD~1..HEAD has no slash; the path-then-diff retry from A6 catches it. No new peel logic needed.

A11. Auto-injection ordering for [project] — if both cfg.memory. inject_max_chars and cfg.project.auto_tree fire at startup, the order is: memory load → tree scan → first ask_ai. The composition in to_messages places [background] (memory) before [project] so the model reads memory facts before file tree. Documented in §3.

A12. Norris interaction — [project] block follows the established [background]/[earlier summary] suppression rule under ctx.norris_active. Planner stays on its goal anchor; the tree can be re-introduced via the goal text if needed. Matches R-C1/R-C4.

PHASE0 is the locked substrate; PHASE1-5 are layered on top. This manifest specifies what Phase 6 adds — tree-sitter syntax highlighting hooks, diff-aware code injection, and project-level context (file-tree summary).

---

1. Scope of Phase 6

Three pillars per PHASE0.md §11 row 6:

1. Tree-sitter syntax highlighting hooks — when an external tree-sitter CLI is detected at startup, assistant code-fence content is filtered through it for ANSI-colorized display. Plain prose streams unchanged. When the CLI is absent, the filter is the identity function (zero overhead, zero hard dependency). Toggleable at runtime with :highlight on|off. Default off until the user opts in (don't surprise existing users with a display change). Per B4: tree-sitter is absent on every fleet host probed; :highlight on when the CLI is missing emits a status that names the install hint (apt install tree-sitter / cargo install tree-sitter-cli) rather than silently falling back to identity.

2. Diff-aware code injection — surface git diffs as first-class context. Two entry points:

   • Meta verb: :diff [args] runs git diff <args> from cwd, appends output to context as exec-output. :diff staged, :diff HEAD~3, :diff main..feature all delegate to git's argument grammar.
   • @-mention extension: @HEAD..feature (a ref-range expression anywhere a @path would go) expands inline as a fenced diff block, mirroring how @README.md already works.

3. Project-level context (file-tree summary) — git ls-files-based tree summary of the cwd, injected as a [project] block in the system prompt. Two entry points:

   • Meta verb: :tree [depth] injects on demand; :tree refresh re-scans.
   • Auto-inject at startup when cfg.project.auto_tree = true — gated like memory injection so existing configs don't change behavior.

Phase 6 is done when:

• With tree-sitter CLI installed and :highlight on, the assistant reply py\nprint("hi")\n shows up with ANSI colors. Without the CLI, :highlight on is a no-op + emits a status warning.
• :diff from a dirty git repo shows the working-tree diff in the exec-output frame; the model sees it on the next ask_ai turn.
• @HEAD~1..HEAD in a prompt expands inline to a fenced diff block.
• :tree injects a [project] <N files>: block visible in ctx:to_messages() (via the system prompt assembly).
• With cfg.project.auto_tree = true, the project block appears on every broker call (subject to max_chars cap).
• Existing configs without cfg.project and with :highlight off (default) behave exactly like Phase 5 (Phase 5 regression coverage).

---

2. Technology Decisions (delta from Phase 5)

Decision	Choice	Rationale
Highlight backend	External tree-sitter CLI (tree-sitter highlight --lang X)	Honors PHASE0 §3: no compiled extensions, no luarocks. Detected once at startup; absence → identity filter. Opt-in via :highlight on so install-state changes don't break users.
Highlight buffering	Accumulate inside fenced code blocks, emit on closing fence; pass-through outside fences	Streaming UX preserved for prose. Code blocks get colorized atomically, accepting a per-block latency (~ block streaming time). Per-chunk highlighting would split a token across tree-sitter invocations and corrupt the output.
Lang detection	First-line fence info-string ( ```py, ```python, ```lua) → normalized via small map (py→python, js→javascript, etc.)	The lang tag mirrors the one we already emit in expand_mentions (#7). No tag → identity (no highlight).
Diff backend	Shell out to git diff <args> via executor.exec	Honors substrate (no libgit2 FFI). The existing exec frame handles capture + stream. git is universally present where aish makes sense.
Diff failure	Bail with status [aish] :diff failed (not a git repo / bad ref); do NOT inject empty output	Avoids polluting context with stale or empty diffs.
Tree backend	git ls-files --cached --others --exclude-standard when cwd is a git repo, else find . -type f -not -path './.*'	Free .gitignore honor in repos; sensible default outside. Both are POSIX-portable.
Tree summary form	Sorted relative paths, grouped by directory at depth ≤ cfg.project.tree_depth (default 3), truncated by char count cfg.project.tree_max_chars (default 4096)	One block, deterministic order, cheap to compute. Matches the [background] memory block convention (Phase 4) so the system prompt's compositional shape stays familiar.
Tree injection point	context.lua: new compose_project(...) adds a [project] <header>\n<body> block to the system content, between [background] and [earlier summary]	Same suppression rule as [background]/[earlier summary]: NOT injected during Norris (R-C1 / R-C4 — planner stays on its anchor).
Tree refresh policy	One scan at startup if auto; :tree refresh to re-scan on demand	Scanning on every ask_ai is wasteful for slow filesystems. Manual refresh is sufficient for v1.
@-mention diff syntax	@<ref>..<ref> (two .. separator) only — recognized via the existing trailing-punct peel logic	Avoids ambiguity with literal paths. @HEAD alone is NOT a diff trigger (would collide with files literally named HEAD).

---

3. Module Changes

File	State after Phase 5	Phase 6 changes
renderer.lua	assistant_delta(text) writes chunks; assistant_flush() finalizes	Add fence-aware filter inside the assistant stream. State machine: outside-fence (pass-through) / inside-fence (buffer, emit on close). On close, pipe buffer through tree-sitter highlight --lang <X> (if highlight enabled), emit result. Toggle exposed as renderer.set_highlight(bool).
executor.lua	extract_cmd_lines, extract_cmd_bg_lines, extract_delegate_lines	No changes. Diff and tree use the existing exec path.
context.lua	system prompt = base + [background] + [earlier summary] + NORRIS suffix	Add self.project = "..." string field + compose_project(self.project) helper. Injection between [background] and [earlier summary] (A11: memory facts read before file tree). Suppressed under Norris (A12, parity with R-C1/R-C4).
repl.lua	meta dispatch + main loop + #13 secrets wiring	New helpers: _detect_treesitter() (run once at startup), _run_git_diff(args), _scan_project_tree(dir, opts). New meta: :highlight, :diff, :tree. Extend expand_mentions to recognize <ref>..<ref> token shape.
config.lua	example blocks for mcp/safety/memory/routing/secrets/etc.	Add commented-out project = { auto_tree = false, tree_depth = 3, tree_max_chars = 4096 } block.

No new module files in v1. Three new helpers in repl.lua keep the file growing but consolidate the Phase 6 surface. If the highlighter filter grows past ~80 LOC, lift it into highlight.lua as a follow-up.

---

4. Pillar 1 — Tree-sitter highlighting

Detection (startup, once)

local function _detect_treesitter()
    local pipe = io.popen("command -v tree-sitter 2>/dev/null && tree-sitter --version 2>/dev/null")
    -- N2 / B3: pipe:close() returns true on LuaJIT regardless of exit
    -- code; we don't use it for the verdict. Presence of an output
    -- line from --version is the actual signal.
    local ok = pipe and pipe:read("*l") and pipe:close()
    return ok
end

If not present, renderer.set_highlight(true) emits a status warning and leaves the filter as a no-op. Don't error; the user can install tree-sitter and re-toggle.

Stream filter

The filter wraps renderer.assistant_delta. State machine (R1 + N1 revisions — outside-state accumulator + SOL anchor):

state    = "outside" | "inside"
tail     = ""                  -- outside-state lookahead buffer (R1)
buf      = ""                  -- only used in "inside"
lang     = nil                 -- captured at fence open

push(chunk):
  if state == "outside":
      combined = tail .. chunk
      -- R1: hold back trailing partial-fence so a split fence
      -- ("``" arrives, then "`python\n") doesn't get emitted
      -- as plain text before we recognize the opener.
      -- N1: fence opens only at start-of-stream OR after a newline
      -- ("^```" or "\n```"). Inline backticks in prose don't open.
      match_pos = find(combined, "(^|\n)```([%w_-]*)\n")
      if match_pos:
          -- everything before the opening is plain text
          emit combined[1 .. fence_start - 1]
          lang = captured_lang
          buf  = combined[fence_end .. end]   -- text after \n
          state = "inside"; tail = ""
          if buf has \n``` inside, fall through to inside-state below
      else:
          -- Hold back the last K chars if they could be the start
          -- of a fence-open. Specifically: tail = the longest suffix
          -- of combined that is a prefix of any well-formed fence
          -- marker ("`", "``", "```", "```l", "```lua", "```lua\n").
          -- Bounded by max-lang-tag-length + 4 (~10 chars in practice).
          tail = longest_partial_fence_suffix(combined, max=10)
          emit combined[1 .. #combined - #tail]
          -- (next push will combine tail with the next chunk and retry)

  if state == "inside":
      buf = buf .. chunk
      -- closing fence: "\n```" anywhere in buf (followed by EOL or end).
      close_pos = find(buf, "\n```")
      if close_pos:
          fence_body = buf[1 .. close_pos - 1]
          closing    = buf[close_pos .. close_pos + 3]   -- "\n```"
          rest       = buf[close_pos + 4 .. end]
          emit highlighted(fence_body, lang)
          emit closing verbatim
          state = "outside"; buf = ""; tail = ""
          if rest != "":
              push(rest)   -- recurse for any plain text after the closing
      else:
          -- still buffering; nothing emitted this push

Edge cases:

• Chunk boundary lands inside an opening marker (e.g., chunk ends with '``', next starts with 'python\n'). The tailbuffer holds'``'`; next push combines and finds the full opener.
• Chunk boundary inside a closing marker. The inside branch already accumulates into buf; find against cumulative buf recovers.
• Inline backticks in prose ("use ``` to mark code"). N1's (^|\n)```` anchor means this does NOT open a fence — \n` is required before the three backticks.

The tail is bounded (max ~10 chars), so streaming UX latency is at most 10 chars worth of buffering when between fenced blocks. The existing assistant_delta's stream_buf for full-text accumulation is unaffected — the filter sits BEFORE emit.

highlighted(body, lang) — B3 + R2 + R4-revised:

Lives in repl.lua (per R2; renderer.lua calls it via the highlight_fn passed to renderer.set_highlight). Has access to _shq (existing helper from #3) and the executor require.

-- repl.lua local. Wired into renderer via:
--   renderer.set_highlight(true, treesitter_present, highlighted)
local function highlighted(body, lang)
    if not highlight_enabled or not lang_map[lang] then return body end

    -- R4: tree-sitter highlight CLI grammar is UNVERIFIED.
    -- Upstream `tree-sitter highlight` canonically takes a path and
    -- infers language from the file extension. At commit-5 implement
    -- time, install tree-sitter and check whether `--lang` exists.
    -- If not, name the tmpfile with the language's canonical extension
    -- (lang_extension[lang]) and pass the path directly:
    --   tmp = os.tmpname() .. lang_extension[lang]
    --   cmd = "tree-sitter highlight " .. _shq(tmp)
    -- Below is the optimistic --lang form for code reading; the actual
    -- implementation must be verified.

    local tmp = os.tmpname()
    local f = io.open(tmp, "wb")
    if not f then return body end
    f:write(body); f:close()
    -- B3: io.popen():close() doesn't expose exit codes in LuaJIT.
    -- Route via executor.exec which uses pty.spawn+waitpid and
    -- returns (out, exit_code) reliably.
    local out, code = executor.exec(
        ("cat %s | tree-sitter highlight --lang %s")
        :format(_shq(tmp), lang_map[lang]))
    os.remove(tmp)
    if code ~= 0 then return body end   -- pass-through on highlighter failure
    return out
end

Why this shape (and not the formulate-time A4 sketch):

• R2 file placement: highlighted lives in repl.lua so it has natural access to _shq + executor. renderer.lua stays free of the executor require; it calls back through highlight_fn.
• B3 exit-code path: LuaJIT (5.1 contract) doesn't expose the exit status via io.popen(...):close(). executor.exec is the only reliable channel in our substrate.
• R4 grammar verification: the --lang flag is the formulate-time assumption; the upstream CLI's highlight subcommand may want a PATH with a recognized extension instead. Implement-time check required before commit 5 ships.
• The tmpfile stays — avoids ARGMAX on printf '%s' BODY | and sidesteps shell-escape edge cases on arbitrary code-block bytes.
• Cost: one syscall round (tmpfile create/remove) + one pty spawn per code block — negligible vs the highlighter latency.

Lang map (v1)

local LANG_MAP = {
    py = "python", python = "python",
    lua = "lua",
    js = "javascript", javascript = "javascript", ts = "typescript",
    sh = "bash", bash = "bash",
    c = "c", h = "c", cpp = "cpp", cc = "cpp",
    rs = "rust", go = "go", java = "java", rb = "ruby",
    md = "markdown", json = "json",
}

Reuses the same map as expand_mentions. Factor into a shared helper once both reference it (small _lang_of_ext() in repl.lua).

Toggle

:highlight (no arg) → flip. :highlight on|off → set explicit. :highlight status → report enabled + whether tree-sitter is present. Default: off (don't change existing-user UX).

---

5. Pillar 2 — Diff-aware code injection

Meta: :diff [args]

• :diff → git diff (working tree vs index)
• :diff HEAD → git diff HEAD
• :diff --cached → git diff --cached (staged-only)
• :diff main..feature → git diff main..feature
• :diff <anything else> → passed verbatim to git diff <anything>

N3: the meta is a thin pass-through to git diff. Don't introduce aliases like staged that would diverge from git's own grammar — the user types the real flag (--cached) and aish doesn't second-guess.

R6: :diff reads libc.getcwd() at meta-invocation time. Compare with :tree / ctx.project which captures the cwd at scan time (A8): after cd /other-project, :diff shows the new project's diff, but ctx.project still holds the old project's tree until :tree refresh.

Implementation — B1-revised (must disable pager + color):

meta.diff = function(args)
    args = (args or ""):gsub("^%s+", ""):gsub("%s+$", "")
    -- B1: forkpty makes git think it's interactive, enabling color
    -- ANSI + DEC keypad/line-clear escapes that pollute the injected
    -- context block. --no-pager kills the keypad sequences; --color=
    -- never kills the color codes. Both are required.
    local cmd = "git --no-pager -c color.ui=never diff " .. args
    local out, code = executor.exec(cmd)
    if code ~= 0 then
        renderer.status(("diff failed (exit %d)"):format(code))
        return
    end
    if out == "" or out:gsub("%s", "") == "" then
        renderer.status("(no diff)")
        return
    end
    ctx:append_exec_output(("[diff %s]\n%s"):format(
        args == "" and "(working tree)" or args, out))
end

The [diff ...]\n<output> framing matches the [bg:N exited] / [delegate X] conventions established in Phase 5 / #6 / #8.

The same --no-pager -c color.ui=never prefix applies to the @<r1>..<r2> resolution path in the next section, and to any future git verbs we add (:log, :show, etc.). Factor into a helper _git_clean_cmd(subcmd) if multiple call sites accumulate.

@-mention: @<ref1>..<ref2> — tiered resolution (A6)

Extends expand_mentions (#7) by adding a SECOND resolution attempt when the first (path lookup) fails AND the token contains ..:

-- Existing path-attempt block ends with content = _read_truncated(path)
-- which returns nil if no such file. Add the diff retry there:

if not content and path:find("..", 1, true) then
    local r1, r2 = path:match("^(.-)%.%.(.+)$")
    if r1 and r2 and r1 ~= "" and r2 ~= "" then
        -- B1: --no-pager + color=never (same as the :diff meta path).
        -- B3: io.popen close() doesn't expose exit codes — use the
        -- file-redirect trick OR executor.exec. Here we want a quick
        -- best-effort and the cost of an extra forkpty is acceptable.
        local out, code = executor.exec(
            ("git --no-pager -c color.ui=never diff %s..%s 2>/dev/null")
                :format(shq(r1), shq(r2)))
        if code == 0 and out:match("%S") then
            content = out
            -- Note: language tag becomes "diff" regardless of path lang
            lang_override = "diff"
        end
    end
end

Output replaces the token with:

```diff path=<r1>..<r2>
<content>
```

Tiered resolution semantics:

• @README.md → file lookup succeeds → file expansion
• @../sibling.txt → file lookup succeeds → file expansion
• @HEAD~1..HEAD → file lookup fails, .. present, ref-range succeeds → diff
• @origin/main..feature → file lookup fails (no such file), .. present, ref-range succeeds → diff. The token has / in r1 but git diff accepts it as a ref; no /-based heuristic needed (resolves Q-D2).
• @nonexistent-file..but-also-not-a-ref → both fail; literal token preserved with the existing [aish] @X: not found status path.

---

6. Pillar 3 — Project file-tree

Meta: :tree [depth]

• :tree → scan + inject with default depth and char cap; if a prior :tree <N> set a depth override, this re-scan uses the config defaults (:tree resets to defaults)
• :tree <N> → override depth for this scan; cached as ctx._project_opts for :tree refresh
• :tree refresh → re-scan with ctx._project_opts (last explicit opts) if present; otherwise config defaults (R7)
• :tree off → clear ctx.project AND ctx._project_opts. Future :tree (no arg) re-scans with config defaults. One-shot semantics — there's no "disabled until re-enabled" flag (R5).

Scan logic

local function _scan_project_tree(dir, opts)
    opts = opts or {}
    local max_chars = opts.max_chars or 4096
    local depth     = opts.depth or 3

    -- Prefer git ls-files for .gitignore honor; fall back to find.
    -- N4: `git -C <dir>` skips the subshell vs `cd && git ...`.
    local in_git = os.execute(("git -C %s rev-parse --git-dir >/dev/null 2>&1"):format(shq(dir))) == 0
    local listcmd
    if in_git then
        listcmd = ("git -C %s ls-files --cached --others --exclude-standard"):format(shq(dir))
    else
        listcmd = ("find %s -maxdepth %d -type f -not -path '*/\\.*' 2>/dev/null"):format(shq(dir), depth + 1)
    end

    local pipe = io.popen(listcmd)
    if not pipe then return nil, "scan failed" end

    local files = {}
    for line in pipe:lines() do
        -- Depth filter: count `/` separators
        local _, slashes = line:gsub("/", "")
        if slashes < depth then files[#files + 1] = line end
    end
    pipe:close()

    table.sort(files)

    -- Build a tree-ish summary, truncate by char count.
    local body = table.concat(files, "\n")
    local truncated = false
    if #body > max_chars then
        body = body:sub(1, max_chars) .. "\n... (truncated)"
        truncated = true
    end
    return body, { file_count = #files, truncated = truncated }
end

Injection

ctx.project = "..." (string), composed into the system prompt between [background] and [earlier conversation summary]:

[project] 142 files (truncated at 4096B):
README.md
broker.lua
config.lua
context.lua
...

Suppressed under Norris (R-C1 / R-C4 — planner stays focused; the project context can be re-introduced via the Norris goal text if needed).

Auto-inject

cfg.project.auto_tree = true runs the scan once at startup and sets ctx.project. Default false (existing configs unchanged).

---

7. UX Surface Summary

Meta	Behavior
:highlight [on/off/status]	Toggle tree-sitter highlighter (no-op when CLI absent)
:diff [args]	git diff <args>, append output to context as [diff ...]
:tree [N/refresh/off]	Scan/refresh/clear project file-tree block

@-mention	Behavior
@path	Existing (#7) file expansion
@<ref1>..<ref2>	New: inline git diff <r1>..<r2> expansion

Config	Default	Effect
cfg.project.auto_tree	false	Inject project tree at startup
cfg.project.tree_depth	3	Depth filter for the scan
cfg.project.tree_max_chars	4096	Truncation cap for the injected block
(no config flag for :highlight)	—	Runtime toggle only; no persistence in v1

---

8. Out of Scope (Phase 6)

• Pure-Lua syntax highlighter — defer to a future phase if tree-sitter CLI absence becomes a practical pain point. v1 says "install tree-sitter or accept plain text".

• bat/glow/chroma integration — only tree-sitter is wired. Other highlighters can be added behind the same :highlight toggle later (config field cfg.highlight.backend = "tree-sitter"|"bat"|...).

• Smart diff context selection — no AI-driven "which diff to show". User explicitly says :diff <range> or @<r1>..<r2>.

• File-tree LRU / smart summarization — v1 is a flat truncated list. Hierarchical roll-up ("docs/ — 8 files") is a v2 polish.

• Watching for file changes — no fs-notify reload. Re-scan via :tree refresh.

• Diff history — :diff doesn't track its previous invocations. Each invocation is independent.

• Inline diff highlighting — the diff lang is in LANG_MAP so tree-sitter highlight --lang diff works, but we don't ship custom ANSI for added/removed lines — tree-sitter's own theme covers it.

• Highlighter on @-mention echo (v2 polish per A7) — :highlight applies to assistant output only. Highlighting user-pasted code as it's echoed by readline would need a separate hook in the readline display path; out of scope here.

• Auto-refresh project tree on cd (v2 polish per A8) — the cd interceptor in executor.maybe_chdir is a clean place to call _scan_project_tree(libc.getcwd(), ...) on every successful cd. Skipped in v1 because the scan can be slow on large trees; manual refresh via :tree refresh is the v1 verb.

• Custom include/exclude globs for project tree (v2 polish per A9) — cfg.project = { include = {...}, exclude = {...} } would extend beyond .gitignore. v1 ships with .gitignore-only honor (via git ls-files --exclude-standard) plus the find fallback for non-repo cwds.

---

9. Risks

Risk	Mitigation
tree-sitter CLI not on fleet → most users get no highlighting	It's opt-in; default off; status warning on toggle when absent.
Highlighter latency on long code blocks (whole-block buffering)	Accepted trade-off vs corrupting output. If painful in practice, add a per-block size cap above which we pass-through unhighlighted.
git diff on huge changesets blows context budget	Diff output reuses enforce_budget eviction (it's just exec output). User can :diff <subdir> to scope. v2 could add a --max-bytes truncation.
git ls-files in a non-git cwd → falls back to find, may pick up node_modules / target / etc.	Document in config example; v2 could honor .aishignore or similar.
@<ref1>..<ref2> collides with paths like @../sibling.txt	A6: tiered resolution — try as path first; only fall through to diff retry when path lookup fails AND token contains ... @../sibling.txt hits the path branch and never reaches the diff retry.
Project tree injection adds tokens to every broker call	Char cap + opt-in auto_tree = false default. Suppressed under Norris.
:highlight on mid-stream produces inconsistent rendering for the in-flight turn	Toggle takes effect from the NEXT assistant turn. Document this.

---

10. Open Questions (Phase 6)

All six formulate-time Qs were resolved in analyze (A4–A9). None remain open as blockers for implementation.

#	Question	Resolution
Q-H1	popen3 for tree-sitter highlight	A4: tmpfile roundtrip — io.popen("w") writes body with stdout redirected to a tmp file, then io.open reads the file. Avoids ARGMAX + shell-escape complexity.
Q-D1	Confirm gate on :diff?	A5: no. git diff is read-only; matches :history / :sessions / :safety check (none gate). Permission DSL (#9) applies only to AI-suggested CMD: lines.
Q-D2	@<r1>..<r2> with refs containing /	A6: tiered resolution — file lookup first, then if it fails AND .. is present, retry as ref-range. @origin/main..feature naturally falls through to the retry; no grammar prefix needed.
Q-T1	cfg.project.auto_tree update on cd	A8: no auto-refresh in v1. :tree refresh is the manual verb; cd-intercept hook is documented as v2 polish in §8.
Q-T2	Custom include/exclude globs	A9: rely on .gitignore via git ls-files in repos; find fallback outside. Custom globs deferred to v2.
Q-H2	Highlighting on @-mention echo	A7: assistant-output only in v1. Echo via readline is a different code path; deferred to v2 (see §8).

---

11. Phase 6 → Phase 7+ Out-of-band

The §11 "Planned Phase Sequence" table in PHASE0.md does not list phases beyond 6. After Phase 6 lands, candidate next iterations (non-binding, for the formulate of Phase 7 to confirm):

• Phase 7: secret-redaction wiring into safety.lua (#52 follow-up filed during Phase 5/13 close); session-multiplex / tmux parity surfaces (out of scope per §12 — explicitly rejected); or other backlog as it accumulates on Gitea.

Phase 6 itself is self-contained — none of its three pillars introduce substrate dependencies on phases not yet planned.

---

12. Implementation Plan (commit-by-commit)

Bottom-up ordering: foundations first (context.lua field + composer), then the diff and tree surfaces that have no display-layer risk, then the highlighter (largest experimental surface — last so the rest of Phase 6 ships even if highlighter slips). Each commit leaves the tree green (existing tests pass + smoke ok) and adds a discrete capability.

Order

1. context.lua — [project] block plumbing. Add self.project (string, nil-allowed) on Context.new. Add compose_project(text) helper mirroring compose_background / compose_summary. In to_messages: insert between compose_background and compose_summary so the read order is memory → project tree → earlier-summary → NORRIS. Suppressed under self.norris_active (parity with R-C1 / R-C4). No behavior change yet — nothing sets ctx.project.

   R8: :reset does NOT clear ctx.project. Phase 4 established that :reset preserves ctx.memory_items (startup-injected facts survive a user-driven context reset); ctx.project follows the same rule. Compare Context:reset at context.lua ~343 — clears turns, pending_exec_output, summary; leaves memory_items and now project alone. Smoke: :to_messages() still empty when project nil; with project set, :reset then :to_messages() still shows the [project] block.

2. repl.lua — _scan_project_tree helper + :tree meta.

   • _scan_project_tree(dir, opts) per §6: git ls-files --cached --others --exclude-standard in a repo, find . -maxdepth N -type f -not -path '*/\.*' outside. Returns (body, info) where info = { file_count, truncated }.
   • :tree [N|refresh|off] meta: scans cwd, sets ctx.project, emits status with file count + truncation note.
   • cfg.project.auto_tree startup hook: if true, run _scan once and set ctx.project (before the main loop opens). Default false (existing configs unchanged).
   • Update HELP with :tree lines.
   • Smoke: in the aish repo, :tree injects a ~32-file block; :to_messages() shows the [project] block in the system prompt.

3. repl.lua — :diff meta + _git_clean_cmd helper (B1).

   • _git_clean_cmd(subcmd_and_args) returns the git --no-pager -c color.ui=never <subcmd_and_args> prefix. Used by :diff and the @<r1>..<r2> path in commit #4.
   • :diff [args] meta per §5 (B1-revised): runs the clean git command via executor.exec, appends [diff <args>]\n<out> to context as exec_output. Empty / non-repo / bad-ref paths emit status and skip.
   • Update HELP with :diff line.
   • Smoke: :diff from a dirty aish checkout injects the working tree diff; :diff staged works; :diff junkref emits status and skips.

4. repl.lua — expand_mentions tiered resolution (A6). Extend the existing path-resolution loop with the diff-retry branch from §5: if _read_truncated returns nil AND the token contains .., parse as <r1>..<r2> and try _git_clean_cmd( "diff <r1>..<r2>"). On success, replace with a fenced diff block. Preserves existing peel-on-trailing-punct logic. Smoke: @HEAD~1..HEAD expands inline; @origin/main..feature works when the ref exists; @../sibling.txt still resolves as file.

5. renderer.lua + repl.lua — tree-sitter highlighter. This commit is the largest single change in Phase 6. Substeps:

   a. _detect_treesitter() in repl.lua: one-shot popen of command -v tree-sitter && tree-sitter --version. Stash result on a local.

   b. renderer.lua — fence-aware state machine wrapping assistant_delta. Exports renderer.set_highlight(enabled, detected, highlight_fn) so repl.lua wires the toggle, cli-availability flag, AND the highlighted callback (R2: keeps executor dependency out of renderer.lua). State: outside (pass-through + tail accumulator per R1) / inside (buffer until closing fence). On close: call highlight_fn(body, lang) and emit. Algorithm per §4; bytes-of-cumulative-buf scan + tail lookahead handles fragment-across-boundary fences (B2 + R1).

   c. highlighted(body, lang) per §4 (B3 + R2 + R4): lives in repl.lua. Write body to os.tmpname(), invoke via executor.exec("cat tmp | tree-sitter highlight --lang X"), capture out + exit code, cleanup tmp, pass-through on failure. R4 implement-time check: verify the --lang flag exists on the installed CLI; if not, switch to tmpfile-with-extension and pass the path directly.

   d. :highlight [on|off|status] meta in repl.lua. :highlight on when CLI absent → status with install hint (B4); :highlight status always reports current toggle + CLI availability.

   e. HELP update. R9: status header bump moves to commit 6 (single owner; no duplication).

6. config.lua + docs/PHASE6 status bump (R9).

   • Add commented-out project = { auto_tree = false, tree_depth = 3, tree_max_chars = 4096 } block in config.lua (parity with the Phase 1-5 example blocks).
   • PHASE6.md status header → Implement (matches Phase 5 cadence — manifest tracks implementation state).

Risk index per commit

Commit	Risk	Mitigation
1 (compose_project)	Composition-order regression breaks Phase 4/5 callers	Order test: empty memory + empty project = identical sys_content to pre-Phase-6 baseline
2 (:tree)	find fallback picks up node_modules / target / build / etc.	Document in status warning; users in non-repo cwds scope via :tree <depth>
3 (:diff)	B1 — color/keypad codes leak if a future caller forgets the helper	All call sites must go through _git_clean_cmd; lint by grep before commit
4 (@..)	False positive on @../sibling.txt when no such file exists	A6's tiered resolution: only retry as diff when file lookup fails. @../sibling.txt resolves as path; if the path is missing, diff retry runs and naturally fails — same outcome as before
5 (highlighter)	Fence detector misclassifies inline ``` triple-backtick in prose	N1: state machine triggers on ^```` at start of stream OR after \n` only. §4 algorithm now encodes this constraint in the pseudocode.
5 (highlighter)	tmpfile race / leak on crash	os.remove(tmp) in normal exit path; OS cleans /tmp/lua_* files on reboot. Single-user trust per PHASE0 §12.
5 (highlighter)	R3: PTY raw-mode toggle on every code-block render (executor.exec -> libc.set_raw(0))	Smoke-test before locking: render an assistant turn with 5+ fenced blocks; watch for cursor flicker, SIGWINCH races, terminal state corruption. If problematic, alternate paths: direct io.popen for stdin-write (accept the lost exit code; treat empty output as failure) or run highlighter via os.execute with shell redirection.
5 (highlighter)	R4: tree-sitter highlight --lang X invocation grammar unverified	Implement-time CLI check (tree-sitter highlight --help). If --lang is wrong, fall back to extension-based: name the tmpfile lua_XXX.<ext> per lang_extension[lang] map and pass the path.
6 (config bump + status)	none — pure docs / commented config

Tests + smoke per commit

Each commit must:

• Pass luajit test_safety.lua (87/87) and luajit test_router_model.lua (31/31)
• Load cleanly: luajit -e 'package.path="./?.lua;./vendor/?.lua;"..package.path; require("repl"); print("ok")'
• Pass a feature-specific smoke (described per row above)

No new test framework dependency. Per-feature unit tests can live as inline luajit -e '...' blocks in commit messages or as a dedicated test_phase6.lua if the surface area justifies it (decide at impl-time).

Things deliberately NOT split into a separate commit

• _shq (shell-quote helper) — already exists in repl.lua from #3. Reuse in commit 5 (highlighter); no new helper.
• Lang map — small enough to copy locally in commit 5 (~15 lines); the existing _lang_of(path) in expand_mentions uses a similar but smaller map. Factor only if a third caller appears.
• Streaming-rehydration interaction with the highlighter — secrets_session rehydrate runs BEFORE the highlight filter in the chunk pipeline. Order: chunk → rehydrator:push → highlight_filter → emit. The highlighter operates on plain text only; rehydrated placeholders resolve to real values which the highlighter sees as code. No special wiring needed.

Open at plan-time (resolve at implement)

• R4 implement-time verification: confirm tree-sitter highlight --lang X works on the installed CLI. If not, switch to extension- based path passing. Block commit 5 ship on this check.
• R3 smoke test: render an assistant turn with 5+ fenced blocks through the highlighter; confirm no cursor flicker / SIGWINCH race / terminal-state corruption from per-block raw-mode toggle. If problematic, alternate paths listed in §12 risk row.
• Whether :highlight status should also probe tree-sitter --print-langs to show which langs are actually available. Nice-to-have; defer unless install paths produce variable lang sets in practice.