aish/docs/PHASE6.md

# aish — Phase 6 Manifest

**Project:** aish — AI-augmented conversational shell
**Document:** Phase 6 Requirements, Architecture & Design Decisions
**Status:** Implement (6 commits landed: c4fc7fd, d1dce83, 4d5f93a, 0d63f01, 11d0e59, this)
**Date:** 2026-05-16

**Review findings (independent agent, 2026-05-16) — 2 BLOCKERs resolved
in-place, 7 CONCERNs folded, 6 NITs applied:**

R1 (BLOCKER, RESOLVED). **§4 fence detector's `outside`-state branch
    drops the leading `'``'` chunk of a split-fence.** The §4
    pseudocode as written ("look for ` ```<lang>\n ` in chunk; if found
    [...] else: emit chunk as-is") emits the partial-fence chunk
    immediately, so the next chunk no longer sees the full marker.
    Contradicts B2's split-fence requirement. **Fix folded into §4:**
    `outside`-state also holds a small tail (up to 10 chars) when the
    chunk's tail could be a fence prefix; flushes on next push. Same
    pattern as the `secrets.lua` streaming rehydrator (`secrets.lua`
    ~213). Pseudocode + algorithm updated.

R2 (BLOCKER, RESOLVED). **`highlighted()` file placement was ambiguous
    in §3 vs §12.** `highlighted()` needs `_shq` (currently a `repl.lua`
    M.run-local closure) and `require("executor")`. **Resolution:**
    `highlighted()` stays in `repl.lua`; `renderer.lua` exposes
    `renderer.set_highlight(enabled, detected, highlight_fn)`. The
    filter state machine in `renderer.lua` calls back through
    `highlight_fn(body, lang)` at fence-close. No `executor` dependency
    in `renderer.lua`; no `_shq` lift. §3 + §12 commit 5 updated to
    state this explicitly.

R3 (CONCERN, FOLDED). **PTY raw-mode toggle per code block.** Each
    `executor.exec` call calls `libc.set_raw(0)` briefly. For an
    assistant turn with N fenced blocks that's N raw-mode toggles
    on the streaming hot path. Smoke-test for cursor/flicker before
    locking in. Added to §12 commit 5 risk row.

R4 (CONCERN, FOLDED — risk noted, needs verify at implement-time).
    **`tree-sitter highlight --lang X` invocation grammar is
    unverified.** The upstream `tree-sitter` CLI's `highlight`
    subcommand canonically takes a path argument and infers language
    from the file extension via `~/.config/tree-sitter/config.json`.
    A `--lang` flag may not exist. Since B4 confirmed zero fleet hosts
    have tree-sitter installed, this can't be probed locally.
    **Resolution:** §4 amended — at commit 5 implement time, VERIFY
    against a real install. If `--lang` is wrong, switch to writing
    the tmpfile with the matching extension (`/tmp/lua_XXX.py`) and
    pass the path. Path-based discovery is the CLI's documented
    primary mode.

R5 (CONCERN, FOLDED). **`:tree off` semantics ambiguous.** §6 listed
    it as "clear ctx.project" but didn't clarify whether subsequent
    `:tree` (no arg) re-uses cached opts or falls back to config
    defaults. Clarified in §6: `:tree off` is a one-shot clear of
    `ctx.project`; subsequent `:tree` re-scans with config defaults
    or the explicit arg if given.

R6 (CONCERN, FOLDED). **cwd-coupling differs between `:diff` and
    `:tree`.** `:diff` reads `libc.getcwd()` at meta invocation
    time; `:tree`'s captured `ctx.project` is fixed at scan time
    (per A8). After `cd /other-project`, `:diff` shows the new
    project's diff but `ctx.project` still holds the old project's
    tree. Documented in §5 (the diff section now cross-refs §6 / A8)
    so the user-facing expectation is clear.

R7 (CONCERN, FOLDED). **`:tree refresh` opts caching unspecified.**
    Should `:tree refresh` re-use the last explicit `:tree <N>` depth
    override, or fall back to `cfg.project.tree_depth`? Resolution:
    cache the last opts on `ctx._project_opts`; `:tree refresh` reuses
    them; falls back to config defaults if no prior call. §6 updated.

R8 (CONCERN, FOLDED). **`:reset` interaction with `ctx.project`.**
    Phase 4 established that `:reset` does NOT clear `ctx.memory_items`
    (parity is desirable — startup-injected facts persist across a
    user-driven context reset). `ctx.project` should follow the same
    rule: `:reset` clears `ctx.turns` and `pending_exec_output` and
    `ctx.summary` (per `Context:reset` at `context.lua` ~343), but
    NOT memory_items and NOT project. Documented in §3 + §12 commit 1.

R9 (CONCERN, FOLDED). **Status-bump duplication between §12 commits 5
    and 6.** Commit 5 sub-step (e) said "PHASE6 status → Implement";
    commit 6 also said the same. Resolved: commit 5e does NOT bump
    the status (only HELP update); commit 6 owns the status bump
    (along with the config example). One owner per change.

R-N1..N6 (NITs, APPLIED):
  N1. §4 algorithm pseudocode now includes the SOL/post-newline
      anchor requirement (mid-line backticks in prose don't open a
      fence). The plan §12 risk row already promised this; now §4
      matches.
  N2. §4 detection block gained a comment explaining the `read("*l")
      and pipe:close()` pattern — close return-value is ignored per
      B3; presence of an output line is the signal.
  N3. §5 `:diff staged → git diff --cached` table row dropped (the
      meta is a thin pass-through; user types the right git flags).
      `:diff --cached` works directly. Surface is honest.
  N4. §6 `_scan_project_tree` switched from `os.execute("cd " .. shq
      .. " && git rev-parse ...")` to `git -C <dir> rev-parse
      --git-dir` — no subshell, more idiomatic.
  N5. §12 "Open at plan-time" first bullet (dir-arg vs hardcoded
      getcwd) dropped — already decided in §6's signature; not open.
  N6. §11 wording on Phase 7+ left as-is (reviewer marked purely
      cosmetic).

**Analyze findings (2026-05-16):**

A1. **renderer.lua surface clean** — `assistant_delta(chunk)` already
    concatenates into a `stream_buf` then `emit()`s the chunk;
    `assistant_flush()` finalizes with a trailing newline if missing.
    The fence-aware highlight filter slots in between chunk receipt and
    `emit` without restructuring; no callers besides `repl.lua` touch
    `stream_buf` so the filter state can live alongside it.

A2. **executor surface clean** — `executor.exec(cmd)` already
    forkpty-spawns, captures + live-streams output, returns `(out, code)`.
    Phase 6's `:diff` and `_scan_project_tree` reuse this path verbatim;
    no new IO model. `git`-rooted commands inherit cwd from the parent
    (which `libc.chdir` already mutates), so a `:diff` after `cd` reads
    the right repo.

A3. **context composition order locked** — current `to_messages` builds
    `sys_content = base + [background] + [earlier summary] + NORRIS suffix`.
    Phase 6 inserts `[project]` between `[background]` and `[earlier
    summary]`. Same Norris-suppression guard already in place
    (`if not self.norris_active`).

A4. **Q-H1 RESOLVED: tmpfile roundtrip** for `tree-sitter highlight`
    write+read. Avoids ARGMAX risk on large code blocks (vs `printf
    BODY | tree-sitter ...`) and shell-escape complexity. Two file
    handles, deterministic cleanup via `os.remove`. Sketch:

    ```lua
    local tmp = os.tmpname()
    local w = io.popen(("tree-sitter highlight --lang %s > %s")
                       :format(lang, tmp), "w")
    w:write(body); local _, _, code = w:close()
    local f = io.open(tmp, "rb"); local out = f:read("*a"); f:close()
    os.remove(tmp)
    if code ~= 0 then return body end  -- pass-through on failure
    return out
    ```

A5. **Q-D1 RESOLVED: no confirm gate on `:diff`.** `git diff` is
    read-only; matches `:history`, `:sessions`, `:safety check` —
    none of which gate. Permission DSL (#9) only applies to AI-suggested
    `CMD:` lines, not user-issued metas.

A6. **Q-D2 RESOLVED: tiered resolution for `@<token>`.** The mention
    parser tries `<token>` as a file path first; if it doesn't resolve
    AND the token contains `..`, retry as a diff range. This keeps
    `@../sibling.txt` (path) working AND allows `@origin/main..feature`
    (ref range — resolves via second attempt since no such file exists).
    No grammar prefix needed.

A7. **Q-H2 RESOLVED: highlighting is assistant-output only in v1.**
    `expand_mentions` content lands in the user-turn payload — visible
    on the terminal via readline echo, not via `assistant_delta`. Filing
    "highlight @path-expanded code in echo" as v2 polish. Reason:
    intercepting readline echo for ANSI injection is non-trivial and
    orthogonal to the stream filter.

A8. **Q-T1 RESOLVED: project tree captured at scan time, not auto-
    refreshed on cd.** `cd /other-project` leaves the existing
    `ctx.project` stale; `:tree refresh` is the manual verb to update.
    Auto-refresh on cd intercept is a v2 polish (the cd interceptor in
    `executor.maybe_chdir` is a clean hook for it).

A9. **Q-T2 RESOLVED: rely on `.gitignore` via `git ls-files`** in repos;
    fall back to `find` with simple excludes outside. Custom
    include/exclude glob lists deferred to v2. Reason: most users live
    inside git repos; `.gitignore` already encodes their notion of
    "noise". Out-of-repo users get the simple fallback and can scope
    via `:tree <depth>`.

A10. **`expand_mentions` punct-peel does NOT strip `/`** — so
     `@HEAD~1..HEAD,` peels the `,` and the underlying token `HEAD~1..HEAD`
     has no slash; the path-then-diff retry from A6 catches it. No new
     peel logic needed.

A11. **Auto-injection ordering for `[project]`** — if both `cfg.memory.
     inject_max_chars` and `cfg.project.auto_tree` fire at startup, the
     order is: memory load → tree scan → first ask_ai. The composition
     in `to_messages` places `[background]` (memory) before `[project]`
     so the model reads memory facts before file tree. Documented in §3.

A12. **Norris interaction** — `[project]` block follows the established
     [background]/[earlier summary] suppression rule under
     `ctx.norris_active`. Planner stays on its goal anchor; the tree
     can be re-introduced via the goal text if needed. Matches R-C1/R-C4.

PHASE0 is the locked substrate; PHASE1-5 are layered on top. This manifest
specifies what Phase 6 adds — **tree-sitter syntax highlighting hooks**,
**diff-aware code injection**, and **project-level context (file-tree
summary)**.

---

## 1. Scope of Phase 6

Three pillars per PHASE0.md §11 row 6:

1. **Tree-sitter syntax highlighting hooks** — when an external
   `tree-sitter` CLI is detected at startup, assistant code-fence
   content is filtered through it for ANSI-colorized display. Plain
   prose streams unchanged. When the CLI is absent, the filter is the
   identity function (zero overhead, zero hard dependency). Toggleable
   at runtime with `:highlight on|off`. Default off until the user
   opts in (don't surprise existing users with a display change).
   Per B4: tree-sitter is **absent on every fleet host probed**;
   `:highlight on` when the CLI is missing emits a status that names
   the install hint (`apt install tree-sitter` / `cargo install
   tree-sitter-cli`) rather than silently falling back to identity.

2. **Diff-aware code injection** — surface git diffs as first-class
   context. Two entry points:

   - Meta verb: `:diff [args]` runs `git diff <args>` from cwd, appends
     output to context as exec-output. `:diff staged`, `:diff HEAD~3`,
     `:diff main..feature` all delegate to git's argument grammar.
   - @-mention extension: `@HEAD..feature` (a ref-range expression
     anywhere a `@path` would go) expands inline as a fenced `diff`
     block, mirroring how `@README.md` already works.

3. **Project-level context (file-tree summary)** — `git ls-files`-based
   tree summary of the cwd, injected as a `[project]` block in the
   system prompt. Two entry points:

   - Meta verb: `:tree [depth]` injects on demand; `:tree refresh`
     re-scans.
   - Auto-inject at startup when `cfg.project.auto_tree = true` —
     gated like memory injection so existing configs don't change
     behavior.

**Phase 6 is done when:**

- With `tree-sitter` CLI installed and `:highlight on`, the assistant
  reply ```py\nprint("hi")\n``` shows up with ANSI colors. Without
  the CLI, `:highlight on` is a no-op + emits a status warning.
- `:diff` from a dirty git repo shows the working-tree diff in the
  exec-output frame; the model sees it on the next ask_ai turn.
- `@HEAD~1..HEAD` in a prompt expands inline to a fenced diff block.
- `:tree` injects a `[project] <N files>:` block visible in
  `ctx:to_messages()` (via the system prompt assembly).
- With `cfg.project.auto_tree = true`, the project block appears on
  every broker call (subject to `max_chars` cap).
- Existing configs without `cfg.project` and with `:highlight off`
  (default) behave exactly like Phase 5 (Phase 5 regression coverage).

---

## 2. Technology Decisions (delta from Phase 5)

| Decision | Choice | Rationale |
|---|---|---|
| Highlight backend | External `tree-sitter` CLI (`tree-sitter highlight --lang X`) | Honors PHASE0 §3: no compiled extensions, no luarocks. Detected once at startup; absence → identity filter. Opt-in via `:highlight on` so install-state changes don't break users. |
| Highlight buffering | Accumulate inside fenced code blocks, emit on closing fence; pass-through outside fences | Streaming UX preserved for prose. Code blocks get colorized atomically, accepting a per-block latency (~ block streaming time). Per-chunk highlighting would split a token across `tree-sitter` invocations and corrupt the output. |
| Lang detection | First-line fence info-string (` ```py`, ` ```python`, ` ```lua`) → normalized via small map (py→python, js→javascript, etc.) | The lang tag mirrors the one we already emit in `expand_mentions` (#7). No tag → identity (no highlight). |
| Diff backend | Shell out to `git diff <args>` via `executor.exec` | Honors substrate (no libgit2 FFI). The existing exec frame handles capture + stream. `git` is universally present where aish makes sense. |
| Diff failure | Bail with status `[aish] :diff failed (not a git repo / bad ref)`; do NOT inject empty output | Avoids polluting context with stale or empty diffs. |
| Tree backend | `git ls-files --cached --others --exclude-standard` when cwd is a git repo, else `find . -type f -not -path './.*'` | Free `.gitignore` honor in repos; sensible default outside. Both are POSIX-portable. |
| Tree summary form | Sorted relative paths, grouped by directory at depth ≤ `cfg.project.tree_depth` (default 3), truncated by char count `cfg.project.tree_max_chars` (default 4096) | One block, deterministic order, cheap to compute. Matches the [background] memory block convention (Phase 4) so the system prompt's compositional shape stays familiar. |
| Tree injection point | `context.lua`: new `compose_project(...)` adds a `[project] <header>\n<body>` block to the system content, between [background] and [earlier summary] | Same suppression rule as [background]/[earlier summary]: NOT injected during Norris (R-C1 / R-C4 — planner stays on its anchor). |
| Tree refresh policy | One scan at startup if auto; `:tree refresh` to re-scan on demand | Scanning on every ask_ai is wasteful for slow filesystems. Manual refresh is sufficient for v1. |
| @-mention diff syntax | `@<ref>..<ref>` (two `..` separator) only — recognized via the existing trailing-punct peel logic | Avoids ambiguity with literal paths. `@HEAD` alone is NOT a diff trigger (would collide with files literally named HEAD). |

---

## 3. Module Changes

| File | State after Phase 5 | Phase 6 changes |
|---|---|---|
| `renderer.lua` | `assistant_delta(text)` writes chunks; `assistant_flush()` finalizes | Add fence-aware filter inside the assistant stream. State machine: outside-fence (pass-through) / inside-fence (buffer, emit on close). On close, pipe buffer through `tree-sitter highlight --lang <X>` (if highlight enabled), emit result. Toggle exposed as `renderer.set_highlight(bool)`. |
| `executor.lua` | `extract_cmd_lines`, `extract_cmd_bg_lines`, `extract_delegate_lines` | No changes. Diff and tree use the existing `exec` path. |
| `context.lua` | system prompt = base + [background] + [earlier summary] + NORRIS suffix | Add `self.project = "..."` string field + `compose_project(self.project)` helper. Injection between [background] and [earlier summary] (A11: memory facts read before file tree). Suppressed under Norris (A12, parity with R-C1/R-C4). |
| `repl.lua` | meta dispatch + main loop + #13 secrets wiring | New helpers: `_detect_treesitter()` (run once at startup), `_run_git_diff(args)`, `_scan_project_tree(dir, opts)`. New meta: `:highlight`, `:diff`, `:tree`. Extend `expand_mentions` to recognize `<ref>..<ref>` token shape. |
| `config.lua` | example blocks for mcp/safety/memory/routing/secrets/etc. | Add commented-out `project = { auto_tree = false, tree_depth = 3, tree_max_chars = 4096 }` block. |

No new module files in v1. Three new helpers in `repl.lua` keep the
file growing but consolidate the Phase 6 surface. If the highlighter
filter grows past ~80 LOC, lift it into `highlight.lua` as a follow-up.

---

## 4. Pillar 1 — Tree-sitter highlighting

### Detection (startup, once)

```lua
local function _detect_treesitter()
    local pipe = io.popen("command -v tree-sitter 2>/dev/null && tree-sitter --version 2>/dev/null")
    -- N2 / B3: pipe:close() returns true on LuaJIT regardless of exit
    -- code; we don't use it for the verdict. Presence of an output
    -- line from --version is the actual signal.
    local ok = pipe and pipe:read("*l") and pipe:close()
    return ok
end
```

If not present, `renderer.set_highlight(true)` emits a status warning
and leaves the filter as a no-op. Don't error; the user can install
tree-sitter and re-toggle.

### Stream filter

The filter wraps `renderer.assistant_delta`. State machine (R1 + N1
revisions — outside-state accumulator + SOL anchor):

```
state    = "outside" | "inside"
tail     = ""                  -- outside-state lookahead buffer (R1)
buf      = ""                  -- only used in "inside"
lang     = nil                 -- captured at fence open

push(chunk):
  if state == "outside":
      combined = tail .. chunk
      -- R1: hold back trailing partial-fence so a split fence
      -- ("``" arrives, then "`python\n") doesn't get emitted
      -- as plain text before we recognize the opener.
      -- N1: fence opens only at start-of-stream OR after a newline
      -- ("^```" or "\n```"). Inline backticks in prose don't open.
      match_pos = find(combined, "(^|\n)```([%w_-]*)\n")
      if match_pos:
          -- everything before the opening is plain text
          emit combined[1 .. fence_start - 1]
          lang = captured_lang
          buf  = combined[fence_end .. end]   -- text after \n
          state = "inside"; tail = ""
          if buf has \n``` inside, fall through to inside-state below
      else:
          -- Hold back the last K chars if they could be the start
          -- of a fence-open. Specifically: tail = the longest suffix
          -- of combined that is a prefix of any well-formed fence
          -- marker ("`", "``", "```", "```l", "```lua", "```lua\n").
          -- Bounded by max-lang-tag-length + 4 (~10 chars in practice).
          tail = longest_partial_fence_suffix(combined, max=10)
          emit combined[1 .. #combined - #tail]
          -- (next push will combine tail with the next chunk and retry)

  if state == "inside":
      buf = buf .. chunk
      -- closing fence: "\n```" anywhere in buf (followed by EOL or end).
      close_pos = find(buf, "\n```")
      if close_pos:
          fence_body = buf[1 .. close_pos - 1]
          closing    = buf[close_pos .. close_pos + 3]   -- "\n```"
          rest       = buf[close_pos + 4 .. end]
          emit highlighted(fence_body, lang)
          emit closing verbatim
          state = "outside"; buf = ""; tail = ""
          if rest != "":
              push(rest)   -- recurse for any plain text after the closing
      else:
          -- still buffering; nothing emitted this push
```

Edge cases:
- Chunk boundary lands inside an opening marker (e.g., chunk ends with
  `'``'`, next starts with `'`python\n'`). The `tail` buffer holds
  `'``'`; next push combines and finds the full opener.
- Chunk boundary inside a closing marker. The `inside` branch already
  accumulates into `buf`; `find` against cumulative `buf` recovers.
- Inline backticks in prose (`"use ``` to mark code"`). N1's
  `(^|\n)```` anchor means this does NOT open a fence — `\n` is
  required before the three backticks.

The `tail` is bounded (max ~10 chars), so streaming UX latency is at
most 10 chars worth of buffering when between fenced blocks. The
existing `assistant_delta`'s `stream_buf` for full-text accumulation
is unaffected — the filter sits BEFORE `emit`.

`highlighted(body, lang)` — **B3 + R2 + R4-revised**:

Lives in `repl.lua` (per R2; `renderer.lua` calls it via the
`highlight_fn` passed to `renderer.set_highlight`). Has access to
`_shq` (existing helper from #3) and the `executor` require.

```lua
-- repl.lua local. Wired into renderer via:
--   renderer.set_highlight(true, treesitter_present, highlighted)
local function highlighted(body, lang)
    if not highlight_enabled or not lang_map[lang] then return body end

    -- R4: tree-sitter highlight CLI grammar is UNVERIFIED.
    -- Upstream `tree-sitter highlight` canonically takes a path and
    -- infers language from the file extension. At commit-5 implement
    -- time, install tree-sitter and check whether `--lang` exists.
    -- If not, name the tmpfile with the language's canonical extension
    -- (lang_extension[lang]) and pass the path directly:
    --   tmp = os.tmpname() .. lang_extension[lang]
    --   cmd = "tree-sitter highlight " .. _shq(tmp)
    -- Below is the optimistic --lang form for code reading; the actual
    -- implementation must be verified.

    local tmp = os.tmpname()
    local f = io.open(tmp, "wb")
    if not f then return body end
    f:write(body); f:close()
    -- B3: io.popen():close() doesn't expose exit codes in LuaJIT.
    -- Route via executor.exec which uses pty.spawn+waitpid and
    -- returns (out, exit_code) reliably.
    local out, code = executor.exec(
        ("cat %s | tree-sitter highlight --lang %s")
        :format(_shq(tmp), lang_map[lang]))
    os.remove(tmp)
    if code ~= 0 then return body end   -- pass-through on highlighter failure
    return out
end
```

Why this shape (and not the formulate-time A4 sketch):

- **R2 file placement**: `highlighted` lives in `repl.lua` so it has
  natural access to `_shq` + `executor`. `renderer.lua` stays free of
  the `executor` require; it calls back through `highlight_fn`.
- **B3 exit-code path**: LuaJIT (5.1 contract) doesn't expose the exit
  status via `io.popen(...):close()`. `executor.exec` is the only
  reliable channel in our substrate.
- **R4 grammar verification**: the `--lang` flag is the formulate-time
  assumption; the upstream CLI's `highlight` subcommand may want a
  PATH with a recognized extension instead. Implement-time check
  required before commit 5 ships.
- The tmpfile stays — avoids ARGMAX on `printf '%s' BODY |` and
  sidesteps shell-escape edge cases on arbitrary code-block bytes.
- Cost: one syscall round (tmpfile create/remove) + one pty spawn per
  code block — negligible vs the highlighter latency.

### Lang map (v1)

```lua
local LANG_MAP = {
    py = "python", python = "python",
    lua = "lua",
    js = "javascript", javascript = "javascript", ts = "typescript",
    sh = "bash", bash = "bash",
    c = "c", h = "c", cpp = "cpp", cc = "cpp",
    rs = "rust", go = "go", java = "java", rb = "ruby",
    md = "markdown", json = "json",
}
```

Reuses the same map as `expand_mentions`. Factor into a shared
helper once both reference it (small `_lang_of_ext()` in repl.lua).

### Toggle

`:highlight` (no arg) → flip. `:highlight on|off` → set explicit.
`:highlight status` → report enabled + whether tree-sitter is present.
Default: off (don't change existing-user UX).

---

## 5. Pillar 2 — Diff-aware code injection

### Meta: `:diff [args]`

- `:diff` → `git diff` (working tree vs index)
- `:diff HEAD` → `git diff HEAD`
- `:diff --cached` → `git diff --cached` (staged-only)
- `:diff main..feature` → `git diff main..feature`
- `:diff <anything else>` → passed verbatim to `git diff <anything>`

N3: the meta is a thin pass-through to `git diff`. Don't introduce
aliases like `staged` that would diverge from git's own grammar — the
user types the real flag (`--cached`) and aish doesn't second-guess.

R6: `:diff` reads `libc.getcwd()` at **meta-invocation** time. Compare
with `:tree` / `ctx.project` which captures the cwd at **scan** time
(A8): after `cd /other-project`, `:diff` shows the new project's diff,
but `ctx.project` still holds the old project's tree until `:tree
refresh`.

Implementation — **B1-revised** (must disable pager + color):

```lua
meta.diff = function(args)
    args = (args or ""):gsub("^%s+", ""):gsub("%s+$", "")
    -- B1: forkpty makes git think it's interactive, enabling color
    -- ANSI + DEC keypad/line-clear escapes that pollute the injected
    -- context block. --no-pager kills the keypad sequences; --color=
    -- never kills the color codes. Both are required.
    local cmd = "git --no-pager -c color.ui=never diff " .. args
    local out, code = executor.exec(cmd)
    if code ~= 0 then
        renderer.status(("diff failed (exit %d)"):format(code))
        return
    end
    if out == "" or out:gsub("%s", "") == "" then
        renderer.status("(no diff)")
        return
    end
    ctx:append_exec_output(("[diff %s]\n%s"):format(
        args == "" and "(working tree)" or args, out))
end
```

The `[diff ...]\n<output>` framing matches the `[bg:N exited]` /
`[delegate X]` conventions established in Phase 5 / #6 / #8.

The same `--no-pager -c color.ui=never` prefix applies to the
`@<r1>..<r2>` resolution path in the next section, and to any
future git verbs we add (`:log`, `:show`, etc.). Factor into a
helper `_git_clean_cmd(subcmd)` if multiple call sites accumulate.

### @-mention: `@<ref1>..<ref2>` — tiered resolution (A6)

Extends `expand_mentions` (#7) by adding a SECOND resolution attempt
when the first (path lookup) fails AND the token contains `..`:

```lua
-- Existing path-attempt block ends with content = _read_truncated(path)
-- which returns nil if no such file. Add the diff retry there:

if not content and path:find("..", 1, true) then
    local r1, r2 = path:match("^(.-)%.%.(.+)$")
    if r1 and r2 and r1 ~= "" and r2 ~= "" then
        -- B1: --no-pager + color=never (same as the :diff meta path).
        -- B3: io.popen close() doesn't expose exit codes — use the
        -- file-redirect trick OR executor.exec. Here we want a quick
        -- best-effort and the cost of an extra forkpty is acceptable.
        local out, code = executor.exec(
            ("git --no-pager -c color.ui=never diff %s..%s 2>/dev/null")
                :format(shq(r1), shq(r2)))
        if code == 0 and out:match("%S") then
            content = out
            -- Note: language tag becomes "diff" regardless of path lang
            lang_override = "diff"
        end
    end
end
```

Output replaces the token with:

````
```diff path=<r1>..<r2>
<content>
```
````

Tiered resolution semantics:
- `@README.md` → file lookup succeeds → file expansion
- `@../sibling.txt` → file lookup succeeds → file expansion
- `@HEAD~1..HEAD` → file lookup fails, `..` present, ref-range succeeds → diff
- `@origin/main..feature` → file lookup fails (no such file), `..` present,
  ref-range succeeds → diff. The token has `/` in `r1` but `git diff` accepts
  it as a ref; no `/`-based heuristic needed (resolves Q-D2).
- `@nonexistent-file..but-also-not-a-ref` → both fail; literal token
  preserved with the existing `[aish] @X: not found` status path.

---

## 6. Pillar 3 — Project file-tree

### Meta: `:tree [depth]`

- `:tree` → scan + inject with default depth and char cap; if a
  prior `:tree <N>` set a depth override, this re-scan uses the
  config defaults (`:tree` resets to defaults)
- `:tree <N>` → override depth for this scan; cached as
  `ctx._project_opts` for `:tree refresh`
- `:tree refresh` → re-scan with `ctx._project_opts` (last explicit
  opts) if present; otherwise config defaults (R7)
- `:tree off` → clear `ctx.project` AND `ctx._project_opts`. Future
  `:tree` (no arg) re-scans with config defaults. One-shot semantics
  — there's no "disabled until re-enabled" flag (R5).

### Scan logic

```lua
local function _scan_project_tree(dir, opts)
    opts = opts or {}
    local max_chars = opts.max_chars or 4096
    local depth     = opts.depth or 3

    -- Prefer git ls-files for .gitignore honor; fall back to find.
    -- N4: `git -C <dir>` skips the subshell vs `cd && git ...`.
    local in_git = os.execute(("git -C %s rev-parse --git-dir >/dev/null 2>&1"):format(shq(dir))) == 0
    local listcmd
    if in_git then
        listcmd = ("git -C %s ls-files --cached --others --exclude-standard"):format(shq(dir))
    else
        listcmd = ("find %s -maxdepth %d -type f -not -path '*/\\.*' 2>/dev/null"):format(shq(dir), depth + 1)
    end

    local pipe = io.popen(listcmd)
    if not pipe then return nil, "scan failed" end

    local files = {}
    for line in pipe:lines() do
        -- Depth filter: count `/` separators
        local _, slashes = line:gsub("/", "")
        if slashes < depth then files[#files + 1] = line end
    end
    pipe:close()

    table.sort(files)

    -- Build a tree-ish summary, truncate by char count.
    local body = table.concat(files, "\n")
    local truncated = false
    if #body > max_chars then
        body = body:sub(1, max_chars) .. "\n... (truncated)"
        truncated = true
    end
    return body, { file_count = #files, truncated = truncated }
end
```

### Injection

`ctx.project = "..."` (string), composed into the system prompt
between [background] and [earlier conversation summary]:

```
[project] 142 files (truncated at 4096B):
README.md
broker.lua
config.lua
context.lua
...
```

Suppressed under Norris (R-C1 / R-C4 — planner stays focused; the
project context can be re-introduced via the Norris goal text if
needed).

### Auto-inject

`cfg.project.auto_tree = true` runs the scan once at startup and
sets `ctx.project`. Default false (existing configs unchanged).

---

## 7. UX Surface Summary

| Meta | Behavior |
|---|---|
| `:highlight [on/off/status]` | Toggle tree-sitter highlighter (no-op when CLI absent) |
| `:diff [args]` | `git diff <args>`, append output to context as `[diff ...]` |
| `:tree [N/refresh/off]` | Scan/refresh/clear project file-tree block |

| @-mention | Behavior |
|---|---|
| `@path` | Existing (#7) file expansion |
| `@<ref1>..<ref2>` | New: inline `git diff <r1>..<r2>` expansion |

| Config | Default | Effect |
|---|---|---|
| `cfg.project.auto_tree` | `false` | Inject project tree at startup |
| `cfg.project.tree_depth` | `3` | Depth filter for the scan |
| `cfg.project.tree_max_chars` | `4096` | Truncation cap for the injected block |
| (no config flag for `:highlight`) | — | Runtime toggle only; no persistence in v1 |

---

## 8. Out of Scope (Phase 6)

- **Pure-Lua syntax highlighter** — defer to a future phase if
  tree-sitter CLI absence becomes a practical pain point. v1 says
  "install tree-sitter or accept plain text".
- **bat/glow/chroma integration** — only `tree-sitter` is wired.
  Other highlighters can be added behind the same `:highlight` toggle
  later (config field `cfg.highlight.backend = "tree-sitter"|"bat"|...`).
- **Smart diff context selection** — no AI-driven "which diff to show".
  User explicitly says `:diff <range>` or `@<r1>..<r2>`.
- **File-tree LRU / smart summarization** — v1 is a flat truncated list.
  Hierarchical roll-up ("docs/ — 8 files") is a v2 polish.
- **Watching for file changes** — no fs-notify reload. Re-scan via
  `:tree refresh`.
- **Diff history** — `:diff` doesn't track its previous invocations.
  Each invocation is independent.
- **Inline diff highlighting** — the `diff` lang is in `LANG_MAP` so
  `tree-sitter highlight --lang diff` works, but we don't ship custom
  ANSI for added/removed lines — tree-sitter's own theme covers it.

- **Highlighter on @-mention echo** (v2 polish per A7) — `:highlight`
  applies to assistant output only. Highlighting user-pasted code as
  it's echoed by readline would need a separate hook in the readline
  display path; out of scope here.

- **Auto-refresh project tree on `cd`** (v2 polish per A8) — the cd
  interceptor in `executor.maybe_chdir` is a clean place to call
  `_scan_project_tree(libc.getcwd(), ...)` on every successful cd.
  Skipped in v1 because the scan can be slow on large trees; manual
  refresh via `:tree refresh` is the v1 verb.

- **Custom include/exclude globs for project tree** (v2 polish per A9) —
  `cfg.project = { include = {...}, exclude = {...} }` would extend
  beyond `.gitignore`. v1 ships with `.gitignore`-only honor (via
  `git ls-files --exclude-standard`) plus the `find` fallback for
  non-repo cwds.

---

## 9. Risks

| Risk | Mitigation |
|---|---|
| `tree-sitter` CLI not on fleet → most users get no highlighting | It's opt-in; default off; status warning on toggle when absent. |
| Highlighter latency on long code blocks (whole-block buffering) | Accepted trade-off vs corrupting output. If painful in practice, add a per-block size cap above which we pass-through unhighlighted. |
| `git diff` on huge changesets blows context budget | Diff output reuses `enforce_budget` eviction (it's just exec output). User can `:diff <subdir>` to scope. v2 could add a `--max-bytes` truncation. |
| `git ls-files` in a non-git cwd → falls back to `find`, may pick up node_modules / target / etc. | Document in config example; v2 could honor `.aishignore` or similar. |
| @`<ref1>..<ref2>` collides with paths like `@../sibling.txt` | A6: tiered resolution — try as path first; only fall through to diff retry when path lookup fails AND token contains `..`. `@../sibling.txt` hits the path branch and never reaches the diff retry. |
| Project tree injection adds tokens to every broker call | Char cap + opt-in `auto_tree = false` default. Suppressed under Norris. |
| `:highlight on` mid-stream produces inconsistent rendering for the in-flight turn | Toggle takes effect from the NEXT assistant turn. Document this. |

---

## 10. Open Questions (Phase 6)

All six formulate-time Qs were resolved in analyze (A4–A9). None remain
open as blockers for implementation.

| # | Question | Resolution |
|---|---|---|
| Q-H1 | popen3 for `tree-sitter highlight` | A4: tmpfile roundtrip — `io.popen("w")` writes body with stdout redirected to a tmp file, then `io.open` reads the file. Avoids ARGMAX + shell-escape complexity. |
| Q-D1 | Confirm gate on `:diff`? | A5: no. `git diff` is read-only; matches `:history` / `:sessions` / `:safety check` (none gate). Permission DSL (#9) applies only to AI-suggested `CMD:` lines. |
| Q-D2 | `@<r1>..<r2>` with refs containing `/` | A6: tiered resolution — file lookup first, then if it fails AND `..` is present, retry as ref-range. `@origin/main..feature` naturally falls through to the retry; no grammar prefix needed. |
| Q-T1 | `cfg.project.auto_tree` update on cd | A8: no auto-refresh in v1. `:tree refresh` is the manual verb; cd-intercept hook is documented as v2 polish in §8. |
| Q-T2 | Custom include/exclude globs | A9: rely on `.gitignore` via `git ls-files` in repos; `find` fallback outside. Custom globs deferred to v2. |
| Q-H2 | Highlighting on @-mention echo | A7: assistant-output only in v1. Echo via readline is a different code path; deferred to v2 (see §8). |

---

## 11. Phase 6 → Phase 7+ Out-of-band

The §11 "Planned Phase Sequence" table in PHASE0.md does not list
phases beyond 6. After Phase 6 lands, candidate next iterations
(non-binding, for the formulate of Phase 7 to confirm):

- **Phase 7**: secret-redaction wiring into `safety.lua` (#52
  follow-up filed during Phase 5/13 close); session-multiplex / tmux
  parity surfaces (out of scope per §12 — explicitly rejected);
  or other backlog as it accumulates on Gitea.

Phase 6 itself is self-contained — none of its three pillars introduce
substrate dependencies on phases not yet planned.

---

## 12. Implementation Plan (commit-by-commit)

Bottom-up ordering: foundations first (context.lua field + composer),
then the diff and tree surfaces that have no display-layer risk, then
the highlighter (largest experimental surface — last so the rest of
Phase 6 ships even if highlighter slips). Each commit leaves the tree
green (existing tests pass + smoke ok) and adds a discrete capability.

### Order

1. **`context.lua` — `[project]` block plumbing.** Add `self.project`
   (string, nil-allowed) on `Context.new`. Add `compose_project(text)`
   helper mirroring `compose_background` / `compose_summary`. In
   `to_messages`: insert between `compose_background` and
   `compose_summary` so the read order is memory → project tree →
   earlier-summary → NORRIS. Suppressed under `self.norris_active`
   (parity with R-C1 / R-C4). No behavior change yet — nothing sets
   `ctx.project`.

   **R8: `:reset` does NOT clear `ctx.project`.** Phase 4 established
   that `:reset` preserves `ctx.memory_items` (startup-injected facts
   survive a user-driven context reset); `ctx.project` follows the
   same rule. Compare `Context:reset` at `context.lua` ~343 — clears
   `turns`, `pending_exec_output`, `summary`; leaves `memory_items`
   and now `project` alone. Smoke: `:to_messages()` still empty when
   project nil; with project set, `:reset` then `:to_messages()`
   still shows the `[project]` block.

2. **`repl.lua` — `_scan_project_tree` helper + `:tree` meta.**
   - `_scan_project_tree(dir, opts)` per §6: `git ls-files --cached
     --others --exclude-standard` in a repo, `find . -maxdepth N
     -type f -not -path '*/\.*'` outside. Returns `(body, info)`
     where `info = { file_count, truncated }`.
   - `:tree [N|refresh|off]` meta: scans cwd, sets `ctx.project`,
     emits status with file count + truncation note.
   - `cfg.project.auto_tree` startup hook: if true, run `_scan` once
     and set `ctx.project` (before the main loop opens). Default
     false (existing configs unchanged).
   - Update HELP with `:tree` lines.
   - Smoke: in the aish repo, `:tree` injects a ~32-file block;
     `:to_messages()` shows the `[project]` block in the system prompt.

3. **`repl.lua` — `:diff` meta + `_git_clean_cmd` helper (B1).**
   - `_git_clean_cmd(subcmd_and_args)` returns the `git --no-pager
     -c color.ui=never <subcmd_and_args>` prefix. Used by `:diff`
     and the `@<r1>..<r2>` path in commit #4.
   - `:diff [args]` meta per §5 (B1-revised): runs the clean git
     command via `executor.exec`, appends `[diff <args>]\n<out>`
     to context as exec_output. Empty / non-repo / bad-ref paths
     emit status and skip.
   - Update HELP with `:diff` line.
   - Smoke: `:diff` from a dirty aish checkout injects the working
     tree diff; `:diff staged` works; `:diff junkref` emits status
     and skips.

4. **`repl.lua` — `expand_mentions` tiered resolution (A6).**
   Extend the existing path-resolution loop with the diff-retry
   branch from §5: if `_read_truncated` returns nil AND the token
   contains `..`, parse as `<r1>..<r2>` and try `_git_clean_cmd(
   "diff <r1>..<r2>")`. On success, replace with a fenced `diff`
   block. Preserves existing peel-on-trailing-punct logic. Smoke:
   `@HEAD~1..HEAD` expands inline; `@origin/main..feature` works
   when the ref exists; `@../sibling.txt` still resolves as file.

5. **`renderer.lua` + `repl.lua` — tree-sitter highlighter.**
   This commit is the largest single change in Phase 6. Substeps:

   a. `_detect_treesitter()` in repl.lua: one-shot popen of
      `command -v tree-sitter && tree-sitter --version`. Stash
      result on a local.

   b. `renderer.lua` — fence-aware state machine wrapping
      `assistant_delta`. Exports `renderer.set_highlight(enabled,
      detected, highlight_fn)` so repl.lua wires the toggle,
      cli-availability flag, AND the `highlighted` callback (R2:
      keeps `executor` dependency out of `renderer.lua`). State:
      `outside` (pass-through + tail accumulator per R1) /
      `inside` (buffer until closing fence). On close: call
      `highlight_fn(body, lang)` and emit. Algorithm per §4;
      bytes-of-cumulative-buf scan + tail lookahead handles
      fragment-across-boundary fences (B2 + R1).

   c. `highlighted(body, lang)` per §4 (B3 + R2 + R4): lives in
      `repl.lua`. Write body to `os.tmpname()`, invoke via
      `executor.exec("cat tmp | tree-sitter highlight --lang X")`,
      capture out + exit code, cleanup tmp, pass-through on failure.
      **R4 implement-time check**: verify the `--lang` flag exists
      on the installed CLI; if not, switch to tmpfile-with-extension
      and pass the path directly.

   d. `:highlight [on|off|status]` meta in repl.lua. `:highlight on`
      when CLI absent → status with install hint (B4); `:highlight
      status` always reports current toggle + CLI availability.

   e. HELP update. **R9: status header bump moves to commit 6**
      (single owner; no duplication).

6. **`config.lua` + docs/PHASE6 status bump (R9).**
   - Add commented-out `project = { auto_tree = false, tree_depth = 3,
     tree_max_chars = 4096 }` block in config.lua (parity with the
     Phase 1-5 example blocks).
   - PHASE6.md status header → **Implement** (matches Phase 5
     cadence — manifest tracks implementation state).

### Risk index per commit

| Commit | Risk | Mitigation |
|---|---|---|
| 1 (compose_project) | Composition-order regression breaks Phase 4/5 callers | Order test: empty memory + empty project = identical sys_content to pre-Phase-6 baseline |
| 2 (:tree) | `find` fallback picks up node_modules / target / build / etc. | Document in status warning; users in non-repo cwds scope via `:tree <depth>` |
| 3 (:diff) | B1 — color/keypad codes leak if a future caller forgets the helper | All call sites must go through `_git_clean_cmd`; lint by grep before commit |
| 4 (@<r1>..<r2>) | False positive on `@../sibling.txt` when no such file exists | A6's tiered resolution: only retry as diff when file lookup fails. `@../sibling.txt` resolves as path; if the path is missing, diff retry runs and naturally fails — same outcome as before |
| 5 (highlighter) | Fence detector misclassifies inline ` ` ``` ` ` triple-backtick in prose | N1: state machine triggers on `^```` at start of stream OR after `\n` only. §4 algorithm now encodes this constraint in the pseudocode. |
| 5 (highlighter) | tmpfile race / leak on crash | `os.remove(tmp)` in normal exit path; OS cleans `/tmp/lua_*` files on reboot. Single-user trust per PHASE0 §12. |
| 5 (highlighter) | R3: PTY raw-mode toggle on every code-block render (`executor.exec` -> `libc.set_raw(0)`) | Smoke-test before locking: render an assistant turn with 5+ fenced blocks; watch for cursor flicker, SIGWINCH races, terminal state corruption. If problematic, alternate paths: direct `io.popen` for stdin-write (accept the lost exit code; treat empty output as failure) or run highlighter via `os.execute` with shell redirection. |
| 5 (highlighter) | R4: `tree-sitter highlight --lang X` invocation grammar unverified | Implement-time CLI check (`tree-sitter highlight --help`). If `--lang` is wrong, fall back to extension-based: name the tmpfile `lua_XXX.<ext>` per `lang_extension[lang]` map and pass the path. |
| 6 (config bump + status) | none — pure docs / commented config |

### Tests + smoke per commit

Each commit must:
- Pass `luajit test_safety.lua` (87/87) and `luajit test_router_model.lua` (31/31)
- Load cleanly: `luajit -e 'package.path="./?.lua;./vendor/?.lua;"..package.path; require("repl"); print("ok")'`
- Pass a feature-specific smoke (described per row above)

No new test framework dependency. Per-feature unit tests can live as
inline `luajit -e '...'` blocks in commit messages or as a dedicated
`test_phase6.lua` if the surface area justifies it (decide at impl-time).

### Things deliberately NOT split into a separate commit

- `_shq` (shell-quote helper) — already exists in repl.lua from #3.
  Reuse in commit 5 (highlighter); no new helper.
- Lang map — small enough to copy locally in commit 5 (~15 lines);
  the existing `_lang_of(path)` in `expand_mentions` uses a similar
  but smaller map. Factor only if a third caller appears.
- Streaming-rehydration interaction with the highlighter — `secrets_session`
  rehydrate runs BEFORE the highlight filter in the chunk pipeline.
  Order: `chunk → rehydrator:push → highlight_filter → emit`. The
  highlighter operates on plain text only; rehydrated placeholders
  resolve to real values which the highlighter sees as code. No
  special wiring needed.

### Open at plan-time (resolve at implement)

- **R4 implement-time verification**: confirm `tree-sitter highlight
  --lang X` works on the installed CLI. If not, switch to extension-
  based path passing. Block commit 5 ship on this check.
- **R3 smoke test**: render an assistant turn with 5+ fenced blocks
  through the highlighter; confirm no cursor flicker / SIGWINCH race
  / terminal-state corruption from per-block raw-mode toggle. If
  problematic, alternate paths listed in §12 risk row.
- Whether `:highlight status` should also probe `tree-sitter --print-langs`
  to show which langs are actually available. Nice-to-have; defer
  unless install paths produce variable lang sets in practice.