aish/docs/PHASE6.md

# aish — Phase 6 Manifest

**Project:** aish — AI-augmented conversational shell
**Document:** Phase 6 Requirements, Architecture & Design Decisions
**Status:** Plan (formulate + analyze + baseline complete; tree at `9f50206`)
**Date:** 2026-05-16

**Analyze findings (2026-05-16):**

A1. **renderer.lua surface clean** — `assistant_delta(chunk)` already
    concatenates into a `stream_buf` then `emit()`s the chunk;
    `assistant_flush()` finalizes with a trailing newline if missing.
    The fence-aware highlight filter slots in between chunk receipt and
    `emit` without restructuring; no callers besides `repl.lua` touch
    `stream_buf` so the filter state can live alongside it.

A2. **executor surface clean** — `executor.exec(cmd)` already
    forkpty-spawns, captures + live-streams output, returns `(out, code)`.
    Phase 6's `:diff` and `_scan_project_tree` reuse this path verbatim;
    no new IO model. `git`-rooted commands inherit cwd from the parent
    (which `libc.chdir` already mutates), so a `:diff` after `cd` reads
    the right repo.

A3. **context composition order locked** — current `to_messages` builds
    `sys_content = base + [background] + [earlier summary] + NORRIS suffix`.
    Phase 6 inserts `[project]` between `[background]` and `[earlier
    summary]`. Same Norris-suppression guard already in place
    (`if not self.norris_active`).

A4. **Q-H1 RESOLVED: tmpfile roundtrip** for `tree-sitter highlight`
    write+read. Avoids ARGMAX risk on large code blocks (vs `printf
    BODY | tree-sitter ...`) and shell-escape complexity. Two file
    handles, deterministic cleanup via `os.remove`. Sketch:

    ```lua
    local tmp = os.tmpname()
    local w = io.popen(("tree-sitter highlight --lang %s > %s")
                       :format(lang, tmp), "w")
    w:write(body); local _, _, code = w:close()
    local f = io.open(tmp, "rb"); local out = f:read("*a"); f:close()
    os.remove(tmp)
    if code ~= 0 then return body end  -- pass-through on failure
    return out
    ```

A5. **Q-D1 RESOLVED: no confirm gate on `:diff`.** `git diff` is
    read-only; matches `:history`, `:sessions`, `:safety check` —
    none of which gate. Permission DSL (#9) only applies to AI-suggested
    `CMD:` lines, not user-issued metas.

A6. **Q-D2 RESOLVED: tiered resolution for `@<token>`.** The mention
    parser tries `<token>` as a file path first; if it doesn't resolve
    AND the token contains `..`, retry as a diff range. This keeps
    `@../sibling.txt` (path) working AND allows `@origin/main..feature`
    (ref range — resolves via second attempt since no such file exists).
    No grammar prefix needed.

A7. **Q-H2 RESOLVED: highlighting is assistant-output only in v1.**
    `expand_mentions` content lands in the user-turn payload — visible
    on the terminal via readline echo, not via `assistant_delta`. Filing
    "highlight @path-expanded code in echo" as v2 polish. Reason:
    intercepting readline echo for ANSI injection is non-trivial and
    orthogonal to the stream filter.

A8. **Q-T1 RESOLVED: project tree captured at scan time, not auto-
    refreshed on cd.** `cd /other-project` leaves the existing
    `ctx.project` stale; `:tree refresh` is the manual verb to update.
    Auto-refresh on cd intercept is a v2 polish (the cd interceptor in
    `executor.maybe_chdir` is a clean hook for it).

A9. **Q-T2 RESOLVED: rely on `.gitignore` via `git ls-files`** in repos;
    fall back to `find` with simple excludes outside. Custom
    include/exclude glob lists deferred to v2. Reason: most users live
    inside git repos; `.gitignore` already encodes their notion of
    "noise". Out-of-repo users get the simple fallback and can scope
    via `:tree <depth>`.

A10. **`expand_mentions` punct-peel does NOT strip `/`** — so
     `@HEAD~1..HEAD,` peels the `,` and the underlying token `HEAD~1..HEAD`
     has no slash; the path-then-diff retry from A6 catches it. No new
     peel logic needed.

A11. **Auto-injection ordering for `[project]`** — if both `cfg.memory.
     inject_max_chars` and `cfg.project.auto_tree` fire at startup, the
     order is: memory load → tree scan → first ask_ai. The composition
     in `to_messages` places `[background]` (memory) before `[project]`
     so the model reads memory facts before file tree. Documented in §3.

A12. **Norris interaction** — `[project]` block follows the established
     [background]/[earlier summary] suppression rule under
     `ctx.norris_active`. Planner stays on its goal anchor; the tree
     can be re-introduced via the goal text if needed. Matches R-C1/R-C4.

PHASE0 is the locked substrate; PHASE1-5 are layered on top. This manifest
specifies what Phase 6 adds — **tree-sitter syntax highlighting hooks**,
**diff-aware code injection**, and **project-level context (file-tree
summary)**.

---

## 1. Scope of Phase 6

Three pillars per PHASE0.md §11 row 6:

1. **Tree-sitter syntax highlighting hooks** — when an external
   `tree-sitter` CLI is detected at startup, assistant code-fence
   content is filtered through it for ANSI-colorized display. Plain
   prose streams unchanged. When the CLI is absent, the filter is the
   identity function (zero overhead, zero hard dependency). Toggleable
   at runtime with `:highlight on|off`. Default off until the user
   opts in (don't surprise existing users with a display change).
   Per B4: tree-sitter is **absent on every fleet host probed**;
   `:highlight on` when the CLI is missing emits a status that names
   the install hint (`apt install tree-sitter` / `cargo install
   tree-sitter-cli`) rather than silently falling back to identity.

2. **Diff-aware code injection** — surface git diffs as first-class
   context. Two entry points:

   - Meta verb: `:diff [args]` runs `git diff <args>` from cwd, appends
     output to context as exec-output. `:diff staged`, `:diff HEAD~3`,
     `:diff main..feature` all delegate to git's argument grammar.
   - @-mention extension: `@HEAD..feature` (a ref-range expression
     anywhere a `@path` would go) expands inline as a fenced `diff`
     block, mirroring how `@README.md` already works.

3. **Project-level context (file-tree summary)** — `git ls-files`-based
   tree summary of the cwd, injected as a `[project]` block in the
   system prompt. Two entry points:

   - Meta verb: `:tree [depth]` injects on demand; `:tree refresh`
     re-scans.
   - Auto-inject at startup when `cfg.project.auto_tree = true` —
     gated like memory injection so existing configs don't change
     behavior.

**Phase 6 is done when:**

- With `tree-sitter` CLI installed and `:highlight on`, the assistant
  reply ```py\nprint("hi")\n``` shows up with ANSI colors. Without
  the CLI, `:highlight on` is a no-op + emits a status warning.
- `:diff` from a dirty git repo shows the working-tree diff in the
  exec-output frame; the model sees it on the next ask_ai turn.
- `@HEAD~1..HEAD` in a prompt expands inline to a fenced diff block.
- `:tree` injects a `[project] <N files>:` block visible in
  `ctx:to_messages()` (via the system prompt assembly).
- With `cfg.project.auto_tree = true`, the project block appears on
  every broker call (subject to `max_chars` cap).
- Existing configs without `cfg.project` and with `:highlight off`
  (default) behave exactly like Phase 5 (Phase 5 regression coverage).

---

## 2. Technology Decisions (delta from Phase 5)

| Decision | Choice | Rationale |
|---|---|---|
| Highlight backend | External `tree-sitter` CLI (`tree-sitter highlight --lang X`) | Honors PHASE0 §3: no compiled extensions, no luarocks. Detected once at startup; absence → identity filter. Opt-in via `:highlight on` so install-state changes don't break users. |
| Highlight buffering | Accumulate inside fenced code blocks, emit on closing fence; pass-through outside fences | Streaming UX preserved for prose. Code blocks get colorized atomically, accepting a per-block latency (~ block streaming time). Per-chunk highlighting would split a token across `tree-sitter` invocations and corrupt the output. |
| Lang detection | First-line fence info-string (` ```py`, ` ```python`, ` ```lua`) → normalized via small map (py→python, js→javascript, etc.) | The lang tag mirrors the one we already emit in `expand_mentions` (#7). No tag → identity (no highlight). |
| Diff backend | Shell out to `git diff <args>` via `executor.exec` | Honors substrate (no libgit2 FFI). The existing exec frame handles capture + stream. `git` is universally present where aish makes sense. |
| Diff failure | Bail with status `[aish] :diff failed (not a git repo / bad ref)`; do NOT inject empty output | Avoids polluting context with stale or empty diffs. |
| Tree backend | `git ls-files --cached --others --exclude-standard` when cwd is a git repo, else `find . -type f -not -path './.*'` | Free `.gitignore` honor in repos; sensible default outside. Both are POSIX-portable. |
| Tree summary form | Sorted relative paths, grouped by directory at depth ≤ `cfg.project.tree_depth` (default 3), truncated by char count `cfg.project.tree_max_chars` (default 4096) | One block, deterministic order, cheap to compute. Matches the [background] memory block convention (Phase 4) so the system prompt's compositional shape stays familiar. |
| Tree injection point | `context.lua`: new `compose_project(...)` adds a `[project] <header>\n<body>` block to the system content, between [background] and [earlier summary] | Same suppression rule as [background]/[earlier summary]: NOT injected during Norris (R-C1 / R-C4 — planner stays on its anchor). |
| Tree refresh policy | One scan at startup if auto; `:tree refresh` to re-scan on demand | Scanning on every ask_ai is wasteful for slow filesystems. Manual refresh is sufficient for v1. |
| @-mention diff syntax | `@<ref>..<ref>` (two `..` separator) only — recognized via the existing trailing-punct peel logic | Avoids ambiguity with literal paths. `@HEAD` alone is NOT a diff trigger (would collide with files literally named HEAD). |

---

## 3. Module Changes

| File | State after Phase 5 | Phase 6 changes |
|---|---|---|
| `renderer.lua` | `assistant_delta(text)` writes chunks; `assistant_flush()` finalizes | Add fence-aware filter inside the assistant stream. State machine: outside-fence (pass-through) / inside-fence (buffer, emit on close). On close, pipe buffer through `tree-sitter highlight --lang <X>` (if highlight enabled), emit result. Toggle exposed as `renderer.set_highlight(bool)`. |
| `executor.lua` | `extract_cmd_lines`, `extract_cmd_bg_lines`, `extract_delegate_lines` | No changes. Diff and tree use the existing `exec` path. |
| `context.lua` | system prompt = base + [background] + [earlier summary] + NORRIS suffix | Add `self.project = "..."` string field + `compose_project(self.project)` helper. Injection between [background] and [earlier summary] (A11: memory facts read before file tree). Suppressed under Norris (A12, parity with R-C1/R-C4). |
| `repl.lua` | meta dispatch + main loop + #13 secrets wiring | New helpers: `_detect_treesitter()` (run once at startup), `_run_git_diff(args)`, `_scan_project_tree(dir, opts)`. New meta: `:highlight`, `:diff`, `:tree`. Extend `expand_mentions` to recognize `<ref>..<ref>` token shape. |
| `config.lua` | example blocks for mcp/safety/memory/routing/secrets/etc. | Add commented-out `project = { auto_tree = false, tree_depth = 3, tree_max_chars = 4096 }` block. |

No new module files in v1. Three new helpers in `repl.lua` keep the
file growing but consolidate the Phase 6 surface. If the highlighter
filter grows past ~80 LOC, lift it into `highlight.lua` as a follow-up.

---

## 4. Pillar 1 — Tree-sitter highlighting

### Detection (startup, once)

```lua
local function _detect_treesitter()
    local pipe = io.popen("command -v tree-sitter 2>/dev/null && tree-sitter --version 2>/dev/null")
    local ok = pipe and pipe:read("*l") and pipe:close()
    return ok
end
```

If not present, `renderer.set_highlight(true)` emits a status warning
and leaves the filter as a no-op. Don't error; the user can install
tree-sitter and re-toggle.

### Stream filter

The filter wraps `renderer.assistant_delta`. State machine:

```
state = "outside" | "inside"
buf   = ""                     -- only used in "inside"
lang  = nil                    -- captured at fence open

push(chunk):
  if state == "outside":
      look for ```<lang>\n in chunk
      if found:
          emit chunk up to fence-open
          state = "inside"; lang = parsed; buf = chunk after fence-open
      else:
          emit chunk as-is

  if state == "inside":
      buf = buf .. chunk
      look for \n``` in buf
      if found:
          fence_body = buf up to closing
          rest        = buf after closing
          emit highlighted(fence_body, lang)
          emit closing fence verbatim
          emit rest as-is (recurse with state="outside")
          state = "outside"; buf = ""
      else:
          -- still buffering; nothing emitted this push
```

Edge cases: chunk boundary lands inside the fence marker itself
(e.g., chunk ends with ` `` `, next starts with `\n`). The state
machine looks at the cumulative `buf`, so partial markers are
recovered correctly.

`highlighted(body, lang)` — **B3-revised** (supersedes A4):

```lua
if not highlight_enabled or not lang_map[lang] then return body end
local tmp = os.tmpname()
local f = io.open(tmp, "wb")
if not f then return body end
f:write(body); f:close()
-- B3: LuaJIT io.popen():close() returns (true, nil, nil) regardless of
-- exit code. Route via executor.exec which uses pty.spawn+waitpid and
-- returns (out, exit_code) reliably.
local out, code = executor.exec(
    ("cat %s | tree-sitter highlight --lang %s")
    :format(_shq(tmp), lang_map[lang]))
os.remove(tmp)
if code ~= 0 then return body end   -- pass-through on highlighter failure
return out
```

Why this shape (and not the formulate-time A4 sketch):

- **A4 assumed Lua 5.2+ popen-close-with-code.** LuaJIT (5.1 contract)
  doesn't expose the exit status via `io.popen(...):close()`. Baseline
  B3 caught this; the only reliable exit-code path in our substrate
  is `executor.exec` (pty.spawn + waitpid).
- The tmpfile stays — avoids ARGMAX on `printf '%s' BODY |` and
  sidesteps shell-escape edge cases on arbitrary code-block bytes.
- Cost is one syscall round (tmpfile create/remove) plus one pty
  spawn per code block — negligible vs the highlighter latency.
- `_shq` is the existing shell-quote helper from #3 (pre/post hooks).

### Lang map (v1)

```lua
local LANG_MAP = {
    py = "python", python = "python",
    lua = "lua",
    js = "javascript", javascript = "javascript", ts = "typescript",
    sh = "bash", bash = "bash",
    c = "c", h = "c", cpp = "cpp", cc = "cpp",
    rs = "rust", go = "go", java = "java", rb = "ruby",
    md = "markdown", json = "json",
}
```

Reuses the same map as `expand_mentions`. Factor into a shared
helper once both reference it (small `_lang_of_ext()` in repl.lua).

### Toggle

`:highlight` (no arg) → flip. `:highlight on|off` → set explicit.
`:highlight status` → report enabled + whether tree-sitter is present.
Default: off (don't change existing-user UX).

---

## 5. Pillar 2 — Diff-aware code injection

### Meta: `:diff [args]`

- `:diff` → `git diff` (working tree vs index)
- `:diff staged` → `git diff --cached`
- `:diff HEAD` → `git diff HEAD`
- `:diff main..feature` → `git diff main..feature`
- `:diff <anything else>` → passed verbatim to `git diff <anything>`

Implementation — **B1-revised** (must disable pager + color):

```lua
meta.diff = function(args)
    args = (args or ""):gsub("^%s+", ""):gsub("%s+$", "")
    -- B1: forkpty makes git think it's interactive, enabling color
    -- ANSI + DEC keypad/line-clear escapes that pollute the injected
    -- context block. --no-pager kills the keypad sequences; --color=
    -- never kills the color codes. Both are required.
    local cmd = "git --no-pager -c color.ui=never diff " .. args
    local out, code = executor.exec(cmd)
    if code ~= 0 then
        renderer.status(("diff failed (exit %d)"):format(code))
        return
    end
    if out == "" or out:gsub("%s", "") == "" then
        renderer.status("(no diff)")
        return
    end
    ctx:append_exec_output(("[diff %s]\n%s"):format(
        args == "" and "(working tree)" or args, out))
end
```

The `[diff ...]\n<output>` framing matches the `[bg:N exited]` /
`[delegate X]` conventions established in Phase 5 / #6 / #8.

The same `--no-pager -c color.ui=never` prefix applies to the
`@<r1>..<r2>` resolution path in the next section, and to any
future git verbs we add (`:log`, `:show`, etc.). Factor into a
helper `_git_clean_cmd(subcmd)` if multiple call sites accumulate.

### @-mention: `@<ref1>..<ref2>` — tiered resolution (A6)

Extends `expand_mentions` (#7) by adding a SECOND resolution attempt
when the first (path lookup) fails AND the token contains `..`:

```lua
-- Existing path-attempt block ends with content = _read_truncated(path)
-- which returns nil if no such file. Add the diff retry there:

if not content and path:find("..", 1, true) then
    local r1, r2 = path:match("^(.-)%.%.(.+)$")
    if r1 and r2 and r1 ~= "" and r2 ~= "" then
        -- B1: --no-pager + color=never (same as the :diff meta path).
        -- B3: io.popen close() doesn't expose exit codes — use the
        -- file-redirect trick OR executor.exec. Here we want a quick
        -- best-effort and the cost of an extra forkpty is acceptable.
        local out, code = executor.exec(
            ("git --no-pager -c color.ui=never diff %s..%s 2>/dev/null")
                :format(shq(r1), shq(r2)))
        if code == 0 and out:match("%S") then
            content = out
            -- Note: language tag becomes "diff" regardless of path lang
            lang_override = "diff"
        end
    end
end
```

Output replaces the token with:

````
```diff path=<r1>..<r2>
<content>
```
````

Tiered resolution semantics:
- `@README.md` → file lookup succeeds → file expansion
- `@../sibling.txt` → file lookup succeeds → file expansion
- `@HEAD~1..HEAD` → file lookup fails, `..` present, ref-range succeeds → diff
- `@origin/main..feature` → file lookup fails (no such file), `..` present,
  ref-range succeeds → diff. The token has `/` in `r1` but `git diff` accepts
  it as a ref; no `/`-based heuristic needed (resolves Q-D2).
- `@nonexistent-file..but-also-not-a-ref` → both fail; literal token
  preserved with the existing `[aish] @X: not found` status path.

---

## 6. Pillar 3 — Project file-tree

### Meta: `:tree [depth]`

- `:tree` → scan + inject with default depth and char cap
- `:tree <N>` → override depth for this scan
- `:tree refresh` → re-scan with cached opts
- `:tree off` → clear `ctx.project`

### Scan logic

```lua
local function _scan_project_tree(dir, opts)
    opts = opts or {}
    local max_chars = opts.max_chars or 4096
    local depth     = opts.depth or 3

    -- Prefer git ls-files for .gitignore honor; fall back to find.
    local in_git = os.execute("cd " .. shq(dir) .. " && git rev-parse --git-dir >/dev/null 2>&1") == 0
    local listcmd
    if in_git then
        listcmd = ("cd %s && git ls-files --cached --others --exclude-standard"):format(shq(dir))
    else
        listcmd = ("find %s -maxdepth %d -type f -not -path '*/\\.*' 2>/dev/null"):format(shq(dir), depth + 1)
    end

    local pipe = io.popen(listcmd)
    if not pipe then return nil, "scan failed" end

    local files = {}
    for line in pipe:lines() do
        -- Depth filter: count `/` separators
        local _, slashes = line:gsub("/", "")
        if slashes < depth then files[#files + 1] = line end
    end
    pipe:close()

    table.sort(files)

    -- Build a tree-ish summary, truncate by char count.
    local body = table.concat(files, "\n")
    local truncated = false
    if #body > max_chars then
        body = body:sub(1, max_chars) .. "\n... (truncated)"
        truncated = true
    end
    return body, { file_count = #files, truncated = truncated }
end
```

### Injection

`ctx.project = "..."` (string), composed into the system prompt
between [background] and [earlier conversation summary]:

```
[project] 142 files (truncated at 4096B):
README.md
broker.lua
config.lua
context.lua
...
```

Suppressed under Norris (R-C1 / R-C4 — planner stays focused; the
project context can be re-introduced via the Norris goal text if
needed).

### Auto-inject

`cfg.project.auto_tree = true` runs the scan once at startup and
sets `ctx.project`. Default false (existing configs unchanged).

---

## 7. UX Surface Summary

| Meta | Behavior |
|---|---|
| `:highlight [on/off/status]` | Toggle tree-sitter highlighter (no-op when CLI absent) |
| `:diff [args]` | `git diff <args>`, append output to context as `[diff ...]` |
| `:tree [N/refresh/off]` | Scan/refresh/clear project file-tree block |

| @-mention | Behavior |
|---|---|
| `@path` | Existing (#7) file expansion |
| `@<ref1>..<ref2>` | New: inline `git diff <r1>..<r2>` expansion |

| Config | Default | Effect |
|---|---|---|
| `cfg.project.auto_tree` | `false` | Inject project tree at startup |
| `cfg.project.tree_depth` | `3` | Depth filter for the scan |
| `cfg.project.tree_max_chars` | `4096` | Truncation cap for the injected block |
| (no config flag for `:highlight`) | — | Runtime toggle only; no persistence in v1 |

---

## 8. Out of Scope (Phase 6)

- **Pure-Lua syntax highlighter** — defer to a future phase if
  tree-sitter CLI absence becomes a practical pain point. v1 says
  "install tree-sitter or accept plain text".
- **bat/glow/chroma integration** — only `tree-sitter` is wired.
  Other highlighters can be added behind the same `:highlight` toggle
  later (config field `cfg.highlight.backend = "tree-sitter"|"bat"|...`).
- **Smart diff context selection** — no AI-driven "which diff to show".
  User explicitly says `:diff <range>` or `@<r1>..<r2>`.
- **File-tree LRU / smart summarization** — v1 is a flat truncated list.
  Hierarchical roll-up ("docs/ — 8 files") is a v2 polish.
- **Watching for file changes** — no fs-notify reload. Re-scan via
  `:tree refresh`.
- **Diff history** — `:diff` doesn't track its previous invocations.
  Each invocation is independent.
- **Inline diff highlighting** — the `diff` lang is in `LANG_MAP` so
  `tree-sitter highlight --lang diff` works, but we don't ship custom
  ANSI for added/removed lines — tree-sitter's own theme covers it.

- **Highlighter on @-mention echo** (v2 polish per A7) — `:highlight`
  applies to assistant output only. Highlighting user-pasted code as
  it's echoed by readline would need a separate hook in the readline
  display path; out of scope here.

- **Auto-refresh project tree on `cd`** (v2 polish per A8) — the cd
  interceptor in `executor.maybe_chdir` is a clean place to call
  `_scan_project_tree(libc.getcwd(), ...)` on every successful cd.
  Skipped in v1 because the scan can be slow on large trees; manual
  refresh via `:tree refresh` is the v1 verb.

- **Custom include/exclude globs for project tree** (v2 polish per A9) —
  `cfg.project = { include = {...}, exclude = {...} }` would extend
  beyond `.gitignore`. v1 ships with `.gitignore`-only honor (via
  `git ls-files --exclude-standard`) plus the `find` fallback for
  non-repo cwds.

---

## 9. Risks

| Risk | Mitigation |
|---|---|
| `tree-sitter` CLI not on fleet → most users get no highlighting | It's opt-in; default off; status warning on toggle when absent. |
| Highlighter latency on long code blocks (whole-block buffering) | Accepted trade-off vs corrupting output. If painful in practice, add a per-block size cap above which we pass-through unhighlighted. |
| `git diff` on huge changesets blows context budget | Diff output reuses `enforce_budget` eviction (it's just exec output). User can `:diff <subdir>` to scope. v2 could add a `--max-bytes` truncation. |
| `git ls-files` in a non-git cwd → falls back to `find`, may pick up node_modules / target / etc. | Document in config example; v2 could honor `.aishignore` or similar. |
| @`<ref1>..<ref2>` collides with paths like `@../sibling.txt` | A6: tiered resolution — try as path first; only fall through to diff retry when path lookup fails AND token contains `..`. `@../sibling.txt` hits the path branch and never reaches the diff retry. |
| Project tree injection adds tokens to every broker call | Char cap + opt-in `auto_tree = false` default. Suppressed under Norris. |
| `:highlight on` mid-stream produces inconsistent rendering for the in-flight turn | Toggle takes effect from the NEXT assistant turn. Document this. |

---

## 10. Open Questions (Phase 6)

All six formulate-time Qs were resolved in analyze (A4–A9). None remain
open as blockers for implementation.

| # | Question | Resolution |
|---|---|---|
| Q-H1 | popen3 for `tree-sitter highlight` | A4: tmpfile roundtrip — `io.popen("w")` writes body with stdout redirected to a tmp file, then `io.open` reads the file. Avoids ARGMAX + shell-escape complexity. |
| Q-D1 | Confirm gate on `:diff`? | A5: no. `git diff` is read-only; matches `:history` / `:sessions` / `:safety check` (none gate). Permission DSL (#9) applies only to AI-suggested `CMD:` lines. |
| Q-D2 | `@<r1>..<r2>` with refs containing `/` | A6: tiered resolution — file lookup first, then if it fails AND `..` is present, retry as ref-range. `@origin/main..feature` naturally falls through to the retry; no grammar prefix needed. |
| Q-T1 | `cfg.project.auto_tree` update on cd | A8: no auto-refresh in v1. `:tree refresh` is the manual verb; cd-intercept hook is documented as v2 polish in §8. |
| Q-T2 | Custom include/exclude globs | A9: rely on `.gitignore` via `git ls-files` in repos; `find` fallback outside. Custom globs deferred to v2. |
| Q-H2 | Highlighting on @-mention echo | A7: assistant-output only in v1. Echo via readline is a different code path; deferred to v2 (see §8). |

---

## 11. Phase 6 → Phase 7+ Out-of-band

The §11 "Planned Phase Sequence" table in PHASE0.md does not list
phases beyond 6. After Phase 6 lands, candidate next iterations
(non-binding, for the formulate of Phase 7 to confirm):

- **Phase 7**: secret-redaction wiring into `safety.lua` (#52
  follow-up filed during Phase 5/13 close); session-multiplex / tmux
  parity surfaces (out of scope per §12 — explicitly rejected);
  or other backlog as it accumulates on Gitea.

Phase 6 itself is self-contained — none of its three pillars introduce
substrate dependencies on phases not yet planned.

---

## 12. Implementation Plan (commit-by-commit)

Bottom-up ordering: foundations first (context.lua field + composer),
then the diff and tree surfaces that have no display-layer risk, then
the highlighter (largest experimental surface — last so the rest of
Phase 6 ships even if highlighter slips). Each commit leaves the tree
green (existing tests pass + smoke ok) and adds a discrete capability.

### Order

1. **`context.lua` — `[project]` block plumbing.** Add `self.project`
   (string, nil-allowed) on `Context.new`. Add `compose_project(text)`
   helper mirroring `compose_background` / `compose_summary`. In
   `to_messages`: insert between `compose_background` and
   `compose_summary` so the read order is memory → project tree →
   earlier-summary → NORRIS. Suppressed under `self.norris_active`
   (parity with R-C1 / R-C4). No behavior change yet — nothing sets
   `ctx.project`. Smoke: `:to_messages()` still empty when project nil.

2. **`repl.lua` — `_scan_project_tree` helper + `:tree` meta.**
   - `_scan_project_tree(dir, opts)` per §6: `git ls-files --cached
     --others --exclude-standard` in a repo, `find . -maxdepth N
     -type f -not -path '*/\.*'` outside. Returns `(body, info)`
     where `info = { file_count, truncated }`.
   - `:tree [N|refresh|off]` meta: scans cwd, sets `ctx.project`,
     emits status with file count + truncation note.
   - `cfg.project.auto_tree` startup hook: if true, run `_scan` once
     and set `ctx.project` (before the main loop opens). Default
     false (existing configs unchanged).
   - Update HELP with `:tree` lines.
   - Smoke: in the aish repo, `:tree` injects a ~32-file block;
     `:to_messages()` shows the `[project]` block in the system prompt.

3. **`repl.lua` — `:diff` meta + `_git_clean_cmd` helper (B1).**
   - `_git_clean_cmd(subcmd_and_args)` returns the `git --no-pager
     -c color.ui=never <subcmd_and_args>` prefix. Used by `:diff`
     and the `@<r1>..<r2>` path in commit #4.
   - `:diff [args]` meta per §5 (B1-revised): runs the clean git
     command via `executor.exec`, appends `[diff <args>]\n<out>`
     to context as exec_output. Empty / non-repo / bad-ref paths
     emit status and skip.
   - Update HELP with `:diff` line.
   - Smoke: `:diff` from a dirty aish checkout injects the working
     tree diff; `:diff staged` works; `:diff junkref` emits status
     and skips.

4. **`repl.lua` — `expand_mentions` tiered resolution (A6).**
   Extend the existing path-resolution loop with the diff-retry
   branch from §5: if `_read_truncated` returns nil AND the token
   contains `..`, parse as `<r1>..<r2>` and try `_git_clean_cmd(
   "diff <r1>..<r2>")`. On success, replace with a fenced `diff`
   block. Preserves existing peel-on-trailing-punct logic. Smoke:
   `@HEAD~1..HEAD` expands inline; `@origin/main..feature` works
   when the ref exists; `@../sibling.txt` still resolves as file.

5. **`renderer.lua` + `repl.lua` — tree-sitter highlighter.**
   This commit is the largest single change in Phase 6. Substeps:

   a. `_detect_treesitter()` in repl.lua: one-shot popen of
      `command -v tree-sitter && tree-sitter --version`. Stash
      result on a local.

   b. `renderer.lua` — fence-aware state machine wrapping
      `assistant_delta`. Exports `renderer.set_highlight(enabled,
      detected)` so repl.lua wires the toggle + cli-availability
      flags. State: `outside` (pass-through) / `inside` (buffer until
      closing fence). On close: call `highlighted(body, lang)` and
      emit. Algorithm per §4; bytes-of-cumulative-buf scan as B2
      requires for fragment-across-boundary fences.

   c. `highlighted(body, lang)` per §4 (B3-revised): write body to
      `os.tmpname()`, invoke via `executor.exec("cat tmp |
      tree-sitter highlight --lang X")`, capture out + exit code,
      cleanup tmp, pass-through on failure.

   d. `:highlight [on|off|status]` meta in repl.lua. `:highlight on`
      when CLI absent → status with install hint (B4); `:highlight
      status` always reports current toggle + CLI availability.

   e. HELP update; PHASE6 status → Implement.

6. **`config.lua` + docs/PHASE6 status bump.**
   - Add commented-out `project = { auto_tree = false, tree_depth = 3,
     tree_max_chars = 4096 }` block in config.lua (parity with the
     Phase 1-5 example blocks).
   - PHASE6.md status header → **Implement** (matches Phase 5
     cadence — manifest tracks implementation state).

### Risk index per commit

| Commit | Risk | Mitigation |
|---|---|---|
| 1 (compose_project) | Composition-order regression breaks Phase 4/5 callers | Order test: empty memory + empty project = identical sys_content to pre-Phase-6 baseline |
| 2 (:tree) | `find` fallback picks up node_modules / target / build / etc. | Document in status warning; users in non-repo cwds scope via `:tree <depth>` |
| 3 (:diff) | B1 — color/keypad codes leak if a future caller forgets the helper | All call sites must go through `_git_clean_cmd`; lint by grep before commit |
| 4 (@<r1>..<r2>) | False positive on `@../sibling.txt` when no such file exists | A6's tiered resolution: only retry as diff when file lookup fails. `@../sibling.txt` resolves as path; if the path is missing, diff retry runs and naturally fails — same outcome as before |
| 5 (highlighter) | Fence detector misclassifies inline ` ` ``` ` ` triple-backtick in prose | State machine triggers on `^```` at the start of a line OR following a newline only; mid-line backticks don't open a fence. Document in §4. |
| 5 (highlighter) | tmpfile race / leak on crash | `os.remove(tmp)` in normal exit path; OS cleans `/tmp/lua_*` files on reboot. Single-user trust per PHASE0 §12. |
| 6 (config bump) | none — pure docs / commented config |

### Tests + smoke per commit

Each commit must:
- Pass `luajit test_safety.lua` (87/87) and `luajit test_router_model.lua` (31/31)
- Load cleanly: `luajit -e 'package.path="./?.lua;./vendor/?.lua;"..package.path; require("repl"); print("ok")'`
- Pass a feature-specific smoke (described per row above)

No new test framework dependency. Per-feature unit tests can live as
inline `luajit -e '...'` blocks in commit messages or as a dedicated
`test_phase6.lua` if the surface area justifies it (decide at impl-time).

### Things deliberately NOT split into a separate commit

- `_shq` (shell-quote helper) — already exists in repl.lua from #3.
  Reuse in commit 5 (highlighter); no new helper.
- Lang map — small enough to copy locally in commit 5 (~15 lines);
  the existing `_lang_of(path)` in `expand_mentions` uses a similar
  but smaller map. Factor only if a third caller appears.
- Streaming-rehydration interaction with the highlighter — `secrets_session`
  rehydrate runs BEFORE the highlight filter in the chunk pipeline.
  Order: `chunk → rehydrator:push → highlight_filter → emit`. The
  highlighter operates on plain text only; rehydrated placeholders
  resolve to real values which the highlighter sees as code. No
  special wiring needed.

### Open at plan-time (resolve at implement)

- Whether `_scan_project_tree` should honor a per-call `opts.dir`
  override (so a future feature like "scan `<other-dir>`" lands cheaply)
  vs hardcoding `libc.getcwd()`. Default to taking `dir` as arg;
  the `:tree` meta passes `libc.getcwd()` explicitly.
- Whether `:highlight status` should also probe `tree-sitter --print-langs`
  to show which langs are actually available. Nice-to-have; defer
  unless install paths produce variable lang sets in practice.