marfrit 11d0e599cd repl + renderer: tree-sitter highlighter (Phase 6 commit #5)
The largest Phase 6 commit — fence-aware stream filter in renderer.lua
+ external tree-sitter dispatch + :highlight meta in repl.lua.

renderer.lua — fence-aware filter wrapping assistant_delta:

  M.set_highlight(enabled, detected, highlight_fn)
      Called by repl.lua at startup AND on every :highlight toggle.
      Stores state in module-locals (off by default).

  State machine inside _hl_push:
    outside: pass chunks through; HOLD trailing partial-fence chars
             (per R1 — local llama.cpp splits ```python as `'``'`
             then `'`python\n'`, so naive pass-through drops the
             leading "``" and never recovers).
    inside:  buffer cumulatively until "\n```" appears; emit
             highlight_fn(body, lang) then the closing fence verbatim.
             Recursive call handles "rest" after the closing fence.

  N1: fences only open at start-of-stream OR after a newline
      (`^```` or `\n```` only). Inline backticks in prose
      ("use ``` to mark code") do not open a fence.

  R3 (PTY raw-mode toggle per highlight call): no change here — every
      executor.exec call already toggles raw-mode (existing behavior
      since Phase 1). The risk is theoretical; smoke-test interactively
      after install if multi-fence renders show flicker.

  assistant_flush handles end-of-stream gracefully: drains any held
  partial-fence tail OR an unterminated inside-fence buffer.

repl.lua — _detect_treesitter + highlighted + :highlight meta:

  _detect_treesitter()  one-shot popen probe of `tree-sitter --version`.
                        Run once at startup; cached as
                        highlight_detected.

  highlighted(body, lang_tag)   R2-placed in repl.lua (has _shq +
                                executor access). Translates the fence
                                tag (`py`, `python`, `lua`, etc.) to
                                a canonical lang via LANG_TAG, picks
                                the canonical extension via LANG_EXTENSION,
                                writes body to a tmpfile with that
                                extension, runs `tree-sitter highlight
                                <tmpfile>` via executor.exec, returns
                                the output. On ANY failure (CLI absent,
                                non-zero exit, empty output), returns
                                `body` unchanged — silent pass-through.

  R4 RESOLVED VIA REAL INSTALL: probed `tree-sitter highlight --help`
      on noether; confirmed:
        - NO `--lang` flag exists (formulate-time assumption wrong)
        - takes a PATH; language inferred from file extension
        - alternative `--scope source.X` exists but also unreliable
          without configured grammars
      Resolution: write tmpfile with `os.tmpname() .. LANG_EXTENSION[lang]`
      and pass the path. Matches the documented upstream contract.

  B4-followup: even with the CLI installed, highlighting requires
      `~/.config/tree-sitter/config.json` parser-directories with
      cloned + built `tree-sitter-<lang>` grammars. Without parsers,
      every call exits non-zero and we silently pass through. The
      :highlight install hint surfaces all three install steps so the
      user knows what's actually needed.

  :highlight [on|off|status] meta:
      no arg     -> flip
      on/off     -> set explicit
      status     -> report toggle + CLI detection state
      When toggled on AND CLI absent: emit a 4-line install hint
        (CLI install, init-config, grammar clone reminder).
      When toggled on AND CLI present: emit a 1-line note that
        parser-directories must be set up for actual highlighting.

HELP gains :highlight entry.

Tested:
  10/10 unit cases on the renderer state machine, including:
    - plain prose passthrough
    - single-chunk fence
    - B2 split fence ("``" + "`python\n" + "x=42" + "\n```")
    - N1 SOL anchor (mid-line ``` does not open)
    - trailing \n properly emitted across chunks
    - SOL-only fence open
    - prose after closing fence preserved
    - two fences in one stream
    - highlight off = passthrough (callback never fires)

  E2E :highlight meta verified:
    :highlight status -> off / detected
    :highlight on     -> toggles + emits parser-dir reminder
    :highlight status -> on / detected
    :highlight off    -> off

Regression: test_safety 87/87, test_router_model 31/31, repl loads.

Pillars 1 + 2 + 3 of Phase 6 now all implemented. Commit #6 is config
example block + status -> Implement.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-16 22:27:04 +00:00

aish

aish — AI-augmented conversational shell.

A single REPL that interleaves shell command execution and language-model conversation, backed by a llama.cpp HTTP broker. Implementation is LuaJIT 2.x with FFI bindings to libcurl, GNU readline, and libc — no C extensions, no build step, one source tree.

Why

Three flows that currently live in three windows fold into one:

  1. "Run this command and show me the output" — fast feedback loop, no copy-paste between terminal and chat.
  2. "Explain or write code based on the output we just looked at" — exec output is automatically injected into the model's context.
  3. "Plan and execute a multi-step task with confirmation gates" — landing in Phase 3 as Chuck Norris autonomous mode.

aish is not a wrapper around bash. It's a first-class interactive environment where the shell is one of several execution channels.

Status

Component State
Repository skeleton in this commit
Phase 0 manifest docs/PHASE0.md — locked
Phase 0 implementation 🔜 next session
Phase 1+ 📋 enumerated in PHASE0.md §11

Every module file currently raises not implemented (Phase 0 pending) when called. luajit main.lua fails loudly at the first un-implemented function, never silently.

Quick orientation

Read this If you want to know
docs/PHASE0.md §12 What aish is and what Phase 0 ships
docs/PHASE0.md §3 Technology decisions (LuaJIT, FFI, readline, libcurl, llama.cpp)
docs/PHASE0.md §4 Directory layout — these file names are stable across all phases
docs/PHASE0.md §5 How input is dispatched (meta / shell / AI)
docs/PHASE0.md §6 Broker contract: /v1/chat/completions, CMD: extraction
docs/PHASE0.md §10 Config schema and resolution order
docs/PHASE0.md §11 Phase sequence (what lands when)
docs/PHASE0.md §13 Open questions, tracked per phase
CLAUDE.md Project conventions for AI-assisted contributors

Directory layout

aish/
├── main.lua              # entry point
├── repl.lua              # readline loop, dispatch, prompt
├── broker.lua            # llama.cpp HTTP client
├── router.lua            # input classifier (meta/shell/AI)
├── executor.lua          # command exec + CMD: extraction
├── context.lua           # in-memory turn history
├── history.lua           # disk persistence (Phase 1+)
├── safety.lua            # destructive-op gate (Phase 3+)
├── renderer.lua          # output formatting
├── config.lua            # default model registry + preferences
├── ffi/
│   ├── curl.lua          # libcurl easy interface
│   ├── readline.lua      # GNU readline
│   ├── pty.lua           # forkpty (Phase 1+)
│   └── libc.lua          # chdir, errno, strerror
└── docs/
    └── PHASE0.md         # locked substrate

Build / runtime dependencies

System packages (Debian / ALARM / Arch names):

  • luajit (>= 2.0)
  • libcurl4 / libcurl-openssl-3 runtime
  • libreadline8 runtime
  • libc6 runtime (always present)

No compilation, no luarocks, no make. Just luajit main.lua.

Running

Once Phase 0 ships:

luajit main.lua                          # uses ~/.config/aish/config.lua
luajit main.lua --config ./config.lua    # explicit config path
AISH_CONFIG=/path/to/config.lua luajit main.lua

Config resolution order is documented in docs/PHASE0.md §10.

Configuration

config.lua is a Lua file returning a single table. The committed config.lua in this repo is both the canonical example and the development-fallback config (lowest precedence). Copy it to ~/.config/aish/config.lua and edit endpoints to your local llama.cpp servers, or point AISH_CONFIG at your own.

The default endpoints assume mfritsche's home network:

  • fastdirac.fritz.box:8081 (Qwen2.5-Coder-7B q4 8k ctx)
  • deepdirac.fritz.box:8080 (Qwen2.5-Coder-7B q4 32k ctx)
  • cloudhossenfelder.fritz.box:8082 (forwards to OpenRouter)

Replace these with your own llama.cpp endpoints if you're not on that LAN.

License

Not yet selected. Default-private until decided.

Project conventions

See CLAUDE.md for contribution conventions, commit style, and the phase-loop discipline this project follows.

S
Description
AI-augmented conversational shell — LuaJIT REPL with llama.cpp broker, shell executor, and routed AI inference.
Readme MIT 2.2 MiB
Languages
Lua 99.8%
Shell 0.2%