Clean up for public release: README rewrite, LICENSE (MIT), patches reorganized

Final deliverables (mesa-panvk-shared-tls-wls + llama.cpp-mali-mmq-warptile) at top of patches/;
iteration trail moved to patches/iterations/. Public-facing README with the 4.8x result and the
two-fix summary. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

LICENSE

@@ -0,0 +1,25 @@
+MIT License
+
+Copyright (c) 2026 Markus Fritsche
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.
+
+The patches under patches/ are modifications to Mesa and llama.cpp, each of
+which is distributed under the MIT license; those modifications inherit the
+upstream licenses of the files they apply to.

README.md

@@ -1,55 +1,65 @@
 # rocky-vulkan-llama
 
-GPU-accelerated **LLM prefill** on the RK3588 **Mali-G610** via the open Vulkan stack
-(panthor kernel driver + Mesa **panvk** + llama.cpp Vulkan backend).
+**Making the RK3588 Mali-G610 GPU a real LLM prefill accelerator — via the open Vulkan stack
+(panthor kernel + Mesa panvk + llama.cpp), no vendor blobs.**
 
-Goal: make the Mali GPU a useful **prompt-processing (prefill) accelerator** for local agents on
-RK3588 boards (boltzmann = Radxa ROCK 5 ITX+; ampere = CoolPi CM5 GenBook), so a ~5k-token agent
-context doesn't cost ~150–250 s of CPU prefill before the model starts generating.
+Two upstreamable fixes take large-batch prefill on a Radxa ROCK 5 ITX+ (Rockchip RK3588) from
+*crashing* to **4.8× faster**, beating the CPU — on a GPU with **no matrix cores**.
 
-> Decode (token generation) is memory-bandwidth-bound (~28 GB/s shared LPDDR5) and the GPU can't
-> help it. **Prefill is compute-bound** — that's the only place the GPU's FLOPS matter. This repo
-> is about prefill.
+## TL;DR results (Qwen2.5-3B-Q4_K_M, pp512, Mali-G610 MC4, Mesa 26.1.3 panvk)
 
-## Dependency stack
+| | prefill t/s | note |
+|---|---|---|
+| stock llama.cpp Vulkan @ ub≥128 | **crash** | `vk::CommandBuffer::end: ErrorOutOfDeviceMemory` |
+| + driver fix | 5.8 | runs, but the desktop-tuned shader is register-spilling |
+| + shader tune (this repo) | **28.1** | **4.8×**, spill-free, ~35 % of roofline, output-verified |
+| (CPU reference, i8mm-off) | ~22 | |
 
-| Layer | Requirement | Notes |
-|-------|-------------|-------|
-| Kernel | **panthor** DRM driver (mainline ≥ 6.x), GPU node bound to panthor, **not** the vendor `mali` kbase blob | On boltzmann: boot `linux-rk3588-marfrit-A1` (mainline 7.0.0-rc3). The vendor BSP kernel (`6.1.75-npu-port`) binds `mali` → panvk cannot enumerate. |
-| CSF firmware | `/lib/firmware/arm/mali/arch10.10/mali_csffw.bin` | Present on boltzmann. |
-| Userspace GL/Vulkan | **Mesa panvk** (`libvulkan_panfrost.so`), Mesa **26.1.3** | `panfrost_icd.json` ICD. Enumerates `Mali-G610 MC4 (panvk)` with `uma:1 fp16:1 int dot:1 matrix cores:none`. |
-| Runtime env | `XDG_RUNTIME_DIR` must be set | panvk init fails without it. |
-| Inference | **llama.cpp** Vulkan backend (`-DGGML_VULKAN=ON`), commit `389ff61` | Build tools: `glslc`, `libvulkan`, cmake. |
+The ceiling was never the hardware — it was a desktop-tuned shader thrashing Mali's tiny register
+file, and a Mesa allocator that never shared per-command-buffer scratch. Same shape as unlocking
+hardware video decode: fix the whole stack (kernel → Mesa → the consumer), not just one layer.
 
-## Status (2026-07-11)
+## The two fixes
 
-- ✅ Full open stack works: panthor binds the GPU, panvk enumerates it, llama.cpp runs on it.
-- ⚠️ **Blocker:** panvk throws `vk::CommandBuffer::end: ErrorOutOfDeviceMemory` on any prefill
-  dispatch above **ubatch 64**. Only `-ub 64` runs; at ub64 prefill is **21 t/s**, which is
-  *overhead-bound* (ub16/32 far slower) — i.e. 21 t/s is the "forced-tiny-chunk" ceiling, not the
-  Mali GEMM ceiling. It ties an (i8mm-disabled) CPU.
-- 🔬 Root cause narrowed to a panvk driver allocation that scales past ub64; see
-  [docs/root-cause.md](docs/root-cause.md). The **fix is a Mesa panvk patch** (this repo).
+1. **Mesa panvk — priv-heap exhaustion** (`patches/mesa-panvk-shared-tls-wls.patch`)
+   CSF panvk allocates a per-command-buffer TLS stack (up to 304 MB) and per-dispatch WLS scratch
+   from a device-wide **sub-4 GB priv VA heap**, never shared. A pipelined LLM graph keeps ~12
+   command buffers in flight → ~4 GB → OOM at any ubatch > 64. Fix: one **device-shared TLS scratch**
+   reused across the serially-executing compute queue, plus grid-independent **WLS reuse**. Peak
+   priv-VA 3985 MB → 908 MB; no crash from ub 64 to 1024. Affects *any* large-workgroup compute on
+   Mali/panvk, not just LLMs.
 
-## Contents
+2. **llama.cpp — Mali-tuned `mul_mm`/`mul_mmq` warptile** (`patches/llama.cpp-mali-mmq-warptile.patch`)
+   Per-thread matmul accumulators = `WM·WN/WARP`; the stock L-variant is 128 → spills 5328 B/thread
+   → occupancy collapses on Mali. Fix (an ARM/panvk warptile branch): shrink to 16 accumulators
+   (spill → 96 B), enlarge the workgroup at fixed registers for shared-memory occupancy, and scope
+   `BK_STEP` to the L pipeline via a spec constant (so decode/M-variant are untouched). Correctness
+   verified by GPU-vs-CPU generation match.
 
-- `patches/` — collected patches (llama.cpp + Mesa panvk).
-- `docs/` — root-cause analysis, benchmark data, build/boot procedure.
-- `mesa/` — Mesa 26.1.3 source (gitignored tarball) for the panvk build.
+Apply on top of Mesa 26.1.3 (panvk build: `-Dvulkan-drivers=panfrost`, minimal) and llama.cpp
+`389ff61`; run on a panthor kernel with the GPU bound to `panthor` (not the vendor `mali` blob).
 
-## Benchmarks (Qwen2.5-3B-Q4_K_M, boltzmann, pp512)
+## How it was found
 
-| Backend | t/s | Notes |
-|---------|-----|-------|
-| CPU (plain build, 8 threads) | 22.4 | `+noi8mm` handicap; i8mm would be higher |
-| GPU (panvk Vulkan, `-ub 64`) | 21.0 | ub≥72 → OOM crash |
-| GPU (panvk, ub128+) | crash | `vk::CommandBuffer::end: ErrorOutOfDeviceMemory` |
+Instrument-driven, with a second model ("Fable") implementing the shader and a hardware-in-the-loop
+measure loop. The panvk TLS-stack size *is* the register-spill metric, which made every tile change
+a single-variable experiment — including the informative negatives (bigger tiles → occupancy-bound
+not compute-bound; `BK_STEP=1` → barrier crossover; f16-accumulators → refused with proof). Full
+trail in `docs/` (`root-cause*.md` → `shader-tuning.md` → `RESULTS.md`).
 
-## Prior art (why the Mali is "framebuffer-only")
+## Repo layout
 
-Arm chose no matrix cores (int8 dotprod on vector ALUs); panvk is graphics-first (Mesa 26.1 sprint
-= Proton/gaming; Vulkan 1.2 conformance only 2025); llama.cpp Vulkan is ~15× slower than CPU on
-Mali+Adreno and unsolved upstream ([ggml #9464]). But **MLC-LLM/TVM** (auto-tuned OpenCL kernels)
-get decent Mali perf → the hardware is capable; the gap is tuned kernels + driver maturity.
+- `patches/` — the two final patches (Mesa + llama.cpp), plus `patches/iterations/` (the tuning trail).
+- `docs/` — root-cause analysis, the review notes, the iteration log, benchmarks, and the fix plan.
 
-[ggml #9464]: https://github.com/ggml-org/llama.cpp/discussions/9464
+## Not fixed here
+
+- **Decode** stays memory-bandwidth-bound (~28 GB/s LPDDR5) — the GPU can't help it; that's a
+  speculation/MoE problem, not a shader one.
+- **Gemma** models produce garbage on llama.cpp's Vulkan backend (a pre-existing gemma-op bug,
+  independent of these patches — they're correct on CPU). Qwen-family works.
+- The last ~15 % to roofline is the separate q8_1 activation-quantize dispatch, not this shader.
+
+## License
+
+MIT (see `LICENSE`). The patches are against Mesa and llama.cpp, both MIT-licensed.

patches/README.md

@@ -0,0 +1,26 @@
+# Patches
+
+## Apply these two (the deliverables)
+
+| patch | target | what |
+|-------|--------|------|
+| `mesa-panvk-shared-tls-wls.patch` | Mesa 26.1.3, `src/panfrost/vulkan/` | Fix the panvk priv-VA-heap exhaustion (OOM at ub>64). Device-shared TLS scratch + WLS reuse. Also contains a small `PANVK_AS_LOG` priv-heap instrument (env-gated). |
+| `llama.cpp-mali-mmq-warptile.patch` | llama.cpp `389ff61`, `ggml/src/ggml-vulkan/` | Mali/panvk `mul_mm`/`mul_mmq` warptile branch + `BK_STEP` spec constant (L-scoped). The 4.8× prefill. Also contains an env-gated `[panvk-tls]`/`PANVK_WLS_LOG` shader-spill trace. |
+
+Env flags used at runtime (both patches): `PANVK_SHARED_TLS=1 PANVK_WLS_REUSE=1`. Trace flags
+(optional): `PANVK_AS_LOG=1`, `PANVK_WLS_LOG=1`.
+
+Build panvk minimally: `meson setup build -Dvulkan-drivers=panfrost -Dgallium-drivers= ...`
+(mirrors a bifrost-style co-install). Point `VK_ICD_FILENAMES` at the built `libvulkan_panfrost.so`.
+Run on a **panthor** kernel with the GPU bound to `panthor`, not the vendor `mali` blob.
+
+## `iterations/` — the tuning trail (not for applying)
+
+The single-variable shader experiments, in order. See `docs/shader-tuning.md` for the numbers.
+
+- `llama.cpp-0001-vk-tunable-submit-thresholds.patch` — exploratory, no effect (kept for the record).
+- `llama.cpp-0002-vk-cmd-cleanup-freq-env.patch` — falsified hypothesis (cmdbufs are in-flight).
+- `llama.cpp-0003-mali-warptile-iter1.patch` — 16-acc tile, spill 5328→96 B, ub128 4×.
+- `llama.cpp-0003-mali-warptile-iter3-champion.patch` — bigger workgroup, same registers, +10%.
+- `llama.cpp-0004-mali-shader-iter5-champion.patch` — scoped BK_STEP=2 (superseded by the final
+  patch, which is iter5's content).