From 107ccddbb1229be254acc67c0c05a52bda7a0b16 Mon Sep 17 00:00:00 2001 From: Markus Fritsche Date: Sat, 11 Jul 2026 20:35:44 +0200 Subject: [PATCH] Clean up for public release: README rewrite, LICENSE (MIT), patches reorganized Final deliverables (mesa-panvk-shared-tls-wls + llama.cpp-mali-mmq-warptile) at top of patches/; iteration trail moved to patches/iterations/. Public-facing README with the 4.8x result and the two-fix summary. Co-Authored-By: Claude Opus 4.8 --- LICENSE | 25 ++++++ README.md | 90 ++++++++++--------- patches/README.md | 26 ++++++ ...pp-0001-vk-tunable-submit-thresholds.patch | 0 ...ama.cpp-0002-vk-cmd-cleanup-freq-env.patch | 0 .../llama.cpp-0003-mali-warptile-iter1.patch | 0 ...pp-0003-mali-warptile-iter3-champion.patch | 0 ....cpp-0004-mali-shader-iter5-champion.patch | 0 ...atch => llama.cpp-mali-mmq-warptile.patch} | 0 ....patch => mesa-panvk-shared-tls-wls.patch} | 0 10 files changed, 101 insertions(+), 40 deletions(-) create mode 100644 LICENSE create mode 100644 patches/README.md rename patches/{ => iterations}/llama.cpp-0001-vk-tunable-submit-thresholds.patch (100%) rename patches/{ => iterations}/llama.cpp-0002-vk-cmd-cleanup-freq-env.patch (100%) rename patches/{ => iterations}/llama.cpp-0003-mali-warptile-iter1.patch (100%) rename patches/{ => iterations}/llama.cpp-0003-mali-warptile-iter3-champion.patch (100%) rename patches/{ => iterations}/llama.cpp-0004-mali-shader-iter5-champion.patch (100%) rename patches/{llama.cpp-0005-mali-shader-CHAMPION.patch => llama.cpp-mali-mmq-warptile.patch} (100%) rename patches/{mesa-panvk-0010-fix-priv-heap-exhaustion-shared-tls-wls.patch => mesa-panvk-shared-tls-wls.patch} (100%) diff --git a/LICENSE b/LICENSE new file mode 100644 index 0000000..0cc364e --- /dev/null +++ b/LICENSE @@ -0,0 +1,25 @@ +MIT License + +Copyright (c) 2026 Markus Fritsche + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. + +The patches under patches/ are modifications to Mesa and llama.cpp, each of +which is distributed under the MIT license; those modifications inherit the +upstream licenses of the files they apply to. diff --git a/README.md b/README.md index b0a6f6a..77b3fcd 100644 --- a/README.md +++ b/README.md @@ -1,55 +1,65 @@ # rocky-vulkan-llama -GPU-accelerated **LLM prefill** on the RK3588 **Mali-G610** via the open Vulkan stack -(panthor kernel driver + Mesa **panvk** + llama.cpp Vulkan backend). +**Making the RK3588 Mali-G610 GPU a real LLM prefill accelerator — via the open Vulkan stack +(panthor kernel + Mesa panvk + llama.cpp), no vendor blobs.** -Goal: make the Mali GPU a useful **prompt-processing (prefill) accelerator** for local agents on -RK3588 boards (boltzmann = Radxa ROCK 5 ITX+; ampere = CoolPi CM5 GenBook), so a ~5k-token agent -context doesn't cost ~150–250 s of CPU prefill before the model starts generating. +Two upstreamable fixes take large-batch prefill on a Radxa ROCK 5 ITX+ (Rockchip RK3588) from +*crashing* to **4.8× faster**, beating the CPU — on a GPU with **no matrix cores**. -> Decode (token generation) is memory-bandwidth-bound (~28 GB/s shared LPDDR5) and the GPU can't -> help it. **Prefill is compute-bound** — that's the only place the GPU's FLOPS matter. This repo -> is about prefill. +## TL;DR results (Qwen2.5-3B-Q4_K_M, pp512, Mali-G610 MC4, Mesa 26.1.3 panvk) -## Dependency stack +| | prefill t/s | note | +|---|---|---| +| stock llama.cpp Vulkan @ ub≥128 | **crash** | `vk::CommandBuffer::end: ErrorOutOfDeviceMemory` | +| + driver fix | 5.8 | runs, but the desktop-tuned shader is register-spilling | +| + shader tune (this repo) | **28.1** | **4.8×**, spill-free, ~35 % of roofline, output-verified | +| (CPU reference, i8mm-off) | ~22 | | -| Layer | Requirement | Notes | -|-------|-------------|-------| -| Kernel | **panthor** DRM driver (mainline ≥ 6.x), GPU node bound to panthor, **not** the vendor `mali` kbase blob | On boltzmann: boot `linux-rk3588-marfrit-A1` (mainline 7.0.0-rc3). The vendor BSP kernel (`6.1.75-npu-port`) binds `mali` → panvk cannot enumerate. | -| CSF firmware | `/lib/firmware/arm/mali/arch10.10/mali_csffw.bin` | Present on boltzmann. | -| Userspace GL/Vulkan | **Mesa panvk** (`libvulkan_panfrost.so`), Mesa **26.1.3** | `panfrost_icd.json` ICD. Enumerates `Mali-G610 MC4 (panvk)` with `uma:1 fp16:1 int dot:1 matrix cores:none`. | -| Runtime env | `XDG_RUNTIME_DIR` must be set | panvk init fails without it. | -| Inference | **llama.cpp** Vulkan backend (`-DGGML_VULKAN=ON`), commit `389ff61` | Build tools: `glslc`, `libvulkan`, cmake. | +The ceiling was never the hardware — it was a desktop-tuned shader thrashing Mali's tiny register +file, and a Mesa allocator that never shared per-command-buffer scratch. Same shape as unlocking +hardware video decode: fix the whole stack (kernel → Mesa → the consumer), not just one layer. -## Status (2026-07-11) +## The two fixes -- ✅ Full open stack works: panthor binds the GPU, panvk enumerates it, llama.cpp runs on it. -- ⚠️ **Blocker:** panvk throws `vk::CommandBuffer::end: ErrorOutOfDeviceMemory` on any prefill - dispatch above **ubatch 64**. Only `-ub 64` runs; at ub64 prefill is **21 t/s**, which is - *overhead-bound* (ub16/32 far slower) — i.e. 21 t/s is the "forced-tiny-chunk" ceiling, not the - Mali GEMM ceiling. It ties an (i8mm-disabled) CPU. -- 🔬 Root cause narrowed to a panvk driver allocation that scales past ub64; see - [docs/root-cause.md](docs/root-cause.md). The **fix is a Mesa panvk patch** (this repo). +1. **Mesa panvk — priv-heap exhaustion** (`patches/mesa-panvk-shared-tls-wls.patch`) + CSF panvk allocates a per-command-buffer TLS stack (up to 304 MB) and per-dispatch WLS scratch + from a device-wide **sub-4 GB priv VA heap**, never shared. A pipelined LLM graph keeps ~12 + command buffers in flight → ~4 GB → OOM at any ubatch > 64. Fix: one **device-shared TLS scratch** + reused across the serially-executing compute queue, plus grid-independent **WLS reuse**. Peak + priv-VA 3985 MB → 908 MB; no crash from ub 64 to 1024. Affects *any* large-workgroup compute on + Mali/panvk, not just LLMs. -## Contents +2. **llama.cpp — Mali-tuned `mul_mm`/`mul_mmq` warptile** (`patches/llama.cpp-mali-mmq-warptile.patch`) + Per-thread matmul accumulators = `WM·WN/WARP`; the stock L-variant is 128 → spills 5328 B/thread + → occupancy collapses on Mali. Fix (an ARM/panvk warptile branch): shrink to 16 accumulators + (spill → 96 B), enlarge the workgroup at fixed registers for shared-memory occupancy, and scope + `BK_STEP` to the L pipeline via a spec constant (so decode/M-variant are untouched). Correctness + verified by GPU-vs-CPU generation match. -- `patches/` — collected patches (llama.cpp + Mesa panvk). -- `docs/` — root-cause analysis, benchmark data, build/boot procedure. -- `mesa/` — Mesa 26.1.3 source (gitignored tarball) for the panvk build. +Apply on top of Mesa 26.1.3 (panvk build: `-Dvulkan-drivers=panfrost`, minimal) and llama.cpp +`389ff61`; run on a panthor kernel with the GPU bound to `panthor` (not the vendor `mali` blob). -## Benchmarks (Qwen2.5-3B-Q4_K_M, boltzmann, pp512) +## How it was found -| Backend | t/s | Notes | -|---------|-----|-------| -| CPU (plain build, 8 threads) | 22.4 | `+noi8mm` handicap; i8mm would be higher | -| GPU (panvk Vulkan, `-ub 64`) | 21.0 | ub≥72 → OOM crash | -| GPU (panvk, ub128+) | crash | `vk::CommandBuffer::end: ErrorOutOfDeviceMemory` | +Instrument-driven, with a second model ("Fable") implementing the shader and a hardware-in-the-loop +measure loop. The panvk TLS-stack size *is* the register-spill metric, which made every tile change +a single-variable experiment — including the informative negatives (bigger tiles → occupancy-bound +not compute-bound; `BK_STEP=1` → barrier crossover; f16-accumulators → refused with proof). Full +trail in `docs/` (`root-cause*.md` → `shader-tuning.md` → `RESULTS.md`). -## Prior art (why the Mali is "framebuffer-only") +## Repo layout -Arm chose no matrix cores (int8 dotprod on vector ALUs); panvk is graphics-first (Mesa 26.1 sprint -= Proton/gaming; Vulkan 1.2 conformance only 2025); llama.cpp Vulkan is ~15× slower than CPU on -Mali+Adreno and unsolved upstream ([ggml #9464]). But **MLC-LLM/TVM** (auto-tuned OpenCL kernels) -get decent Mali perf → the hardware is capable; the gap is tuned kernels + driver maturity. +- `patches/` — the two final patches (Mesa + llama.cpp), plus `patches/iterations/` (the tuning trail). +- `docs/` — root-cause analysis, the review notes, the iteration log, benchmarks, and the fix plan. -[ggml #9464]: https://github.com/ggml-org/llama.cpp/discussions/9464 +## Not fixed here + +- **Decode** stays memory-bandwidth-bound (~28 GB/s LPDDR5) — the GPU can't help it; that's a + speculation/MoE problem, not a shader one. +- **Gemma** models produce garbage on llama.cpp's Vulkan backend (a pre-existing gemma-op bug, + independent of these patches — they're correct on CPU). Qwen-family works. +- The last ~15 % to roofline is the separate q8_1 activation-quantize dispatch, not this shader. + +## License + +MIT (see `LICENSE`). The patches are against Mesa and llama.cpp, both MIT-licensed. diff --git a/patches/README.md b/patches/README.md new file mode 100644 index 0000000..edc2748 --- /dev/null +++ b/patches/README.md @@ -0,0 +1,26 @@ +# Patches + +## Apply these two (the deliverables) + +| patch | target | what | +|-------|--------|------| +| `mesa-panvk-shared-tls-wls.patch` | Mesa 26.1.3, `src/panfrost/vulkan/` | Fix the panvk priv-VA-heap exhaustion (OOM at ub>64). Device-shared TLS scratch + WLS reuse. Also contains a small `PANVK_AS_LOG` priv-heap instrument (env-gated). | +| `llama.cpp-mali-mmq-warptile.patch` | llama.cpp `389ff61`, `ggml/src/ggml-vulkan/` | Mali/panvk `mul_mm`/`mul_mmq` warptile branch + `BK_STEP` spec constant (L-scoped). The 4.8× prefill. Also contains an env-gated `[panvk-tls]`/`PANVK_WLS_LOG` shader-spill trace. | + +Env flags used at runtime (both patches): `PANVK_SHARED_TLS=1 PANVK_WLS_REUSE=1`. Trace flags +(optional): `PANVK_AS_LOG=1`, `PANVK_WLS_LOG=1`. + +Build panvk minimally: `meson setup build -Dvulkan-drivers=panfrost -Dgallium-drivers= ...` +(mirrors a bifrost-style co-install). Point `VK_ICD_FILENAMES` at the built `libvulkan_panfrost.so`. +Run on a **panthor** kernel with the GPU bound to `panthor`, not the vendor `mali` blob. + +## `iterations/` — the tuning trail (not for applying) + +The single-variable shader experiments, in order. See `docs/shader-tuning.md` for the numbers. + +- `llama.cpp-0001-vk-tunable-submit-thresholds.patch` — exploratory, no effect (kept for the record). +- `llama.cpp-0002-vk-cmd-cleanup-freq-env.patch` — falsified hypothesis (cmdbufs are in-flight). +- `llama.cpp-0003-mali-warptile-iter1.patch` — 16-acc tile, spill 5328→96 B, ub128 4×. +- `llama.cpp-0003-mali-warptile-iter3-champion.patch` — bigger workgroup, same registers, +10%. +- `llama.cpp-0004-mali-shader-iter5-champion.patch` — scoped BK_STEP=2 (superseded by the final + patch, which is iter5's content). diff --git a/patches/llama.cpp-0001-vk-tunable-submit-thresholds.patch b/patches/iterations/llama.cpp-0001-vk-tunable-submit-thresholds.patch similarity index 100% rename from patches/llama.cpp-0001-vk-tunable-submit-thresholds.patch rename to patches/iterations/llama.cpp-0001-vk-tunable-submit-thresholds.patch diff --git a/patches/llama.cpp-0002-vk-cmd-cleanup-freq-env.patch b/patches/iterations/llama.cpp-0002-vk-cmd-cleanup-freq-env.patch similarity index 100% rename from patches/llama.cpp-0002-vk-cmd-cleanup-freq-env.patch rename to patches/iterations/llama.cpp-0002-vk-cmd-cleanup-freq-env.patch diff --git a/patches/llama.cpp-0003-mali-warptile-iter1.patch b/patches/iterations/llama.cpp-0003-mali-warptile-iter1.patch similarity index 100% rename from patches/llama.cpp-0003-mali-warptile-iter1.patch rename to patches/iterations/llama.cpp-0003-mali-warptile-iter1.patch diff --git a/patches/llama.cpp-0003-mali-warptile-iter3-champion.patch b/patches/iterations/llama.cpp-0003-mali-warptile-iter3-champion.patch similarity index 100% rename from patches/llama.cpp-0003-mali-warptile-iter3-champion.patch rename to patches/iterations/llama.cpp-0003-mali-warptile-iter3-champion.patch diff --git a/patches/llama.cpp-0004-mali-shader-iter5-champion.patch b/patches/iterations/llama.cpp-0004-mali-shader-iter5-champion.patch similarity index 100% rename from patches/llama.cpp-0004-mali-shader-iter5-champion.patch rename to patches/iterations/llama.cpp-0004-mali-shader-iter5-champion.patch diff --git a/patches/llama.cpp-0005-mali-shader-CHAMPION.patch b/patches/llama.cpp-mali-mmq-warptile.patch similarity index 100% rename from patches/llama.cpp-0005-mali-shader-CHAMPION.patch rename to patches/llama.cpp-mali-mmq-warptile.patch diff --git a/patches/mesa-panvk-0010-fix-priv-heap-exhaustion-shared-tls-wls.patch b/patches/mesa-panvk-shared-tls-wls.patch similarity index 100% rename from patches/mesa-panvk-0010-fix-priv-heap-exhaustion-shared-tls-wls.patch rename to patches/mesa-panvk-shared-tls-wls.patch