Markus Fritsche
107ccddbb1
Final deliverables (mesa-panvk-shared-tls-wls + llama.cpp-mali-mmq-warptile) at top of patches/; iteration trail moved to patches/iterations/. Public-facing README with the 4.8x result and the two-fix summary. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
66 lines
3.6 KiB
Markdown
66 lines
3.6 KiB
Markdown
# rocky-vulkan-llama
|
||
|
||
**Making the RK3588 Mali-G610 GPU a real LLM prefill accelerator — via the open Vulkan stack
|
||
(panthor kernel + Mesa panvk + llama.cpp), no vendor blobs.**
|
||
|
||
Two upstreamable fixes take large-batch prefill on a Radxa ROCK 5 ITX+ (Rockchip RK3588) from
|
||
*crashing* to **4.8× faster**, beating the CPU — on a GPU with **no matrix cores**.
|
||
|
||
## TL;DR results (Qwen2.5-3B-Q4_K_M, pp512, Mali-G610 MC4, Mesa 26.1.3 panvk)
|
||
|
||
| | prefill t/s | note |
|
||
|---|---|---|
|
||
| stock llama.cpp Vulkan @ ub≥128 | **crash** | `vk::CommandBuffer::end: ErrorOutOfDeviceMemory` |
|
||
| + driver fix | 5.8 | runs, but the desktop-tuned shader is register-spilling |
|
||
| + shader tune (this repo) | **28.1** | **4.8×**, spill-free, ~35 % of roofline, output-verified |
|
||
| (CPU reference, i8mm-off) | ~22 | |
|
||
|
||
The ceiling was never the hardware — it was a desktop-tuned shader thrashing Mali's tiny register
|
||
file, and a Mesa allocator that never shared per-command-buffer scratch. Same shape as unlocking
|
||
hardware video decode: fix the whole stack (kernel → Mesa → the consumer), not just one layer.
|
||
|
||
## The two fixes
|
||
|
||
1. **Mesa panvk — priv-heap exhaustion** (`patches/mesa-panvk-shared-tls-wls.patch`)
|
||
CSF panvk allocates a per-command-buffer TLS stack (up to 304 MB) and per-dispatch WLS scratch
|
||
from a device-wide **sub-4 GB priv VA heap**, never shared. A pipelined LLM graph keeps ~12
|
||
command buffers in flight → ~4 GB → OOM at any ubatch > 64. Fix: one **device-shared TLS scratch**
|
||
reused across the serially-executing compute queue, plus grid-independent **WLS reuse**. Peak
|
||
priv-VA 3985 MB → 908 MB; no crash from ub 64 to 1024. Affects *any* large-workgroup compute on
|
||
Mali/panvk, not just LLMs.
|
||
|
||
2. **llama.cpp — Mali-tuned `mul_mm`/`mul_mmq` warptile** (`patches/llama.cpp-mali-mmq-warptile.patch`)
|
||
Per-thread matmul accumulators = `WM·WN/WARP`; the stock L-variant is 128 → spills 5328 B/thread
|
||
→ occupancy collapses on Mali. Fix (an ARM/panvk warptile branch): shrink to 16 accumulators
|
||
(spill → 96 B), enlarge the workgroup at fixed registers for shared-memory occupancy, and scope
|
||
`BK_STEP` to the L pipeline via a spec constant (so decode/M-variant are untouched). Correctness
|
||
verified by GPU-vs-CPU generation match.
|
||
|
||
Apply on top of Mesa 26.1.3 (panvk build: `-Dvulkan-drivers=panfrost`, minimal) and llama.cpp
|
||
`389ff61`; run on a panthor kernel with the GPU bound to `panthor` (not the vendor `mali` blob).
|
||
|
||
## How it was found
|
||
|
||
Instrument-driven, with a second model ("Fable") implementing the shader and a hardware-in-the-loop
|
||
measure loop. The panvk TLS-stack size *is* the register-spill metric, which made every tile change
|
||
a single-variable experiment — including the informative negatives (bigger tiles → occupancy-bound
|
||
not compute-bound; `BK_STEP=1` → barrier crossover; f16-accumulators → refused with proof). Full
|
||
trail in `docs/` (`root-cause*.md` → `shader-tuning.md` → `RESULTS.md`).
|
||
|
||
## Repo layout
|
||
|
||
- `patches/` — the two final patches (Mesa + llama.cpp), plus `patches/iterations/` (the tuning trail).
|
||
- `docs/` — root-cause analysis, the review notes, the iteration log, benchmarks, and the fix plan.
|
||
|
||
## Not fixed here
|
||
|
||
- **Decode** stays memory-bandwidth-bound (~28 GB/s LPDDR5) — the GPU can't help it; that's a
|
||
speculation/MoE problem, not a shader one.
|
||
- **Gemma** models produce garbage on llama.cpp's Vulkan backend (a pre-existing gemma-op bug,
|
||
independent of these patches — they're correct on CPU). Qwen-family works.
|
||
- The last ~15 % to roofline is the separate q8_1 activation-quantize dispatch, not this shader.
|
||
|
||
## License
|
||
|
||
MIT (see `LICENSE`). The patches are against Mesa and llama.cpp, both MIT-licensed.
|