rocky-vulkan-llama

Making the RK3588 Mali-G610 GPU a real LLM prefill accelerator — via the open Vulkan stack (panthor kernel + Mesa panvk + llama.cpp), no vendor blobs.

Two upstreamable fixes take large-batch prefill on a Radxa ROCK 5 ITX+ (Rockchip RK3588) from crashing to 4.8× faster, beating the CPU — on a GPU with no matrix cores.

TL;DR results (Qwen2.5-3B-Q4_K_M, pp512, Mali-G610 MC4, Mesa 26.1.3 panvk)

	prefill t/s	note
stock llama.cpp Vulkan @ ub≥128	crash	vk::CommandBuffer::end: ErrorOutOfDeviceMemory
+ driver fix	5.8	runs, but the desktop-tuned shader is register-spilling
+ shader tune (this repo)	28.1	4.8×, spill-free, ~35 % of roofline, output-verified
(CPU reference, i8mm-off)	~22

The ceiling was never the hardware — it was a desktop-tuned shader thrashing Mali's tiny register file, and a Mesa allocator that never shared per-command-buffer scratch. Same shape as unlocking hardware video decode: fix the whole stack (kernel → Mesa → the consumer), not just one layer.

The two fixes

1. Mesa panvk — priv-heap exhaustion (patches/mesa-panvk-shared-tls-wls.patch) CSF panvk allocates a per-command-buffer TLS stack (up to 304 MB) and per-dispatch WLS scratch from a device-wide sub-4 GB priv VA heap, never shared. A pipelined LLM graph keeps ~12 command buffers in flight → ~4 GB → OOM at any ubatch > 64. Fix: one device-shared TLS scratch reused across the serially-executing compute queue, plus grid-independent WLS reuse. Peak priv-VA 3985 MB → 908 MB; no crash from ub 64 to 1024. Affects any large-workgroup compute on Mali/panvk, not just LLMs.

2. llama.cpp — Mali-tuned mul_mm/mul_mmq warptile (patches/llama.cpp-mali-mmq-warptile.patch) Per-thread matmul accumulators = WM·WN/WARP; the stock L-variant is 128 → spills 5328 B/thread → occupancy collapses on Mali. Fix (an ARM/panvk warptile branch): shrink to 16 accumulators (spill → 96 B), enlarge the workgroup at fixed registers for shared-memory occupancy, and scope BK_STEP to the L pipeline via a spec constant (so decode/M-variant are untouched). Correctness verified by GPU-vs-CPU generation match.

Apply on top of Mesa 26.1.3 (panvk build: -Dvulkan-drivers=panfrost, minimal) and llama.cpp 389ff61; run on a panthor kernel with the GPU bound to panthor (not the vendor mali blob).

How it was found

Instrument-driven, with a second model ("Fable") implementing the shader and a hardware-in-the-loop measure loop. The panvk TLS-stack size is the register-spill metric, which made every tile change a single-variable experiment — including the informative negatives (bigger tiles → occupancy-bound not compute-bound; BK_STEP=1 → barrier crossover; f16-accumulators → refused with proof). Full trail in docs/ (root-cause*.md → shader-tuning.md → RESULTS.md).

Repo layout

• patches/ — the two final patches (Mesa + llama.cpp), plus patches/iterations/ (the tuning trail).
• docs/ — root-cause analysis, the review notes, the iteration log, benchmarks, and the fix plan.

Not fixed here

• Decode stays memory-bandwidth-bound (~28 GB/s LPDDR5) — the GPU can't help it; that's a speculation/MoE problem, not a shader one.
• Gemma models produce garbage on llama.cpp's Vulkan backend (a pre-existing gemma-op bug, independent of these patches — they're correct on CPU). Qwen-family works.
• The last ~15 % to roofline is the separate q8_1 activation-quantize dispatch, not this shader.

License

MIT (see LICENSE). The patches are against Mesa and llama.cpp, both MIT-licensed.