# rocky-vulkan-llama

**Making the RK3588 Mali-G610 GPU a real LLM prefill accelerator — via the open Vulkan stack
(panthor kernel + Mesa panvk + llama.cpp), no vendor blobs.**

Two upstreamable fixes take large-batch prefill on a Radxa ROCK 5 ITX+ (Rockchip RK3588) from
*crashing* to **4.8× faster**, beating the CPU — on a GPU with **no matrix cores**.

## TL;DR results (Qwen2.5-3B-Q4_K_M, pp512, Mali-G610 MC4, Mesa 26.1.3 panvk)

| | prefill t/s | note |
|---|---|---|
| stock llama.cpp Vulkan @ ub≥128 | **crash** | `vk::CommandBuffer::end: ErrorOutOfDeviceMemory` |
| + driver fix | 5.8 | runs, but the desktop-tuned shader is register-spilling |
| + shader tune (this repo) | **28.1** | **4.8×**, spill-free, ~35 % of roofline, output-verified |
| (CPU reference, i8mm-off) | ~22 | |

The ceiling was never the hardware — it was a desktop-tuned shader thrashing Mali's tiny register
file, and a Mesa allocator that never shared per-command-buffer scratch. Same shape as unlocking
hardware video decode: fix the whole stack (kernel → Mesa → the consumer), not just one layer.

## The two fixes

1. **Mesa panvk — priv-heap exhaustion** (`patches/mesa-panvk-shared-tls-wls.patch`)
   CSF panvk allocates a per-command-buffer TLS stack (up to 304 MB) and per-dispatch WLS scratch
   from a device-wide **sub-4 GB priv VA heap**, never shared. A pipelined LLM graph keeps ~12
   command buffers in flight → ~4 GB → OOM at any ubatch > 64. Fix: one **device-shared TLS scratch**
   reused across the serially-executing compute queue, plus grid-independent **WLS reuse**. Peak
   priv-VA 3985 MB → 908 MB; no crash from ub 64 to 1024. Affects *any* large-workgroup compute on
   Mali/panvk, not just LLMs.

2. **llama.cpp — Mali-tuned `mul_mm`/`mul_mmq` warptile** (`patches/llama.cpp-mali-mmq-warptile.patch`)
   Per-thread matmul accumulators = `WM·WN/WARP`; the stock L-variant is 128 → spills 5328 B/thread
   → occupancy collapses on Mali. Fix (an ARM/panvk warptile branch): shrink to 16 accumulators
   (spill → 96 B), enlarge the workgroup at fixed registers for shared-memory occupancy, and scope
   `BK_STEP` to the L pipeline via a spec constant (so decode/M-variant are untouched). Correctness
   verified by GPU-vs-CPU generation match.

Apply on top of Mesa 26.1.3 (panvk build: `-Dvulkan-drivers=panfrost`, minimal) and llama.cpp
`389ff61`; run on a panthor kernel with the GPU bound to `panthor` (not the vendor `mali` blob).

## How it was found

Instrument-driven, with a second model ("Fable") implementing the shader and a hardware-in-the-loop
measure loop. The panvk TLS-stack size *is* the register-spill metric, which made every tile change
a single-variable experiment — including the informative negatives (bigger tiles → occupancy-bound
not compute-bound; `BK_STEP=1` → barrier crossover; f16-accumulators → refused with proof). Full
trail in `docs/` (`root-cause*.md` → `shader-tuning.md` → `RESULTS.md`).

## Repo layout

- `patches/` — the two final patches (Mesa + llama.cpp), plus `patches/iterations/` (the tuning trail).
- `docs/` — root-cause analysis, the review notes, the iteration log, benchmarks, and the fix plan.

## Not fixed here

- **Decode** stays memory-bandwidth-bound (~28 GB/s LPDDR5) — the GPU can't help it; that's a
  speculation/MoE problem, not a shader one.
- **Gemma** models produce garbage on llama.cpp's Vulkan backend (a pre-existing gemma-op bug,
  independent of these patches — they're correct on CPU). Qwen-family works.
- The last ~15 % to roofline is the separate q8_1 activation-quantize dispatch, not this shader.

## License

MIT (see `LICENSE`). The patches are against Mesa and llama.cpp, both MIT-licensed.