Claude (noether)
16b96537fd
Documents the RK3588 Mali-G610 GPU-prefill stack (panthor + panvk + llama.cpp Vulkan), the panvk ub64 OOM blocker, and collects the ggml-vulkan tunable-submit patch. Mesa panvk fix pending root-cause confirmation (see docs/root-cause.md). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
56 lines
3.2 KiB
Markdown
56 lines
3.2 KiB
Markdown
# rocky-vulkan-llama
|
||
|
||
GPU-accelerated **LLM prefill** on the RK3588 **Mali-G610** via the open Vulkan stack
|
||
(panthor kernel driver + Mesa **panvk** + llama.cpp Vulkan backend).
|
||
|
||
Goal: make the Mali GPU a useful **prompt-processing (prefill) accelerator** for local agents on
|
||
RK3588 boards (boltzmann = Radxa ROCK 5 ITX+; ampere = CoolPi CM5 GenBook), so a ~5k-token agent
|
||
context doesn't cost ~150–250 s of CPU prefill before the model starts generating.
|
||
|
||
> Decode (token generation) is memory-bandwidth-bound (~28 GB/s shared LPDDR5) and the GPU can't
|
||
> help it. **Prefill is compute-bound** — that's the only place the GPU's FLOPS matter. This repo
|
||
> is about prefill.
|
||
|
||
## Dependency stack
|
||
|
||
| Layer | Requirement | Notes |
|
||
|-------|-------------|-------|
|
||
| Kernel | **panthor** DRM driver (mainline ≥ 6.x), GPU node bound to panthor, **not** the vendor `mali` kbase blob | On boltzmann: boot `linux-rk3588-marfrit-A1` (mainline 7.0.0-rc3). The vendor BSP kernel (`6.1.75-npu-port`) binds `mali` → panvk cannot enumerate. |
|
||
| CSF firmware | `/lib/firmware/arm/mali/arch10.10/mali_csffw.bin` | Present on boltzmann. |
|
||
| Userspace GL/Vulkan | **Mesa panvk** (`libvulkan_panfrost.so`), Mesa **26.1.3** | `panfrost_icd.json` ICD. Enumerates `Mali-G610 MC4 (panvk)` with `uma:1 fp16:1 int dot:1 matrix cores:none`. |
|
||
| Runtime env | `XDG_RUNTIME_DIR` must be set | panvk init fails without it. |
|
||
| Inference | **llama.cpp** Vulkan backend (`-DGGML_VULKAN=ON`), commit `389ff61` | Build tools: `glslc`, `libvulkan`, cmake. |
|
||
|
||
## Status (2026-07-11)
|
||
|
||
- ✅ Full open stack works: panthor binds the GPU, panvk enumerates it, llama.cpp runs on it.
|
||
- ⚠️ **Blocker:** panvk throws `vk::CommandBuffer::end: ErrorOutOfDeviceMemory` on any prefill
|
||
dispatch above **ubatch 64**. Only `-ub 64` runs; at ub64 prefill is **21 t/s**, which is
|
||
*overhead-bound* (ub16/32 far slower) — i.e. 21 t/s is the "forced-tiny-chunk" ceiling, not the
|
||
Mali GEMM ceiling. It ties an (i8mm-disabled) CPU.
|
||
- 🔬 Root cause narrowed to a panvk driver allocation that scales past ub64; see
|
||
[docs/root-cause.md](docs/root-cause.md). The **fix is a Mesa panvk patch** (this repo).
|
||
|
||
## Contents
|
||
|
||
- `patches/` — collected patches (llama.cpp + Mesa panvk).
|
||
- `docs/` — root-cause analysis, benchmark data, build/boot procedure.
|
||
- `mesa/` — Mesa 26.1.3 source (gitignored tarball) for the panvk build.
|
||
|
||
## Benchmarks (Qwen2.5-3B-Q4_K_M, boltzmann, pp512)
|
||
|
||
| Backend | t/s | Notes |
|
||
|---------|-----|-------|
|
||
| CPU (plain build, 8 threads) | 22.4 | `+noi8mm` handicap; i8mm would be higher |
|
||
| GPU (panvk Vulkan, `-ub 64`) | 21.0 | ub≥72 → OOM crash |
|
||
| GPU (panvk, ub128+) | crash | `vk::CommandBuffer::end: ErrorOutOfDeviceMemory` |
|
||
|
||
## Prior art (why the Mali is "framebuffer-only")
|
||
|
||
Arm chose no matrix cores (int8 dotprod on vector ALUs); panvk is graphics-first (Mesa 26.1 sprint
|
||
= Proton/gaming; Vulkan 1.2 conformance only 2025); llama.cpp Vulkan is ~15× slower than CPU on
|
||
Mali+Adreno and unsolved upstream ([ggml #9464]). But **MLC-LLM/TVM** (auto-tuned OpenCL kernels)
|
||
get decent Mali perf → the hardware is capable; the gap is tuned kernels + driver maturity.
|
||
|
||
[ggml #9464]: https://github.com/ggml-org/llama.cpp/discussions/9464
|