Claude (noether)
97629e935c
Live instrumentation on panthor: WLS reuse verified working (644 REUSE/46 ALLOC) but the crash is dominated by the TLS *stack* alloc in emit_tls() at EndCommandBuffer — L-variant matmul shader spills 10x more registers (tls_size 544->5328 B/thread => 38MB->304MB stack), x~12 cmdbufs retained in the tls pool => ~3GB, over the 4GB priv heap. Remaining fix is structural (free tls-pool BOs post-submit and/or fix the x19 sparse-core overallocation). See docs/root-cause-final.md. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
rocky-vulkan-llama
GPU-accelerated LLM prefill on the RK3588 Mali-G610 via the open Vulkan stack (panthor kernel driver + Mesa panvk + llama.cpp Vulkan backend).
Goal: make the Mali GPU a useful prompt-processing (prefill) accelerator for local agents on RK3588 boards (boltzmann = Radxa ROCK 5 ITX+; ampere = CoolPi CM5 GenBook), so a ~5k-token agent context doesn't cost ~150–250 s of CPU prefill before the model starts generating.
Decode (token generation) is memory-bandwidth-bound (~28 GB/s shared LPDDR5) and the GPU can't help it. Prefill is compute-bound — that's the only place the GPU's FLOPS matter. This repo is about prefill.
Dependency stack
| Layer | Requirement | Notes |
|---|---|---|
| Kernel | panthor DRM driver (mainline ≥ 6.x), GPU node bound to panthor, not the vendor mali kbase blob |
On boltzmann: boot linux-rk3588-marfrit-A1 (mainline 7.0.0-rc3). The vendor BSP kernel (6.1.75-npu-port) binds mali → panvk cannot enumerate. |
| CSF firmware | /lib/firmware/arm/mali/arch10.10/mali_csffw.bin |
Present on boltzmann. |
| Userspace GL/Vulkan | Mesa panvk (libvulkan_panfrost.so), Mesa 26.1.3 |
panfrost_icd.json ICD. Enumerates Mali-G610 MC4 (panvk) with uma:1 fp16:1 int dot:1 matrix cores:none. |
| Runtime env | XDG_RUNTIME_DIR must be set |
panvk init fails without it. |
| Inference | llama.cpp Vulkan backend (-DGGML_VULKAN=ON), commit 389ff61 |
Build tools: glslc, libvulkan, cmake. |
Status (2026-07-11)
- ✅ Full open stack works: panthor binds the GPU, panvk enumerates it, llama.cpp runs on it.
- ⚠️ Blocker: panvk throws
vk::CommandBuffer::end: ErrorOutOfDeviceMemoryon any prefill dispatch above ubatch 64. Only-ub 64runs; at ub64 prefill is 21 t/s, which is overhead-bound (ub16/32 far slower) — i.e. 21 t/s is the "forced-tiny-chunk" ceiling, not the Mali GEMM ceiling. It ties an (i8mm-disabled) CPU. - 🔬 Root cause narrowed to a panvk driver allocation that scales past ub64; see docs/root-cause.md. The fix is a Mesa panvk patch (this repo).
Contents
patches/— collected patches (llama.cpp + Mesa panvk).docs/— root-cause analysis, benchmark data, build/boot procedure.mesa/— Mesa 26.1.3 source (gitignored tarball) for the panvk build.
Benchmarks (Qwen2.5-3B-Q4_K_M, boltzmann, pp512)
| Backend | t/s | Notes |
|---|---|---|
| CPU (plain build, 8 threads) | 22.4 | +noi8mm handicap; i8mm would be higher |
GPU (panvk Vulkan, -ub 64) |
21.0 | ub≥72 → OOM crash |
| GPU (panvk, ub128+) | crash | vk::CommandBuffer::end: ErrorOutOfDeviceMemory |
Prior art (why the Mali is "framebuffer-only")
Arm chose no matrix cores (int8 dotprod on vector ALUs); panvk is graphics-first (Mesa 26.1 sprint = Proton/gaming; Vulkan 1.2 conformance only 2025); llama.cpp Vulkan is ~15× slower than CPU on Mali+Adreno and unsolved upstream (ggml #9464). But MLC-LLM/TVM (auto-tuned OpenCL kernels) get decent Mali perf → the hardware is capable; the gap is tuned kernels + driver maturity.