From 16b96537fda741d904b41504c0343868bc904af5 Mon Sep 17 00:00:00 2001 From: "Claude (noether)" Date: Sat, 11 Jul 2026 13:33:00 +0200 Subject: [PATCH] =?UTF-8?q?rocky-vulkan-llama:=20skeleton=20=E2=80=94=20de?= =?UTF-8?q?ps,=20root-cause=20diagnosis,=20llama.cpp=20submit=20patch?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Documents the RK3588 Mali-G610 GPU-prefill stack (panthor + panvk + llama.cpp Vulkan), the panvk ub64 OOM blocker, and collects the ggml-vulkan tunable-submit patch. Mesa panvk fix pending root-cause confirmation (see docs/root-cause.md). Co-Authored-By: Claude Opus 4.8 --- .gitignore | 5 ++ README.md | 55 +++++++++++++++++ docs/root-cause.md | 61 +++++++++++++++++++ ...pp-0001-vk-tunable-submit-thresholds.patch | 29 +++++++++ 4 files changed, 150 insertions(+) create mode 100644 .gitignore create mode 100644 README.md create mode 100644 docs/root-cause.md create mode 100644 patches/llama.cpp-0001-vk-tunable-submit-thresholds.patch diff --git a/.gitignore b/.gitignore new file mode 100644 index 0000000..e9dfdbb --- /dev/null +++ b/.gitignore @@ -0,0 +1,5 @@ +# Mesa source is fetched, not vendored — keep only patches +mesa/*.tar.xz +mesa/mesa-*/ +build/ +*.log diff --git a/README.md b/README.md new file mode 100644 index 0000000..b0a6f6a --- /dev/null +++ b/README.md @@ -0,0 +1,55 @@ +# rocky-vulkan-llama + +GPU-accelerated **LLM prefill** on the RK3588 **Mali-G610** via the open Vulkan stack +(panthor kernel driver + Mesa **panvk** + llama.cpp Vulkan backend). + +Goal: make the Mali GPU a useful **prompt-processing (prefill) accelerator** for local agents on +RK3588 boards (boltzmann = Radxa ROCK 5 ITX+; ampere = CoolPi CM5 GenBook), so a ~5k-token agent +context doesn't cost ~150–250 s of CPU prefill before the model starts generating. + +> Decode (token generation) is memory-bandwidth-bound (~28 GB/s shared LPDDR5) and the GPU can't +> help it. **Prefill is compute-bound** — that's the only place the GPU's FLOPS matter. This repo +> is about prefill. + +## Dependency stack + +| Layer | Requirement | Notes | +|-------|-------------|-------| +| Kernel | **panthor** DRM driver (mainline ≥ 6.x), GPU node bound to panthor, **not** the vendor `mali` kbase blob | On boltzmann: boot `linux-rk3588-marfrit-A1` (mainline 7.0.0-rc3). The vendor BSP kernel (`6.1.75-npu-port`) binds `mali` → panvk cannot enumerate. | +| CSF firmware | `/lib/firmware/arm/mali/arch10.10/mali_csffw.bin` | Present on boltzmann. | +| Userspace GL/Vulkan | **Mesa panvk** (`libvulkan_panfrost.so`), Mesa **26.1.3** | `panfrost_icd.json` ICD. Enumerates `Mali-G610 MC4 (panvk)` with `uma:1 fp16:1 int dot:1 matrix cores:none`. | +| Runtime env | `XDG_RUNTIME_DIR` must be set | panvk init fails without it. | +| Inference | **llama.cpp** Vulkan backend (`-DGGML_VULKAN=ON`), commit `389ff61` | Build tools: `glslc`, `libvulkan`, cmake. | + +## Status (2026-07-11) + +- ✅ Full open stack works: panthor binds the GPU, panvk enumerates it, llama.cpp runs on it. +- ⚠️ **Blocker:** panvk throws `vk::CommandBuffer::end: ErrorOutOfDeviceMemory` on any prefill + dispatch above **ubatch 64**. Only `-ub 64` runs; at ub64 prefill is **21 t/s**, which is + *overhead-bound* (ub16/32 far slower) — i.e. 21 t/s is the "forced-tiny-chunk" ceiling, not the + Mali GEMM ceiling. It ties an (i8mm-disabled) CPU. +- 🔬 Root cause narrowed to a panvk driver allocation that scales past ub64; see + [docs/root-cause.md](docs/root-cause.md). The **fix is a Mesa panvk patch** (this repo). + +## Contents + +- `patches/` — collected patches (llama.cpp + Mesa panvk). +- `docs/` — root-cause analysis, benchmark data, build/boot procedure. +- `mesa/` — Mesa 26.1.3 source (gitignored tarball) for the panvk build. + +## Benchmarks (Qwen2.5-3B-Q4_K_M, boltzmann, pp512) + +| Backend | t/s | Notes | +|---------|-----|-------| +| CPU (plain build, 8 threads) | 22.4 | `+noi8mm` handicap; i8mm would be higher | +| GPU (panvk Vulkan, `-ub 64`) | 21.0 | ub≥72 → OOM crash | +| GPU (panvk, ub128+) | crash | `vk::CommandBuffer::end: ErrorOutOfDeviceMemory` | + +## Prior art (why the Mali is "framebuffer-only") + +Arm chose no matrix cores (int8 dotprod on vector ALUs); panvk is graphics-first (Mesa 26.1 sprint += Proton/gaming; Vulkan 1.2 conformance only 2025); llama.cpp Vulkan is ~15× slower than CPU on +Mali+Adreno and unsolved upstream ([ggml #9464]). But **MLC-LLM/TVM** (auto-tuned OpenCL kernels) +get decent Mali perf → the hardware is capable; the gap is tuned kernels + driver maturity. + +[ggml #9464]: https://github.com/ggml-org/llama.cpp/discussions/9464 diff --git a/docs/root-cause.md b/docs/root-cause.md new file mode 100644 index 0000000..56a273e --- /dev/null +++ b/docs/root-cause.md @@ -0,0 +1,61 @@ +# Root cause: panvk OOM on prefill dispatches above ubatch 64 + +## Symptom +llama.cpp Vulkan on Mali-G610 (panvk, Mesa 26.1.3, panthor kernel) crashes on prompt processing: + +``` +terminate called after throwing an instance of 'vk::OutOfDeviceMemoryError' + what(): vk::CommandBuffer::end: ErrorOutOfDeviceMemory + ...ggml_vk_ctx_end -> ggml_vk_build_graph -> ggml_backend_vk_graph_compute +``` + +- **Hard ceiling = ubatch 64.** ub64 works (21 t/s pp512). ub72/80/96/128/256/512 all crash — even + a single-dispatch `pp72`. Independent of prompt length; purely per-dispatch ubatch size. +- The 31 GB unified memory is nowhere near full — this is NOT real memory exhaustion. + +## What we ruled out (empirically) +- **Data-buffer allocation size:** `GGML_VK_ALLOW_SYSMEM_FALLBACK`, `GGML_VK_PREFER_HOST_MEMORY`, + `GGML_VK_SUBALLOCATION_BLOCK_SIZE=64M`, `GGML_VK_FORCE_MAX_ALLOCATION_SIZE=256M` — all still crash. +- **Command-buffer accumulation (llama.cpp side):** patched `ggml-vulkan` to make the submit + thresholds tunable (`GGML_VK_NODES_PER_SUBMIT`, `GGML_VK_MAX_MUL_MAT_BYTES_PER_SUBMIT`; caps the + `*=2` doubling). Even `NODES_PER_SUBMIT=4` (submit every 4 nodes) still crashes at ub≥72. So the + crash is a **single dispatch**, not accumulated commands. (Patch kept — it's a useful upstream + knob — see `patches/llama.cpp-0001-*`.) + +## panvk source review (mesa-26.1.3/src/panfrost/vulkan/csf/) +The `end()` OOM is almost certainly a **deferred report** of a recording-time allocation failure +(`panvk_catch_indirect_alloc_failure` → `VK_ERROR_OUT_OF_DEVICE_MEMORY`). Candidate allocations, +with static-analysis verdicts: + +| Allocation | Where | Scales with ubatch? | Verdict | +|-----------|-------|---------------------|---------| +| `RENDER_DESC_RINGBUF_SIZE = 512*1024` | `panvk_cmd_buffer.h:86`, used in `panvk_vX_cmd_draw.c` | fixed | **draw-only**, compute path doesn't use it → probably not it | +| WLS / TLS: `panvk_cmd_alloc_dev_mem(cmdbuf, tls, wls_total_size, 4096)` | `panvk_vX_cmd_dispatch.c:104`, finalized in `cmd_buffer.c:1009` | `pan_calc_total_wls_size = wls.size * instances * ncores`; `instances` capped per-core | bounded by cores **unless `wls.size` (shader shared-mem) jumps for the larger-M `mul_mm` variant** → prime suspect | +| CS chunks: `alloc_cs_buffer` = 64 KB each from `cs` pool | `cmd_buffer.c:727` | grows in 64 KB chunks, unbounded | grows fine → probably not it | +| `driver_set` / descriptors | `panvk_vX_cmd_dispatch.c:50` | per-binding, constant | not it | + +## Leading hypothesis +At M > 64 tokens, llama.cpp's Vulkan backend selects a **larger-tile `mul_mm` GEMM variant** with +bigger per-workgroup shared memory (`wls.size`). `pan_calc_total_wls_size` multiplies that by +`instances * ncores`, and the resulting single `tls`-pool allocation exceeds a panvk/panthor BO or +pool limit → caught → reported as `OutOfDeviceMemory` at `end()`. The Mali reports only 32 KB +shared memory (`llama-bench`: `shared memory: 32768`), so a shader tile that assumes desktop-sized +shared memory would blow the WLS budget. + +## Open question for review +1. Is the WLS/`wls_total_size` allocation actually the one that fails, or is it a different + pool/BO limit (e.g. a panthor VM/heap cap, or `panvk_as_alloc` on the priv_heap)? +2. Right fix: (a) grow the offending panvk allocation/pool, (b) cap panvk's advertised + `maxComputeSharedMemorySize`/`maxComputeWorkGroupInvocations` so llama.cpp picks a smaller-tile + shader that fits, or (c) fix llama.cpp shader selection for `shared memory == 32768` devices? +3. Should we **instrument first** (log every `panvk_cmd_alloc_*` size + the failing pool on one + panthor boot) before patching, rather than guess? + +## Reproduce +``` +# on panthor kernel, GPU bound to panthor, panvk enumerating: +export XDG_RUNTIME_DIR=/run/user/$(id -u) +cd ~/src/llama.cpp/build-vulkan/bin +./llama-bench -m ~/models/qwen2.5-3b-instruct-q4_k_m.gguf -p 512 -n 0 -ngl 99 -b 512 -ub 64 # 21 t/s +./llama-bench -m ~/models/qwen2.5-3b-instruct-q4_k_m.gguf -p 512 -n 0 -ngl 99 -b 512 -ub 128 # OOM crash +``` diff --git a/patches/llama.cpp-0001-vk-tunable-submit-thresholds.patch b/patches/llama.cpp-0001-vk-tunable-submit-thresholds.patch new file mode 100644 index 0000000..44283b7 --- /dev/null +++ b/patches/llama.cpp-0001-vk-tunable-submit-thresholds.patch @@ -0,0 +1,29 @@ +diff --git a/ggml/src/ggml-vulkan/ggml-vulkan.cpp b/ggml/src/ggml-vulkan/ggml-vulkan.cpp +index 0a79310..8732cf9 100644 +--- a/ggml/src/ggml-vulkan/ggml-vulkan.cpp ++++ b/ggml/src/ggml-vulkan/ggml-vulkan.cpp +@@ -14684,11 +14684,14 @@ static ggml_status ggml_backend_vk_graph_compute(ggml_backend_t backend, ggml_cg + // (and scaled down based on model size, so smaller models submit earlier). + // Also submit at least every 100 nodes, in case there are workloads without as much matmul. + int nodes_per_submit = 100; ++ if (const char* e = getenv("GGML_VK_NODES_PER_SUBMIT")) { nodes_per_submit = atoi(e); } + int submitted_nodes = 0; + int submit_count = 0; + uint64_t mul_mat_bytes = 0; + uint64_t total_mul_mat_bytes = 0; +- uint64_t mul_mat_bytes_per_submit = std::min(uint64_t(100*1000*1000), ctx->last_total_mul_mat_bytes / 40u); ++ uint64_t mul_mat_bytes_cap = 100*1000*1000; ++ if (const char* e = getenv("GGML_VK_MAX_MUL_MAT_BYTES_PER_SUBMIT")) { mul_mat_bytes_cap = strtoull(e, nullptr, 10); } ++ uint64_t mul_mat_bytes_per_submit = std::min(mul_mat_bytes_cap, ctx->last_total_mul_mat_bytes / 40u); + for (int i = 0; i < cgraph->n_nodes; i++) { + if (first_node_in_batch) { + submit_node_idx = i; +@@ -14925,7 +14928,7 @@ static ggml_status ggml_backend_vk_graph_compute(ggml_backend_t backend, ggml_cg + submitted_nodes = 0; + mul_mat_bytes = 0; + if (submit_count < 3) { +- mul_mat_bytes_per_submit *= 2; ++ mul_mat_bytes_per_submit = std::min(mul_mat_bytes_per_submit * 2, mul_mat_bytes_cap); + } + submit_count++; + }