rocky-vulkan-llama: skeleton — deps, root-cause diagnosis, llama.cpp submit patch
Documents the RK3588 Mali-G610 GPU-prefill stack (panthor + panvk + llama.cpp Vulkan), the panvk ub64 OOM blocker, and collects the ggml-vulkan tunable-submit patch. Mesa panvk fix pending root-cause confirmation (see docs/root-cause.md). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
Claude (noether)
@@ -0,0 +1,5 @@
|
||||
# Mesa source is fetched, not vendored — keep only patches
|
||||
mesa/*.tar.xz
|
||||
mesa/mesa-*/
|
||||
build/
|
||||
*.log
|
||||
@@ -0,0 +1,55 @@
|
||||
# rocky-vulkan-llama
|
||||
|
||||
GPU-accelerated **LLM prefill** on the RK3588 **Mali-G610** via the open Vulkan stack
|
||||
(panthor kernel driver + Mesa **panvk** + llama.cpp Vulkan backend).
|
||||
|
||||
Goal: make the Mali GPU a useful **prompt-processing (prefill) accelerator** for local agents on
|
||||
RK3588 boards (boltzmann = Radxa ROCK 5 ITX+; ampere = CoolPi CM5 GenBook), so a ~5k-token agent
|
||||
context doesn't cost ~150–250 s of CPU prefill before the model starts generating.
|
||||
|
||||
> Decode (token generation) is memory-bandwidth-bound (~28 GB/s shared LPDDR5) and the GPU can't
|
||||
> help it. **Prefill is compute-bound** — that's the only place the GPU's FLOPS matter. This repo
|
||||
> is about prefill.
|
||||
|
||||
## Dependency stack
|
||||
|
||||
| Layer | Requirement | Notes |
|
||||
|-------|-------------|-------|
|
||||
| Kernel | **panthor** DRM driver (mainline ≥ 6.x), GPU node bound to panthor, **not** the vendor `mali` kbase blob | On boltzmann: boot `linux-rk3588-marfrit-A1` (mainline 7.0.0-rc3). The vendor BSP kernel (`6.1.75-npu-port`) binds `mali` → panvk cannot enumerate. |
|
||||
| CSF firmware | `/lib/firmware/arm/mali/arch10.10/mali_csffw.bin` | Present on boltzmann. |
|
||||
| Userspace GL/Vulkan | **Mesa panvk** (`libvulkan_panfrost.so`), Mesa **26.1.3** | `panfrost_icd.json` ICD. Enumerates `Mali-G610 MC4 (panvk)` with `uma:1 fp16:1 int dot:1 matrix cores:none`. |
|
||||
| Runtime env | `XDG_RUNTIME_DIR` must be set | panvk init fails without it. |
|
||||
| Inference | **llama.cpp** Vulkan backend (`-DGGML_VULKAN=ON`), commit `389ff61` | Build tools: `glslc`, `libvulkan`, cmake. |
|
||||
|
||||
## Status (2026-07-11)
|
||||
|
||||
- ✅ Full open stack works: panthor binds the GPU, panvk enumerates it, llama.cpp runs on it.
|
||||
- ⚠️ **Blocker:** panvk throws `vk::CommandBuffer::end: ErrorOutOfDeviceMemory` on any prefill
|
||||
dispatch above **ubatch 64**. Only `-ub 64` runs; at ub64 prefill is **21 t/s**, which is
|
||||
*overhead-bound* (ub16/32 far slower) — i.e. 21 t/s is the "forced-tiny-chunk" ceiling, not the
|
||||
Mali GEMM ceiling. It ties an (i8mm-disabled) CPU.
|
||||
- 🔬 Root cause narrowed to a panvk driver allocation that scales past ub64; see
|
||||
[docs/root-cause.md](docs/root-cause.md). The **fix is a Mesa panvk patch** (this repo).
|
||||
|
||||
## Contents
|
||||
|
||||
- `patches/` — collected patches (llama.cpp + Mesa panvk).
|
||||
- `docs/` — root-cause analysis, benchmark data, build/boot procedure.
|
||||
- `mesa/` — Mesa 26.1.3 source (gitignored tarball) for the panvk build.
|
||||
|
||||
## Benchmarks (Qwen2.5-3B-Q4_K_M, boltzmann, pp512)
|
||||
|
||||
| Backend | t/s | Notes |
|
||||
|---------|-----|-------|
|
||||
| CPU (plain build, 8 threads) | 22.4 | `+noi8mm` handicap; i8mm would be higher |
|
||||
| GPU (panvk Vulkan, `-ub 64`) | 21.0 | ub≥72 → OOM crash |
|
||||
| GPU (panvk, ub128+) | crash | `vk::CommandBuffer::end: ErrorOutOfDeviceMemory` |
|
||||
|
||||
## Prior art (why the Mali is "framebuffer-only")
|
||||
|
||||
Arm chose no matrix cores (int8 dotprod on vector ALUs); panvk is graphics-first (Mesa 26.1 sprint
|
||||
= Proton/gaming; Vulkan 1.2 conformance only 2025); llama.cpp Vulkan is ~15× slower than CPU on
|
||||
Mali+Adreno and unsolved upstream ([ggml #9464]). But **MLC-LLM/TVM** (auto-tuned OpenCL kernels)
|
||||
get decent Mali perf → the hardware is capable; the gap is tuned kernels + driver maturity.
|
||||
|
||||
[ggml #9464]: https://github.com/ggml-org/llama.cpp/discussions/9464
|
||||
@@ -0,0 +1,61 @@
|
||||
# Root cause: panvk OOM on prefill dispatches above ubatch 64
|
||||
|
||||
## Symptom
|
||||
llama.cpp Vulkan on Mali-G610 (panvk, Mesa 26.1.3, panthor kernel) crashes on prompt processing:
|
||||
|
||||
```
|
||||
terminate called after throwing an instance of 'vk::OutOfDeviceMemoryError'
|
||||
what(): vk::CommandBuffer::end: ErrorOutOfDeviceMemory
|
||||
...ggml_vk_ctx_end -> ggml_vk_build_graph -> ggml_backend_vk_graph_compute
|
||||
```
|
||||
|
||||
- **Hard ceiling = ubatch 64.** ub64 works (21 t/s pp512). ub72/80/96/128/256/512 all crash — even
|
||||
a single-dispatch `pp72`. Independent of prompt length; purely per-dispatch ubatch size.
|
||||
- The 31 GB unified memory is nowhere near full — this is NOT real memory exhaustion.
|
||||
|
||||
## What we ruled out (empirically)
|
||||
- **Data-buffer allocation size:** `GGML_VK_ALLOW_SYSMEM_FALLBACK`, `GGML_VK_PREFER_HOST_MEMORY`,
|
||||
`GGML_VK_SUBALLOCATION_BLOCK_SIZE=64M`, `GGML_VK_FORCE_MAX_ALLOCATION_SIZE=256M` — all still crash.
|
||||
- **Command-buffer accumulation (llama.cpp side):** patched `ggml-vulkan` to make the submit
|
||||
thresholds tunable (`GGML_VK_NODES_PER_SUBMIT`, `GGML_VK_MAX_MUL_MAT_BYTES_PER_SUBMIT`; caps the
|
||||
`*=2` doubling). Even `NODES_PER_SUBMIT=4` (submit every 4 nodes) still crashes at ub≥72. So the
|
||||
crash is a **single dispatch**, not accumulated commands. (Patch kept — it's a useful upstream
|
||||
knob — see `patches/llama.cpp-0001-*`.)
|
||||
|
||||
## panvk source review (mesa-26.1.3/src/panfrost/vulkan/csf/)
|
||||
The `end()` OOM is almost certainly a **deferred report** of a recording-time allocation failure
|
||||
(`panvk_catch_indirect_alloc_failure` → `VK_ERROR_OUT_OF_DEVICE_MEMORY`). Candidate allocations,
|
||||
with static-analysis verdicts:
|
||||
|
||||
| Allocation | Where | Scales with ubatch? | Verdict |
|
||||
|-----------|-------|---------------------|---------|
|
||||
| `RENDER_DESC_RINGBUF_SIZE = 512*1024` | `panvk_cmd_buffer.h:86`, used in `panvk_vX_cmd_draw.c` | fixed | **draw-only**, compute path doesn't use it → probably not it |
|
||||
| WLS / TLS: `panvk_cmd_alloc_dev_mem(cmdbuf, tls, wls_total_size, 4096)` | `panvk_vX_cmd_dispatch.c:104`, finalized in `cmd_buffer.c:1009` | `pan_calc_total_wls_size = wls.size * instances * ncores`; `instances` capped per-core | bounded by cores **unless `wls.size` (shader shared-mem) jumps for the larger-M `mul_mm` variant** → prime suspect |
|
||||
| CS chunks: `alloc_cs_buffer` = 64 KB each from `cs` pool | `cmd_buffer.c:727` | grows in 64 KB chunks, unbounded | grows fine → probably not it |
|
||||
| `driver_set` / descriptors | `panvk_vX_cmd_dispatch.c:50` | per-binding, constant | not it |
|
||||
|
||||
## Leading hypothesis
|
||||
At M > 64 tokens, llama.cpp's Vulkan backend selects a **larger-tile `mul_mm` GEMM variant** with
|
||||
bigger per-workgroup shared memory (`wls.size`). `pan_calc_total_wls_size` multiplies that by
|
||||
`instances * ncores`, and the resulting single `tls`-pool allocation exceeds a panvk/panthor BO or
|
||||
pool limit → caught → reported as `OutOfDeviceMemory` at `end()`. The Mali reports only 32 KB
|
||||
shared memory (`llama-bench`: `shared memory: 32768`), so a shader tile that assumes desktop-sized
|
||||
shared memory would blow the WLS budget.
|
||||
|
||||
## Open question for review
|
||||
1. Is the WLS/`wls_total_size` allocation actually the one that fails, or is it a different
|
||||
pool/BO limit (e.g. a panthor VM/heap cap, or `panvk_as_alloc` on the priv_heap)?
|
||||
2. Right fix: (a) grow the offending panvk allocation/pool, (b) cap panvk's advertised
|
||||
`maxComputeSharedMemorySize`/`maxComputeWorkGroupInvocations` so llama.cpp picks a smaller-tile
|
||||
shader that fits, or (c) fix llama.cpp shader selection for `shared memory == 32768` devices?
|
||||
3. Should we **instrument first** (log every `panvk_cmd_alloc_*` size + the failing pool on one
|
||||
panthor boot) before patching, rather than guess?
|
||||
|
||||
## Reproduce
|
||||
```
|
||||
# on panthor kernel, GPU bound to panthor, panvk enumerating:
|
||||
export XDG_RUNTIME_DIR=/run/user/$(id -u)
|
||||
cd ~/src/llama.cpp/build-vulkan/bin
|
||||
./llama-bench -m ~/models/qwen2.5-3b-instruct-q4_k_m.gguf -p 512 -n 0 -ngl 99 -b 512 -ub 64 # 21 t/s
|
||||
./llama-bench -m ~/models/qwen2.5-3b-instruct-q4_k_m.gguf -p 512 -n 0 -ngl 99 -b 512 -ub 128 # OOM crash
|
||||
```
|
||||
@@ -0,0 +1,29 @@
|
||||
diff --git a/ggml/src/ggml-vulkan/ggml-vulkan.cpp b/ggml/src/ggml-vulkan/ggml-vulkan.cpp
|
||||
index 0a79310..8732cf9 100644
|
||||
--- a/ggml/src/ggml-vulkan/ggml-vulkan.cpp
|
||||
+++ b/ggml/src/ggml-vulkan/ggml-vulkan.cpp
|
||||
@@ -14684,11 +14684,14 @@ static ggml_status ggml_backend_vk_graph_compute(ggml_backend_t backend, ggml_cg
|
||||
// (and scaled down based on model size, so smaller models submit earlier).
|
||||
// Also submit at least every 100 nodes, in case there are workloads without as much matmul.
|
||||
int nodes_per_submit = 100;
|
||||
+ if (const char* e = getenv("GGML_VK_NODES_PER_SUBMIT")) { nodes_per_submit = atoi(e); }
|
||||
int submitted_nodes = 0;
|
||||
int submit_count = 0;
|
||||
uint64_t mul_mat_bytes = 0;
|
||||
uint64_t total_mul_mat_bytes = 0;
|
||||
- uint64_t mul_mat_bytes_per_submit = std::min(uint64_t(100*1000*1000), ctx->last_total_mul_mat_bytes / 40u);
|
||||
+ uint64_t mul_mat_bytes_cap = 100*1000*1000;
|
||||
+ if (const char* e = getenv("GGML_VK_MAX_MUL_MAT_BYTES_PER_SUBMIT")) { mul_mat_bytes_cap = strtoull(e, nullptr, 10); }
|
||||
+ uint64_t mul_mat_bytes_per_submit = std::min(mul_mat_bytes_cap, ctx->last_total_mul_mat_bytes / 40u);
|
||||
for (int i = 0; i < cgraph->n_nodes; i++) {
|
||||
if (first_node_in_batch) {
|
||||
submit_node_idx = i;
|
||||
@@ -14925,7 +14928,7 @@ static ggml_status ggml_backend_vk_graph_compute(ggml_backend_t backend, ggml_cg
|
||||
submitted_nodes = 0;
|
||||
mul_mat_bytes = 0;
|
||||
if (submit_count < 3) {
|
||||
- mul_mat_bytes_per_submit *= 2;
|
||||
+ mul_mat_bytes_per_submit = std::min(mul_mat_bytes_per_submit * 2, mul_mat_bytes_cap);
|
||||
}
|
||||
submit_count++;
|
||||
}
|
||||
Reference in New Issue
Block a user