Claude (noether)
16b96537fd
Documents the RK3588 Mali-G610 GPU-prefill stack (panthor + panvk + llama.cpp Vulkan), the panvk ub64 OOM blocker, and collects the ggml-vulkan tunable-submit patch. Mesa panvk fix pending root-cause confirmation (see docs/root-cause.md). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
62 lines
4.0 KiB
Markdown
62 lines
4.0 KiB
Markdown
# Root cause: panvk OOM on prefill dispatches above ubatch 64
|
|
|
|
## Symptom
|
|
llama.cpp Vulkan on Mali-G610 (panvk, Mesa 26.1.3, panthor kernel) crashes on prompt processing:
|
|
|
|
```
|
|
terminate called after throwing an instance of 'vk::OutOfDeviceMemoryError'
|
|
what(): vk::CommandBuffer::end: ErrorOutOfDeviceMemory
|
|
...ggml_vk_ctx_end -> ggml_vk_build_graph -> ggml_backend_vk_graph_compute
|
|
```
|
|
|
|
- **Hard ceiling = ubatch 64.** ub64 works (21 t/s pp512). ub72/80/96/128/256/512 all crash — even
|
|
a single-dispatch `pp72`. Independent of prompt length; purely per-dispatch ubatch size.
|
|
- The 31 GB unified memory is nowhere near full — this is NOT real memory exhaustion.
|
|
|
|
## What we ruled out (empirically)
|
|
- **Data-buffer allocation size:** `GGML_VK_ALLOW_SYSMEM_FALLBACK`, `GGML_VK_PREFER_HOST_MEMORY`,
|
|
`GGML_VK_SUBALLOCATION_BLOCK_SIZE=64M`, `GGML_VK_FORCE_MAX_ALLOCATION_SIZE=256M` — all still crash.
|
|
- **Command-buffer accumulation (llama.cpp side):** patched `ggml-vulkan` to make the submit
|
|
thresholds tunable (`GGML_VK_NODES_PER_SUBMIT`, `GGML_VK_MAX_MUL_MAT_BYTES_PER_SUBMIT`; caps the
|
|
`*=2` doubling). Even `NODES_PER_SUBMIT=4` (submit every 4 nodes) still crashes at ub≥72. So the
|
|
crash is a **single dispatch**, not accumulated commands. (Patch kept — it's a useful upstream
|
|
knob — see `patches/llama.cpp-0001-*`.)
|
|
|
|
## panvk source review (mesa-26.1.3/src/panfrost/vulkan/csf/)
|
|
The `end()` OOM is almost certainly a **deferred report** of a recording-time allocation failure
|
|
(`panvk_catch_indirect_alloc_failure` → `VK_ERROR_OUT_OF_DEVICE_MEMORY`). Candidate allocations,
|
|
with static-analysis verdicts:
|
|
|
|
| Allocation | Where | Scales with ubatch? | Verdict |
|
|
|-----------|-------|---------------------|---------|
|
|
| `RENDER_DESC_RINGBUF_SIZE = 512*1024` | `panvk_cmd_buffer.h:86`, used in `panvk_vX_cmd_draw.c` | fixed | **draw-only**, compute path doesn't use it → probably not it |
|
|
| WLS / TLS: `panvk_cmd_alloc_dev_mem(cmdbuf, tls, wls_total_size, 4096)` | `panvk_vX_cmd_dispatch.c:104`, finalized in `cmd_buffer.c:1009` | `pan_calc_total_wls_size = wls.size * instances * ncores`; `instances` capped per-core | bounded by cores **unless `wls.size` (shader shared-mem) jumps for the larger-M `mul_mm` variant** → prime suspect |
|
|
| CS chunks: `alloc_cs_buffer` = 64 KB each from `cs` pool | `cmd_buffer.c:727` | grows in 64 KB chunks, unbounded | grows fine → probably not it |
|
|
| `driver_set` / descriptors | `panvk_vX_cmd_dispatch.c:50` | per-binding, constant | not it |
|
|
|
|
## Leading hypothesis
|
|
At M > 64 tokens, llama.cpp's Vulkan backend selects a **larger-tile `mul_mm` GEMM variant** with
|
|
bigger per-workgroup shared memory (`wls.size`). `pan_calc_total_wls_size` multiplies that by
|
|
`instances * ncores`, and the resulting single `tls`-pool allocation exceeds a panvk/panthor BO or
|
|
pool limit → caught → reported as `OutOfDeviceMemory` at `end()`. The Mali reports only 32 KB
|
|
shared memory (`llama-bench`: `shared memory: 32768`), so a shader tile that assumes desktop-sized
|
|
shared memory would blow the WLS budget.
|
|
|
|
## Open question for review
|
|
1. Is the WLS/`wls_total_size` allocation actually the one that fails, or is it a different
|
|
pool/BO limit (e.g. a panthor VM/heap cap, or `panvk_as_alloc` on the priv_heap)?
|
|
2. Right fix: (a) grow the offending panvk allocation/pool, (b) cap panvk's advertised
|
|
`maxComputeSharedMemorySize`/`maxComputeWorkGroupInvocations` so llama.cpp picks a smaller-tile
|
|
shader that fits, or (c) fix llama.cpp shader selection for `shared memory == 32768` devices?
|
|
3. Should we **instrument first** (log every `panvk_cmd_alloc_*` size + the failing pool on one
|
|
panthor boot) before patching, rather than guess?
|
|
|
|
## Reproduce
|
|
```
|
|
# on panthor kernel, GPU bound to panthor, panvk enumerating:
|
|
export XDG_RUNTIME_DIR=/run/user/$(id -u)
|
|
cd ~/src/llama.cpp/build-vulkan/bin
|
|
./llama-bench -m ~/models/qwen2.5-3b-instruct-q4_k_m.gguf -p 512 -n 0 -ngl 99 -b 512 -ub 64 # 21 t/s
|
|
./llama-bench -m ~/models/qwen2.5-3b-instruct-q4_k_m.gguf -p 512 -n 0 -ngl 99 -b 512 -ub 128 # OOM crash
|
|
```
|