# Root cause: panvk OOM on prefill dispatches above ubatch 64

## Symptom
llama.cpp Vulkan on Mali-G610 (panvk, Mesa 26.1.3, panthor kernel) crashes on prompt processing:

```
terminate called after throwing an instance of 'vk::OutOfDeviceMemoryError'
  what():  vk::CommandBuffer::end: ErrorOutOfDeviceMemory
  ...ggml_vk_ctx_end -> ggml_vk_build_graph -> ggml_backend_vk_graph_compute
```

- **Hard ceiling = ubatch 64.** ub64 works (21 t/s pp512). ub72/80/96/128/256/512 all crash — even
  a single-dispatch `pp72`. Independent of prompt length; purely per-dispatch ubatch size.
- The 31 GB unified memory is nowhere near full — this is NOT real memory exhaustion.

## What we ruled out (empirically)
- **Data-buffer allocation size:** `GGML_VK_ALLOW_SYSMEM_FALLBACK`, `GGML_VK_PREFER_HOST_MEMORY`,
  `GGML_VK_SUBALLOCATION_BLOCK_SIZE=64M`, `GGML_VK_FORCE_MAX_ALLOCATION_SIZE=256M` — all still crash.
- **Command-buffer accumulation (llama.cpp side):** patched `ggml-vulkan` to make the submit
  thresholds tunable (`GGML_VK_NODES_PER_SUBMIT`, `GGML_VK_MAX_MUL_MAT_BYTES_PER_SUBMIT`; caps the
  `*=2` doubling). Even `NODES_PER_SUBMIT=4` (submit every 4 nodes) still crashes at ub≥72. So the
  crash is a **single dispatch**, not accumulated commands. (Patch kept — it's a useful upstream
  knob — see `patches/llama.cpp-0001-*`.)

## panvk source review (mesa-26.1.3/src/panfrost/vulkan/csf/)
The `end()` OOM is almost certainly a **deferred report** of a recording-time allocation failure
(`panvk_catch_indirect_alloc_failure` → `VK_ERROR_OUT_OF_DEVICE_MEMORY`). Candidate allocations,
with static-analysis verdicts:

| Allocation | Where | Scales with ubatch? | Verdict |
|-----------|-------|---------------------|---------|
| `RENDER_DESC_RINGBUF_SIZE = 512*1024` | `panvk_cmd_buffer.h:86`, used in `panvk_vX_cmd_draw.c` | fixed | **draw-only**, compute path doesn't use it → probably not it |
| WLS / TLS: `panvk_cmd_alloc_dev_mem(cmdbuf, tls, wls_total_size, 4096)` | `panvk_vX_cmd_dispatch.c:104`, finalized in `cmd_buffer.c:1009` | `pan_calc_total_wls_size = wls.size * instances * ncores`; `instances` capped per-core | bounded by cores **unless `wls.size` (shader shared-mem) jumps for the larger-M `mul_mm` variant** → prime suspect |
| CS chunks: `alloc_cs_buffer` = 64 KB each from `cs` pool | `cmd_buffer.c:727` | grows in 64 KB chunks, unbounded | grows fine → probably not it |
| `driver_set` / descriptors | `panvk_vX_cmd_dispatch.c:50` | per-binding, constant | not it |

## Leading hypothesis
At M > 64 tokens, llama.cpp's Vulkan backend selects a **larger-tile `mul_mm` GEMM variant** with
bigger per-workgroup shared memory (`wls.size`). `pan_calc_total_wls_size` multiplies that by
`instances * ncores`, and the resulting single `tls`-pool allocation exceeds a panvk/panthor BO or
pool limit → caught → reported as `OutOfDeviceMemory` at `end()`. The Mali reports only 32 KB
shared memory (`llama-bench`: `shared memory: 32768`), so a shader tile that assumes desktop-sized
shared memory would blow the WLS budget.

## Open question for review
1. Is the WLS/`wls_total_size` allocation actually the one that fails, or is it a different
   pool/BO limit (e.g. a panthor VM/heap cap, or `panvk_as_alloc` on the priv_heap)?
2. Right fix: (a) grow the offending panvk allocation/pool, (b) cap panvk's advertised
   `maxComputeSharedMemorySize`/`maxComputeWorkGroupInvocations` so llama.cpp picks a smaller-tile
   shader that fits, or (c) fix llama.cpp shader selection for `shared memory == 32768` devices?
3. Should we **instrument first** (log every `panvk_cmd_alloc_*` size + the failing pool on one
   panthor boot) before patching, rather than guess?

## Reproduce
```
# on panthor kernel, GPU bound to panthor, panvk enumerating:
export XDG_RUNTIME_DIR=/run/user/$(id -u)
cd ~/src/llama.cpp/build-vulkan/bin
./llama-bench -m ~/models/qwen2.5-3b-instruct-q4_k_m.gguf -p 512 -n 0 -ngl 99 -b 512 -ub 64   # 21 t/s
./llama-bench -m ~/models/qwen2.5-3b-instruct-q4_k_m.gguf -p 512 -n 0 -ngl 99 -b 512 -ub 128  # OOM crash
```