Claude (noether)
dbf97a0fc1
Device-shared TLS scratch + grid-independent WLS reuse collapse per-cmdbuf 304MB TLS and per-dispatch WLS into ~1 BO each. Peak priv-VA 3985MB->908MB, no OOM at any ubatch (was crash at >=72). Verified on panthor. Real upstreamable Mesa panvk bug fix (mesa-panvk-0010). BUT prefill speedup NOT achieved: larger ubatch is 3x SLOWER (ub64=21, ub128=5.8, ub256=7.4 t/s) because llama.cpp's L-variant matmul shader spills ~10x more registers (the 304MB stack) -> poor Mali occupancy. ub64 @ 21 t/s stays the ceiling, ties CPU. Remaining lever = ggml-vulkan L-shader Mali tuning (separate effort). See docs/RESULTS.md. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2.5 KiB
Results — driver bug FIXED; prefill-speedup goal blocked by shader (2026-07-11)
The panvk OOM is fixed ✅ (real, upstreamable Mesa bug)
Root cause: CSF panvk allocates per-command-buffer TLS-stack + per-dispatch WLS scratch from a
device-wide sub-4 GB priv VA heap, and never shares them. On a pipelined LLM graph ~12 in-flight
cmdbufs each carry a 304 MB TLS stack → ~4 GB → vk::CommandBuffer::end: ErrorOutOfDeviceMemory at
any ubatch > 64. Neither cleanup-frequency nor free-on-reset helps (cmdbufs are in-flight, can't be
reset). Confirmed by instrumenting the priv heap: peak 3985 MB, 12× 304 MB TLS, 0 frees.
Fix (3 Mesa panvk patches + the WLS one):
- Device-shared TLS scratch (
PANVK_SHARED_TLS): one grow-only TLS BO reused across all cmdbufs on the serial compute queue (they never run concurrently —pan_scratch.cinvariant), retired-BO keep-alive list freed at device destroy. Collapses N_inflight×304 MB → 1×304 MB. - Grid-independent WLS instances + grow-only WLS reuse (
PANVK_WLS_REUSE): one WLS BO per distinct pipeline instead of one per dispatch. - tls_pool
big_bo_pool→NULL (free WLS big BOs on reset, not recycle).
Measured: with PANVK_SHARED_TLS=1 PANVK_WLS_REUSE=1, no crash at ub 64→1024 (was: crash at
≥72), peak priv-VA 3985 MB → 908 MB (~4.4× lower), 58 emit_tls calls collapse to 1 live TLS BO.
This bug affects any large-workgroup compute on Mali/panvk, not just LLMs — worth upstreaming.
But the prefill SPEEDUP goal is NOT achieved ❌ (llama.cpp shader, not the driver)
Unlocking ub>64 doesn't help, because larger ubatch is slower on Mali:
| pp256, Qwen2.5-3B-Q4 | t/s | shader |
|---|---|---|
| ub64 | 21.2 | M-variant matmul |
| ub128 | 5.8 | L-variant (3.6× slower) |
| ub256 | 7.4 | L-variant (2.8× slower) |
llama.cpp switches M→L matmul at n=64; the L-variant spills ~10× more registers (the 304 MB stack), wrecking GPU occupancy on Mali. So ub64 @ ~21 t/s remains the ceiling — ties an i8mm-off CPU, and the GPU is still not a prefill win.
Conclusion & remaining lever
- Driver: done. A genuine panvk fix, verified, upstreamable.
- Speedup: needs llama.cpp Vulkan shader tuning — make the L-variant
mul_mmMali-friendly (cut register spill; tile for 16-wide warp / 32 KB shared mem) so large-batch prefill is actually faster than the M-variant. That's a separate, sizeable ggml-vulkan effort (Phase 5), independent of the now-fixed driver. Until then, the Mali GPU ties the CPU on prefill and isn't worth deploying for it.