Files

T

Claude (noether) e74bae69fe panvk WLS fix v1 + confirmed mechanism (partial: reuse halves but doesn't eliminate)

Instrument-first panthor boot confirms Fable's diagnosis: ~326 per-dispatch WLS allocs
exhaust the sub-4GB priv VA heap at ub>=128. Grow-only single-max reuse (fix v1) halves the
pressure (326->715 dispatches) but WLS sizes vary 128x (155KB-19MB) so a single buffer is
insufficient. Refined fix pending 2nd review.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

2026-07-11 14:06:47 +02:00

3.4 KiB

Raw Blame History

Fable review — CONFIRMED root cause & fix (2026-07-11)

Second-model (Fable) review of root-cause.md, verified against Mesa 26.1.3 source. It corrected my WLS-size hypothesis: right allocation, wrong mechanism.

Confirmed mechanism: sub-4 GB priv-VA-heap exhaustion from per-dispatch WLS

CSF panvk allocates a fresh WLS (workgroup-local-storage) buffer per vkCmdDispatch and never reuses it — there is a literal TODO admitting this at csf/panvk_vX_cmd_dispatch.c:99. The JM backend (jm/panvk_vX_cmd_dispatch.c:48-57) and gallium (pan_job.c:438) both reuse one WLS per batch; CSF is the outlier.

On CSF (PAN_ARCH >= 10) the VM is created without AUTO_VA; all internal/priv BOs draw VA from priv_heap = [PANVK_VA_RESERVE_BOTTOM (32 MB), 1ull<<32) ≈ 4064 MB (csf/panvk_vX_device.c:466-476). User VkDeviceMemory uses a separate ≥4 GB heap — which is why every GGML_VK_* data-buffer knob changed nothing.
Pool BOs are freed only at cmdbuf reset (csf/panvk_vX_cmd_buffer.c:818-820), not at submit. So "submit every 4 nodes still crashes" did not prove single-dispatch — the accumulation lives in the driver pool, which submits don't drain. That supports the aggregate diagnosis.
Failure chain: util_vma_heap_alloc → 0 → panvk_as_alloc (panvk_device.h:188) → panvk_priv_bo.c:62-70 → VK_ERROR_OUT_OF_DEVICE_MEMORY → caught by panvk_catch_indirect_alloc_failure (panvk_cmd_alloc.h:24-29) → deferred to vkEndCommandBuffer. Exactly the observed signature, 31 GB RAM untouched.

Why exactly 64 → 72

ggml_vk_guess_matmul_pipeline (ggml-vulkan.cpp:7267-7273): n≤64 → M warptile, n>64 → L. Shared mem: M-mmq 128×33×2 = 8448 B → pow2 → 16 KB; L-mmq 256×33×2 = 16896 B → 32 KB. Per-dispatch WLS = pow2(size) × instances × core_id_range, with core_id_range = 19 (live core mask 0x50005 → sparse, highest core ID 18 → ×19, ~5× overallocation by construction). ~10-20 MB (M) vs ~20-40 MB (L) per dispatch. A Qwen-3B ubatch graph records ~300 dispatches: ub64 ≈ 3-3.5 GB (just under 4064 MB); ub72 flips to L → ~2× → over the heap. The shared-mem-fit theory is ruled out — llama.cpp gates L/M on maxComputeSharedMemorySize and would silently downgrade, not crash.

The fix (implemented): reuse a grow-only WLS buffer per cmdbuf

Mirrors JM/gallium. Adds wls_ptr/wls_size to struct panvk_tls_state; a dispatch reuses the buffer when wls_total_size <= state.tls.wls_size, else allocates and remembers it. Collapses ~300 allocations to ~1 (a few if the size grows M→L). Safe on reset (panvk_reset_cmdbuf memsets state). Gated behind PANVK_WLS_REUSE for A/B testing; PANVK_WLS_LOG traces per-dispatch sizes. See patches/mesa-panvk-0001-csf-reuse-wls-across-dispatches.patch.

Rejected alternatives: (b) lowering advertised maxComputeSharedMemorySize — collateral damage to flash-attn/other ops, and the L shader fits 32 KB fine; growing the priv heap — riskier (WLS may not cross a 4 GB boundary, pan_desc.c:933-935).

Predictions to verify on the panthor boot (instrument-first)

PANVK_WLS_LOG off-reuse: ~300 allocations of ~20-40 MB; failure when running total ≈ 3.9 GB.
Reuse on: one WLS allocation; ub128/256/512 no longer crash.
Free checks: a larger model at ub64 should also crash without reuse (more dispatches); ub64 without reuse sits measurably close to the 4 GB edge.

3.4 KiB Raw Blame History Unescape Escape