diff --git a/docs/root-cause-final.md b/docs/root-cause-final.md index 3893b08..8ae6f59 100644 --- a/docs/root-cause-final.md +++ b/docs/root-cause-final.md @@ -41,3 +41,21 @@ shared mem) is the smaller half. Together they cross the heap ceiling at ub>64. Mechanism fully characterized and instrumented. WLS half fixed & verified. The TLS-stack half needs a pool-lifecycle change (option 2) and/or the core-overallocation fix (option 3) — a larger, upstream-worthy change. Not a guess anymore: every number above is measured on the live GPU. + +## Phase 1 executed (2026-07-11) — consumer-side levers FALSIFIED +Instrument-driven test on panthor (patched panvk w/ priv-VA counter + `GGML_VK_CMD_CLEANUP_FREQ`): +- **`cleanup_frequency` 10→2: no effect.** Peak priv-VA ~3800 MB at both; still OOM. `resetCommandPool` + doesn't reduce live VA — the tls pool RECYCLES big BOs into a free-list (`panvk_pool_reset`, + `big_bo_pool` branch) instead of freeing; varying 155 KB–304 MB sizes never match → VA grows + monotonically (**0 frees in 128 allocs**). +- **free-on-reset (tls_pool big_bo_pool→NULL) + freq=3: still OOM.** Only **1 free / 174 allocs**. + The cmdbufs are **IN-FLIGHT** (submitted, fences pending till graph end) so they can't be reset; + post-hoc freeing can't touch the in-flight peak. +- **Confirmed peak: 12 live 304 MB TLS-stack BOs ≈ 3.65 GB → OOM.** The failing alloc is a 304 MB + TLS stack at used=3.98 GB. + +**Conclusion:** the accumulation is IN-FLIGHT command buffers (~12 per pp512, from pipelined +ubatch forwards), each carrying a 304 MB TLS stack. Neither cleanup_frequency nor free-on-reset +helps. The only viable levers reduce the *in-flight peak*: (a) **cut the ×19 sparse-core +overallocation** (304→64 MB; 12×64=768 MB → fits) or (b) **reduce in-flight cmdbuf depth** in +llama.cpp (fewer/bigger submits per graph, or submit+sync granularity). Phase 4.