Claude (noether)
5.0 KiB
Shader tuning — Mali-tuned mul_mm warptile (the throughput unlock)
After the driver fix (no OOM) revealed the real ceiling was llama.cpp's desktop-tuned matmul shader, we tune it for Mali. Fable implements the tile; hardware-in-the-loop measures on the G610.
The model (Fable, verified against our instrument)
Per-thread matmul accumulators = WM·WN/WARP (mul_mm.comp: sums[WMITER·TM·WNITER·TN/2],
WNITER=WM·WN/(WARP·TM·TN·WMITER)). That count IS the register-spill driver, measured directly by
our panvk [panvk-tls] tls_size instrument. Stock on G610 (WARP=16):
- L: 32·64/16 = 128 acc → 5328 B spill (matches measured L exactly)
- M: 16·32/16 = 32 acc → 544 B spill (matches measured M exactly)
Fix = shrink WM·WN/WARP so the tile fits Mali's register file (no spill → occupancy recovers).
Iteration 1 — L-variant WM=WN=16 (16 acc). Qwen2.5-3B-Q4_K_M, pp512
| ubatch | shader | before | after it1 | tls_size |
|---|---|---|---|---|
| 64 | M (control, untouched) | 21.0 | 20.6 t/s | 544 (unchanged) |
| 128 | L (new tile) | 5.8 | 23.4 t/s (4.0×) | 5328 → 96 |
| 256 | L (new tile) | 7.4 | 23.8 t/s (3.2×) | 5328 → 96 |
Spill collapsed 55×; large ubatch now beats the ub64/M ceiling AND the CPU. Confirms: the ceiling
was the shader, NOT the hardware. ~24 t/s ≈ 25–30% of roofline (~80–150 t/s ideal) → headroom left.
Patch: patches/llama.cpp-0003-mali-warptile-iter1.patch (adds VK_VENDOR_ID_ARM + is_mali warptile
branch, L tuples only). Iteration 2: grow the tile toward the register ceiling for arithmetic reuse.
Iteration 2 — grow tile to 32 acc (WM=32,WN=16) — REGRESSED
| ubatch | iter1 | iter2 | tls_size |
|---|---|---|---|
| 128 | 23.4 | 9.7 t/s (2.4× slower) | 96 → 624 |
| 256 | 23.8 | 10.1 t/s (2.4× slower) | 96 → 624 |
Bigger tile improved arithmetic intensity (AI 1.6→2.67) but tanked throughput. We're OCCUPANCY-bound, not AI-bound: 8704 B shared + 624 B spill cut resident workgroups per core. "Smaller is better" wins the head-to-head — iter1 (16 acc) stays champion at ~23.5 t/s. Iteration 3: go smaller / maximize concurrent workgroups, not bigger.
Iteration 3 — bigger workgroup, SAME registers (BLOCK 256, still 16 acc) — WINS
| ubatch | iter1 | iter3 | tls_size |
|---|---|---|---|
| 128 | 23.4 | 25.8 t/s (+10%) | 96 (held) |
| 256 | 23.8 | 26.2 t/s (+10%) | 96 (held) |
Confirms shared-occupancy is the lever (not registers, not AI). Fable back-derived max_threads_per_core=2048 and Valhall reg-bands from our TLS data: at iter1's shape shared-mem/wg capped occupancy ~640 threads; bigger tiles lower shared-per-thread (shared∝BM+BN, threads∝BM·BN) → ~768 threads → +10%. Progression so far: 5.8 (broken) → 23.4 (i1) → 26.2 (i3), ~30% of roofline. Iter4: aggressive BM=128 (BLOCK 512, ~1024 threads) to saturate the register ceiling; then the only remaining lever is a mul_mm.comp register-pressure cut (f16 acc / fewer dequant temporaries).
Iteration 4 — BK_STEP 4→2 (2× occupancy via halved shared) — L path WINS, M regressed
Fable course-corrected: the Q4_K prefill shader is mul_mmq.comp (int-dot), shared =
BK_STEP·(20·BM+36·BN). Under BK_STEP=4 iter3 was the warptile occupancy ceiling (2 wg/512 threads);
BK_STEP=2 → 4 wg/1024 threads (2× occupancy).
| ubatch | iter3 | iter4 | |
|---|---|---|---|
| 64 (M) | 20.6 | 15.5 | ↓ (M shares mul_mmq.comp; extra barriers hurt the saturated M path) |
| 128 (L) | 25.8 | 28.1 (+9%) | ↑ tls→{16,48} |
| 256 (L) | 26.2 | 28.5 (+9%) | ↑ |
BK_STEP is a global #define → hit M too. Fix: scope BK_STEP to the L pipeline (spec constant / |
|||
| per-variant define). Then BK_STEP=1 for full 2048-thread occupancy. Progression (L path): | |||
| 5.8 → 23.4 → 26.2 → 28.1 t/s (~4.8× from broken, ~35% of roofline). Then f16-acc (iter5) to | |||
| break the register band toward 2048 threads. |
Iteration 5 — scope BK_STEP to L via spec constant #11 — CLEAN CHAMPION
BK_STEP became layout(constant_id=11) in mul_mmq.comp; only the Mali l_warptile_mmq_int[_k]
tuples carry the 12th element (=2), so L gets BK_STEP=2 while M/S inherit default 4.
| ubatch | iter4 | iter5 | |
|---|---|---|---|
| 64 (M) | 15.5 | 20.63 | ↑ RESTORED (control intact) |
| 128 (L) | 28.1 | 28.08 | held (L win kept) |
| 256 (L) | 28.5 | 28.46 | held |
| L-path 28 t/s WITHOUT the M regression. Shippable champion. Progression (L): 5.8→23.4→26.2→28.1. |
Iteration 6 — BK_STEP=1 (full 2048 occupancy attempt) — REGRESSED (barrier crossover)
| ubatch | iter5 | iter6 | |
|---|---|---|---|
| 128 (L) | 28.1 | 23.1 | ↓ 4× K-loop barriers outran the occupancy gain |
| 256 (L) | 28.5 | 23.3 | ↓ |
| BK_STEP=2 is the occupancy sweet spot; reverted to iter5 champion (28 t/s). Shared-based occupancy | |||
| exhausted. Next lever: f16 accumulators (register-based occupancy, no extra barriers) to break | |||
| the 33–64 reg band toward 2048 resident threads. |