Claude (noether)
272604b10d
Confirms the ceiling was the desktop-tuned shader, not the hardware. Per-thread acc = WM*WN/WARP (matches our tls_size instrument to the byte). Large ubatch now beats the ub64 ceiling + CPU. ~25-30% of roofline; iter2 grows the tile. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
1.6 KiB
Shader tuning — Mali-tuned mul_mm warptile (the throughput unlock)
After the driver fix (no OOM) revealed the real ceiling was llama.cpp's desktop-tuned matmul shader, we tune it for Mali. Fable implements the tile; hardware-in-the-loop measures on the G610.
The model (Fable, verified against our instrument)
Per-thread matmul accumulators = WM·WN/WARP (mul_mm.comp: sums[WMITER·TM·WNITER·TN/2],
WNITER=WM·WN/(WARP·TM·TN·WMITER)). That count IS the register-spill driver, measured directly by
our panvk [panvk-tls] tls_size instrument. Stock on G610 (WARP=16):
- L: 32·64/16 = 128 acc → 5328 B spill (matches measured L exactly)
- M: 16·32/16 = 32 acc → 544 B spill (matches measured M exactly)
Fix = shrink WM·WN/WARP so the tile fits Mali's register file (no spill → occupancy recovers).
Iteration 1 — L-variant WM=WN=16 (16 acc). Qwen2.5-3B-Q4_K_M, pp512
| ubatch | shader | before | after it1 | tls_size |
|---|---|---|---|---|
| 64 | M (control, untouched) | 21.0 | 20.6 t/s | 544 (unchanged) |
| 128 | L (new tile) | 5.8 | 23.4 t/s (4.0×) | 5328 → 96 |
| 256 | L (new tile) | 7.4 | 23.8 t/s (3.2×) | 5328 → 96 |
Spill collapsed 55×; large ubatch now beats the ub64/M ceiling AND the CPU. Confirms: the ceiling
was the shader, NOT the hardware. ~24 t/s ≈ 25–30% of roofline (~80–150 t/s ideal) → headroom left.
Patch: patches/llama.cpp-0003-mali-warptile-iter1.patch (adds VK_VENDOR_ID_ARM + is_mali warptile
branch, L tuples only). Iteration 2: grow the tile toward the register ceiling for arithmetic reuse.