Gemma-on-Mali-GPU bring-up + panthor kernel matrix (2026-07-12)

Goal: run gemma-4-26B-A4B (Q4 QAT, MoE 4B-active) on boltzmann's Mali-G610 via panvk for prefill acceleration. (Decode stays DDR-bandwidth-walled regardless — unified memory, ~28 GB/s shared CPU/GPU; the GPU only helps compute-bound prefill.)

This session did NOT reach a clean coherence+speed number on the GPU. It DID map the kernel/boot landscape and eliminate several hypotheses. Recorded here so the trail survives compaction.

boltzmann kernel matrix (critical reference)

Boot chain: edk2 UEFI → GRUB (NOT extlinux/u-boot; /boot/extlinux/extlinux.conf is stale and ignored). GRUB cfg /boot/grub/grub.cfg, GRUB_DEFAULT="BredOS Linux 6.1.75-npu-port ...", GRUB_TIMEOUT=5 menu. grubenv honors next_entry (grub.cfg lines ~14-17) so grub-reboot one-shot works: boots the chosen entry once, auto-reverts to the npu-port default. Safe for remote/headless (a bad boot self-heals on the next reboot). lmcp.service is enabled → MCP returns after reboot; SSH-via-hertz (ssh mfritsche@boltzmann) is the fallback.

uname -r	GRUB entry	GPU driver	panvk compute	notes
`6.1.75-npu-port`	default	mali blob + RKNPU	n/a (no Vulkan)	the NPU/DeepSeek kernel; GPU on vendor blob
`7.0.0-rc3`	`BredOS Linux (mainline 7.0.0-rc3)`	panthor 1.7.0	RUNS (91% CPU, state Rl)	plain `linux-rockchip-mainline`, Mar-29 build. The champion panvk kernel (by elimination).
`7.0.0-rc3-ARCH+`	`BredOS Linux marfrit-A1 (NVMe-fixed ...)`	panthor 1.7.0	HANGS (0% CPU, state Ss, no output, no dmesg fault)	marfrit-A1, NVMe-hardened. Chose it first for NVMe safety; it hangs every GPU dispatch.
`7.0.0-rc3-dirty`	—	—	—	orphaned module tree, no vmlinuz → NOT GRUB-bootable

To select plain mainline for one boot: sudo grub-reboot 'BredOS Linux (mainline 7.0.0-rc3)' then verify sudo grub-editenv /boot/grub/grubenv list | grep next_entry, then sudo reboot.

Findings / eliminated hypotheses

A1 hangs panvk compute; plain 7.0.0-rc3 runs it. On A1, llama.cpp Vulkan blocks at 0% CPU at model load for EVERY build (July mainline + the May champion that hit 28 t/s on a prior boot) and EVERY model (gemma-26B, gemma-E2B, even the champion's own qwen2.5-3b). vulkaninfo --summary works on both (enumerate path OK); it's the compute-submit path that wedges — a fence that never signals. On plain 7.0.0-rc3 the first CLI run actually executed (91% CPU, state Rl).
"Rebuild panvk vs A1 headers" is DISPROVEN — do not do it. Diffed Mesa's bundled include/drm-uapi/panthor_drm.h vs the system/kernel panthor_drm.h: byte-identical for every ioctl/struct panvk uses (BO create, VM bind, group submit, sync). The ONLY delta is an additive DRM_PANTHOR_TIMESTAMP_* query family that old panvk never calls. A uABI mismatch would also produce -EINVAL, not a hang. So the champion Mesa's ioctls are already correct for this kernel; rebuilding changes nothing.
Firmware is fine. Both kernels: CSF FW using interface v1.5.0, Initialized panthor 1.7.0, same firmware git sha 95a25d71.... Not a firmware-load issue.
Same panthor version on both → the A1-vs-plain difference is NOT the driver version. Both report panthor 1.7.0. OPEN QUESTION why A1 hangs and plain runs (kernel build config? the "+A1" NVMe patch touching something? job-init path? or the single plain-mainline success was order/GPU-state dependent). Unresolved.
Model-size wall on the Mali. gemma-26B Q4 (~13 GB) cannot be fully offloaded — panvk's private VA heaps are sub-4GB (see root-cause docs), so -ngl 99 on the big model hangs on allocation. Even if correctness is proven, full offload of the 26B is impossible; only a fraction of layers fit → prefill benefit is partial-layer at best. The champion 28 t/s result was a fully-fitting 3B model. Small models (qwen2.5-3b 1.9GB, gemma-4-E2B 2.8GB) DO fit.

Discipline notes (learned the hard way)

Never pkill a panvk job mid-compute. It wedges the GPU context; subsequent Vulkan init on the next process blocks at 0% CPU with a 0-byte log (looks identical to the A1 hang). Let a running GPU job finish, or reboot to clear GPU state.
llama-cli -no-cnv still dropped into interactive mode in the July build (shows /exit help + > prompts) — muddies output capture. Prefer llama-server + /completion for a clean coherence/speed read on the GPU.
Detached launches via ssh ... setsid bash -c "..." & were unreliable (log never created). Use the boltzmann MCP shell_bg tool instead — it detaches reliably.

Correctness/QK-norm context (from earlier characterization)

The "gemma/Qwen3 garbage on Vulkan" is a known upstream regression class (llama.cpp issue #20610: garbage since build b8184; hits RTX/AMD/Mali; quant/coopmat/dequant-shaped), NOT a Mali-only QK-norm mystery. So the correctness gate is "find the correct build," and the STOCK July mainline Vulkan build was pulled specifically to test whether upstream already renders gemma correctly — never got a clean read due to the kernel/GPU-state issues above. Note the champion SPEED shader (mul_mmq warptile) is NOT in the July build yet; correctness was always the first gate, speed second.

Next steps (clean, in order)

One-shot reboot into plain 7.0.0-rc3 (fresh GPU state).
shell_bg a SINGLE llama-server on the GPU with a small QK-norm model that fully fits (gemma-4-E2B 2.8GB is ideal — same gemma4 arch as the target), -ngl 99. Do not kill it.
/completion POST → read coherence (correct vs garbage) + prefill/decode t/s.
If gemma4-E2B is correct on GPU: the 26B correctness question is answered (arch works); the remaining blocker is purely the VA/size limit for the big model.
Only then consider porting the champion mul_mmq shader onto the July build for prefill speed.

Build artifacts (on boltzmann, not in this repo)

~/src/llama.cpp-latest/build-vulkan/ — STOCK July mainline, GGML_VULKAN=ON (gemma4 arch).
~/src/llama.cpp/build-vulkan/ — May champion build (has the mul_mmq warptile shader) but predates gemma4 arch (can't load gemma4 models).
panvk ICD: ~/rocky-panvk-icd.json → the patched champion Mesa libvulkan_panfrost.so (shared-TLS/WLS OOM fix). Env: PANVK_SHARED_TLS=1 PANVK_WLS_REUSE=1.
Models: ~/models/gemma4-a4b/ (Q4 QAT 13GB, UD-Q2 9.8GB, MTP draft), ~/models/gemma-4-E2B-it-Q4_K_M.gguf, ~/models/qwen2.5-3b-instruct-q4_k_m.gguf.

RESULTS (2026-07-12, plain 7.0.0-rc3, clean boot, gemma-4-E2B fully offloaded)

CORRECTNESS GATE: PASSED. On plain 7.0.0-rc3 + fresh GPU state, a single uninterrupted llama-server (stock July mainline build-vulkan, -ngl 99, panvk) loaded gemma-4-E2B and /completion "The capital of France is" returned " Paris." — coherent, not garbage. So the "gemma/Qwen3 garbage on Vulkan" does NOT reproduce in this stock build on Mali. gemma4 arch renders correctly on the GPU. The 26B's only blocker is size (Mali VA), not correctness.

SPEED: stock panvk on Mali is ~1 t/s prefill — unusably slow WITHOUT the champion shader. First request (6-tok prompt): prefill 0.91 t/s, decode 2.63 t/s. This is exactly the hole the champion mul_mmq warptile shader fills (5.8 -> 28 t/s on qwen2.5-3b). The stock build has no Mali tuning. So: correctness is free on stock; speed requires porting patches/llama.cpp-mali-mmq-warptile.patch onto the July build.

Resolved: A1 (7.0.0-rc3-ARCH+) hangs panvk compute; plain 7.0.0-rc3 runs it. Same panthor 1.7.0 on both, so the differentiator is the kernel BUILD, not the driver version (root cause of A1's hang still unknown but moot — use plain mainline). GPU wedges if you pkill a job mid-compute; reboot clears.

Next: port the champion shader onto build-vulkan and re-measure gemma-4-E2B prefill on GPU; the correctness question is now closed (works).

CHAMPION SHADER PORTED TO JULY BUILD + honest CPU comparison (2026-07-12)

Ported the champion mul_mmq warptile onto stock July mainline via a REDUCED patch: the 4 essential ggml-vulkan hunks (VK_VENDOR_ID_ARM, GGML_VK_CMD_CLEANUP_FREQ env, is_mali warptile override {256,64,64,...}, Mali wg_denoms/align) + the mul_mmq.comp BK_STEP spec-constant. Dropped the 2 submit-cadence hunks (they hit July's restructured ggml_backend_vk_graph_compute; secondary). July already had the BK_STEP scaffolding + l_warptile_mmq_int variants upstream, and warptile is std::vector<uint32_t> so the 12th element (BK_STEP=2) compiles + plumbs as spec-constant 11. Reduced patch applies clean; build OK; correctness preserved (gemma-E2B -> ' Paris.').

Measured (gemma-4-E2B, 242-tok prompt, warm)

path	prefill	decode
GPU stock July (no champion)	0.9 t/s	3.2 t/s
GPU champion shader	26.5 t/s	4.3 t/s
CPU (4x A76)	28.1 t/s	23.7 t/s

VERDICT — the champion shader works, but it does NOT beat CPU on this box

Champion vs stock GPU: 29x prefill (0.9 -> 26.5). The shader port is a real success and hits its documented ~28 t/s. Bug reproduced-and-fixed on the July build.
Champion GPU vs CPU: prefill is a TIE (26.5 vs 28.1), and CPU wins decode decisively (23.7 vs 4.3, the unified-DDR bandwidth wall the GPU can't beat).
Correction to an earlier optimistic claim: the champion's documented "4.8x" is stock-GPU -> champion-GPU, NOT vs CPU. The real result is GPU reaches PARITY with CPU on prefill, not a win.
The only way GPU prefill would beat CPU is a model big enough for the tile efficiency to pull ahead — but those don't fit on the Mali (sub-4GB VA heaps). The models that FIT are too small for the GPU to win. Squeeze: fits => too small to win; big enough to win => doesn't fit.
Net on boltzmann: CPU (llama.cpp, Q4 QAT) remains the right path for gemma. The GPU champion is a correct, impressive shader that lands at prefill-parity / decode-loss — not an upgrade.

Op note: SIGKILL of even an IDLE GPU llama-server wedges the panvk context (next GPU init hangs at 0% CPU) -> reboot to clear. Confirmed twice.

10 KiB Raw Permalink Blame History