Gemma-on-Mali-GPU bring-up + panthor kernel matrix (2026-07-12)

Goal: run gemma-4-26B-A4B (Q4 QAT, MoE 4B-active) on boltzmann's Mali-G610 via panvk for prefill acceleration. (Decode stays DDR-bandwidth-walled regardless — unified memory, ~28 GB/s shared CPU/GPU; the GPU only helps compute-bound prefill.)

This session did NOT reach a clean coherence+speed number on the GPU. It DID map the kernel/boot landscape and eliminate several hypotheses. Recorded here so the trail survives compaction.

boltzmann kernel matrix (critical reference)

Boot chain: edk2 UEFI → GRUB (NOT extlinux/u-boot; /boot/extlinux/extlinux.conf is stale and ignored). GRUB cfg /boot/grub/grub.cfg, GRUB_DEFAULT="BredOS Linux 6.1.75-npu-port ...", GRUB_TIMEOUT=5 menu. grubenv honors next_entry (grub.cfg lines ~14-17) so grub-reboot one-shot works: boots the chosen entry once, auto-reverts to the npu-port default. Safe for remote/headless (a bad boot self-heals on the next reboot). lmcp.service is enabled → MCP returns after reboot; SSH-via-hertz (ssh mfritsche@boltzmann) is the fallback.

uname -r	GRUB entry	GPU driver	panvk compute	notes
`6.1.75-npu-port`	default	mali blob + RKNPU	n/a (no Vulkan)	the NPU/DeepSeek kernel; GPU on vendor blob
`7.0.0-rc3`	`BredOS Linux (mainline 7.0.0-rc3)`	panthor 1.7.0	RUNS (91% CPU, state Rl)	plain `linux-rockchip-mainline`, Mar-29 build. The champion panvk kernel (by elimination).
`7.0.0-rc3-ARCH+`	`BredOS Linux marfrit-A1 (NVMe-fixed ...)`	panthor 1.7.0	HANGS (0% CPU, state Ss, no output, no dmesg fault)	marfrit-A1, NVMe-hardened. Chose it first for NVMe safety; it hangs every GPU dispatch.
`7.0.0-rc3-dirty`	—	—	—	orphaned module tree, no vmlinuz → NOT GRUB-bootable

To select plain mainline for one boot: sudo grub-reboot 'BredOS Linux (mainline 7.0.0-rc3)' then verify sudo grub-editenv /boot/grub/grubenv list | grep next_entry, then sudo reboot.

Findings / eliminated hypotheses

A1 hangs panvk compute; plain 7.0.0-rc3 runs it. On A1, llama.cpp Vulkan blocks at 0% CPU at model load for EVERY build (July mainline + the May champion that hit 28 t/s on a prior boot) and EVERY model (gemma-26B, gemma-E2B, even the champion's own qwen2.5-3b). vulkaninfo --summary works on both (enumerate path OK); it's the compute-submit path that wedges — a fence that never signals. On plain 7.0.0-rc3 the first CLI run actually executed (91% CPU, state Rl).
"Rebuild panvk vs A1 headers" is DISPROVEN — do not do it. Diffed Mesa's bundled include/drm-uapi/panthor_drm.h vs the system/kernel panthor_drm.h: byte-identical for every ioctl/struct panvk uses (BO create, VM bind, group submit, sync). The ONLY delta is an additive DRM_PANTHOR_TIMESTAMP_* query family that old panvk never calls. A uABI mismatch would also produce -EINVAL, not a hang. So the champion Mesa's ioctls are already correct for this kernel; rebuilding changes nothing.
Firmware is fine. Both kernels: CSF FW using interface v1.5.0, Initialized panthor 1.7.0, same firmware git sha 95a25d71.... Not a firmware-load issue.
Same panthor version on both → the A1-vs-plain difference is NOT the driver version. Both report panthor 1.7.0. OPEN QUESTION why A1 hangs and plain runs (kernel build config? the "+A1" NVMe patch touching something? job-init path? or the single plain-mainline success was order/GPU-state dependent). Unresolved.
Model-size wall on the Mali. gemma-26B Q4 (~13 GB) cannot be fully offloaded — panvk's private VA heaps are sub-4GB (see root-cause docs), so -ngl 99 on the big model hangs on allocation. Even if correctness is proven, full offload of the 26B is impossible; only a fraction of layers fit → prefill benefit is partial-layer at best. The champion 28 t/s result was a fully-fitting 3B model. Small models (qwen2.5-3b 1.9GB, gemma-4-E2B 2.8GB) DO fit.

Discipline notes (learned the hard way)

Never pkill a panvk job mid-compute. It wedges the GPU context; subsequent Vulkan init on the next process blocks at 0% CPU with a 0-byte log (looks identical to the A1 hang). Let a running GPU job finish, or reboot to clear GPU state.
llama-cli -no-cnv still dropped into interactive mode in the July build (shows /exit help + > prompts) — muddies output capture. Prefer llama-server + /completion for a clean coherence/speed read on the GPU.
Detached launches via ssh ... setsid bash -c "..." & were unreliable (log never created). Use the boltzmann MCP shell_bg tool instead — it detaches reliably.

Correctness/QK-norm context (from earlier characterization)

The "gemma/Qwen3 garbage on Vulkan" is a known upstream regression class (llama.cpp issue #20610: garbage since build b8184; hits RTX/AMD/Mali; quant/coopmat/dequant-shaped), NOT a Mali-only QK-norm mystery. So the correctness gate is "find the correct build," and the STOCK July mainline Vulkan build was pulled specifically to test whether upstream already renders gemma correctly — never got a clean read due to the kernel/GPU-state issues above. Note the champion SPEED shader (mul_mmq warptile) is NOT in the July build yet; correctness was always the first gate, speed second.

Next steps (clean, in order)

One-shot reboot into plain 7.0.0-rc3 (fresh GPU state).
shell_bg a SINGLE llama-server on the GPU with a small QK-norm model that fully fits (gemma-4-E2B 2.8GB is ideal — same gemma4 arch as the target), -ngl 99. Do not kill it.
/completion POST → read coherence (correct vs garbage) + prefill/decode t/s.
If gemma4-E2B is correct on GPU: the 26B correctness question is answered (arch works); the remaining blocker is purely the VA/size limit for the big model.
Only then consider porting the champion mul_mmq shader onto the July build for prefill speed.

Build artifacts (on boltzmann, not in this repo)

~/src/llama.cpp-latest/build-vulkan/ — STOCK July mainline, GGML_VULKAN=ON (gemma4 arch).
~/src/llama.cpp/build-vulkan/ — May champion build (has the mul_mmq warptile shader) but predates gemma4 arch (can't load gemma4 models).
panvk ICD: ~/rocky-panvk-icd.json → the patched champion Mesa libvulkan_panfrost.so (shared-TLS/WLS OOM fix). Env: PANVK_SHARED_TLS=1 PANVK_WLS_REUSE=1.
Models: ~/models/gemma4-a4b/ (Q4 QAT 13GB, UD-Q2 9.8GB, MTP draft), ~/models/gemma-4-E2B-it-Q4_K_M.gguf, ~/models/qwen2.5-3b-instruct-q4_k_m.gguf.

RESULTS (2026-07-12, plain 7.0.0-rc3, clean boot, gemma-4-E2B fully offloaded)

CORRECTNESS GATE: PASSED. On plain 7.0.0-rc3 + fresh GPU state, a single uninterrupted llama-server (stock July mainline build-vulkan, -ngl 99, panvk) loaded gemma-4-E2B and /completion "The capital of France is" returned " Paris." — coherent, not garbage. So the "gemma/Qwen3 garbage on Vulkan" does NOT reproduce in this stock build on Mali. gemma4 arch renders correctly on the GPU. The 26B's only blocker is size (Mali VA), not correctness.

SPEED: stock panvk on Mali is ~1 t/s prefill — unusably slow WITHOUT the champion shader. First request (6-tok prompt): prefill 0.91 t/s, decode 2.63 t/s. This is exactly the hole the champion mul_mmq warptile shader fills (5.8 -> 28 t/s on qwen2.5-3b). The stock build has no Mali tuning. So: correctness is free on stock; speed requires porting patches/llama.cpp-mali-mmq-warptile.patch onto the July build.

Resolved: A1 (7.0.0-rc3-ARCH+) hangs panvk compute; plain 7.0.0-rc3 runs it. Same panthor 1.7.0 on both, so the differentiator is the kernel BUILD, not the driver version (root cause of A1's hang still unknown but moot — use plain mainline). GPU wedges if you pkill a job mid-compute; reboot clears.

Next: port the champion shader onto build-vulkan and re-measure gemma-4-E2B prefill on GPU; the correctness question is now closed (works).

8.0 KiB Raw Blame History