rocky-vulkan-llama/docs/gemma-gpu-kernel-bringup-2026-07-12.md

Gemma-on-Mali-GPU bring-up + panthor kernel matrix (2026-07-12)

Goal: run gemma-4-26B-A4B (Q4 QAT, MoE 4B-active) on boltzmann's Mali-G610 via panvk for prefill acceleration. (Decode stays DDR-bandwidth-walled regardless — unified memory, ~28 GB/s shared CPU/GPU; the GPU only helps compute-bound prefill.)

This session did NOT reach a clean coherence+speed number on the GPU. It DID map the kernel/boot landscape and eliminate several hypotheses. Recorded here so the trail survives compaction.

boltzmann kernel matrix (critical reference)

Boot chain: edk2 UEFI → GRUB (NOT extlinux/u-boot; /boot/extlinux/extlinux.conf is stale and ignored). GRUB cfg /boot/grub/grub.cfg, GRUB_DEFAULT="BredOS Linux 6.1.75-npu-port ...", GRUB_TIMEOUT=5 menu. grubenv honors next_entry (grub.cfg lines ~14-17) so grub-reboot one-shot works: boots the chosen entry once, auto-reverts to the npu-port default. Safe for remote/headless (a bad boot self-heals on the next reboot). lmcp.service is enabled → MCP returns after reboot; SSH-via-hertz (ssh mfritsche@boltzmann) is the fallback.

uname -r	GRUB entry	GPU driver	panvk compute	notes
6.1.75-npu-port	default	mali blob + RKNPU	n/a (no Vulkan)	the NPU/DeepSeek kernel; GPU on vendor blob
7.0.0-rc3	BredOS Linux (mainline 7.0.0-rc3)	panthor 1.7.0	RUNS (91% CPU, state Rl)	plain linux-rockchip-mainline, Mar-29 build. The champion panvk kernel (by elimination).
7.0.0-rc3-ARCH+	BredOS Linux marfrit-A1 (NVMe-fixed ...)	panthor 1.7.0	HANGS (0% CPU, state Ss, no output, no dmesg fault)	marfrit-A1, NVMe-hardened. Chose it first for NVMe safety; it hangs every GPU dispatch.
7.0.0-rc3-dirty	—	—	—	orphaned module tree, no vmlinuz → NOT GRUB-bootable

To select plain mainline for one boot: sudo grub-reboot 'BredOS Linux (mainline 7.0.0-rc3)' then verify sudo grub-editenv /boot/grub/grubenv list | grep next_entry, then sudo reboot.

Findings / eliminated hypotheses

1. A1 hangs panvk compute; plain 7.0.0-rc3 runs it. On A1, llama.cpp Vulkan blocks at 0% CPU at model load for EVERY build (July mainline + the May champion that hit 28 t/s on a prior boot) and EVERY model (gemma-26B, gemma-E2B, even the champion's own qwen2.5-3b). vulkaninfo --summary works on both (enumerate path OK); it's the compute-submit path that wedges — a fence that never signals. On plain 7.0.0-rc3 the first CLI run actually executed (91% CPU, state Rl).

2. "Rebuild panvk vs A1 headers" is DISPROVEN — do not do it. Diffed Mesa's bundled include/drm-uapi/panthor_drm.h vs the system/kernel panthor_drm.h: byte-identical for every ioctl/struct panvk uses (BO create, VM bind, group submit, sync). The ONLY delta is an additive DRM_PANTHOR_TIMESTAMP_* query family that old panvk never calls. A uABI mismatch would also produce -EINVAL, not a hang. So the champion Mesa's ioctls are already correct for this kernel; rebuilding changes nothing.

3. Firmware is fine. Both kernels: CSF FW using interface v1.5.0, Initialized panthor 1.7.0, same firmware git sha 95a25d71.... Not a firmware-load issue.

4. Same panthor version on both → the A1-vs-plain difference is NOT the driver version. Both report panthor 1.7.0. OPEN QUESTION why A1 hangs and plain runs (kernel build config? the "+A1" NVMe patch touching something? job-init path? or the single plain-mainline success was order/GPU-state dependent). Unresolved.

5. Model-size wall on the Mali. gemma-26B Q4 (~13 GB) cannot be fully offloaded — panvk's private VA heaps are sub-4GB (see root-cause docs), so -ngl 99 on the big model hangs on allocation. Even if correctness is proven, full offload of the 26B is impossible; only a fraction of layers fit → prefill benefit is partial-layer at best. The champion 28 t/s result was a fully-fitting 3B model. Small models (qwen2.5-3b 1.9GB, gemma-4-E2B 2.8GB) DO fit.

Discipline notes (learned the hard way)

• Never pkill a panvk job mid-compute. It wedges the GPU context; subsequent Vulkan init on the next process blocks at 0% CPU with a 0-byte log (looks identical to the A1 hang). Let a running GPU job finish, or reboot to clear GPU state.
• llama-cli -no-cnv still dropped into interactive mode in the July build (shows /exit help + > prompts) — muddies output capture. Prefer llama-server + /completion for a clean coherence/speed read on the GPU.
• Detached launches via ssh ... setsid bash -c "..." & were unreliable (log never created). Use the boltzmann MCP shell_bg tool instead — it detaches reliably.

Correctness/QK-norm context (from earlier characterization)

The "gemma/Qwen3 garbage on Vulkan" is a known upstream regression class (llama.cpp issue #20610: garbage since build b8184; hits RTX/AMD/Mali; quant/coopmat/dequant-shaped), NOT a Mali-only QK-norm mystery. So the correctness gate is "find the correct build," and the STOCK July mainline Vulkan build was pulled specifically to test whether upstream already renders gemma correctly — never got a clean read due to the kernel/GPU-state issues above. Note the champion SPEED shader (mul_mmq warptile) is NOT in the July build yet; correctness was always the first gate, speed second.

Next steps (clean, in order)

1. One-shot reboot into plain 7.0.0-rc3 (fresh GPU state).
2. shell_bg a SINGLE llama-server on the GPU with a small QK-norm model that fully fits (gemma-4-E2B 2.8GB is ideal — same gemma4 arch as the target), -ngl 99. Do not kill it.
3. /completion POST → read coherence (correct vs garbage) + prefill/decode t/s.
4. If gemma4-E2B is correct on GPU: the 26B correctness question is answered (arch works); the remaining blocker is purely the VA/size limit for the big model.
5. Only then consider porting the champion mul_mmq shader onto the July build for prefill speed.

Build artifacts (on boltzmann, not in this repo)

• ~/src/llama.cpp-latest/build-vulkan/ — STOCK July mainline, GGML_VULKAN=ON (gemma4 arch).
• ~/src/llama.cpp/build-vulkan/ — May champion build (has the mul_mmq warptile shader) but predates gemma4 arch (can't load gemma4 models).
• panvk ICD: ~/rocky-panvk-icd.json → the patched champion Mesa libvulkan_panfrost.so (shared-TLS/WLS OOM fix). Env: PANVK_SHARED_TLS=1 PANVK_WLS_REUSE=1.
• Models: ~/models/gemma4-a4b/ (Q4 QAT 13GB, UD-Q2 9.8GB, MTP draft), ~/models/gemma-4-E2B-it-Q4_K_M.gguf, ~/models/qwen2.5-3b-instruct-q4_k_m.gguf.