# Gemma-on-Mali-GPU bring-up + panthor kernel matrix (2026-07-12)

Goal: run `gemma-4-26B-A4B` (Q4 QAT, MoE 4B-active) on boltzmann's Mali-G610 via panvk
for **prefill** acceleration. (Decode stays DDR-bandwidth-walled regardless — unified memory,
~28 GB/s shared CPU/GPU; the GPU only helps compute-bound prefill.)

This session did NOT reach a clean coherence+speed number on the GPU. It DID map the kernel/boot
landscape and eliminate several hypotheses. Recorded here so the trail survives compaction.

## boltzmann kernel matrix (critical reference)

Boot chain: **edk2 UEFI → GRUB** (NOT extlinux/u-boot; `/boot/extlinux/extlinux.conf` is stale
and ignored). GRUB cfg `/boot/grub/grub.cfg`, `GRUB_DEFAULT="BredOS Linux 6.1.75-npu-port ..."`,
`GRUB_TIMEOUT=5` menu. grubenv honors `next_entry` (grub.cfg lines ~14-17) so **`grub-reboot`
one-shot works**: boots the chosen entry once, auto-reverts to the npu-port default. Safe for
remote/headless (a bad boot self-heals on the next reboot). `lmcp.service` is enabled → MCP
returns after reboot; SSH-via-hertz (`ssh mfritsche@boltzmann`) is the fallback.

| uname -r | GRUB entry | GPU driver | panvk compute | notes |
|----------|-----------|-----------|---------------|-------|
| `6.1.75-npu-port` | default | **mali blob** + RKNPU | n/a (no Vulkan) | the NPU/DeepSeek kernel; GPU on vendor blob |
| `7.0.0-rc3` | `BredOS Linux (mainline 7.0.0-rc3)` | **panthor 1.7.0** | **RUNS** (91% CPU, state Rl) | plain `linux-rockchip-mainline`, Mar-29 build. The champion panvk kernel (by elimination). |
| `7.0.0-rc3-ARCH+` | `BredOS Linux marfrit-A1 (NVMe-fixed ...)` | **panthor 1.7.0** | **HANGS** (0% CPU, state Ss, no output, no dmesg fault) | marfrit-A1, NVMe-hardened. Chose it first for NVMe safety; it hangs every GPU dispatch. |
| `7.0.0-rc3-dirty` | — | — | — | orphaned module tree, **no vmlinuz → NOT GRUB-bootable** |

To select plain mainline for one boot:
`sudo grub-reboot 'BredOS Linux (mainline 7.0.0-rc3)'` then verify
`sudo grub-editenv /boot/grub/grubenv list | grep next_entry`, then `sudo reboot`.

## Findings / eliminated hypotheses

1. **A1 hangs panvk compute; plain 7.0.0-rc3 runs it.** On A1, llama.cpp Vulkan blocks at
   0% CPU at model load for EVERY build (July mainline + the May champion that hit 28 t/s on a
   prior boot) and EVERY model (gemma-26B, gemma-E2B, even the champion's own qwen2.5-3b).
   `vulkaninfo --summary` works on both (enumerate path OK); it's the **compute-submit** path
   that wedges — a fence that never signals. On plain 7.0.0-rc3 the first CLI run actually
   executed (91% CPU, state Rl).

2. **"Rebuild panvk vs A1 headers" is DISPROVEN — do not do it.** Diffed Mesa's bundled
   `include/drm-uapi/panthor_drm.h` vs the system/kernel `panthor_drm.h`: **byte-identical for
   every ioctl/struct panvk uses** (BO create, VM bind, group submit, sync). The ONLY delta is
   an *additive* `DRM_PANTHOR_TIMESTAMP_*` query family that old panvk never calls. A uABI
   mismatch would also produce -EINVAL, not a hang. So the champion Mesa's ioctls are already
   correct for this kernel; rebuilding changes nothing.

3. **Firmware is fine.** Both kernels: `CSF FW using interface v1.5.0`, `Initialized panthor
   1.7.0`, same firmware git sha `95a25d71...`. Not a firmware-load issue.

4. **Same panthor version on both → the A1-vs-plain difference is NOT the driver version.**
   Both report panthor 1.7.0. OPEN QUESTION why A1 hangs and plain runs (kernel build config?
   the "+A1" NVMe patch touching something? job-init path? or the single plain-mainline success
   was order/GPU-state dependent). Unresolved.

5. **Model-size wall on the Mali.** gemma-26B Q4 (~13 GB) cannot be fully offloaded — panvk's
   private VA heaps are sub-4GB (see root-cause docs), so `-ngl 99` on the big model hangs on
   allocation. Even if correctness is proven, full offload of the 26B is impossible; only a
   fraction of layers fit → prefill benefit is partial-layer at best. The champion 28 t/s result
   was a fully-fitting 3B model. Small models (qwen2.5-3b 1.9GB, gemma-4-E2B 2.8GB) DO fit.

## Discipline notes (learned the hard way)

- **Never `pkill` a panvk job mid-compute.** It wedges the GPU context; subsequent Vulkan init
  on the next process blocks at 0% CPU with a 0-byte log (looks identical to the A1 hang). Let a
  running GPU job finish, or reboot to clear GPU state.
- `llama-cli -no-cnv` still dropped into **interactive mode** in the July build (shows `/exit`
  help + `> ` prompts) — muddies output capture. Prefer **llama-server + `/completion`** for a
  clean coherence/speed read on the GPU.
- Detached launches via `ssh ... setsid bash -c "..." &` were unreliable (log never created).
  Use the boltzmann MCP `shell_bg` tool instead — it detaches reliably.

## Correctness/QK-norm context (from earlier characterization)

The "gemma/Qwen3 garbage on Vulkan" is a **known upstream regression class** (llama.cpp issue
#20610: garbage since build b8184; hits RTX/AMD/Mali; quant/coopmat/dequant-shaped), NOT a
Mali-only QK-norm mystery. So the correctness gate is "find the correct build," and the STOCK
July mainline Vulkan build was pulled specifically to test whether upstream already renders gemma
correctly — never got a clean read due to the kernel/GPU-state issues above. Note the champion
SPEED shader (`mul_mmq` warptile) is NOT in the July build yet; correctness was always the first
gate, speed second.

## Next steps (clean, in order)

1. One-shot reboot into plain `7.0.0-rc3` (fresh GPU state).
2. `shell_bg` a SINGLE llama-server on the GPU with a small QK-norm model that fully fits
   (`gemma-4-E2B` 2.8GB is ideal — same gemma4 arch as the target), `-ngl 99`. **Do not kill it.**
3. `/completion` POST → read coherence (correct vs garbage) + prefill/decode t/s.
4. If gemma4-E2B is correct on GPU: the 26B correctness question is answered (arch works); the
   remaining blocker is purely the VA/size limit for the big model.
5. Only then consider porting the champion `mul_mmq` shader onto the July build for prefill speed.

## Build artifacts (on boltzmann, not in this repo)

- `~/src/llama.cpp-latest/build-vulkan/` — STOCK July mainline, `GGML_VULKAN=ON` (gemma4 arch).
- `~/src/llama.cpp/build-vulkan/` — May champion build (has the `mul_mmq` warptile shader) but
  predates gemma4 arch (can't load gemma4 models).
- panvk ICD: `~/rocky-panvk-icd.json` → the patched champion Mesa `libvulkan_panfrost.so`
  (shared-TLS/WLS OOM fix). Env: `PANVK_SHARED_TLS=1 PANVK_WLS_REUSE=1`.
- Models: `~/models/gemma4-a4b/` (Q4 QAT 13GB, UD-Q2 9.8GB, MTP draft), `~/models/gemma-4-E2B-it-Q4_K_M.gguf`,
  `~/models/qwen2.5-3b-instruct-q4_k_m.gguf`.