Markus Fritsche
d32c653c9a
GPU-on-gemma session findings: boltzmann kernel/boot matrix (edk2 UEFI->GRUB, grub-reboot one-shot), marfrit-A1 hangs GPU dispatch at 0% CPU while plain mainline runs (91% CPU), Mesa panthor uABI identical for all panvk paths (only additive TIMESTAMP delta) so header rebuild is futile, 26B too big for Mali VA (sub-4GB heaps), don't pkill mid-GPU-job (wedges context).
100 lines
6.6 KiB
Markdown
100 lines
6.6 KiB
Markdown
# Gemma-on-Mali-GPU bring-up + panthor kernel matrix (2026-07-12)
|
|
|
|
Goal: run `gemma-4-26B-A4B` (Q4 QAT, MoE 4B-active) on boltzmann's Mali-G610 via panvk
|
|
for **prefill** acceleration. (Decode stays DDR-bandwidth-walled regardless — unified memory,
|
|
~28 GB/s shared CPU/GPU; the GPU only helps compute-bound prefill.)
|
|
|
|
This session did NOT reach a clean coherence+speed number on the GPU. It DID map the kernel/boot
|
|
landscape and eliminate several hypotheses. Recorded here so the trail survives compaction.
|
|
|
|
## boltzmann kernel matrix (critical reference)
|
|
|
|
Boot chain: **edk2 UEFI → GRUB** (NOT extlinux/u-boot; `/boot/extlinux/extlinux.conf` is stale
|
|
and ignored). GRUB cfg `/boot/grub/grub.cfg`, `GRUB_DEFAULT="BredOS Linux 6.1.75-npu-port ..."`,
|
|
`GRUB_TIMEOUT=5` menu. grubenv honors `next_entry` (grub.cfg lines ~14-17) so **`grub-reboot`
|
|
one-shot works**: boots the chosen entry once, auto-reverts to the npu-port default. Safe for
|
|
remote/headless (a bad boot self-heals on the next reboot). `lmcp.service` is enabled → MCP
|
|
returns after reboot; SSH-via-hertz (`ssh mfritsche@boltzmann`) is the fallback.
|
|
|
|
| uname -r | GRUB entry | GPU driver | panvk compute | notes |
|
|
|----------|-----------|-----------|---------------|-------|
|
|
| `6.1.75-npu-port` | default | **mali blob** + RKNPU | n/a (no Vulkan) | the NPU/DeepSeek kernel; GPU on vendor blob |
|
|
| `7.0.0-rc3` | `BredOS Linux (mainline 7.0.0-rc3)` | **panthor 1.7.0** | **RUNS** (91% CPU, state Rl) | plain `linux-rockchip-mainline`, Mar-29 build. The champion panvk kernel (by elimination). |
|
|
| `7.0.0-rc3-ARCH+` | `BredOS Linux marfrit-A1 (NVMe-fixed ...)` | **panthor 1.7.0** | **HANGS** (0% CPU, state Ss, no output, no dmesg fault) | marfrit-A1, NVMe-hardened. Chose it first for NVMe safety; it hangs every GPU dispatch. |
|
|
| `7.0.0-rc3-dirty` | — | — | — | orphaned module tree, **no vmlinuz → NOT GRUB-bootable** |
|
|
|
|
To select plain mainline for one boot:
|
|
`sudo grub-reboot 'BredOS Linux (mainline 7.0.0-rc3)'` then verify
|
|
`sudo grub-editenv /boot/grub/grubenv list | grep next_entry`, then `sudo reboot`.
|
|
|
|
## Findings / eliminated hypotheses
|
|
|
|
1. **A1 hangs panvk compute; plain 7.0.0-rc3 runs it.** On A1, llama.cpp Vulkan blocks at
|
|
0% CPU at model load for EVERY build (July mainline + the May champion that hit 28 t/s on a
|
|
prior boot) and EVERY model (gemma-26B, gemma-E2B, even the champion's own qwen2.5-3b).
|
|
`vulkaninfo --summary` works on both (enumerate path OK); it's the **compute-submit** path
|
|
that wedges — a fence that never signals. On plain 7.0.0-rc3 the first CLI run actually
|
|
executed (91% CPU, state Rl).
|
|
|
|
2. **"Rebuild panvk vs A1 headers" is DISPROVEN — do not do it.** Diffed Mesa's bundled
|
|
`include/drm-uapi/panthor_drm.h` vs the system/kernel `panthor_drm.h`: **byte-identical for
|
|
every ioctl/struct panvk uses** (BO create, VM bind, group submit, sync). The ONLY delta is
|
|
an *additive* `DRM_PANTHOR_TIMESTAMP_*` query family that old panvk never calls. A uABI
|
|
mismatch would also produce -EINVAL, not a hang. So the champion Mesa's ioctls are already
|
|
correct for this kernel; rebuilding changes nothing.
|
|
|
|
3. **Firmware is fine.** Both kernels: `CSF FW using interface v1.5.0`, `Initialized panthor
|
|
1.7.0`, same firmware git sha `95a25d71...`. Not a firmware-load issue.
|
|
|
|
4. **Same panthor version on both → the A1-vs-plain difference is NOT the driver version.**
|
|
Both report panthor 1.7.0. OPEN QUESTION why A1 hangs and plain runs (kernel build config?
|
|
the "+A1" NVMe patch touching something? job-init path? or the single plain-mainline success
|
|
was order/GPU-state dependent). Unresolved.
|
|
|
|
5. **Model-size wall on the Mali.** gemma-26B Q4 (~13 GB) cannot be fully offloaded — panvk's
|
|
private VA heaps are sub-4GB (see root-cause docs), so `-ngl 99` on the big model hangs on
|
|
allocation. Even if correctness is proven, full offload of the 26B is impossible; only a
|
|
fraction of layers fit → prefill benefit is partial-layer at best. The champion 28 t/s result
|
|
was a fully-fitting 3B model. Small models (qwen2.5-3b 1.9GB, gemma-4-E2B 2.8GB) DO fit.
|
|
|
|
## Discipline notes (learned the hard way)
|
|
|
|
- **Never `pkill` a panvk job mid-compute.** It wedges the GPU context; subsequent Vulkan init
|
|
on the next process blocks at 0% CPU with a 0-byte log (looks identical to the A1 hang). Let a
|
|
running GPU job finish, or reboot to clear GPU state.
|
|
- `llama-cli -no-cnv` still dropped into **interactive mode** in the July build (shows `/exit`
|
|
help + `> ` prompts) — muddies output capture. Prefer **llama-server + `/completion`** for a
|
|
clean coherence/speed read on the GPU.
|
|
- Detached launches via `ssh ... setsid bash -c "..." &` were unreliable (log never created).
|
|
Use the boltzmann MCP `shell_bg` tool instead — it detaches reliably.
|
|
|
|
## Correctness/QK-norm context (from earlier characterization)
|
|
|
|
The "gemma/Qwen3 garbage on Vulkan" is a **known upstream regression class** (llama.cpp issue
|
|
#20610: garbage since build b8184; hits RTX/AMD/Mali; quant/coopmat/dequant-shaped), NOT a
|
|
Mali-only QK-norm mystery. So the correctness gate is "find the correct build," and the STOCK
|
|
July mainline Vulkan build was pulled specifically to test whether upstream already renders gemma
|
|
correctly — never got a clean read due to the kernel/GPU-state issues above. Note the champion
|
|
SPEED shader (`mul_mmq` warptile) is NOT in the July build yet; correctness was always the first
|
|
gate, speed second.
|
|
|
|
## Next steps (clean, in order)
|
|
|
|
1. One-shot reboot into plain `7.0.0-rc3` (fresh GPU state).
|
|
2. `shell_bg` a SINGLE llama-server on the GPU with a small QK-norm model that fully fits
|
|
(`gemma-4-E2B` 2.8GB is ideal — same gemma4 arch as the target), `-ngl 99`. **Do not kill it.**
|
|
3. `/completion` POST → read coherence (correct vs garbage) + prefill/decode t/s.
|
|
4. If gemma4-E2B is correct on GPU: the 26B correctness question is answered (arch works); the
|
|
remaining blocker is purely the VA/size limit for the big model.
|
|
5. Only then consider porting the champion `mul_mmq` shader onto the July build for prefill speed.
|
|
|
|
## Build artifacts (on boltzmann, not in this repo)
|
|
|
|
- `~/src/llama.cpp-latest/build-vulkan/` — STOCK July mainline, `GGML_VULKAN=ON` (gemma4 arch).
|
|
- `~/src/llama.cpp/build-vulkan/` — May champion build (has the `mul_mmq` warptile shader) but
|
|
predates gemma4 arch (can't load gemma4 models).
|
|
- panvk ICD: `~/rocky-panvk-icd.json` → the patched champion Mesa `libvulkan_panfrost.so`
|
|
(shared-TLS/WLS OOM fix). Env: `PANVK_SHARED_TLS=1 PANVK_WLS_REUSE=1`.
|
|
- Models: `~/models/gemma4-a4b/` (Q4 QAT 13GB, UD-Q2 9.8GB, MTP draft), `~/models/gemma-4-E2B-it-Q4_K_M.gguf`,
|
|
`~/models/qwen2.5-3b-instruct-q4_k_m.gguf`.
|