Kernel bring-up: A1 hangs panvk compute, plain 7.0.0-rc3 runs it; rebuild-vs-A1 disproven (uABI byte-identical); both panthor 1.7.0
GPU-on-gemma session findings: boltzmann kernel/boot matrix (edk2 UEFI->GRUB, grub-reboot one-shot), marfrit-A1 hangs GPU dispatch at 0% CPU while plain mainline runs (91% CPU), Mesa panthor uABI identical for all panvk paths (only additive TIMESTAMP delta) so header rebuild is futile, 26B too big for Mali VA (sub-4GB heaps), don't pkill mid-GPU-job (wedges context).
This commit is contained in:
Markus Fritsche
@@ -0,0 +1,99 @@
|
||||
# Gemma-on-Mali-GPU bring-up + panthor kernel matrix (2026-07-12)
|
||||
|
||||
Goal: run `gemma-4-26B-A4B` (Q4 QAT, MoE 4B-active) on boltzmann's Mali-G610 via panvk
|
||||
for **prefill** acceleration. (Decode stays DDR-bandwidth-walled regardless — unified memory,
|
||||
~28 GB/s shared CPU/GPU; the GPU only helps compute-bound prefill.)
|
||||
|
||||
This session did NOT reach a clean coherence+speed number on the GPU. It DID map the kernel/boot
|
||||
landscape and eliminate several hypotheses. Recorded here so the trail survives compaction.
|
||||
|
||||
## boltzmann kernel matrix (critical reference)
|
||||
|
||||
Boot chain: **edk2 UEFI → GRUB** (NOT extlinux/u-boot; `/boot/extlinux/extlinux.conf` is stale
|
||||
and ignored). GRUB cfg `/boot/grub/grub.cfg`, `GRUB_DEFAULT="BredOS Linux 6.1.75-npu-port ..."`,
|
||||
`GRUB_TIMEOUT=5` menu. grubenv honors `next_entry` (grub.cfg lines ~14-17) so **`grub-reboot`
|
||||
one-shot works**: boots the chosen entry once, auto-reverts to the npu-port default. Safe for
|
||||
remote/headless (a bad boot self-heals on the next reboot). `lmcp.service` is enabled → MCP
|
||||
returns after reboot; SSH-via-hertz (`ssh mfritsche@boltzmann`) is the fallback.
|
||||
|
||||
| uname -r | GRUB entry | GPU driver | panvk compute | notes |
|
||||
|----------|-----------|-----------|---------------|-------|
|
||||
| `6.1.75-npu-port` | default | **mali blob** + RKNPU | n/a (no Vulkan) | the NPU/DeepSeek kernel; GPU on vendor blob |
|
||||
| `7.0.0-rc3` | `BredOS Linux (mainline 7.0.0-rc3)` | **panthor 1.7.0** | **RUNS** (91% CPU, state Rl) | plain `linux-rockchip-mainline`, Mar-29 build. The champion panvk kernel (by elimination). |
|
||||
| `7.0.0-rc3-ARCH+` | `BredOS Linux marfrit-A1 (NVMe-fixed ...)` | **panthor 1.7.0** | **HANGS** (0% CPU, state Ss, no output, no dmesg fault) | marfrit-A1, NVMe-hardened. Chose it first for NVMe safety; it hangs every GPU dispatch. |
|
||||
| `7.0.0-rc3-dirty` | — | — | — | orphaned module tree, **no vmlinuz → NOT GRUB-bootable** |
|
||||
|
||||
To select plain mainline for one boot:
|
||||
`sudo grub-reboot 'BredOS Linux (mainline 7.0.0-rc3)'` then verify
|
||||
`sudo grub-editenv /boot/grub/grubenv list | grep next_entry`, then `sudo reboot`.
|
||||
|
||||
## Findings / eliminated hypotheses
|
||||
|
||||
1. **A1 hangs panvk compute; plain 7.0.0-rc3 runs it.** On A1, llama.cpp Vulkan blocks at
|
||||
0% CPU at model load for EVERY build (July mainline + the May champion that hit 28 t/s on a
|
||||
prior boot) and EVERY model (gemma-26B, gemma-E2B, even the champion's own qwen2.5-3b).
|
||||
`vulkaninfo --summary` works on both (enumerate path OK); it's the **compute-submit** path
|
||||
that wedges — a fence that never signals. On plain 7.0.0-rc3 the first CLI run actually
|
||||
executed (91% CPU, state Rl).
|
||||
|
||||
2. **"Rebuild panvk vs A1 headers" is DISPROVEN — do not do it.** Diffed Mesa's bundled
|
||||
`include/drm-uapi/panthor_drm.h` vs the system/kernel `panthor_drm.h`: **byte-identical for
|
||||
every ioctl/struct panvk uses** (BO create, VM bind, group submit, sync). The ONLY delta is
|
||||
an *additive* `DRM_PANTHOR_TIMESTAMP_*` query family that old panvk never calls. A uABI
|
||||
mismatch would also produce -EINVAL, not a hang. So the champion Mesa's ioctls are already
|
||||
correct for this kernel; rebuilding changes nothing.
|
||||
|
||||
3. **Firmware is fine.** Both kernels: `CSF FW using interface v1.5.0`, `Initialized panthor
|
||||
1.7.0`, same firmware git sha `95a25d71...`. Not a firmware-load issue.
|
||||
|
||||
4. **Same panthor version on both → the A1-vs-plain difference is NOT the driver version.**
|
||||
Both report panthor 1.7.0. OPEN QUESTION why A1 hangs and plain runs (kernel build config?
|
||||
the "+A1" NVMe patch touching something? job-init path? or the single plain-mainline success
|
||||
was order/GPU-state dependent). Unresolved.
|
||||
|
||||
5. **Model-size wall on the Mali.** gemma-26B Q4 (~13 GB) cannot be fully offloaded — panvk's
|
||||
private VA heaps are sub-4GB (see root-cause docs), so `-ngl 99` on the big model hangs on
|
||||
allocation. Even if correctness is proven, full offload of the 26B is impossible; only a
|
||||
fraction of layers fit → prefill benefit is partial-layer at best. The champion 28 t/s result
|
||||
was a fully-fitting 3B model. Small models (qwen2.5-3b 1.9GB, gemma-4-E2B 2.8GB) DO fit.
|
||||
|
||||
## Discipline notes (learned the hard way)
|
||||
|
||||
- **Never `pkill` a panvk job mid-compute.** It wedges the GPU context; subsequent Vulkan init
|
||||
on the next process blocks at 0% CPU with a 0-byte log (looks identical to the A1 hang). Let a
|
||||
running GPU job finish, or reboot to clear GPU state.
|
||||
- `llama-cli -no-cnv` still dropped into **interactive mode** in the July build (shows `/exit`
|
||||
help + `> ` prompts) — muddies output capture. Prefer **llama-server + `/completion`** for a
|
||||
clean coherence/speed read on the GPU.
|
||||
- Detached launches via `ssh ... setsid bash -c "..." &` were unreliable (log never created).
|
||||
Use the boltzmann MCP `shell_bg` tool instead — it detaches reliably.
|
||||
|
||||
## Correctness/QK-norm context (from earlier characterization)
|
||||
|
||||
The "gemma/Qwen3 garbage on Vulkan" is a **known upstream regression class** (llama.cpp issue
|
||||
#20610: garbage since build b8184; hits RTX/AMD/Mali; quant/coopmat/dequant-shaped), NOT a
|
||||
Mali-only QK-norm mystery. So the correctness gate is "find the correct build," and the STOCK
|
||||
July mainline Vulkan build was pulled specifically to test whether upstream already renders gemma
|
||||
correctly — never got a clean read due to the kernel/GPU-state issues above. Note the champion
|
||||
SPEED shader (`mul_mmq` warptile) is NOT in the July build yet; correctness was always the first
|
||||
gate, speed second.
|
||||
|
||||
## Next steps (clean, in order)
|
||||
|
||||
1. One-shot reboot into plain `7.0.0-rc3` (fresh GPU state).
|
||||
2. `shell_bg` a SINGLE llama-server on the GPU with a small QK-norm model that fully fits
|
||||
(`gemma-4-E2B` 2.8GB is ideal — same gemma4 arch as the target), `-ngl 99`. **Do not kill it.**
|
||||
3. `/completion` POST → read coherence (correct vs garbage) + prefill/decode t/s.
|
||||
4. If gemma4-E2B is correct on GPU: the 26B correctness question is answered (arch works); the
|
||||
remaining blocker is purely the VA/size limit for the big model.
|
||||
5. Only then consider porting the champion `mul_mmq` shader onto the July build for prefill speed.
|
||||
|
||||
## Build artifacts (on boltzmann, not in this repo)
|
||||
|
||||
- `~/src/llama.cpp-latest/build-vulkan/` — STOCK July mainline, `GGML_VULKAN=ON` (gemma4 arch).
|
||||
- `~/src/llama.cpp/build-vulkan/` — May champion build (has the `mul_mmq` warptile shader) but
|
||||
predates gemma4 arch (can't load gemma4 models).
|
||||
- panvk ICD: `~/rocky-panvk-icd.json` → the patched champion Mesa `libvulkan_panfrost.so`
|
||||
(shared-TLS/WLS OOM fix). Env: `PANVK_SHARED_TLS=1 PANVK_WLS_REUSE=1`.
|
||||
- Models: `~/models/gemma4-a4b/` (Q4 QAT 13GB, UD-Q2 9.8GB, MTP draft), `~/models/gemma-4-E2B-it-Q4_K_M.gguf`,
|
||||
`~/models/qwen2.5-3b-instruct-q4_k_m.gguf`.
|
||||
Reference in New Issue
Block a user