Vulkan backend: garbage output for Qwen3 & Gemma architectures (Qwen2.5 works) — QK-norm suspect #2

New Issue

2026-07-11T19:17:30Z

marfrit commented

2026-07-11 19:17:30 +00:00

The llama.cpp Vulkan backend (Mesa panvk, Mali-G610, panthor) produces garbage output for Qwen3- and Gemma-generation architectures, while Qwen2.5-era dense models work perfectly. Isolated cleanly; not this repo's driver/shader patches.

Evidence (same build, same prompts, `-ngl 99` vs `-ngl 0`)

model	arch	GPU (Vulkan)	CPU
Qwen2.5-3B-Q4 (dense)	qwen2	coherent (28 t/s champion)	coherent
Qwen3-8B-Q8 (dense)	qwen3	garbage (`sos\nsos\n...`)	coherent (`<think>\nOkay...`)
Qwen3-30B-A3B-Q4 (MoE)	qwen3moe	garbage (`????`), q4 and q8 KV	—
gemma-4-E2B-Q4	gemma	garbage (`<unused49>`)	coherent (`Tokyo`)

Ruled out

Model files — all coherent on CPU with identical prompt/template.
These patches — Qwen2.5 is correct on the full champion (driver + shader); Qwen3 fails even on the stock M-variant shader (prompt_n<64, untouched path).
KV-cache quant — Qwen3-30B fails at both q8_0 and q4_0 KV.
MoE-specific — dense Qwen3-8B fails too, so it is architecture-wide (Qwen3 + Gemma), not the expert-routing path alone.

Suspect

A newer-architecture op computed wrong on the Vulkan backend. Both Qwen3 and Gemma add query-key RMSNorm (per-head QK normalization); Gemma also has logit soft-capping and non-standard attention scaling. Prime suspect: QK-norm on Vulkan (wrong shape/reduction), since it is the op common to both failing families and absent from Qwen2.5.

Repro / bisect

Run any Qwen3 or Gemma GGUF with -ngl 99 (garbage) vs -ngl 0 (correct). To localize: dump per-op intermediate tensors GPU vs CPU (or GGML_VULKAN_CHECK_RESULTS) and find the first diverging node — expect it at the QK-norm / attention block.

Impact

Blocks the GPU champion (this repo) for all modern models — Qwen3, Gemma, and every MoE — and therefore the Cataract heterogeneous-prefill plan (its MoE centerpiece is garbage on the GPU but correct on the NPU). A fix is upstream-worthy and high-leverage: it reopens the entire llama.cpp Vulkan backend to Qwen3/Gemma on any GPU, not just Mali/panvk. Arguably higher value than further shader tuning.

The llama.cpp **Vulkan backend** (Mesa panvk, Mali-G610, panthor) produces **garbage output** for Qwen3- and Gemma-generation architectures, while Qwen2.5-era dense models work perfectly. Isolated cleanly; **not** this repo's driver/shader patches. ## Evidence (same build, same prompts, `-ngl 99` vs `-ngl 0`) | model | arch | GPU (Vulkan) | CPU | |-------|------|--------------|-----| | Qwen2.5-3B-Q4 (dense) | qwen2 | **coherent** (28 t/s champion) | coherent | | Qwen3-8B-Q8 (dense) | qwen3 | **garbage** (`sos\nsos\n...`) | **coherent** (`<think>\nOkay...`) | | Qwen3-30B-A3B-Q4 (MoE)| qwen3moe | **garbage** (`????`), q4 *and* q8 KV | — | | gemma-4-E2B-Q4 | gemma | **garbage** (`<unused49>`) | **coherent** (`Tokyo`) | ## Ruled out - **Model files** — all coherent on CPU with identical prompt/template. - **These patches** — Qwen2.5 is correct on the full champion (driver + shader); Qwen3 fails even on the *stock* M-variant shader (prompt_n<64, untouched path). - **KV-cache quant** — Qwen3-30B fails at both `q8_0` and `q4_0` KV. - **MoE-specific** — dense Qwen3-8B fails too, so it is architecture-wide (Qwen3 + Gemma), not the expert-routing path alone. ## Suspect A newer-architecture op computed wrong on the Vulkan backend. Both Qwen3 and Gemma add **query-key RMSNorm** (per-head QK normalization); Gemma also has **logit soft-capping** and non-standard attention scaling. Prime suspect: QK-norm on Vulkan (wrong shape/reduction), since it is the op common to *both* failing families and absent from Qwen2.5. ## Repro / bisect Run any Qwen3 or Gemma GGUF with `-ngl 99` (garbage) vs `-ngl 0` (correct). To localize: dump per-op intermediate tensors GPU vs CPU (or `GGML_VULKAN_CHECK_RESULTS`) and find the first diverging node — expect it at the QK-norm / attention block. ## Impact Blocks the GPU champion (this repo) for **all modern models** — Qwen3, Gemma, and every MoE — and therefore the Cataract heterogeneous-prefill plan (its MoE centerpiece is garbage on the GPU but correct on the NPU). A fix is **upstream-worthy and high-leverage**: it reopens the entire llama.cpp Vulkan backend to Qwen3/Gemma on *any* GPU, not just Mali/panvk. Arguably higher value than further shader tuning.

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: marfrit/rocky-vulkan-llama#2

Vulkan backend: garbage output for Qwen3 & Gemma architectures (Qwen2.5 works) — QK-norm suspect #2

Evidence (same build, same prompts, -ngl 99 vs -ngl 0)

Ruled out

Suspect

Repro / bisect

Impact

Evidence (same build, same prompts, `-ngl 99` vs `-ngl 0`)