# Gemma 4 12B on the RK3588 NPU — optimized serving config

Branch `mtp-b9549` (upstream b9570 + RKNPU2 backend + perf work).

Hardware: Radxa ROCK 5 ITX+ (Rockchip RK3588), 32 GB RAM, 4× Cortex-A76 (cpu4-7) +
4× Cortex-A55 (cpu0-3), vendor NPU via `librknnrt`.

## Models

Both required. Place in `~/models/`.

| file | size | role |
|------|-----:|------|
| `gemma-4-12b-it-Q8_0.gguf` | 11.8 GiB | target. `arch=gemma4`, dense, `n_ctx_train=131072` |
| `mtp-gemma-4-12b-it-Q8_0.gguf` | 444 MiB | MTP drafter. `arch=gemma4-assistant`, 4 layers, `nextn_predict_layers=4` |

Both from `unsloth/gemma-4-12b-it-GGUF` (drafter under `MTP/`). The drafter is welded to the
target: its graph reads the target's `tok_embd` and aliases K/V pointers, so it only loads
in-process alongside the matching target.

## systemd unit

`~/.config/systemd/user/llama-server-gemma4-npu.service`, enabled with `loginctl enable-linger`.

```ini
[Service]
Type=simple
LimitNOFILE=65536
CPUAffinity=4-7
WorkingDirectory=%h
ExecStart=%h/src/rk-llama.cpp/build-mtp/bin/llama-server \
     -m %h/models/gemma-4-12b-it-Q8_0.gguf \
     -md %h/models/mtp-gemma-4-12b-it-Q8_0.gguf \
     --spec-type draft-mtp --spec-draft-n-max 4 \
     --alias gemma-4-12b-q8 \
     -c 131072 -t 4 \
     --host 0.0.0.0 --port 8085
Restart=on-failure
TimeoutStartSec=300
```

(`%h` is the systemd specifier for the service user home directory.)

Build: `cmake -B build-mtp -DLLAMA_RKNPU2=ON -DGGML_NATIVE=ON -DGGML_OPENMP=ON`.

Non-obvious, all load-bearing:
- `LimitNOFILE=65536` — RKNPU exhausts the default 1024 fds during init and **segfaults**.
- `CPUAffinity=4-7` — A76 only. `-t 8` across both clusters is *slower*; the A55s straggle at
  the per-token barrier.
- `--spec-draft-n-max 4` — `nextn_predict_layers=4`, so there is no head to predict a 5th token.
- `-c 131072` — `n_ctx_train`. Raising it needs RoPE scaling.
- The NPU is **single-tenant**. Stop this unit before running any other NPU process.

## Measurements (this version, `02a883fea`)

`llama-bench`, `taskset -c 4-7`, `-t 4 -r 3`, no speculation, host idle:

| backend | pp512 | tg128 |
|---------|------:|------:|
| CPU only (`LLAMA_RKNPU2=OFF`) |  8.70 ± 0.01 | 1.91 ± 0.00 |
| RKNPU (model only)            | 38.31 ± 0.03 | 2.35 ± 0.00 |

Taken on an idle host (`-r 3`). NPU: **4.4× prefill**, **+23% decode**. Decode gains little because it is memory-bandwidth
bound, not compute bound — which is what makes speculation the real decode lever.

`llama-server`, `-c 8192`, `temp 0`, distinct prompts, decode t/s:

| config | prose | file echo (`--reasoning-budget 0`) |
|--------|------:|----------------------------------:|
| RKNPU, no speculation | 2.118 | 2.055 |
| RKNPU + MTP (`-n-max 4`) | **3.502** (+65%) | **5.207** (+153%) |

MTP draft acceptance is workload-dependent: 0.50 prose, 0.77–0.83 reasoning traces, 0.90–0.94
verbatim file echo. Live at `-c 131072`: ~4.0–4.7 t/s.

Speculative decoding is lossless — greedy output was verified **byte-identical** to the
non-speculative run.

## Why this configuration

**Dense Q8_0, not MoE.** `ggml-rknpu2` implements `GGML_OP_MUL_MAT` only. It has no
`MUL_MAT_ID`, the op llama.cpp uses for MoE expert routing, so on any MoE every expert FFN
falls back to CPU and the NPU's 4.4× prefill evaporates. Q8_0 is also the NPU's native W8A8
pipeline. A Q8_0 30B MoE would need ~30.3 GiB of weights on a 31 GiB box regardless.

**Gemma 4 12B.** Dense, Q8_0, native 131072 context, and its KV is cheap: 8 global layers at
1 KV head (16 KiB/token) plus a fixed ~320 MiB for the 40 sliding-window layers. ~15 GiB
resident at full context.

**MTP over the alternatives.** Three drafting strategies were measured:

| strategy | draft size | acceptance | result |
|----------|-----------:|-----------:|-------:|
| `gemma-4-E2B` draft model | 2.9 GiB | 0.49 | **−5.3%** |
| `ngram-simple` prompt-lookup | none | 0.33–0.52 | +48% (copy-heavy) |
| **MTP drafter** | 0.44 GiB | 0.50–0.94 | **+65% … +153%** |

Acceptance is not the variable — E2B and MTP accept at the same rate and land 70 points apart.
What decides it is the *cost of a wrong guess*: a 25×-smaller drafter makes rejected tokens
nearly free. `ngram-simple` is the fallback for models without an MTP head.

**Do not stack.** `--spec-type draft-mtp,ngram-cache` is *slower* than `draft-mtp` alone
(5.073 vs 5.207); the two drafters compete for the same budget. Also: `ngram-cache` is not
prompt-lookup — it reads a static cache file (`-lcs`) and drafts from an empty corpus without one.

## Operational notes

- gemma-4 is a **reasoning model**: it fills `reasoning_content` before `content`. A tight
  `max_tokens` returns an empty `content` — that is not a failure. Benchmarks of copy behaviour
  need `--reasoning-budget 0`.
- Tool calls work without `--jinja` (the gguf carries a template with tool support).
- Rollback: branch `perf-rknpu2` and a non-MTP `build/` tree are the fallback; the backend
  there is identical minus speculation.