marfrit/rk-llama.cpp
1
0
Fork 0
Code Issues Pull Requests Actions 31 Packages Projects Releases Wiki Activity

docs: re-measure CPU/NPU baselines on an idle host

Browse Source 
The first CPU-only numbers were taken while an unrelated ollama llama-server
was inferencing at ~450% CPU on the same A76 cores. Re-ran both configs after
confirming four consecutive idle samples, at -r 3.

  CPU only (LLAMA_RKNPU2=OFF):  pp512  8.70 +/- 0.01   tg128 1.91 +/- 0.00
  RKNPU (model only):           pp512 38.31 +/- 0.03   tg128 2.35 +/- 0.00

Contention moved the CPU figures by ~0.5% (within noise); NPU was unaffected.

Note: --device none does NOT disable the rknpu2 backend (it intercepts MUL_MAT
via supports_op rather than registering as an offload device). A real CPU-only
baseline requires configuring with -DLLAMA_RKNPU2=OFF.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EWpfhDgYNA21tETDP9ueBE
This commit is contained in:
mfritsche
2026-07-10 07:13:39 +02:00
parent 5c66767159
commit 3258224a57
1 changed files with 4 additions and 4 deletions
Download Patch File Download Diff File Expand all files Collapse all files
gemma4-optimized.md
+4 -4
View File
@@ -49,14 +49,14 @@ Non-obvious, all load-bearing:
## Measurements (this version, `02a883fea`)
`llama-bench`, `taskset -c 4-7`, `-t 4 -r 2`, no speculation:
`llama-bench`, `taskset -c 4-7`, `-t 4 -r 3`, no speculation, host idle:
| backend | pp512 | tg128 |
|---------|------:|------:|
| CPU only (`LLAMA_RKNPU2=OFF`) | 8.74 ± 0.00 | 1.92 ± 0.00 |
| RKNPU (model only) | 38.31 ± 0.04 | 2.35 ± 0.00 |
| CPU only (`LLAMA_RKNPU2=OFF`) | 8.70 ± 0.01 | 1.91 ± 0.00 |
| RKNPU (model only) | 38.31 ± 0.03 | 2.35 ± 0.00 |
NPU: **4.4× prefill**, **+22% decode**. Decode gains little because it is memory-bandwidth
Taken on an idle host (`-r 3`). NPU: **4.4× prefill**, **+23% decode**. Decode gains little because it is memory-bandwidth
bound, not compute bound — which is what makes speculation the real decode lever.
`llama-server`, `-c 8192`, `temp 0`, distinct prompts, decode t/s: