From 3258224a57c5c6ea5cad63c2bfa2f6a5298c1b90 Mon Sep 17 00:00:00 2001 From: mfritsche Date: Fri, 10 Jul 2026 07:13:39 +0200 Subject: [PATCH] docs: re-measure CPU/NPU baselines on an idle host The first CPU-only numbers were taken while an unrelated ollama llama-server was inferencing at ~450% CPU on the same A76 cores. Re-ran both configs after confirming four consecutive idle samples, at -r 3. CPU only (LLAMA_RKNPU2=OFF): pp512 8.70 +/- 0.01 tg128 1.91 +/- 0.00 RKNPU (model only): pp512 38.31 +/- 0.03 tg128 2.35 +/- 0.00 Contention moved the CPU figures by ~0.5% (within noise); NPU was unaffected. Note: --device none does NOT disable the rknpu2 backend (it intercepts MUL_MAT via supports_op rather than registering as an offload device). A real CPU-only baseline requires configuring with -DLLAMA_RKNPU2=OFF. Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_01EWpfhDgYNA21tETDP9ueBE --- gemma4-optimized.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/gemma4-optimized.md b/gemma4-optimized.md index 39c590985..115efb1e4 100644 --- a/gemma4-optimized.md +++ b/gemma4-optimized.md @@ -49,14 +49,14 @@ Non-obvious, all load-bearing: ## Measurements (this version, `02a883fea`) -`llama-bench`, `taskset -c 4-7`, `-t 4 -r 2`, no speculation: +`llama-bench`, `taskset -c 4-7`, `-t 4 -r 3`, no speculation, host idle: | backend | pp512 | tg128 | |---------|------:|------:| -| CPU only (`LLAMA_RKNPU2=OFF`) | 8.74 ± 0.00 | 1.92 ± 0.00 | -| RKNPU (model only) | 38.31 ± 0.04 | 2.35 ± 0.00 | +| CPU only (`LLAMA_RKNPU2=OFF`) | 8.70 ± 0.01 | 1.91 ± 0.00 | +| RKNPU (model only) | 38.31 ± 0.03 | 2.35 ± 0.00 | -NPU: **4.4× prefill**, **+22% decode**. Decode gains little because it is memory-bandwidth +Taken on an idle host (`-r 3`). NPU: **4.4× prefill**, **+23% decode**. Decode gains little because it is memory-bandwidth bound, not compute bound — which is what makes speculation the real decode lever. `llama-server`, `-c 8192`, `temp 0`, distinct prompts, decode t/s: