Replace the hostname with the hardware it actually is (Radxa ROCK 5 ITX+,
RK3588), and swap hardcoded home paths for the systemd %h specifier so the
unit stays correct while being user-agnostic.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EWpfhDgYNA21tETDP9ueBE
The first CPU-only numbers were taken while an unrelated ollama llama-server
was inferencing at ~450% CPU on the same A76 cores. Re-ran both configs after
confirming four consecutive idle samples, at -r 3.
CPU only (LLAMA_RKNPU2=OFF): pp512 8.70 +/- 0.01 tg128 1.91 +/- 0.00
RKNPU (model only): pp512 38.31 +/- 0.03 tg128 2.35 +/- 0.00
Contention moved the CPU figures by ~0.5% (within noise); NPU was unaffected.
Note: --device none does NOT disable the rknpu2 backend (it intercepts MUL_MAT
via supports_op rather than registering as an offload device). A real CPU-only
baseline requires configuring with -DLLAMA_RKNPU2=OFF.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EWpfhDgYNA21tETDP9ueBE
Documents the live systemd unit, the two required GGUFs, and why this
configuration was chosen over the alternatives.
Measured on this version (02a883fea), gemma-4-12b-it-Q8_0, A76-pinned:
CPU only (LLAMA_RKNPU2=OFF): pp512 8.74 tg128 1.92
RKNPU (model only): pp512 38.31 tg128 2.35
RKNPU + MTP drafter: decode 3.502 prose / 5.207 file-echo
MTP chosen over an E2B draft model (-5.3%) and ngram-simple prompt-lookup
(+48%): acceptance is comparable across drafters, but a 25x-smaller drafter
makes rejected tokens nearly free. Speculative output verified byte-identical
to non-speculative greedy.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EWpfhDgYNA21tETDP9ueBE
mfritsche