ggml-rknpu2: graceful abort with W8A8 hint on NPU IOVA exhaustion

rknn_create_mem returns null when the RK3588 NPU's ~4 GiB IOVA aperture is
full -- this backend maps every weight tensor into a single 4 GiB IOMMU window
(rkopnu currently ignores iommu_domain_id, so the IOMMUDomainManager's 2 GiB
domain-spreading is a no-op and everything lands in one 4 GiB window). The old
GGML_ASSERT reported a cryptic "Failed to allocate tensor memory via RKNN API"
with no guidance.

Replace it with a GGML_ABORT that names the aperture and points at the fix:
use a quantized GGUF, or set RKNPU_HYBRID=W8A8_STANDARD for on-the-fly INT8
weights (which fits >4 GiB dense f16 models -- e.g. Qwen2.5-3B-f16 runs at
~130 t/s pp512 that way). Also null-check the grow re-alloc path, which was a
latent copy of the same crash.

No behaviour change for models that fit; only the failure message improves.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EWpfhDgYNA21tETDP9ueBE

README.md

@@ -12,6 +12,82 @@
 
 LLM inference in C/C++
 
+---
+
+## Rockchip-specific environment variables, parameters, and recipe
+
+> This is a Rockchip **RK3588 NPU** fork of llama.cpp. The `performance-enhancement`
+> branch is the consolidated, benchmarked state: it adds two RK3588 NPU backends plus
+> tuning knobs and benchmark docs. Base llama.cpp usage below is unchanged.
+
+### Two NPU backends (enable exactly one)
+
+| backend | build flag | kernel driver | userspace | notes |
+|---------|-----------|---------------|-----------|-------|
+| **rknpu2** | `-DLLAMA_RKNPU2=ON` | vendor BSP **or** mainline via [rkopnu](https://github.com/marfrit/rkopnu) | closed `librknnrt.so` | drives the NPU through Rockchip's runtime; fastest (vendor parity) |
+| **rocket** | `-DGGML_ROCKET=ON` | mainline `drivers/accel/rocket` | none (fully open) | open ggml backend on the mainline accel driver; ~1.33× over CPU |
+
+### Recipe — rknpu2 backend on a **mainline** kernel
+
+The fully-open kernel path running the closed runtime at vendor speed:
+
+1. **Kernel:** build + load [**rkopnu**](https://github.com/marfrit/rkopnu) (presents the
+   vendor `rknpu` ioctl ABI on a mainline kernel; blacklist the in-tree `rocket`).
+2. **Runtime:** `sudo install -m0644 librknnrt.so /usr/lib && sudo ldconfig`
+   (librknnrt ships in this tree at `ggml/src/ggml-rknpu2/libs/`).
+3. **Build:** `cmake -B build -G Ninja -DLLAMA_RKNPU2=ON -DGGML_ROCKET=OFF && ninja -C build llama-server`
+4. **Run** — raise the fd ceiling (librknnrt exports ~1500+ dmabuf fds per prefill) and pin
+   to the A76 big cluster:
+   ```sh
+   ulimit -n 65536
+   LD_LIBRARY_PATH=build/bin taskset -c 4-7 ./build/bin/llama-server \
+       -m model.gguf -c 131072 -t 4 -fit off --host 0.0.0.0 --port 8090
+   ```
+   librknnrt round-robins single-core matmuls across all three NPU cores automatically.
+
+### Environment variables
+
+**rknpu2 backend** (`ggml/src/ggml-rknpu2/`):
+
+| variable | default | effect |
+|----------|---------|--------|
+| `RKNPU_DEVICE` | `RK3588` | target SoC profile |
+| `RKNPU_CORES` | all | comma-separated NPU core indices to use, e.g. `0,1,2` |
+| `RKNPU_DOMAINS` | all | restrict which op domains offload to the NPU |
+| `RKNPU_HYBRID` | built-in | custom per-layer CPU/NPU split pattern |
+
+**rocket backend** (`ggml/src/ggml-rocket/`):
+
+| variable | default | effect |
+|----------|---------|--------|
+| `GGML_ROCKET_MIN_BATCH` | `ROCKET_MIN_BATCH` | minimum M (batch) to offload; smaller matmuls (e.g. decode, M<32) stay on CPU |
+| `GGML_ROCKET_PROF` | off | on backend teardown, dump per-matmul timing: setup / submit / npu_wait / free / tiles / calls |
+| `GGML_ROCKET_DEBUG` | off | verbose per-matmul offload-decision logging |
+| `ROCKET_NOREPACK_TENSORS` | unset | measurement only: comma-separated tensor-name substrings to force off the `CPU_REPACK` buftype so they reach the NPU. Routing *more* tensors to the NPU is generally **slower** (the CPU NEON repack kernel beats the NPU for those shapes) — for A/B experiments, not production |
+
+### Useful parameters
+
+- `-fit off` — don't auto-shrink unset args to device memory when you set `-c` explicitly.
+- `taskset -c 4-7` / `-t 4` — pin to the four A76 cores; the CPU-side quantize/repack is the
+  prefill bottleneck, so keep it off the little A55 cluster.
+
+### Benchmarks
+
+Full data in [`docs/rknpu2-campaign-results.md`](docs/rknpu2-campaign-results.md). Summary
+(gemma-4-E2B-it-Q8_0, prefill, RK3588 NPU @ 1 GHz):
+
+| stack | prefill tok/s |
+|-------|---------------|
+| CPU (NEON, A76×4) | ~34.9 |
+| rocket backend (mainline, open) | ~46–49 |
+| **rknpu2 (vendor runtime, via rkopnu on mainline)** | **~51.5** (Rock 5 ITX) / ~49.5 (CoolPi GenBook, DDR-bound) |
+
+A persistent `llama-server` at **128K context** on the rkopnu+rknpu2 path is soak-tested at
+150/150 completions with all three NPU cores balanced and no crashes. Board-to-board
+differences are DDR-bandwidth-bound, not clock.
+
+---
+
 ## Recent API changes
 
 - [Changelog for `libllama` API](https://github.com/ggml-org/llama.cpp/issues/9289)

common/arg.cpp

@@ -2348,10 +2348,12 @@ common_params_context common_params_parser_init(common_params & params, llama_ex
             }
         }
     ).set_env("LLAMA_ARG_N_CPU_MOE"));
-    GGML_ASSERT(params.n_gpu_layers < 0); // string_format would need to be extended for a default >= 0
+    // NPU/accel backends (e.g. rknpu2, DEVICE_TYPE_ACCEL) may set a default n_gpu_layers >= 0,
+    // so handle that case in the help string instead of asserting a negative default.
     add_opt(common_arg(
         {"-ngl", "--gpu-layers", "--n-gpu-layers"}, "N",
-        string_format("max. number of layers to store in VRAM, either an exact number, 'auto', or 'all' (default: %s)", params.n_gpu_layers == -1 ? "auto" : "all"),
+        string_format("max. number of layers to store in VRAM, either an exact number, 'auto', or 'all' (default: %s)",
+            params.n_gpu_layers == -1 ? "auto" : (params.n_gpu_layers <= -2 ? "all" : std::to_string(params.n_gpu_layers).c_str())),
         [](common_params & params, const std::string & value) {
             if (value == "auto") {
                 params.n_gpu_layers = -1;
@@ -3616,11 +3618,11 @@ common_params_context common_params_parser_init(common_params & params, llama_ex
             params.speculative.draft.devices = parse_device_list(value);
         }
     ).set_spec().set_examples({LLAMA_EXAMPLE_SPECULATIVE, LLAMA_EXAMPLE_SERVER, LLAMA_EXAMPLE_CLI}));
-    GGML_ASSERT(params.speculative.draft.n_gpu_layers < 0); // string_format would need to be extended for a default >= 0
+    // NPU/accel backends may set a default draft n_gpu_layers >= 0; handle it in the help string.
     add_opt(common_arg(
         {"--spec-draft-ngl", "-ngld", "--gpu-layers-draft", "--n-gpu-layers-draft"}, "N",
         string_format("max. number of draft model layers to store in VRAM, either an exact number, 'auto', or 'all' (default: %s)",
-            params.speculative.draft.n_gpu_layers == -1 ? "auto" : "all"),
+            params.speculative.draft.n_gpu_layers == -1 ? "auto" : (params.speculative.draft.n_gpu_layers <= -2 ? "all" : std::to_string(params.speculative.draft.n_gpu_layers).c_str())),
         [](common_params & params, const std::string & value) {
             if (value == "auto") {
                 params.speculative.draft.n_gpu_layers = -1;

docs/rknpu2-campaign-results.md

@@ -1,7 +1,7 @@
 # RKNPU2 perf campaign — results (E1–E6)
 
 Model: ornith-1.0-9b-Q8_0.gguf (Q8_0 → whole model on NPU)
-Host: boltzmann.fritz.box (RK3588). Harness: `ulimit -n 65536`, llama-bench, `-t 4`.
+Host: an RK3588 board (8× Cortex-A76/A55, LPDDR5). Harness: `ulimit -n 65536`, llama-bench, `-t 4`.
 Branch: perf-rknpu2 (base compat commit 39622ccb3).
 
 ## Throughput (t/s), baseline → each step

ggml/src/ggml-cpu/repack.cpp

@@ -13,6 +13,7 @@
 
 #include <cmath>
 #include <cstring>
+#include <cstdlib>
 #include <cassert>
 #include <cstdio>  // for GGML_ASSERT
 
@@ -4526,6 +4527,22 @@ template <typename BLOC_TYPE, int64_t INTER_SIZE, int64_t NB_COLS, ggml_type PAR
 }  // namespace ggml::cpu::repack
 
 static const ggml::cpu::tensor_traits * ggml_repack_get_optimal_repack_type(const struct ggml_tensor * cur) {
+    // Selective rocket offload (marfrit): keep listed weights OUT of CPU_REPACK so the
+    // scheduler's host-buffer offload gate can route them to the rocket NPU backend.
+    // ROCKET_NOREPACK_TENSORS = space/comma-separated name substrings, e.g. "ffn_down".
+    {
+        const char * skip = getenv("ROCKET_NOREPACK_TENSORS");
+        if (skip && cur->name[0]) {
+            const char * p = skip;
+            while (*p) {
+                while (*p == ' ' || *p == ',') p++;
+                char tok[64]; int n = 0;
+                while (*p && *p != ' ' && *p != ',' && n < 63) tok[n++] = *p++;
+                tok[n] = 0;
+                if (n > 0 && strstr(cur->name, tok)) return nullptr;
+            }
+        }
+    }
     // instance for Q4
     static const ggml::cpu::repack::tensor_traits<block_q4_0, 4, 4, GGML_TYPE_Q8_0> q4_0_4x4_q8_0;
     static const ggml::cpu::repack::tensor_traits<block_q4_0, 8, 4, GGML_TYPE_Q8_0> q4_0_4x8_q8_0;

ggml/src/ggml-rknpu2/ggml-rknpu2.cpp

@@ -296,6 +296,9 @@ struct ggml_backend_rknpu_buffer_context {
                 it->second.iommu_domain_id = g_domain_manager.assign_domain_memory(size);
                 rknn_matmul_ctx new_ctx = g_domain_manager.get_allocator_context(it->second.iommu_domain_id);
                 it->second.mem = rknn_create_mem(new_ctx, size);
+                if (it->second.mem == nullptr) {
+                    GGML_ABORT("rknpu2: rknn_create_mem failed for %zu bytes (grow re-alloc) -- the NPU's ~4 GiB IOVA aperture is exhausted; this backend maps all weights into a single 4 GiB IOMMU window. Use a quantized GGUF, or set RKNPU_HYBRID=W8A8_STANDARD for on-the-fly INT8 weights.", size);
+                }
                 it->second.size = size;
             }
             return it->second;
@@ -311,7 +314,9 @@ struct ggml_backend_rknpu_buffer_context {
         alloc.size = size;
         alloc.iommu_domain_id = domain_id;
 
-        GGML_ASSERT(alloc.mem != nullptr && "Failed to allocate tensor memory via RKNN API");
+        if (alloc.mem == nullptr) {
+            GGML_ABORT("rknpu2: rknn_create_mem failed for %zu bytes -- the NPU's ~4 GiB IOVA aperture is exhausted; this backend maps all weights into a single 4 GiB IOMMU window. Use a quantized GGUF, or set RKNPU_HYBRID=W8A8_STANDARD for on-the-fly INT8 weights.", size);
+        }
         tensor_allocs[tensor_offset] = alloc;
 
         return alloc;