From 698c6b731c853bd676e1141b43152058916ce3c5 Mon Sep 17 00:00:00 2001 From: mfritsche Date: Tue, 14 Jul 2026 11:45:05 +0200 Subject: [PATCH] =?UTF-8?q?ggml-rocket:=20min=5Fbatch=20floor=20=E2=80=94?= =?UTF-8?q?=20keep=20M<32=20(decode)=20on=20CPU?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit llama-bench A/B on boltzmann (gemma-4-E2B-Q8_0, A76-pinned, -t 4) showed the NPU path HALVED decode: tg128 CPU 7.97 vs Rocket 3.88 t/s, because every M=1 matmul paid the full per-tile BO-alloc/mmap/submit/wait/close round-trip while decode is bandwidth-bound (the vendor rknpu2 campaign found the NPU only +7% there even at full offload). Gate supports_op on M>=ROCKET_MIN_BATCH(32), mirroring ggml-blas, so small batches stay on CPU. After: pp512 40.32 (was 40.52, +12% vs CPU 35.95), tg128 8.34 (was 3.88, now >= CPU). Strictly >= CPU on both phases instead of a 2x decode regression. Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_01EWpfhDgYNA21tETDP9ueBE --- ggml/src/ggml-rocket/ggml-rocket.cpp | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/ggml/src/ggml-rocket/ggml-rocket.cpp b/ggml/src/ggml-rocket/ggml-rocket.cpp index eeb7690af..1a868afc6 100644 --- a/ggml/src/ggml-rocket/ggml-rocket.cpp +++ b/ggml/src/ggml-rocket/ggml-rocket.cpp @@ -40,6 +40,7 @@ extern "C" { #define ROCKET_TILE_M 16 #define ROCKET_TILE_N 128 #define ROCKET_K_MAX 8192 +#define ROCKET_MIN_BATCH 32 // below this, CPU wins (decode is bandwidth-bound) struct ggml_backend_rocket_context { int fd = -1; @@ -350,7 +351,12 @@ static bool ggml_backend_rocket_device_supports_op(ggml_backend_dev_t dev, const case GGML_OP_MUL_MAT: { const int64_t K = src1->ne[0]; // == src0->ne[0] + const int64_t M = op->ne[1]; // batch rows (tokens) + // Offload only when the NPU can amortise its per-tile submit + // overhead: decode (M=1) is bandwidth-bound and strictly faster on + // CPU, so keep small batches there (mirrors ggml-blas min_batch). + if (M < ROCKET_MIN_BATCH) return false; if (src1->type != GGML_TYPE_F32) return false; if (!ggml_is_contiguous(src0)) return false; if (!ggml_is_contiguous(src1)) return false;