marfrit/rk-llama.cpp
1
0
Fork 0
Code Issues Pull Requests Actions 31 Packages Projects Releases Wiki Activity

rknpu2: M-bucketing — ceil-to-128 for M>128 instead of next_power_of_two

Browse Source 
Experiment: next_power_of_two pads prefill M up to the next pow2 (300->512), wasting matmul work. Switch to ceil-to-128 for M>128 (300->384); keep pow2 for M<=128. Byte-identical output (padding only). Baseline pp300/pp512/pp700/tg128: 30.66/40.42/40.17/2.58 t/s. Measured: 37.01/40.25/40.03/2.61 t/s. pp300 +20.7%; pp512/pp700 flat (ubatch<=512 caps M; 700=512+188, both bucketings map 188->256); tg unchanged.
This commit is contained in:
mfritsche
2026-07-09 00:38:02 +02:00
parent 39622ccb32
commit 04d2c6e548
1 changed files with 5 additions and 2 deletions
Download Patch File Download Diff File Expand all files Collapse all files
ggml/src/ggml-rknpu2/ggml-rknpu2.cpp
+5 -2
View File
@@ -488,9 +488,12 @@ static enum ggml_status ggml_backend_rknpu_graph_compute(ggml_backend_t backend,
continue; continue;
} }
// Using next power of two for M for efficient caching // M bucketing: pow2 up to 128, then ceil-to-128 for larger M.
// Avoids next_power_of_two doubling waste at large prefill (300->512, 700->1024).
int M_op = M; int M_op = M;
if (M > 1) { if (M > 128) {
M_op = ((M + 127) / 128) * 128;
} else if (M > 1) {
M_op = rknpu2_calibration::next_power_of_two(M); M_op = rknpu2_calibration::next_power_of_two(M);
} }