New MUL_MAT backend for the RK3588 NPU via the mainline 'rocket' DRM-accel
driver (/dev/accel/accel0), with no vendor RKNPU2 userspace. Modeled on
ggml-blas: an ACCEL device with CPU-resident weights; the scheduler offloads
supported MUL_MAT nodes and leaves the rest on CPU.
Per plane: dequantize weight to F32 (ggml to_float), requantize per-output-
channel to int8, quantize F32 activation per-tensor to int8, run the tiled
int8 GEMM on the NPU (rkt_npu_matmul, ported from the rosenblatt npu-probe:
Mesa-rocket regcmd builder + NVDLA CBUF tiling + librocket DRM ioctls), then
dequantize the int8 result. out_scale is a Cauchy-Schwarz bound on |output|
so the int8 output never clips.
Empirically-verified HW limits gate supports_op: TILE_M=16, TILE_N=128,
K<=8192 (int8 K-limit), K%16==0, F32 activations, dequantizable weight.
Job-completion wait uses a finite 6s absolute deadline so a hung NPU job
errors instead of deadlocking.
Enable with -DGGML_ROCKET=ON (and -DLLAMA_RKNPU2=OFF so the vendor backend
does not claim the weight buffers). Verified end-to-end: gemma-4-E2B-it-Q8_0
generates coherent text on boltzmann's mainline rocket kernel, matmuls
offloaded to the NPU, ~3.9 tok/s eval.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EWpfhDgYNA21tETDP9ueBE
mfritsche
8e49f43486