bo_free() (munmap+close) fixes the per-tile mmap leak in rkt_npu_matmul; add
rocket_munmap_bo to librocket. rkt_matmul rejects K>8192 (int8 K-limit). Keeps
the canonical primitive in sync with the ggml-rocket backend copy in rk-llama.cpp.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EWpfhDgYNA21tETDP9ueBE
Links against rk-llama.cpp's real libggml and uses ggml_mul_mat(Q8_0 weight,
F32 activation) computed by ggml's CPU backend as ground truth — the exact op
gemma4 runs. NPU path: dequantize_row_q8_0 -> per-output-channel int8 requant,
per-tensor F32 activation quant, rkt_npu_matmul with per-channel out_scales.
Hardware result (M=16 N=64 K=256): max|err|=0.266 vs tol=1.745 (4 out-steps),
ref_rms=10.6 -> PASS. Confirms the U6 primitive is a drop-in for a ggml MUL_MAT
over Q8_0 weights, matching llama.cpp numerics within quant tolerance.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EWpfhDgYNA21tETDP9ueBE
Adds optional out_scales[N] (per-output-channel requant scale) to
rkt_npu_matmul; per N-tile it uses the tile's channel scale (exact per-channel
with tile_n=1). This is what gemma4's per-channel weight quantization needs.
Verified on NPU: per-channel test (out_scales[oc]=1/(oc+1)) -> row 144 160 176
192 208 224 240 255 matches CPU exactly. All 5 gemmtest cases pass.
Next U6: ggml glue (Q8_0 weights -> per-oc int8 + per-oc out_scale array;
F32 activations -> int8 + activation scale; call this primitive).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Extracts the verified 2D-tiling matmul into a clean library function:
rkt_npu_matmul(fd, X, Wc, bias, M,N,K, zero-points, scales, tile_m,tile_n, Y)
-> tiles via rkt_gemm_plan, runs each single-op tile (scaled builder + bias
BO + INT64_MAX prep_bo), assembles Y. This is the function a ggml MUL_MAT
backend calls. gemmtest.c is now a test driver: small / large-M(4096) /
2D-tiled(2048x256) / gemma-scale(K=2048) all PASS on the NPU.
Next (U6): fork rk-llama.cpp ggml-rknpu2.cpp, replace rknn_matmul_run
(line 777) with rkt_npu_matmul, map ggml Q8_0 tensors -> our int8 X/Wc +
per-tensor scales, wire buffer-type + supports_op(MUL_MAT).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
gemmtest generalized to tile BOTH M and N via rkt_gemm_plan: each (m x n,
full-K) tile extracts its own rows/cols, packs standalone, runs one verified
single-op matmul, assembles into Y. Verified: M=2048 N=256 K=16 (4 tiles) and
gemma-scale M=12 K=2048 N=256 (6x128 tiles) -> output matches CPU exactly.
So an arbitrary (M,N,K) INT8 matmul now runs correctly on the mainline rocket
NPU. Per-op sizing: m limited by CBUF input banks (K=16 -> m<=1024; large K
shrinks m since weights eat banks), K*n <= ~327k. Next: weight-reuse across
M-tiles (Mesa reuse_weights_cbuf) for perf (large-K forces tiny m today), then
U6 ggml-backend wiring (MUL_MAT dispatch -> this primitive).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
gemmtest.c computes Y[M][N]=X[M][K]*W[K][N] for M larger than one CBUF tile by
splitting M via rkt_gemm_plan and running each chunk as a verified single-op
matmul (fresh BOs/tile, weights packed once), assembling the output. M=4096
via 4x1024-row tiles -> output matches CPU exactly (0/32768 wrong).
Key HW limit found: a single rocket op writes correct output only up to
M~1024 rows (K=16); M=2048 silently drops rows past ~971 (CBUF input capacity),
so tile_m<=1024 is the safe per-op bound. Also: fresh BOs per tile are needed
(reusing in/out/reg BOs across jobs corrupted later tiles). This is the tiling
orchestration U6/ggml will use; N-tiling (weight-column) + weight-reuse
optimization are next.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
rkt_build_matmul_regcmd_scaled() takes per-tensor input/weights/output scales
and a bias BO dma address; rkt_build_matmul_regcmd() is now a thin wrapper
(identity scales, no bias) so all existing callers + the golden test are
unchanged (verified: selftest ALL PASS, golden IDENTICAL). The OUT_CVT
scale/shift path was already hardware-verified (conv_scale=4 -> 192 matches
CPU). This threads real quantization params through for U6/gemma4, replacing
the identity-only assumption. Bias correction (rkt_compute_biases) can now be
wired via the bias_dma param instead of the hwtest regcmd patch.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The whole path works end-to-end on real silicon (boltzmann, 7.0.0-rc3 rocket,
/dev/accel/accel0): our golden regcmd builder + rkt_operands tiling + librocket
submission -> a 4x8x16 INT8 matmul whose per-element output (incl requant +
int8 saturation) matches the CPU reference exactly (144,160,...,255).
ROOT CAUSE of the long all-zero saga: rocket_prep_bo()'s timeout_ns is an
ABSOLUTE CLOCK_MONOTONIC deadline (kernel drm_timeout_abs_to_jiffies), not a
relative duration. hwtest passed 3e9 ('3s') -> read as a deadline ~9h in the
past -> effective timeout 0 -> we read the output BO (memset 0) BEFORE the NPU
wrote it. The -EBUSY we dismissed WAS the 'not done' signal. Mesa passes
INT64_MAX (block until the DPU-write fence signals). Nothing was wrong with the
regcmd, operands, addresses, driver, or hardware -- diagnosed via the working
Mesa Teflon reference ('if tflite works it has the proper spells', per Markus;
divergence found by Fable/rknpu-specialist).
Fixes: hwtest prep_bo INT64_MAX; close all BO handles (kills drm_mm_takedown
WARN); librocket rocket_close_bo added + prep_bo doc corrected (abs deadline).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
rkt_operands.{c,h}: faithful ports of Mesa's operand layout — rkt_pack_weights
(rkt_coefs.c tiling+sign), rkt_compute_biases (zero-point correction),
rkt_pack_input (feature tiling from rkt_ml_subgraph_invoke), rkt_unpack_output
(de-tile from read_outputs). hwtest.c v2 lays out a real 4x8x16 INT8 matmul,
submits our golden regcmd with real BO addresses, and checks vs a CPU ref.
State: SUBMIT accepted; job still does NOT complete. Root cause located in the
kernel driver (rocket_job.c): completion is a DPU interrupt (INTERRUPT_MASK
DPU_0|DPU_1) and JOB_TIMEOUT_MS=500 — our job runs but the DPU never raises
done within 500ms, so drm_sched resets the core and prep_bo returns -EBUSY.
Interface/regcmd/operands all match Mesa; the stall is at hardware execution.
Next: known-good Mesa/Teflon reference run on this NPU to disambiguate operand
tiling vs early-driver/DTB corner, or NPU status-register readback post-timeout.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
hwtest.c allocates BOs, builds the golden matmul regcmd with their real NPU
DMA addresses, patches in a bias BO, and submits to /dev/accel/accel0 on the
live mainline rocket kernel (boltzmann, 7.0.0-rc3-ARCH+).
Result: DRM_IOCTL_ROCKET_SUBMIT rc=0 (accepted) — the driver queues a job
built from our register buffer, coupling the Rosenblatt builder to real
silicon for the first time. Job does NOT complete (prep_bo -EBUSY, output
zero): the CNA weight path uses DCOMP decompression and we fed zeros, which
stalls the weight DMA. Completion needs real NPU-tiled/compressed weights
(rkt_coefs.c port) — next unit. No NPU hardware fault; cores stay bound; the
drm_mm_takedown WARN at fd-close is driver cleanup, not a hang.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
rkt_gemm_plan() partitions an MxN (full-K) output into <=tile_m x tile_n
sub-matmuls; rkt_gemm_op_fits() confirms a shape fits one rocket op via the
verified builder. First slice of the CPU+NPU backend: lets a gemma-scale GEMM
be expressed as many NPU ops. K-tiling (DPU partial-sum accumulation) deferred.
selftest: exact coverage of 64x200 by 8 tiles, every tile fits one op (K=64),
impossible-K reported unfit. Golden match unaffected. ALL PASS.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Adds userspace/npu-probe/verify/: a reference oracle built from Mesa's
UNMODIFIED rkt_task.c + rkt_regcmd.c (vendored, MIT) linked against a thin
shim (rkt_ml.h), which emits the ground-truth regcmd buffer for our test
matmul. 'make diff' byte-compares it against rkt_build_matmul_regcmd.
First run caught three real bugs, now fixed:
1. REG_DPU_RDMA_RDMA_S_POINTER emitted at target 0x1001, must be 0x2001
2. an 18-word CNA register run (CONV_CON2..FC_CON1) emitted twice
3. the whole DPU_RDMA block at 0x1001 instead of 0x2001 (EMIT_RDMA added)
Values were correct throughout; only block-targets and a copy-paste were
wrong. CNA stage rewritten to Mesa's single-pass order; coredpu RDMA regs
repointed. selftest histogram now counts the RDMA (0x2001) block.
Result: 130 words, IDENTICAL — GOLDEN MATCH. The builder is now proven, not
just faithful.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Ports Mesa rkt_task.c (fill_task + single-tile rkt_split_tasks branch) into
rkt_matmul.c so the declared public API is now real: a plain (M,N,K) GEMM is
mapped to a 1x1 conv and every low-level CNA/CORE/DPU field (banks, atomic
count, line/surface strides, data entries, kernel/channel alignment) is
derived from the NVDLA CBUF geometry (CBUF_BANKS=12, ENTRY=128, ATOMIC=16),
then handed to the two emit stages. Single-tile only: shapes needing CBUF
task-splitting return -1 rather than mis-emit.
selftest now drives the unified builder as the primary path; it produces a
buffer structurally identical to the hand-filled stage path (148 words, same
PC/CNA/CORE/DPU histogram) and refuses an oversized split-required shape.
Clean -Wall -Wextra, ALL PASS.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Makefile (make test / make lib) and selftest.c — a host-side smoke gate
for the CNA+CORE+DPU+PC regcmd builder. Not a golden-byte check (needs
Mesa/NPU); verifies the stages assemble without overflow, are
deterministic, report capacity exhaustion, and emit a sane per-block
target histogram. 148 words on a 4x8x16 single-tile GEMM, ALL PASS.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
rkt_matmul_coredpu.{c,h}: faithful port of Mesa rocket fill_first_regcmd's
CORE + DPU + PC sections (89 emits incl. float requant math via fui(), the
elementwise/add-tensor path, RDMA config, and the PC op-enable tail). Compiles
clean (-Wall -Wextra). Ported via the curl-driven dspark transformer (pi-run was
wedged on an MCP-connect hang) in 3 slices + hand-integrated the non-mechanical
spots (DMA addrs, float scales, raw emits, num_tasks/offset locals).
Builder now complete end to end: librocket + rkt_matmul_cna + rkt_matmul_coredpu
all build. Still faithful-to-Mesa, NOT golden-verified (needs Mesa build/NPU).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EWpfhDgYNA21tETDP9ueBE
Codename: Frank Rosenblatt — Mark I Perceptron 1958, the first
hardware neural network. This project lights up the RK3588 NPU on
mainline Linux so the OSS world finally owns the silicon-side of
inference on that chip.
Phase-1 scope: small LLM running CPU + NPU mix on boltzmann (Rock 5
ITX+). Backend: llama.cpp with a new rknpu ggml backend offloading
INT8 GEMM (attention + FFN matmuls) to the NPU's tile-MAC array while
leaving dequant / RoPE / softmax / sampling / embedding on A76 NEON.
Target model: qwen2.5-1.5B-instruct Q4_K_M GGUF.
Scaffold layout: README.md (frame + 9+1-phase plan), TODO.md (rolling
punch-list), docs/{npu-mainline-status,architecture}.md, kernel/ for
DT bindings + driver tweaks, userspace/{npu-probe,llm-runtime}/,
fleet/boltzmann.yaml.
Next: Phase-1 substrate audit — fill the TBDs in docs/npu-mainline-status.md
with the actual state of Tomeu Vizoso's rknpu / DRM-accel work on
the boltzmann-running kernel.
Markus Fritsche
marfrit