Complete pp512/tg128 pairs from the coordinator's clean run on the
a7ace5d prod boot (30B and 3B-f16 benched together, superseding the
pp512-only numbers in 7e32abd):
Qwen2.5-3B-Instruct f16 (5.75 GiB, RKNPU): pp512 = 76.92 tok/s, tg128 = 3.36 tok/s
Qwen3-30B-A3B-Q4_K_XL (16.47 GiB, RKNPU): pp512 = 16.68 tok/s, tg128 = 4.63 tok/s
Worth calling out explicitly: the 30B MoE decodes faster than the
dense 3B-f16 (4.63 vs 3.36 tok/s) because only ~3B params are active
per token, vs the 3B-f16's fully memory-bound dense weight read on
every token. Decode tracks active-parameter count, not total resident
weight -- the 30B is genuinely usable on-NPU at these numbers, not
just "it fits."
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EWpfhDgYNA21tETDP9ueBE
a7ace5d's commit message says "NOT YET LIVE-VALIDATED" -- it now is.
Markus rebooted boltzmann on the a7ace5d build (md5 29714de2) and
re-ran the 30B prefill that previously wedged the NPU permanently:
Qwen3-30B-A3B-Instruct-2507 UD-Q4_K_XL (16.47 GiB, ~5 domains):
pp512 = 16.70 tok/s, exit 0, clean. Fence signals, job completes.
Zero SError/panic/hung-task/VA-insert in dmesg for the whole boot.
The pm_lock-around-domain-switch hypothesis in a7ace5d was correct:
the D-state that looked like a permanent hang was the uninterruptible
NPU-job wait now actually completing instead of wedging forever.
Full scoreboard on a7ace5d, all on the same boot: 4B-Q8 unchanged
(single-domain, the pre-existing production path), 3B-f16 = 76.75
tok/s native (b831a2d), 30B-A3B = 16.70 tok/s native (a7ace5d). The
~4GiB single-domain ceiling is broken and MoE cross-domain access
works. a7ace5d is production.
Deliberately a new commit rather than amending a7ace5d: it's already
pushed to github+gitea, and rewriting pushed history over a doc update
isn't worth the force-push risk. README updated with the same numbers
for anyone landing here without the commit log.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EWpfhDgYNA21tETDP9ueBE
NOT YET LIVE-VALIDATED. Written and built after the 30B/multi-domain
prefill hang reported against b831a2d ("honor vendor iommu_domain_id"),
from source analysis only (no live reboot access at patch time) --
needs a coordinated reboot-test before it earns "fixes the hang" in
the commit log. Recording the reasoning here so the next person (or
session) doesn't have to re-derive it.
What's confirmed by source reading, not speculation:
- rkopnu_ioctl_submit's own dma_fence_wait_timeout(..., 5000ms) is
correctly bounded (dma-fence.c: dma_fence_default_wait() uses a
real schedule_timeout(), always returns). It is NOT the permanent
hang -- but on timeout it fell through to unconditionally report
*success* (args->task_counter = args->task_number; ret = 0)
regardless of whether the job ever actually finished. Fixed: now
returns -ETIMEDOUT (or the fence's own error) and leaves the job's
own scheduler-held kref alone (drm_sched_job_cleanup() is only safe
for jobs that never got pushed; this one did).
- drm_sched_stop()'s unbounded dma_fence_wait() (sched_main.c, for
jobs whose HW-fence callback removal races) is unreachable here:
credit_limit=1 per core means at most one job is ever pending, so
that loop never has a second job to wait on.
- rockchip-iommu.c's attach/detach path (rk_iommu_enable/disable,
force_reset, stall) is bounded throughout via readx_poll_timeout()
with real timeout constants -- no raw unbounded register poll found
there. iommu_attach_group() failure is already handled correctly by
rocket_job_run() (signals the fence with the error immediately), so
a bounded HW-level attach timeout was already not a hang source.
- rocket_device_pm_get()/put() (rocket_device.c) serialize ALL 3
cores' power-domain resume/release through one device-global
rdev->pm_lock, specifically because (per that file's own comments)
a sibling core's independent clock/power-domain gating can race
another core's in-flight DMA on the shared NIU/PMU. That lock is
NOT held around the IOMMU domain attach/detach in rocket_job_run()
or rocket_reset() -- and rk_iommu_attach_device()/rk_iommu_disable()
do their own clk_bulk_enable()/disable() on the IOMMU submodule's
clocks, sharing the same upstream PMU/clock-tree infrastructure.
This is the same hazard class already fixed here for power-domain
gating, just not yet extended to IOMMU domain switching -- and the
30B case is exactly the scenario that exercises it hard (frequent
cross-domain MoE expert routing across 3 concurrently-segmenting
cores, each independently deciding to attach/detach at any moment).
Fix: take rdev->pm_lock around the iommu_detach_group()/
iommu_attach_group() calls in both rocket_job_run() and rocket_reset(),
serializing domain switches against power-unit transitions (and against
each other, device-wide) the same way power-domain gating already is.
Safe to acquire: rocket_device_pm_get()/put_noidle() immediately above
each site already acquire-and-release pm_lock without holding it across
their own return, so this is a fresh, non-recursive acquisition. Lock
order is always job_lock -> pm_lock in both call sites, never the
reverse, so no new AB-BA risk.
This does not explain (and does not fix, if true) a genuine hardware-
level unbounded hang -- I found no evidence of one in the code I could
read, which is what makes the PM/clock-race theory the leading
candidate. If the 30B prefill still hangs after this, the next step is
live /proc/PID/stack (needs lockdown adjusted or a debug kernel) or
ftrace on iommu_attach_group/rk_iommu_enable/pm_runtime_resume_and_get
to catch the actual stuck frame.
Builds clean (make -j8, vermagic 7.0.0-rc3-npuclk+ matches the running
kernel), checkpatch --strict clean (only pre-existing false-positive
blank-context-line whitespace hits, and the pre-existing missing-
prototype warning for rkopnu_ioctl_submit -- both present on
unmodified rocket_job.c too, verified by diffing a clean checkout).
Not yet inserted into the running kernel (never rmmod rkopnu -- reboot
required to swap modules).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EWpfhDgYNA21tETDP9ueBE
The vendor rknpu ABI (rknpu_mem_create.iommu_domain_id,
rknpu_submit.iommu_domain_id) has always let a client spread
allocations across multiple IOMMU paging domains, and our own
ggml-rknpu2 glue (IOMMUDomainManager) already assigns tensors across
up to 16 domains for exactly this reason: each domain is an
independent page-table root with its own ~4GiB IOVA aperture (capped
by the vendor rknn_matmul API's int32 B-matrix offsets, not by IOMMU
hardware). rkopnu was reading and silently dropping this field in both
ioctls, collapsing every allocation into the single domain created
once at open() -- so userspace's domain-spreading was a no-op and any
model with >~4GiB of NPU-resident weights hit ENOSPC ("MEM_CREATE VA
insert failed").
Replace the single per-fd domain with a lazily-created 16-slot pool
keyed by the vendor domain id, each owning its own drm_mm IOVA
allocator (previously one drm_mm shared per-fd; now one per domain,
matching each domain's own independent address space). Wire
iommu_domain_id through in rkopnu_ioctl_mem_create and
rkopnu_ioctl_submit. The domain attach/detach-on-change logic in
rocket_job_run() already existed and needed no changes -- it was
comparing core->attached_domain against job->domain per core with a
skip-if-unchanged optimization, just never fed more than one domain
value before now.
Also fixes two latent bugs surfaced while making domain creation
lazy/fallible (rocket_iommu_domain_get() could never fail before this;
now it can, e.g. -ENOMEM from iommu_paging_domain_alloc):
- rocket_iommu_domain_put(NULL) would crash; made it NULL-tolerant
like the dma_fence_put()/kfree() calls next to it, since
rocket_job_cleanup() unconditionally puts job->domain.
- MEM_CREATE's error paths never dropped the domain reference on
failure (harmless before, since the fd's one domain outlived any
failed allocation regardless; now leaks a whole IOMMU paging domain
per failed MEM_CREATE under memory pressure).
Verified on boltzmann (RK3588, 7.0.0-rc3-npuclk+): Qwen2.5-3B-Instruct
f16 (5.75 GiB, spans >=2 domains) now loads and runs natively on the
NPU at pp512=76.75 t/s -- previously an unconditional GGML_ASSERT abort
in ggml-rknpu2.cpp at ~4GiB. Qwen3-4B-Q8_0 (single-domain, the
existing production path) is unaffected: no regression.
Known issue, not yet fixed: Qwen3-30B-A3B-Instruct-2507 UD-Q4_K_XL
(17.69 GB, ~5 domains) maps cleanly (refcnt to 1221 dmabufs, zero
VA-insert failures, zero IOMMU faults -- the domain packing itself is
correct) but prefill deadlocks: the benchmark process goes permanently
D-state on dma_fence_default_wait, drm_sched's TDR does not visibly
recover it, and the NPU has to be recovered by reboot (never rmmod
rkopnu -- silent deadlock). Root cause not yet confirmed (lockdown
blocked /proc/PID/stack); a hardening patch is staged separately
pending live validation. Until resolved, avoid >4GiB NPU-resident
models with rapid cross-domain access patterns (dense models spanning
2-3 domains sequentially, like the 3B-f16 case above, are fine).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EWpfhDgYNA21tETDP9ueBE
The DRM driver name stays "rknpu" (librknnrt identifies its device by that),
but the platform_driver was still named "rocket" from the fork -> sysfs
(/sys/bus/platform/drivers/rocket), dmesg ("rocket fdab0000.npu:"), and a name
collision with the in-tree rocket platform_driver. Rename to "rkopnu" so
lsmod/sysfs/dmesg all agree and the two drivers can coexist as distinct
platform_drivers. The shared OF compatible means rocket still has to be
blacklisted for rkopnu to win the NPU bind (documented). README updated.
- Strip the perf-trace debug instrumentation (dbg_* counters + HWSUB/rkopnu-trace
dev_err); it served its purpose (proved the drm_sched glue is ~3% of a submit).
No functional change.
- README.md: full build+setup guide (kernel requirements, DT enablement, rocket
blacklist, out-of-tree build, librknnrt + rk-llama.cpp rknpu2 userspace, perf
notes, limitations).
- dt/rk3588-npu-enable.dtso: the necessary DT patch -- mainline ships the RKNN
cores + IOMMUs status=disabled; this enables them + OPP table (example board:
CoolPi CM5 GenBook; supplies are board-specific).
- config/rkopnu.config: kernel config fragment (DRM_ACCEL etc.).
- LICENSE: GPL-2.0.
A third party can now recreate the mainline-NPU stack from this repo alone.
Post-review hardening (Fable) on top of the close-detach fix:
- rocket_job_run: move the IOMMU attach inside core->job_lock so it is
serialized against the detach in rocket_job_close()/rocket_reset() (the
attach was unlocked -> residual attach-vs-detach race in the crash-restart
path that produced the reboots).
- rocket_job_run: on iommu_attach_group failure, unwind the PM ref and
error-signal the fence instead of leaving a phantom 'submitted' job that
never gets an IRQ (was leaking the PM ref + a ~2s stall per failure).
- rkopnu_ioctl_submit: bound the synchronous wait (dma_fence_wait_timeout 5s)
so a wedged NPU fails the request instead of an unkillable D-state task that
hangs reboot.
- rkopnu_ioctl_submit: take a gem reference around the task-BO vmap (raw
userspace pointer + concurrent MEM_DESTROY = UAF).
- rocket_reset: only detach when a domain is attached.
- rkopnu_ioctl mem_create: dev_warn on VA-insert failure (diagnose IOMMU
aperture exhaustion/fragmentation at large context).
- .gitignore build artifacts (they had leaked into git and broke the ampere
cross-build via a stale KSYMTAB_DATA .mod.c).
The Phase 4a attach-once optimization keeps a client's IOMMU domain attached
to the cores that ran its jobs, detaching only on reset. On normal file close
(e.g. a persistent llama-server restarting), rocket_postclose frees the domain
while it is still attached: rockchip-iommu WARNs at rk_iommu_domain_free, and
core->attached_domain is left dangling -> use-after-free when the next client's
job compares/attaches it -> kernel panic / self-reboot. One-shot clients
(llama-completion) closed cleanly enough to hide it; a crash-restarting
llama-server on ampere reliably reboots the box.
Fix: in rocket_job_close (after the sched entities are destroyed, so no more
jobs run) detach this file's domain from any core still holding it and clear
core->attached_domain, before the domain is freed. Preserves the per-job
attach-once benefit (detach happens once, at close, not per submit).
Root cause of both a latent oops AND the perf gap: SUBMIT let drm_sched pick
an arbitrary core, but librknnrt targets a specific physical core via core_mask
(round-robining single-core matmuls across all 3) and only fills that core's
subcore_task[] slot. A mismatched core read the wrong (often garbage) slot ->
out-of-bounds rk_tasks[ts] -> kernel oops (dodged by luck in earlier runs), and
left core 2 permanently idle -> ~1/3 of NPU throughput on the floor.
Fix: per-core drm_sched entities; pin each SUBMIT to rknpu_wait_core_index(
core_mask)'s core so hw_submit reads the matching subcore slot. Plus a bounds
guard in hw_submit (fall back to the whole-job range if a subcore range escapes
[task_start, task_start+task_number)) so a bad index can never oops again.
Reverted the Phase 4b task-BO vmap cache: with all 3 cores now active,
librknnrt drives them from concurrent threads each with its own task BO; a
shared per-file vmap slot races (one thread's vunmap strands another's in-flight
rk_tasks). Back to per-submit vmap (thread-safe; the ioctl holds the map across
its blocking fence wait).
Added ktime latency instrumentation (worker/npu/thread segments). Trace proves
the drm_sched glue is negligible: worker=90us + thread=22us vs npu=3800us per
submit. The gap was parallelism, not submit-path overhead.
Measured (gemma-4-E2B Q8, 1204-tok prefill, cores 4-7, contenders stopped):
21.73 -> 39.33 -> 44.38 -> 51.49 tok/s; vendor RKNPU2 warm = 51.62. 3 cores
balanced (676 IRQs each), 0 oops, 0 timeouts.
librknnrt reuses one task BO for the whole session; the SUBMIT ioctl was
re-vmapping (and vunmapping) it on every one of ~2028 submits/prefill,
rebuilding its kernel PTEs each time. Cache the vmap in rocket_file_priv,
keyed on the gem pointer, holding a gem reference so it can't be freed under
us; only (re)map on a BO change; release it in rocket_postclose.
Measured (boltzmann, gemma-4-E2B Q8_0, 1204-tok prefill, cores 4-7,
contenders stopped): 39.33 -> 44.38 tok/s (vendor RKNPU2 warm = 51.62).
Was iommu_attach_group+detach per submit (TLB flush x2028 submits) = the main
gap vs vendor 52 tok/s. Vendor only switches on domain change. Now: attach only
when core->attached_domain != job->domain (never, for one librknnrt client);
leave attached on completion; clear the cache on reset. Correctness unchanged.
Root cause (Fable, source-grounded): rocket_job_irq_handler rejected the IRQ with
IRQ_NONE unless raw_status had DPU_0|DPU_1 (bits 8/9), but librknnrt's INT8 matmul
completion lands on bits 30/31 (observed int_raw=0xc0000000) -> handler never ran
-> timeout. Fix: wake on any nonzero raw status; bottom-half clears 0xffffffff
(0x1ffff missed 30/31). Plus retry cap (>3 timeouts -> NOMINAL) so a stuck job
can't infinite-loop into the postclose teardown race (drm_mm/list Oops -> shutdown
hang -> power-cycle). NPU already computes; this delivers the done-signal.
Two real bugs fixed vs source: (1) RK3588 (num_irqs>1) reads the per-core task
range from subcore_task[core], not args->task_start/number -> my last_task/int_mask
was wrong (int_mask=0). Now hw_submit selects subcore[core] (or subcore[core+2] for
3-core), uses last->int_mask for INT_MASK, first->int_mask for INT_CLEAR. (2) IRQ
handler INT_CLEAR reverted 0xffffffff->0x1ffff (RKNPU_INT_CLEAR; my 0xffffffff
likely caused the shutdown hang). Job stores task-array base + subcore[] + core_mask;
vmap kept alive across the sync fence-wait.
Diagnostic showed int_raw=0xc0000000 on timeout (NPU RAN, raised done on bits
30/31) but task int_mask=0 masked it off -> no completion IRQ -> timeout.
Fix: INT_MASK=0xc0000000 (unmask the real done bits), INT_CLEAR=0xffffffff in
hw_submit AND the IRQ handler (was 0x1ffff, left 30/31 asserted). The vendor PC
sequence already makes the NPU compute; this delivers the completion.
dev_err the actual regcmd/amount/int_mask/task_number/base written, and read
INT_RAW_STATUS/INT_STATUS on timeout: nonzero raw = NPU ran but int_mask wrong;
zero = NPU never started (bad regcmd/base/enable). Splits the not-firing bug.
The raw writel to pc_iomem+0x1004/0x3004 oopsed (outside rocket's 4KB PC
sub-mapping). Those vendor offsets are the CNA(0x1000)/CORE(0x3000) blocks;
rocket's rocket_cna_writel/rocket_core_writel S_POINTER writes are exactly the
vendor enable writes. Restored them, kept the vendor PC data programming.
Rewrote rocket_job_hw_submit to match rknpu_job_subcore_commit_pc exactly for
RK3588: per-core enable writes (0x1004/0x3004), PC_DATA_ADDR=first regcmd_addr,
amount=(regcfg_amount+4+1)/2-1, INT_MASK=last->int_mask, INT_CLEAR=first->int_mask,
TASK_CONTROL=((0x6|pp)<<12)|task_number, PC_DMA_BASE_ADDR=args->task_base_addr,
OP_EN 1->0 start pulse. One program runs the whole task range (task_count=1).
Fixes vs the first cut: was writing DMA_BASE=0, no OP_EN toggle, fixed DPU int
mask, wrong amount formula -> NPU never fired. rocket_job carries the rk_* params.
rkopnu_ioctl_submit (in rocket_job.c): reads rknpu_task[] from the task BO
(obj_addr), translates regcmd_addr->regcmd + regcfg_amount->regcmd_count into
rocket_task[], pushes onto rocket's drm_sched (rocket_job_push -> rocket_job_run
-> rocket_job_hw_submit programs the same RK3588 PC regs), and blocks on the
inference_done_fence (rknpu ABI is synchronous). Single-core first cut; multi-core
core_mask is Phase 3b. Builds. Boot-test next: does the NPU run matmuls.
Boot-tested: librknnrt allocates 4416 BOs (MEM_CREATE), maps/syncs/destroys all,
zero IOMMU faults, drives a full prefill's worth of memory ops, and stops exactly
at SUBMIT ('E RKNN: failed to submit!, op name: MatMul' = -ENOSYS stub). CPU
fallback 30.97 tok/s. Memory ioctls proven at scale. Next: Phase 3 SUBMIT.
Boot-tested (rocket blacklisted): rkopnu loads, binds all 3 rknn-core nodes,
registers [drm] rknpu on card2. librknnrt opens card2, dispatches ACTION
(version probe OK), fails at MEM_CREATE (-ENOSYS stub) exactly as designed:
'E RKNN: failed to malloc npu memory ... Function not implemented'.
Baked real HW version 1179210309 into GET_HW_VERSION (read from cores at bind).
Phase 1 done. Next: Phase 2 memory ioctls (GEM/shmem + IOMMU).
Builds rkopnu.ko (rocket clone, renamed module) against 7.0.0-rc3-npuclk+.
Baseline to swap rocket's uAPI for the rknpu 6-ioctl ABI librknnrt calls.
Includes rknpu_ioctl.h (target ABI) + README roadmap.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EWpfhDgYNA21tETDP9ueBE
Markus Fritsche