Markus Fritsche
b831a2d931
The vendor rknpu ABI (rknpu_mem_create.iommu_domain_id,
rknpu_submit.iommu_domain_id) has always let a client spread
allocations across multiple IOMMU paging domains, and our own
ggml-rknpu2 glue (IOMMUDomainManager) already assigns tensors across
up to 16 domains for exactly this reason: each domain is an
independent page-table root with its own ~4GiB IOVA aperture (capped
by the vendor rknn_matmul API's int32 B-matrix offsets, not by IOMMU
hardware). rkopnu was reading and silently dropping this field in both
ioctls, collapsing every allocation into the single domain created
once at open() -- so userspace's domain-spreading was a no-op and any
model with >~4GiB of NPU-resident weights hit ENOSPC ("MEM_CREATE VA
insert failed").
Replace the single per-fd domain with a lazily-created 16-slot pool
keyed by the vendor domain id, each owning its own drm_mm IOVA
allocator (previously one drm_mm shared per-fd; now one per domain,
matching each domain's own independent address space). Wire
iommu_domain_id through in rkopnu_ioctl_mem_create and
rkopnu_ioctl_submit. The domain attach/detach-on-change logic in
rocket_job_run() already existed and needed no changes -- it was
comparing core->attached_domain against job->domain per core with a
skip-if-unchanged optimization, just never fed more than one domain
value before now.
Also fixes two latent bugs surfaced while making domain creation
lazy/fallible (rocket_iommu_domain_get() could never fail before this;
now it can, e.g. -ENOMEM from iommu_paging_domain_alloc):
- rocket_iommu_domain_put(NULL) would crash; made it NULL-tolerant
like the dma_fence_put()/kfree() calls next to it, since
rocket_job_cleanup() unconditionally puts job->domain.
- MEM_CREATE's error paths never dropped the domain reference on
failure (harmless before, since the fd's one domain outlived any
failed allocation regardless; now leaks a whole IOMMU paging domain
per failed MEM_CREATE under memory pressure).
Verified on boltzmann (RK3588, 7.0.0-rc3-npuclk+): Qwen2.5-3B-Instruct
f16 (5.75 GiB, spans >=2 domains) now loads and runs natively on the
NPU at pp512=76.75 t/s -- previously an unconditional GGML_ASSERT abort
in ggml-rknpu2.cpp at ~4GiB. Qwen3-4B-Q8_0 (single-domain, the
existing production path) is unaffected: no regression.
Known issue, not yet fixed: Qwen3-30B-A3B-Instruct-2507 UD-Q4_K_XL
(17.69 GB, ~5 domains) maps cleanly (refcnt to 1221 dmabufs, zero
VA-insert failures, zero IOMMU faults -- the domain packing itself is
correct) but prefill deadlocks: the benchmark process goes permanently
D-state on dma_fence_default_wait, drm_sched's TDR does not visibly
recover it, and the NPU has to be recovered by reboot (never rmmod
rkopnu -- silent deadlock). Root cause not yet confirmed (lockdown
blocked /proc/PID/stack); a hardening patch is staged separately
pending live validation. Until resolved, avoid >4GiB NPU-resident
models with rapid cross-domain access patterns (dense models spanning
2-3 domains sequentially, like the 3B-f16 case above, are fine).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EWpfhDgYNA21tETDP9ueBE
90 lines
3.6 KiB
C
90 lines
3.6 KiB
C
/* SPDX-License-Identifier: GPL-2.0-only */
|
|
/* Copyright 2024-2025 Tomeu Vizoso <tomeu@tomeuvizoso.net> */
|
|
|
|
#ifndef __ROCKET_DRV_H__
|
|
#define __ROCKET_DRV_H__
|
|
|
|
#include <drm/drm_mm.h>
|
|
#include <drm/gpu_scheduler.h>
|
|
#include <linux/iosys-map.h>
|
|
|
|
#include "rocket_device.h"
|
|
|
|
extern const struct dev_pm_ops rocket_pm_ops;
|
|
|
|
/*
|
|
* rkopnu multi-domain: the vendor rknpu ABI (rknpu_mem_create.iommu_domain_id,
|
|
* rknpu_submit.iommu_domain_id -- see rknpu_ioctl.h) lets a client spread
|
|
* allocations across multiple IOMMU paging domains. Each domain is a fully
|
|
* independent page-table root, so each gets its own ~4GiB IOVA aperture (the
|
|
* vendor rknn_matmul API's B-matrix offsets are int32, which is the real
|
|
* ceiling -- not the IOMMU hardware, which is per-domain). Before this, every
|
|
* allocation on a given fd landed in the single domain created at open(), so
|
|
* userspace's domain-spreading (ggml-rknpu2's IOMMUDomainManager) was a
|
|
* no-op and any model with >~4GiB of NPU-resident weights hit ENOSPC. Each
|
|
* domain now owns its own drm_mm IOVA allocator (mm/mm_lock below), matching
|
|
* its own independent address space -- a shared allocator across domains
|
|
* would double-book identical IOVA ranges in different page tables, which is
|
|
* fine as far as the IOMMU is concerned but wrong for this driver's
|
|
* bookkeeping (VA aperture accounting, exhaustion diagnostics).
|
|
*/
|
|
struct rocket_iommu_domain {
|
|
struct iommu_domain *domain;
|
|
struct kref kref;
|
|
|
|
/* vendor iommu_domain_id this slot was created for (0..ROCKET_MAX_IOMMU_DOMAINS-1) */
|
|
s32 id;
|
|
|
|
struct drm_mm mm;
|
|
struct mutex mm_lock;
|
|
};
|
|
|
|
/*
|
|
* Matches ggml-rknpu2's IOMMUDomainManager, which dynamically assigns
|
|
* domain ids 0..15 (see assign_domain_memory() in ggml-rknpu2.cpp). Any
|
|
* out-of-range or unset (old-caller) id falls back to domain 0.
|
|
*/
|
|
#define ROCKET_MAX_IOMMU_DOMAINS 16
|
|
|
|
struct rocket_file_priv {
|
|
struct rocket_device *rdev;
|
|
|
|
/*
|
|
* Lazily-created pool of per-fd IOMMU domains, indexed by the vendor
|
|
* iommu_domain_id. Slots are NULL until first referenced by a
|
|
* MEM_CREATE or SUBMIT that names them. domains_lock protects the
|
|
* pool array itself (slot install/lookup); each domain's own
|
|
* mm_lock protects its drm_mm, and its kref its lifetime.
|
|
*/
|
|
struct mutex domains_lock;
|
|
struct rocket_iommu_domain *domains[ROCKET_MAX_IOMMU_DOMAINS];
|
|
|
|
struct drm_sched_entity sched_entity;
|
|
|
|
/* rkopnu correctness: one drm_sched entity bound to each single core, so a
|
|
* SUBMIT is pinned to the physical core selected by core_mask (the vendor
|
|
* commits the PC to that exact core, whose subcore_task[] slot is the only
|
|
* one librknnrt filled). Letting drm_sched pick an arbitrary core reads the
|
|
* wrong subcore slot -> out-of-bounds task index -> oops. */
|
|
struct drm_sched_entity core_entity[3];
|
|
};
|
|
|
|
/*
|
|
* Returns a referenced (kref_get'd) domain for domain_id, creating it on
|
|
* first use. Caller must rocket_iommu_domain_put() when done. Can return
|
|
* ERR_PTR (e.g. -ENOMEM from iommu_paging_domain_alloc) since domain
|
|
* creation is now lazy instead of guaranteed-at-open -- callers that were
|
|
* written against the old infallible single-domain version need updating,
|
|
* not just recompiling.
|
|
*/
|
|
struct rocket_iommu_domain *rocket_iommu_domain_get(struct rocket_file_priv *rocket_priv, s32 domain_id);
|
|
void rocket_iommu_domain_put(struct rocket_iommu_domain *domain);
|
|
|
|
/* Tears down every domain this fd ever created. Call from postclose, after
|
|
* DRM core has already released this file's GEM objects (so every domain's
|
|
* drm_mm is provably empty -- see rocket_open()/rocket_postclose() in
|
|
* rocket_drv.c for the ordering this relies on). */
|
|
void rocket_iommu_domains_close(struct rocket_file_priv *rocket_priv);
|
|
|
|
#endif
|