diff --git a/README.md b/README.md index 4180746..6d40ed3 100644 --- a/README.md +++ b/README.md @@ -5,8 +5,7 @@ hardware neural network (400 photocells, stepper-motor-tunable analog weights). This project lights up the RK3588 NPU on mainline Linux, so the OSS world finally owns the silicon-side of inference on that chip. -**Scope (Phase 1):** small LLM running CPU + NPU mix on `boltzmann` (Rock 5 -ITX+, RK3588, 32 GB DDR4). Backend: `llama.cpp` with a new `rknpu` device that +**Scope (Phase 1):** small LLM running CPU + NPU mix on the Rock 5 ITX+ (RK3588, 32 GB DDR4). Backend: `llama.cpp` with a new `rknpu` device that offloads the heavy GEMM (matmul in attention + FFN) to the NPU's INT8 path while leaving dequant / RoPE / softmax / sampling / embedding lookup on the A76 NEON cores. @@ -35,10 +34,10 @@ point; the open path to it is. | # | Phase | Deliverable | |---|---|---| -| 1 | **Substrate** | Audit mainline NPU driver state (Tomeu Vizoso's rknpu / DRM-accel series); `/dev/accel/*` probe on boltzmann; running kernel + module inventory. `docs/npu-mainline-status.md` snapshot. | +| 1 | **Substrate** | Audit mainline NPU driver state (Tomeu Vizoso's rknpu / DRM-accel series); `/dev/accel/*` probe on the Rock 5 ITX+; running kernel + module inventory. `docs/npu-mainline-status.md` snapshot. | | 2 | Formulate | Pick the exact matmul shape that fits the NPU's tile-MAC array. Identify the smallest-possible op-set llama.cpp can offload. | | 3 | Analyze | Read the RKNPU2 SDK + Tomeu's rknpu uAPI to learn: register layout, DMA tensor format, INT8 quant scheme. Don't lift code — extract the spec. | -| 4 | Baseline | llama.cpp pure-CPU tok/s on boltzmann for qwen-1.5B Q4_K_M. Three runs, median. Reproducible bench script in `benchmarks/`. | +| 4 | Baseline | llama.cpp pure-CPU tok/s on the Rock 5 ITX+ for qwen-1.5B Q4_K_M. Three runs, median. Reproducible bench script in `benchmarks/`. | | 5 | Plan | rknpu backend interface design — where it plugs into ggml's compute graph; memory mapping strategy (dmabuf vs userptr); fallback path. | | 6 | Review | Janet (ARM/DRM specialist agent) reviews the NPU register-write + DMA fence strategy. Cold-eyes pass. | | 7 | Implement | rknpu ggml backend skeleton + first INT8 matmul. Bit-exact against CPU reference (Q4_K dequant + fp32 matmul). | @@ -66,7 +65,7 @@ rosenblatt/ │ ├── npu-probe/ smallest-possible "open device + run trivial matmul" sanity │ └── llm-runtime/ llama.cpp fork with rknpu backend ├── fleet/ -│ └── boltzmann.yaml host manifest (kernel + NPU driver pin, baseline measurement) +│ └── rock5-itx-plus.yaml host manifest (kernel + NPU driver pin, baseline measurement) └── benchmarks/ reproducible bench scripts + recorded results (JSON + plots) ``` @@ -74,13 +73,13 @@ rosenblatt/ ## Host -Primary: **boltzmann** (Rock 5 ITX+, RK3588, 32 GB DDR4-2666, NVMe rootfs). +Primary: **Rock 5 ITX+** (RK3588, 32 GB DDR4-2666, NVMe rootfs). - Already runs mainline ~v7.0 with most peripheral drivers working. - Has the Quark UEFI / Neutron kernel stack — NPU is the next missing peripheral. - Other RK3588 hosts (`ampere` = CoolPi GenBook) come later for port-validation. Why not `ampere`: laptop, intermittent power, in-use for other campaigns. -Boltzmann is always-on with 32 GB headroom — right substrate for kernel +The Rock 5 ITX+ is always-on with 32 GB headroom — right substrate for kernel hacking with serial-console fallback (when [Quark](https://git.reauktion.de/marfrit/quark) exposes one). --- @@ -109,7 +108,7 @@ We're writing the second half. Phase-1 closeout: `docs/phases.md` + `docs/npu-mainline-status.md`. Headline: mainline driver name is **`rocket`** (not `rknpu`); it's -already shipped in boltzmann's kernel as a built module. Phase-2 +already shipped in the Rock 5 ITX+'s kernel as a built module. Phase-2 unblock is small (DT enable + IOMMU v1.0 mitigation + modprobe), not a driver port. diff --git a/REPLICATION.md b/REPLICATION.md index c86b22e..0ad7369 100644 --- a/REPLICATION.md +++ b/REPLICATION.md @@ -2,7 +2,7 @@ Run `llama.cpp` LLM inference on the **RK3588 NPU via the mainline `rocket` DRM-accel driver** — no vendor `librknnrt` blob — at 1 GHz. Measured on -`boltzmann` (Radxa Rock 5 ITX+, RK3588, 32 GB): gemma-4-E2B-it-Q8_0 +the Rock 5 ITX+ (RK3588, 32 GB): gemma-4-E2B-it-Q8_0 **pp512 ≈ 46.7 t/s**, within ~9 % of the vendor closed stack (51.4), both DDR-bandwidth-bound. @@ -150,7 +150,7 @@ output — but pp512-neutral, being bandwidth-bound). --- -## 3. Results & findings (boltzmann, gemma-4-E2B-it-Q8_0, NPU @ 1 GHz) +## 3. Results & findings (the Rock 5 ITX+, gemma-4-E2B-it-Q8_0, NPU @ 1 GHz) | backend | pp512 (t/s) | tg128 | |---|---|---| diff --git a/TODO.md b/TODO.md index f1cf68d..a277990 100644 --- a/TODO.md +++ b/TODO.md @@ -6,7 +6,7 @@ Rolling punch-list. Older items at bottom (move done → DONE.md when noisy). ## Phase 1 — substrate audit -- [ ] On boltzmann: `uname -r` → record in `fleet/boltzmann.yaml:kernel.running_version` +- [ ] On the Rock 5 ITX+: `uname -r` → record in `fleet/rock5-itx-plus.yaml:kernel.running_version` - [ ] `find / -path '*accel*' -name '*.ko' 2>/dev/null` — check if accel framework is built - [ ] `ls /dev/accel/ /dev/dri/` — what's exposed? - [ ] `lsmod | grep -iE 'rknpu|accel'` — what's loaded? @@ -56,11 +56,11 @@ clear answer to "do we drive via accel uAPI or write our own MMIO driver." ## Phase 4 — baseline -- [ ] Build vanilla llama.cpp on boltzmann (mainline branch) +- [ ] Build vanilla llama.cpp on the Rock 5 ITX+ (mainline branch) - [ ] Pull qwen2.5-1.5b-instruct Q4_K_M GGUF - [ ] `llama-bench -m qwen2.5-1.5b -p 512 -n 128` × 3 runs -- [ ] Capture JSON to `benchmarks/$(date +%F)_boltzmann_qwen1.5b_cpu_baseline.json` -- [ ] Record into `fleet/boltzmann.yaml:baseline_measurement` +- [ ] Capture JSON to `benchmarks/$(date +%F)_rock5-itx-plus_qwen1.5b_cpu_baseline.json` +- [ ] Record into `fleet/rock5-itx-plus.yaml:baseline_measurement` --- @@ -70,7 +70,7 @@ Phase-1 audit (2026-05-19) reframed Phase-2 from "design rknpu backend interface" to a concrete bringup sequence. See `docs/npu-mainline-status.md` for full context. -- [ ] Patch boltzmann board DTS / overlay to flip +- [ ] Patch the Rock 5 ITX+ board DTS / overlay to flip `npu@fdab0000`, `npu@fdac0000`, `npu@fdad0000` from `status = "disabled"` → `"okay"`. Rebuild DTB. - [ ] **Mitigate IOMMU v1.0 hazard before first NPU job** (32 GB host). @@ -102,8 +102,8 @@ interface" to a concrete bringup sequence. See - [ ] Mirror Tomeu's branch — superseded: code is now in-tree. Keep `git.kernel.org/.../torvalds/linux.git` checkout pinned to - the boltzmann kernel rev for in-tree reading. -- [ ] Set up serial console on boltzmann for kernel-panic recovery + the the Rock 5 ITX+ kernel rev for in-tree reading. +- [ ] Set up serial console on the Rock 5 ITX+ for kernel-panic recovery (Quark umbrella; check current state) — **becomes load-bearing once we start poking IOMMU code.** - [x] Add `project_rosenblatt_overview.md` + `project_rocket_upstream_state.md` @@ -111,7 +111,7 @@ interface" to a concrete bringup sequence. See - [ ] Decide repo home: marfrit/rosenblatt on git.reauktion.de (probably yes, after Phase-1 substrate is captured). - [ ] **Resolve board-name discrepancy.** README and - `fleet/boltzmann.yaml` say boltzmann is a "Rock 5 ITX+" / + `fleet/rock5-itx-plus.yaml` say the manifest records the board type / `rock-5-itx-plus`; the running DT reports `model = "Radxa ROCK 5 ITX"`, `compatible = "radxa,rock-5-itx"`. Confirm physical board model (Radxa sells both SKUs) and diff --git a/docs/architecture.md b/docs/architecture.md index d392328..60b5e06 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -74,7 +74,7 @@ Two paths: Phase-1 choice: (2) — straightforward, makes the NPU path the only thing happening at inference time, easier to profile. The memory cost -on 1.5B is ~1.5 GB INT8 weights vs ~900 MB Q4_K_M — boltzmann has +on 1.5B is ~1.5 GB INT8 weights vs ~900 MB Q4_K_M — the Rock 5 ITX+ has 32 GB, this isn't the constraint. Phase-2+: revisit (1) if we go for larger models or if INT8 turns out diff --git a/docs/npu-mainline-status.md b/docs/npu-mainline-status.md index d76160d..f0f2bdf 100644 --- a/docs/npu-mainline-status.md +++ b/docs/npu-mainline-status.md @@ -6,11 +6,11 @@ > blocker hits. **Audit date:** 2026-05-19 -**Kernel under audit:** boltzmann, `7.0.0-rc3-ARCH+ #2 SMP PREEMPT_DYNAMIC Wed Apr 29 11:16:17 CEST 2026 aarch64` (image `linux-rk3588-marfrit-A1`) +**Kernel under audit:** the Rock 5 ITX+, `7.0.0-rc3-ARCH+ #2 SMP PREEMPT_DYNAMIC Wed Apr 29 11:16:17 CEST 2026 aarch64` (image `linux-rk3588-marfrit-A1`) --- -## Local-host findings (boltzmann) +## Local-host findings (the Rock 5 ITX+) The custom `linux-rk3588-marfrit-A1` build is mainline-rc3-based and **already ships Tomeu Vizoso's upstream in-tree NPU accel driver** as a loadable module: @@ -67,15 +67,15 @@ forward. Tomeu Vizoso's RK3588 NPU work landed in **Linux 6.18 (Nov 2025)** under the codename **`rocket`** (not `rknpu`). The matching userspace shipped in **Mesa 25.3** (Rocket Gallium driver + Teflon TFLite delegate). -Boltzmann at 7.0.0-rc3 is one major release past the merge. +The Rock 5 ITX+ at 7.0.0-rc3 is one major release past the merge. | Component | Mainline state | URL | Notes | |---|---|---|---| | `drivers/accel/rocket/` | **in-tree since 6.18** | · [kernel docs](https://docs.kernel.org/accel/rocket/index.html) | Author Tomeu Vizoso. Files: `rocket_core.c`, `_device.c`, `_drv.c`, `_gem.c`, `_job.c`, `_registers.h`. Kconfig `DRM_ACCEL_ROCKET`. | | DT binding `rockchip,rk3588-rknn-core.yaml` | in-tree (6.18 pull) | [v8 06/10 patch](https://www.mail-archive.com/dri-devel@lists.freedesktop.org/msg554406.html) | Path `Documentation/devicetree/bindings/npu/`. Requires 3 reg blocks (pc/cna/core), 4 clocks, IOMMU, power-domain, resets. | | `pd_npu` power-domain in `rk3588-base.dtsi` | in-tree (6.18 pull) | [v8 07/10](https://www.mail-archive.com/dri-devel@lists.freedesktop.org/msg554409.html) | Label added so board files can wire a regulator instead of leaving the domain permanently on. | -| `npu@fdab/c/d_0000` nodes in `rk3588.dtsi` | in-tree (boltzmann's DT has all 3) | observed at `/sys/firmware/devicetree/base/npu@fdab0000` etc. | All `compatible = "rockchip,rk3588-rknn-core"`, **`status = "disabled"`** in the DT boltzmann is booting. | -| Per-board enable on Rock 5 ITX+ (boltzmann) | **not done** | n/a | Three `rknn-core` nodes ship disabled; need a board overlay or DTS edit to set `status = "okay"` before `rocket` can probe. | +| `npu@fdab/c/d_0000` nodes in `rk3588.dtsi` | in-tree (the Rock 5 ITX+'s DT has all 3) | observed at `/sys/firmware/devicetree/base/npu@fdab0000` etc. | All `compatible = "rockchip,rk3588-rknn-core"`, **`status = "disabled"`** in the DT the Rock 5 ITX+ is booting. | +| Per-board enable on the Rock 5 ITX+ | **not done** | n/a | Three `rknn-core` nodes ship disabled; need a board overlay or DTS edit to set `status = "okay"` before `rocket` can probe. | | Userspace consumer (Mesa Rocket + Teflon) | merged in **Mesa 25.3** | [Tomeu blog "we are in mainline"](https://blog.tomeuvizoso.net/2025/07/rockchip-npu-update-6-we-are-in-mainline.html) · [Mesa Teflon docs](https://docs.mesa3d.org/teflon.html) | TFLite delegate; the authoritative reference for how jobs are encoded and submitted over the rocket uAPI. | | v8 series cover letter (final revision) | **merged via DRM-misc-next → 6.18** | [v8 cover](https://www.mail-archive.com/dri-devel@lists.freedesktop.org/msg554401.html) · [Patchew](https://patchew.org/linux/20250713-6-10-rocket-v8-0-64fa3115e910@tomeuvizoso.net/) · [LWN coverage](https://lwn.net/Articles/1029800/) | 10 patches. | | BSP `rockchip-linux/rknpu2` (vendor) | out-of-tree, deprecated, superseded by `airockchip/rknn-toolkit2` | · | BSP `rknpu` driver under `drivers/rknpu/` in 5.10/6.1 BSP branches. Register-map partly readable from source; TRM fills gaps. Use as spec-extraction reference per `feedback_megabitchip_semantic_match`. | @@ -89,7 +89,7 @@ Boltzmann at 7.0.0-rc3 is one major release past the merge. The Phase-1 exit question — "accel uAPI vs own MMIO driver" — answers itself with the upstream state we just inventoried: -- `rocket.ko` is already on disk on boltzmann +- `rocket.ko` is already on disk on the Rock 5 ITX+ (`/lib/modules/7.0.0-rc3-ARCH+/kernel/drivers/accel/rocket/rocket.ko`, `intree: Y`) and aliased to the exact `rockchip,rk3588-rknn-core` compatible string the DT uses. @@ -99,7 +99,7 @@ itself with the upstream state we just inventoried: reimplementing the (already-reviewed) `pd_npu` power-domain sequencing, and owning forward-compatibility burden with zero upstream path. -**Unblock path:** patch the boltzmann board DTS (or apply an overlay) to +**Unblock path:** patch the the Rock 5 ITX+ board DTS (or apply an overlay) to flip the three `npu@fdab/c/d_0000` nodes to `status = "okay"`, rebuild the DTB, reboot, `modprobe rocket`, expect `/dev/accel/accel0..2` to appear and corresponding `dmesg` probe lines. That's a Phase-2 task. @@ -145,7 +145,7 @@ VOP2 uses v2.0. Both map 40-bit physical pages, but **v1.0 does not support placing the first-level page table (DTE) above 4 GB**. The current DT binding `rockchip,rk3568-iommu` does **not** distinguish -the two — same compat for both IP versions. On `boltzmann` all three +the two — same compat for both IP versions. On the Rock 5 ITX+ all three NPU IOMMUs are bound to this compat (with `rockchip,rk3588-iommu` fallback), so they will be driven by the v2.0 code path: @@ -155,7 +155,7 @@ fallback), so they will be driven by the v2.0 code path: /sys/firmware/devicetree/base/iommu@fdada000 compatible: rockchip,rk3588-iommu rockchip,rk3568-iommu status: (none → okay default) ``` -With `MemTotal = 32 GB` on boltzmann, two failure modes are reachable +With `MemTotal = 32 GB` on the Rock 5 ITX+, two failure modes are reachable the moment we enable the NPU: 1. **DTE allocated above `0x100000000`** → v1.0 hardware silently @@ -193,7 +193,7 @@ discriminator-compat approach is correct but not yet upstream. **Implication for Rosenblatt Phase 2:** the README's Phase-2 unblock plan ("enable DT nodes → `modprobe rocket` → `/dev/accel/accel0..2`") will **almost certainly trigger one of the two failure modes** on -32 GB boltzmann. Three viable workarounds: +32 GB the Rock 5 ITX+. Three viable workarounds: - **A. Boot with `mem=4GB`** (degraded but valid Phase-2 path; lets us validate the rocket bringup end-to-end without addressing the IOMMU diff --git a/docs/phases.md b/docs/phases.md index 0976513..6d9d84a 100644 --- a/docs/phases.md +++ b/docs/phases.md @@ -20,7 +20,7 @@ for RK3588 NPU on mainline`. **Closed:** 2026-05-19 **Deliverable:** `docs/npu-mainline-status.md` table fully populated; -`fleet/boltzmann.yaml` kernel/NPU-driver block filled with live data; +`fleet/rock5-itx-plus.yaml` kernel/NPU-driver block filled with live data; clear answer to the "accel uAPI vs. own MMIO driver" question. ### Findings @@ -30,7 +30,7 @@ clear answer to the "accel uAPI vs. own MMIO driver" question. Kconfig `DRM_ACCEL_ROCKET`. Driver author + history kept the same shape, but the **upstream name is `rocket`, not `rknpu`** — searching for `rknpu` in mainline misses everything. -2. **Boltzmann already ships the driver** — `linux-rk3588-marfrit-A1` +2. **The Rock 5 ITX+ already ships the driver** — `linux-rk3588-marfrit-A1` (7.0.0-rc3-ARCH+) is post-6.18 and contains `rocket.ko` at `/lib/modules/.../drivers/accel/rocket/rocket.ko`, marked `intree: Y`. Module aliases to `rockchip,rk3588-rknn-core`, @@ -65,7 +65,7 @@ clear answer to the "accel uAPI vs. own MMIO driver" question. 8. **IOMMU v1.0 hazard surfaced from `linux-rockchip` thread** (Midgy BALON / Simon Xue, 2026-04-03). The NPU IOMMU is v1.0 IP bound to generic `rockchip,rk3568-iommu` — driven via the v2.0 - code path. v1.0 can't allocate its DTE above 4 GB. Boltzmann + code path. v1.0 can't allocate its DTE above 4 GB. The Rock 5 ITX+ has 32 GB. Naive enable will silently fault. Discriminator-compat patch series planned but **not landed in mainline master as of 2026-05-19** (verified via cgit on @@ -88,7 +88,7 @@ list is short: DT enable + IOMMU mitigation + `modprobe rocket`. Captured in `TODO.md` "Phase-2 unblock" section. Highlights: - Apply `kernel/dt-overlays/rk3588-rosenblatt-npu-enable.dtso` (or - equivalent board-DTS patch) to boltzmann. + equivalent board-DTS patch) to the Rock 5 ITX+. - Mitigate IOMMU v1.0 hazard before first NPU job: `mem=4G` boot or local discriminator-compat carry. - `modprobe rocket`, confirm `/dev/accel/accel0`, no IOMMU faults. diff --git a/fleet/boltzmann.yaml b/fleet/rock5-itx-plus.yaml similarity index 83% rename from fleet/boltzmann.yaml rename to fleet/rock5-itx-plus.yaml index fa0083f..39c073c 100644 --- a/fleet/boltzmann.yaml +++ b/fleet/rock5-itx-plus.yaml @@ -1,14 +1,14 @@ -# rosenblatt fleet manifest — boltzmann (Rock 5 ITX+, RK3588) +# rosenblatt fleet manifest — rock5-itx-plus (Rock 5 ITX+, RK3588) # # Phase-1 audit host. Always-on, 32 GB DDR4, NVMe rootfs. NPU silicon # present + accessible via Rockchip-BSP vendor module today; mainline # path TBD (see docs/npu-mainline-status.md). -host: boltzmann +host: rock5-itx-plus arch: arm64 soc: rockchip/rk3588 board: rock-5-itx-plus -distro: archlinuxarm # ALARM aarch64; boltzmann is the umbrella RK3588 host +distro: archlinuxarm # ALARM aarch64; rock5-itx-plus is the umbrella RK3588 host role: primary-development # not yet primary-target (laptop targets land later) hardware: @@ -57,7 +57,7 @@ baseline_measurement: target: | llama.cpp pure-CPU tok/s on qwen2.5-1.5b-instruct-q4_k_m.gguf, 3 runs, median wallclock. Use llama-bench from llama.cpp/build/bin. - ground_truth_file: benchmarks/2026-XX-XX_boltzmann_qwen1.5b_cpu_baseline.json + ground_truth_file: benchmarks/2026-XX-XX_rock5-itx-plus_qwen1.5b_cpu_baseline.json bringup_sequence: 1: substrate audit (docs/npu-mainline-status.md table filled) @@ -71,6 +71,6 @@ bringup_sequence: backup_host: ampere # CoolPi GenBook — port-validation target. Phase-2+ scope. reverse_dependencies: - - Quark (boltzmann UEFI) — must stay bootable across kernel-rev experiments - - Neutron (boltzmann kernel build) — provides the kernel we tweak for rknpu - - Volta (boltzmann umbrella) — Rosenblatt is the third Volta-child after Quark + Neutron + - Quark (rock5-itx-plus UEFI) — must stay bootable across kernel-rev experiments + - Neutron (rock5-itx-plus kernel build) — provides the kernel we tweak for rknpu + - Volta (rock5-itx-plus umbrella) — Rosenblatt is the third Volta-child after Quark + Neutron diff --git a/kernel/README.md b/kernel/README.md index a28be38..7963e95 100644 --- a/kernel/README.md +++ b/kernel/README.md @@ -2,7 +2,7 @@ Phase-1 audit closed the "do we need a driver?" question: Tomeu Vizoso's `rocket` driver is in mainline since Linux 6.18, and -boltzmann's `linux-rk3588-marfrit-A1` build (7.0.0-rc3-ARCH+) +the Rock 5 ITX+'s `linux-rk3588-marfrit-A1` build (7.0.0-rc3-ARCH+) already ships it as a module. See `docs/npu-mainline-status.md` for the full audit. @@ -29,4 +29,4 @@ Anticipated next additions: Patch handling routes through the `kernel-agent` subagent (per `feedback_agent_routing.md`); board / host wiring lives in -`fleet/boltzmann.yaml`. +`fleet/rock5-itx-plus.yaml`. diff --git a/kernel/dt-overlays/rk3588-rosenblatt-npu-enable.dtso b/kernel/dt-overlays/rk3588-rosenblatt-npu-enable.dtso index 0635aef..5cd7850 100644 --- a/kernel/dt-overlays/rk3588-rosenblatt-npu-enable.dtso +++ b/kernel/dt-overlays/rk3588-rosenblatt-npu-enable.dtso @@ -3,11 +3,11 @@ * Rosenblatt: enable all three RK3588 RKNN cores + their IOMMUs. * * Apply on top of any rk3588 board DT that uses the mainline - * rk3588-base.dtsi labels. Verified against boltzmann + * rk3588-base.dtsi labels. Verified against the Rock 5 ITX+ * (model "Radxa ROCK 5 ITX", compatible "radxa,rock-5-itx"). * * Pre-conditions before applying on a system with >4 GB RAM - * (boltzmann has 32 GB): + * (the Rock 5 ITX+ has 32 GB): * - The IOMMU v1.0 hazard MUST be mitigated first. * See docs/npu-mainline-status.md "IOMMU v1.0 hazard". * Either boot with `mem=4G`, OR carry the discriminator-compat