Rosenblatt: project scaffold for RK3588 NPU on mainline
Codename: Frank Rosenblatt — Mark I Perceptron 1958, the first
hardware neural network. This project lights up the RK3588 NPU on
mainline Linux so the OSS world finally owns the silicon-side of
inference on that chip.
Phase-1 scope: small LLM running CPU + NPU mix on boltzmann (Rock 5
ITX+). Backend: llama.cpp with a new rknpu ggml backend offloading
INT8 GEMM (attention + FFN matmuls) to the NPU's tile-MAC array while
leaving dequant / RoPE / softmax / sampling / embedding on A76 NEON.
Target model: qwen2.5-1.5B-instruct Q4_K_M GGUF.
Scaffold layout: README.md (frame + 9+1-phase plan), TODO.md (rolling
punch-list), docs/{npu-mainline-status,architecture}.md, kernel/ for
DT bindings + driver tweaks, userspace/{npu-probe,llm-runtime}/,
fleet/boltzmann.yaml.
Next: Phase-1 substrate audit — fill the TBDs in docs/npu-mainline-status.md
with the actual state of Tomeu Vizoso's rknpu / DRM-accel work on
the boltzmann-running kernel.
This commit is contained in:
@@ -0,0 +1,107 @@
|
||||
# Rosenblatt
|
||||
|
||||
**Codename:** Frank Rosenblatt built the Mark I Perceptron in 1958 — the first
|
||||
hardware neural network (400 photocells, stepper-motor-tunable analog weights).
|
||||
This project lights up the RK3588 NPU on mainline Linux, so the OSS world
|
||||
finally owns the silicon-side of inference on that chip.
|
||||
|
||||
**Scope (Phase 1):** small LLM running CPU + NPU mix on `boltzmann` (Rock 5
|
||||
ITX+, RK3588, 32 GB DDR4). Backend: `llama.cpp` with a new `rknpu` device that
|
||||
offloads the heavy GEMM (matmul in attention + FFN) to the NPU's INT8 path
|
||||
while leaving dequant / RoPE / softmax / sampling / embedding lookup on the
|
||||
A76 NEON cores.
|
||||
|
||||
**Target model (Phase 1):**
|
||||
`qwen2.5-1.5B-instruct` Q4_K_M GGUF. Fits in NPU's accessible memory
|
||||
budget, has chat tuning, public license. Stretch: `qwen2.5-3B`,
|
||||
`gemma3-2B`.
|
||||
|
||||
**Out of scope (Phase 1, capture separately if pursued):**
|
||||
- Vision helper (object detection / OCR / face-blur) — different op mix,
|
||||
re-scope after Phase-1 numbers
|
||||
- RKNPU vendor SDK adoption — we want mainline-clean, not vendor-blob
|
||||
- Other Rockchip NPUs (RK3576 has the same NPU IP block — should port for
|
||||
free once the RK3588 path lands, but defer until Phase-1 closes)
|
||||
|
||||
**Not goal: parity with rknn-llm vendor stack on day 1.** Vendor has
|
||||
hand-tuned tensor layouts + quantization; we'll be slower at first. Goal
|
||||
is *credible* — defined as ≥1 tok/s sustained on qwen-1.5B Q4 with the
|
||||
NPU actually doing the bulk of the GEMM work. The number itself isn't the
|
||||
point; the open path to it is.
|
||||
|
||||
---
|
||||
|
||||
## Phases (9 + 1 loop)
|
||||
|
||||
| # | Phase | Deliverable |
|
||||
|---|---|---|
|
||||
| 1 | **Substrate** | Audit mainline NPU driver state (Tomeu Vizoso's rknpu / DRM-accel series); `/dev/accel/*` probe on boltzmann; running kernel + module inventory. `docs/npu-mainline-status.md` snapshot. |
|
||||
| 2 | Formulate | Pick the exact matmul shape that fits the NPU's tile-MAC array. Identify the smallest-possible op-set llama.cpp can offload. |
|
||||
| 3 | Analyze | Read the RKNPU2 SDK + Tomeu's rknpu uAPI to learn: register layout, DMA tensor format, INT8 quant scheme. Don't lift code — extract the spec. |
|
||||
| 4 | Baseline | llama.cpp pure-CPU tok/s on boltzmann for qwen-1.5B Q4_K_M. Three runs, median. Reproducible bench script in `benchmarks/`. |
|
||||
| 5 | Plan | rknpu backend interface design — where it plugs into ggml's compute graph; memory mapping strategy (dmabuf vs userptr); fallback path. |
|
||||
| 6 | Review | Janet (ARM/DRM specialist agent) reviews the NPU register-write + DMA fence strategy. Cold-eyes pass. |
|
||||
| 7 | Implement | rknpu ggml backend skeleton + first INT8 matmul. Bit-exact against CPU reference (Q4_K dequant + fp32 matmul). |
|
||||
| 8 | Verify | Compare tok/s vs Phase-4 baseline. Profile: % time in NPU vs % in CPU vs % stalled on DMA. |
|
||||
| 9 | Closing | Writeup at `dokuwiki.reauktion.de/doku.php?id=rosenblatt`. Benchmarks rendered. Send-to-upstream cover letter draft if quality is there. |
|
||||
| 10 | Memory | `project_rosenblatt.md` in claude-memory: what worked, what to avoid for the next NPU campaign (RK3576 port). |
|
||||
|
||||
Per `feedback_dev_process.md`: rewind to Phase 1 on blocker, Phase 4 on
|
||||
direction change, Phase 0 on scope change.
|
||||
|
||||
---
|
||||
|
||||
## Repo layout
|
||||
|
||||
```
|
||||
rosenblatt/
|
||||
├── README.md this file
|
||||
├── TODO.md rolling punch-list
|
||||
├── docs/
|
||||
│ ├── npu-mainline-status.md Phase-1 audit
|
||||
│ ├── architecture.md CPU+NPU split, ggml backend shape
|
||||
│ └── phases.md per-phase log (analog to ~/src/bin/phases/)
|
||||
├── kernel/ mainline-bound patches (DT bindings, rknpu driver tweaks)
|
||||
├── userspace/
|
||||
│ ├── npu-probe/ smallest-possible "open device + run trivial matmul" sanity
|
||||
│ └── llm-runtime/ llama.cpp fork with rknpu backend
|
||||
├── fleet/
|
||||
│ └── boltzmann.yaml host manifest (kernel + NPU driver pin, baseline measurement)
|
||||
└── benchmarks/ reproducible bench scripts + recorded results (JSON + plots)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Host
|
||||
|
||||
Primary: **boltzmann** (Rock 5 ITX+, RK3588, 32 GB DDR4-2666, NVMe rootfs).
|
||||
- Already runs mainline ~v7.0 with most peripheral drivers working.
|
||||
- Has the Quark UEFI / Neutron kernel stack — NPU is the next missing peripheral.
|
||||
- Other RK3588 hosts (`ampere` = CoolPi GenBook) come later for port-validation.
|
||||
|
||||
Why not `ampere`: laptop, intermittent power, in-use for other campaigns.
|
||||
Boltzmann is always-on with 32 GB headroom — right substrate for kernel
|
||||
hacking with serial-console fallback (when [Quark](https://git.reauktion.de/marfrit/quark) exposes one).
|
||||
|
||||
---
|
||||
|
||||
## Codename rationale
|
||||
|
||||
Rosenblatt's Mark I was custom analog hardware doing fixed-function matmul-
|
||||
adjacent work (weighted-sum + threshold), with weights tunable per slot via
|
||||
mechanical control. The RK3588 NPU is fixed-function INT8 matmul/conv hardware
|
||||
with weights loaded per inference. Same shape, 67 years later, with the same
|
||||
"how do we drive this thing from a general-purpose computer?" problem. The
|
||||
1958 paper's answer was: build a control panel. The 2026 answer is: a DRM
|
||||
accelerator driver + a userspace runtime that maps tensor ops to MMIO + DMA.
|
||||
We're writing the second half.
|
||||
|
||||
---
|
||||
|
||||
## Status
|
||||
|
||||
| Phase | State | Date |
|
||||
|---|---|---|
|
||||
| 0 — bootstrap | done | 2026-05-19 |
|
||||
| 1 — substrate audit | open | |
|
||||
| 2..10 | pending | |
|
||||
Reference in New Issue
Block a user