marfrit
24adc74812
Codename: Frank Rosenblatt — Mark I Perceptron 1958, the first
hardware neural network. This project lights up the RK3588 NPU on
mainline Linux so the OSS world finally owns the silicon-side of
inference on that chip.
Phase-1 scope: small LLM running CPU + NPU mix on boltzmann (Rock 5
ITX+). Backend: llama.cpp with a new rknpu ggml backend offloading
INT8 GEMM (attention + FFN matmuls) to the NPU's tile-MAC array while
leaving dequant / RoPE / softmax / sampling / embedding on A76 NEON.
Target model: qwen2.5-1.5B-instruct Q4_K_M GGUF.
Scaffold layout: README.md (frame + 9+1-phase plan), TODO.md (rolling
punch-list), docs/{npu-mainline-status,architecture}.md, kernel/ for
DT bindings + driver tweaks, userspace/{npu-probe,llm-runtime}/,
fleet/boltzmann.yaml.
Next: Phase-1 substrate audit — fill the TBDs in docs/npu-mainline-status.md
with the actual state of Tomeu Vizoso's rknpu / DRM-accel work on
the boltzmann-running kernel.
Rosenblatt
Codename: Frank Rosenblatt built the Mark I Perceptron in 1958 — the first hardware neural network (400 photocells, stepper-motor-tunable analog weights). This project lights up the RK3588 NPU on mainline Linux, so the OSS world finally owns the silicon-side of inference on that chip.
Scope (Phase 1): small LLM running CPU + NPU mix on boltzmann (Rock 5
ITX+, RK3588, 32 GB DDR4). Backend: llama.cpp with a new rknpu device that
offloads the heavy GEMM (matmul in attention + FFN) to the NPU's INT8 path
while leaving dequant / RoPE / softmax / sampling / embedding lookup on the
A76 NEON cores.
Target model (Phase 1):
qwen2.5-1.5B-instruct Q4_K_M GGUF. Fits in NPU's accessible memory
budget, has chat tuning, public license. Stretch: qwen2.5-3B,
gemma3-2B.
Out of scope (Phase 1, capture separately if pursued):
- Vision helper (object detection / OCR / face-blur) — different op mix, re-scope after Phase-1 numbers
- RKNPU vendor SDK adoption — we want mainline-clean, not vendor-blob
- Other Rockchip NPUs (RK3576 has the same NPU IP block — should port for free once the RK3588 path lands, but defer until Phase-1 closes)
Not goal: parity with rknn-llm vendor stack on day 1. Vendor has hand-tuned tensor layouts + quantization; we'll be slower at first. Goal is credible — defined as ≥1 tok/s sustained on qwen-1.5B Q4 with the NPU actually doing the bulk of the GEMM work. The number itself isn't the point; the open path to it is.
Phases (9 + 1 loop)
| # | Phase | Deliverable |
|---|---|---|
| 1 | Substrate | Audit mainline NPU driver state (Tomeu Vizoso's rknpu / DRM-accel series); /dev/accel/* probe on boltzmann; running kernel + module inventory. docs/npu-mainline-status.md snapshot. |
| 2 | Formulate | Pick the exact matmul shape that fits the NPU's tile-MAC array. Identify the smallest-possible op-set llama.cpp can offload. |
| 3 | Analyze | Read the RKNPU2 SDK + Tomeu's rknpu uAPI to learn: register layout, DMA tensor format, INT8 quant scheme. Don't lift code — extract the spec. |
| 4 | Baseline | llama.cpp pure-CPU tok/s on boltzmann for qwen-1.5B Q4_K_M. Three runs, median. Reproducible bench script in benchmarks/. |
| 5 | Plan | rknpu backend interface design — where it plugs into ggml's compute graph; memory mapping strategy (dmabuf vs userptr); fallback path. |
| 6 | Review | Janet (ARM/DRM specialist agent) reviews the NPU register-write + DMA fence strategy. Cold-eyes pass. |
| 7 | Implement | rknpu ggml backend skeleton + first INT8 matmul. Bit-exact against CPU reference (Q4_K dequant + fp32 matmul). |
| 8 | Verify | Compare tok/s vs Phase-4 baseline. Profile: % time in NPU vs % in CPU vs % stalled on DMA. |
| 9 | Closing | Writeup at dokuwiki.reauktion.de/doku.php?id=rosenblatt. Benchmarks rendered. Send-to-upstream cover letter draft if quality is there. |
| 10 | Memory | project_rosenblatt.md in claude-memory: what worked, what to avoid for the next NPU campaign (RK3576 port). |
Per feedback_dev_process.md: rewind to Phase 1 on blocker, Phase 4 on
direction change, Phase 0 on scope change.
Repo layout
rosenblatt/
├── README.md this file
├── TODO.md rolling punch-list
├── docs/
│ ├── npu-mainline-status.md Phase-1 audit
│ ├── architecture.md CPU+NPU split, ggml backend shape
│ └── phases.md per-phase log (analog to ~/src/bin/phases/)
├── kernel/ mainline-bound patches (DT bindings, rknpu driver tweaks)
├── userspace/
│ ├── npu-probe/ smallest-possible "open device + run trivial matmul" sanity
│ └── llm-runtime/ llama.cpp fork with rknpu backend
├── fleet/
│ └── boltzmann.yaml host manifest (kernel + NPU driver pin, baseline measurement)
└── benchmarks/ reproducible bench scripts + recorded results (JSON + plots)
Host
Primary: boltzmann (Rock 5 ITX+, RK3588, 32 GB DDR4-2666, NVMe rootfs).
- Already runs mainline ~v7.0 with most peripheral drivers working.
- Has the Quark UEFI / Neutron kernel stack — NPU is the next missing peripheral.
- Other RK3588 hosts (
ampere= CoolPi GenBook) come later for port-validation.
Why not ampere: laptop, intermittent power, in-use for other campaigns.
Boltzmann is always-on with 32 GB headroom — right substrate for kernel
hacking with serial-console fallback (when Quark exposes one).
Codename rationale
Rosenblatt's Mark I was custom analog hardware doing fixed-function matmul- adjacent work (weighted-sum + threshold), with weights tunable per slot via mechanical control. The RK3588 NPU is fixed-function INT8 matmul/conv hardware with weights loaded per inference. Same shape, 67 years later, with the same "how do we drive this thing from a general-purpose computer?" problem. The 1958 paper's answer was: build a control panel. The 2026 answer is: a DRM accelerator driver + a userspace runtime that maps tensor ops to MMIO + DMA. We're writing the second half.
Status
| Phase | State | Date |
|---|---|---|
| 0 — bootstrap | done | 2026-05-19 |
| 1 — substrate audit | open | |
| 2..10 | pending |