319 lines
16 KiB
Markdown
319 lines
16 KiB
Markdown
# Fourier — Hardware-Assisted Video Decoding on the Rockchip Fleet
|
||
|
||
One umbrella project to bring up working HW video decode on all four Rockchip
|
||
devices in Markus' fleet. Named 2026-04-24.
|
||
|
||
## Target fleet
|
||
|
||
| Host | SoC | Role | Parent umbrella |
|
||
|-----------|--------|---------------------------|-----------------|
|
||
| fresnel | RK3399 | Pinebook Pro fleet laptop | — |
|
||
| ohm | RK3566 | PineTab2 tablet | — |
|
||
| boltzmann | RK3588 | Rock 5 ITX+ (always on) | Volta |
|
||
| ampere | RK3588 | CoolPi GenBook laptop | Coulomb |
|
||
|
||
Current priority: **ohm**. The other three follow in the order of easiest-win
|
||
(fresnel, mature mainline) → incremental (ampere, RK3588 landed Feb 2026) →
|
||
hardest (boltzmann, currently on vendor kernel).
|
||
|
||
## Related Rockchip projects
|
||
|
||
Fourier touches hardware that already has its own umbrellas. Keep display/boot
|
||
concerns in those projects; keep video decode in Fourier.
|
||
|
||
- **Coulomb** — ampere / CoolPi GenBook stack. Subprojects:
|
||
- **Bin** — u-boot, display bringup (eDP + keyboard)
|
||
- **MegabitChip** — DDR init blob matching-decomp / HIL
|
||
- **RockHard** — mainline kernel (Collabora TF-A, LP5-3200 OC)
|
||
- **Volta** — boltzmann / Rock 5 ITX+ stack. Subprojects:
|
||
- **Quark** — edk2-rk3588 UEFI firmware
|
||
- **Neutron** — mainline / vendor kernel, UEFI-booted
|
||
- **fresnel** — Pinebook Pro, custom overclocked kernel package (no umbrella)
|
||
- **ohm** — PineTab2, DanctNIX base, UART capture rig for ampere DDR work
|
||
|
||
Cross-cutting notes in `/home/mfritsche/.claude/projects/-home-mfritsche-claude/memory/`
|
||
— see `MEMORY.md` for the index.
|
||
|
||
## Meet His
|
||
|
||
When Fourier needs infra muscle (wake a sleeping host, reach a VPN peer, bump
|
||
DHCP on the Fritz, find the right lmcp token, set up distcc for a kernel
|
||
build), summon **His** — the Home Infrastructure Specialist.
|
||
|
||
- As a subagent: `Agent(subagent_type: "his", prompt: "...")` — takes over a
|
||
task end-to-end and returns a report.
|
||
- As a skill: `Skill(skill: "his")` — loads the runbook cheatsheet into the
|
||
current session.
|
||
|
||
His knows the distcc workers (tesla, CT108, dcc1), the Fritz!DECT plug blast
|
||
radii (careful with Himbeere + Office), the `/opt/herding/` layout on hertz,
|
||
the lmcp endpoint token map, and the VPN naming (`<host>.vpn` via shannon's
|
||
dnsmasq). Canonical runbook: `/home/mfritsche/claude/CLAUDE.md` on noether.
|
||
|
||
For Fourier specifically, expect to lean on His for: waking ohm over LAN +
|
||
VPN, confirming ampere is at .vpn (not the stale .88.x LAN IP), setting up
|
||
cross-builds of patched ffmpeg via distcc, and nudging kernel packages into
|
||
the marfrit repo.
|
||
|
||
## Two kernel paths
|
||
|
||
Rockchip hardware decode has **two incompatible kernel driver families**.
|
||
Userspace config diverges accordingly — don't mix flags.
|
||
|
||
### Path A — mainline V4L2 stateless
|
||
|
||
- Kernel: `rkvdec` / `rkvdec2` stateless V4L2 drivers (mainline or out-of-tree
|
||
patch series).
|
||
- /dev node: V4L2 m2m `/dev/videoN` with compressed H.264/HEVC/VP9 queues +
|
||
uncompressed output queue.
|
||
- FFmpeg: `ffmpeg-v4l2-request-git` from AUR (Jernej's patchset). Upstream
|
||
v2 patchset posted on ffmpeg-devel 2024-08, **not yet merged** as of
|
||
2026-04-24.
|
||
- GStreamer: `v4l2codecs` plugin in gst-plugins-bad. 1.26 added Rockchip
|
||
buffer formats; 1.28 added the new HEVC UAPI controls.
|
||
- mpv: `mpv --hwdec=drm --vo=gpu-next` (wayland compositor) or
|
||
`--vo=dmabuf-wayland` (wlroots).
|
||
- Kodi: native V4L2-request API support, independent of FFmpeg.
|
||
- VA-API: `libva-v4l2-request` bridge for Firefox / Chromium paths.
|
||
|
||
### Path B — Rockchip MPP vendor
|
||
|
||
- Kernel: `mpp_rkvdec` / `mpp_rkvdec2` from Rockchip BSP (the rkr* kernel
|
||
series).
|
||
- /dev node: `/dev/mpp_service` chardev, not V4L2.
|
||
- FFmpeg: build with `--enable-rkmpp` against `librockchip-mpp`.
|
||
- GStreamer: Rockchip-specific plugins (`rkximagesink`, `mppvideodec`, …).
|
||
- Present today on **boltzmann** (kernel 6.1.75-rkr3).
|
||
|
||
## rkvdec mainline status (2026-04-24)
|
||
|
||
| SoC | Block | Mainline status | Codecs (mainline) |
|
||
|---------|--------------------|------------------------------|------------------------------------------------------|
|
||
| RK3399 | Hantro G1/G2 | mature | MPEG-2, VP8, H.264, HEVC, VP9 |
|
||
| RK3566 | rkvdec2 (VDPU346) | **not yet** (Collabora TODO) | rkvdec1: MPEG-2/VP8/H.264; HEVC/VP9 need out-of-tree |
|
||
| RK3588 | VDPU381 | **merged 2026-02-27** | H.264, H.265 (no VP9/AV1 upstream yet) |
|
||
| RK3576 | VDPU383 | merged 2026-02-27 | H.264, H.265 |
|
||
|
||
The RK3588 / RK3576 landing was the Feb 2026 Collabora drop. VDPU346 for the
|
||
RK356x family is on the roadmap but not merged; ohm depends on DanctNIX
|
||
carrying out-of-tree rkvdec2 patches if we want HEVC/VP9.
|
||
|
||
## ohm — first priority, known recipe
|
||
|
||
From Martin Chang's blog (clehaxze.tw, 2023, still mirrored by DanctNIX):
|
||
|
||
```sh
|
||
sudo pacman -S mpv
|
||
yay -S ffmpeg-v4l2-request-git
|
||
mpv --hwdec=drm --vo=gpu-next --wayland-disable-vsync=yes input.mp4
|
||
# wlroots compositor variant:
|
||
mpv --hwdec=drm --vo=dmabuf-wayland input.mp4
|
||
```
|
||
|
||
2023 coverage: **MPEG-2, VP8, H.264**. 1080p60 H.264 hit ~80% of one CPU core
|
||
— decode was on the hardware; the CPU was in the compositor path.
|
||
|
||
**Open question**: does the current linux-pinetab2 6.15-danctnix2 carry
|
||
out-of-tree rkvdec2 patches for HEVC/VP9, and if so, does the 2023 recipe
|
||
still work verbatim? Next live recon on ohm answers this.
|
||
|
||
### Plan (tasks #18–24 in the noether task list)
|
||
|
||
1. **Live recon** — kernel, `/dev/video*`, `v4l2-ctl --list-devices`, dmesg
|
||
rkvdec, `/lib/firmware/rockchip/`, installed ffmpeg/gstreamer/mpv/kodi
|
||
2. **SW baseline** — 1080p H.264 / HEVC / VP9 benchmark with ffmpeg and mpv
|
||
hwdec=no, record CPU% + fps
|
||
3. **Driver binding** — confirm V4L2 stateless decode node exposed; if not,
|
||
diagnose kconfig / DT / firmware
|
||
4. **GStreamer v4l2codecs** — cleanest proof; `v4l2slh264dec` pipeline with
|
||
`fakesink`, then with display sink
|
||
5. **FFmpeg v4l2-request** — install `ffmpeg-v4l2-request-git` (AUR) or
|
||
verify a DanctNIX-shipped ffmpeg with v4l2-request enabled
|
||
6. **Kodi + mpv validation** — 1080p HEVC at <30% of one core target
|
||
7. **Document** — freeze the final recipe in this README
|
||
|
||
**Current blocker**: ohm offline. Last known VPN state — only `nc` connected;
|
||
LAN gives "No route to host" for `ohm.fritz.box` (192.168.88.168). Ask
|
||
Markus to wake / reach out when ready.
|
||
|
||
### Acceptance criterion
|
||
|
||
**1080p @ 30 fps, no dropped frames.** Applies to every device and every
|
||
codec we claim support for. Same bar everywhere — don't grade RK3566 on a
|
||
curve vs RK3588. Dropped-frame count comes from `mpv --msg-level=all=info`
|
||
or `ffmpeg -f null -`; "no dropped frames" means zero across a 60 s clip.
|
||
|
||
### Test corpus
|
||
|
||
**Big Buck Bunny** is the canonical open test clip (Blender Foundation 2008,
|
||
CC-BY), ubiquitous in HW-decode testing across a decade of embedded
|
||
silicon. Various encodes readily available at
|
||
[peach.blender.org](https://peach.blender.org/download/) and mirrors.
|
||
|
||
Plan: park a fixed set on doppler under `/moviedata/fourier-test/` so Gerbera
|
||
re-indexes it and every fleet device on LAN + VPN streams identical bits.
|
||
Canonical encodes:
|
||
|
||
| File | Codec | Res | Notes |
|
||
|-------------------------------|--------|-------|------------------------------|
|
||
| bbb_1080p30_h264.mp4 | H.264 | 1080p | covers all four devices |
|
||
| bbb_1080p30_hevc.mp4 | HEVC | 1080p | ampere/boltzmann full; ohm pending rkvdec2 |
|
||
| bbb_1080p30_vp9.webm | VP9 | 1080p | ohm pending; RK3588 pending upstream |
|
||
| bbb_1080p30_mpeg2.ts | MPEG-2 | 1080p | Path A fresnel + rkvdec1 baseline |
|
||
| bbb_1080p30_vp8.webm | VP8 | 1080p | fresnel + RK356x rkvdec1 baseline |
|
||
| bbb_2160p60_hevc.mp4 | HEVC | 4K60 | RK3588 stretch goal (ampere/boltzmann only) |
|
||
|
||
Stretch: add **Tears of Steel** (2012, also CC-BY) for HDR / high-bitrate
|
||
HEVC once the stretch goal is viable.
|
||
|
||
## After ohm
|
||
|
||
- **ampere** (RK3588) — once RockHard kernel tracks a tree with VDPU381
|
||
merged (>= Linux 6.14-ish with the Feb 2026 backport), add `v4l2codecs` and
|
||
`ffmpeg-v4l2-request` packaging to RockHard's output. Should be a small
|
||
step.
|
||
- **boltzmann** (RK3588) — dual-path decision. Short-term: use Path B with
|
||
rkmpp (the kernel already exposes it). Long-term: migrate Neutron to a
|
||
mainline kernel with VDPU381 and move to Path A for symmetry with ampere.
|
||
- **fresnel** (RK3399) — Hantro is mature upstream; likely only needs
|
||
userspace install + recipe validation. Endeavour OS package names TBD.
|
||
|
||
Patches upstreamed along the way (v4l2-request to FFmpeg, VDPU346 to
|
||
linux-media) count double — they benefit the whole fleet.
|
||
|
||
## Packaging — marfrit-packages
|
||
|
||
New fleet hosts should get Fourier userspace via `pacman -S` / `apt install`,
|
||
not per-device AUR rebuilds. Mirror plan in `marfrit-packages` (layout:
|
||
`arch/<pkg>/PKGBUILD`, `debian/<pkg>/build-deb.sh + debian/`):
|
||
|
||
| Package | Source | CI runner | Target |
|
||
|------------------------------|-----------------------------------------|-----------|---------|
|
||
| `ffmpeg-v4l2-request-git` | fork of AUR (Jernej's patchset on ffmpeg tip) | **fermi** (Arch ARM aarch64) | Path A: ohm, fresnel, ampere |
|
||
| `ffmpeg-v4l2-request` | same patches against Debian ffmpeg src | **feynman** (Debian aarch64) | Path A Debian hosts |
|
||
| `gst-plugins-bad-fourier` | only if stock distro package is <1.28 | fermi / feynman | 1.28 HEVC UAPI uplift |
|
||
| `libva-v4l2-request` | upstream tag | fermi / feynman | VA-API bridge |
|
||
|
||
The Arch package is the priority (ohm + fresnel are Arch-based). Debian
|
||
package comes second (relevant if a fleet host ever runs Debian — kepler or
|
||
similar). Both are `provides=(ffmpeg) conflicts=(ffmpeg)`, so install
|
||
deliberately replaces the distro ffmpeg on a given host.
|
||
|
||
CI trigger on each push. Artifacts land in packages.reauktion.de
|
||
(`reference_marfrit_repo_bootstrap.md` in noether memory has the client
|
||
bootstrap).
|
||
|
||
Out of scope: rkmpp-based FFmpeg (Path B). Boltzmann stays on rkr* until
|
||
Neutron migrates; that host doesn't need a marfrit-packages entry until
|
||
migration.
|
||
|
||
## Working agreements
|
||
|
||
Standing rules for how we run this project — inherited from the broader
|
||
collaboration canon (`feedback_*` / `project_*` in noether's memory system),
|
||
captured here so a cold-start Fourier session doesn't re-learn them.
|
||
|
||
### ReCAP — ReContextualization After Pruning
|
||
|
||
Claude's context gets compacted when long sessions hit the window limit, and
|
||
any live-session state not in durable storage vanishes. Our counter:
|
||
|
||
- Memory files (`/home/mfritsche/.claude/projects/-home-mfritsche-claude/memory/`,
|
||
`MEMORY.md` is the index) are the long-term substrate. Don't keep
|
||
load-bearing facts only in conversation.
|
||
- Project READMEs (this file) carry the research dossier + current plan.
|
||
Update them when state changes, not later.
|
||
- Task list on noether carries active work. Status transitions are cheap —
|
||
mark in_progress when starting, completed when done.
|
||
- After a `/compact`, re-read relevant memory files + README + recent git log
|
||
before reasoning. Don't infer from conversation alone.
|
||
|
||
See `project_recap.md` in memory for the full protocol.
|
||
|
||
### Commit per experiment
|
||
|
||
Every experiment that touches the tree — a kconfig change, a DT tweak, a
|
||
ffmpeg build flag, a test clip benchmark — gets its own commit on a WIP
|
||
branch, with a short message naming what changed and what was observed.
|
||
|
||
- Dirty trees are tech debt. If the tree is dirty for >30 min, commit.
|
||
- Include benchmark numbers / dmesg excerpts / observation in the commit
|
||
body, not just the code diff.
|
||
- Rebase / squash later if a clean series matters; don't delay the commit
|
||
waiting for it.
|
||
|
||
See `feedback_commit_cadence.md` in memory.
|
||
|
||
### Ask before flash / reboot
|
||
|
||
This project spans fleet laptops and always-on hosts. We have **real blast
|
||
radius**.
|
||
|
||
- Never flash anything without a verified, tested backup. Fritz!Box 7490 was
|
||
bricked by flashing with an empty backup on 2026-04-12 — we don't repeat
|
||
that class of mistake. See `feedback_flash_critical.md`.
|
||
- Shared hardware reboots pause the user's other work. Ask before rebooting
|
||
data (Proxmox), hertz (home-LAN spine), boltzmann (always-on build host).
|
||
See `feedback_no_bulldoze_reboots.md`.
|
||
- For kernel / u-boot / firmware changes: simulate first where possible
|
||
(QEMU, module-style load) before flashing silicon. See
|
||
`feedback_simulate_first.md`.
|
||
- **Never** run `update-initramfs` or equivalent on a remote host that boots
|
||
from a non-standard root (ZFS, btrfs-with-subvols). See
|
||
`feedback_initramfs.md`.
|
||
|
||
### Off-machine backups
|
||
|
||
Fleet laptops back up to data via backintime (Anacron mode, daily attempts,
|
||
VPN-guarded, 30-day staleness alert by email). Hertz backs up to data
|
||
weekly / quarterly via `backup-hertz.sh` (vitruvius dashboard shows
|
||
progress). Data itself lives on ZFS RAIDZ2 + snapshots.
|
||
|
||
- Before any invasive change on a fleet laptop, confirm its last successful
|
||
backintime snapshot in `/opt/herding/var/backintime-status/<host>` on
|
||
hertz.
|
||
- New per-device state (kernel patch series, ffmpeg build tree) lives in a
|
||
Gitea repo + gets backed up with the rest of the working tree — don't
|
||
leave unique work on a single disk.
|
||
- External photos / family data on hertz is mirrored via the hertz backup to
|
||
data. Restore recipe is in the two-step `restore-hertz-step1-sd.sh` /
|
||
`restore-hertz-step2-lxd.sh` on data.
|
||
|
||
### See also — broader feedback canon
|
||
|
||
- **Test the observer first** (`feedback_observer_first.md`) — before
|
||
drawing decode-performance conclusions, confirm the rig (v4l2-ctl reports
|
||
sensible caps? ffmpeg actually routing through v4l2request?).
|
||
- **Three strikes then verify** (`feedback_three_strikes.md`) — after two
|
||
failed fixes, stop guessing; verify the binary / wire / protocol.
|
||
- **TRM or nothing** (`feedback_trm_or_nothing.md`) — register writes need
|
||
documentation backing. Relevant for any DT or driver patches we ship.
|
||
- **Trust Markus' eyes** (`feedback_trust_user_eyes.md`) — if he reports
|
||
"it plays smoothly", that's primary evidence. Don't over-qualify.
|
||
- **No budget framing** (`feedback_no_budget_framing.md`) — don't pre-shrink
|
||
scope citing "session cost"; Markus sets pace.
|
||
|
||
## References
|
||
|
||
### Mainline kernel state
|
||
- [Rockchip RK3588 / RK3576 H.264 and H.265 video decoders gain mainline Linux support (CNX Software, 2026-02-27)](https://www.cnx-software.com/2026/02/27/rockchip-rk3588-rk3576-h-264-and-h-265-video-decoders-mainline-linux/)
|
||
- [RK3588 and RK3576 video decoders support merged in the upstream Linux Kernel (Collabora)](https://www.collabora.com/news-and-blog/news-and-events/rk3588-and-rk3576-video-decoders-support-merged-in-the-upstream-linux-kernel.html)
|
||
- [Upstream support for Rockchip's RK3588: Progress and future plans (Collabora)](https://www.collabora.com/news-and-blog/news-and-events/rockchip-rk3588-upstream-support-progress-future-plans.html)
|
||
- [media: rkvdec: Add support for VDPU381 and VDPU383 (LWN)](https://lwn.net/Articles/1053556/)
|
||
- [RKVDEC2 Driver Posted For Accelerated Video Decoding On Newer Rockchip SoCs (Phoronix)](https://www.phoronix.com/news/RKVDEC2-Rockchip-Video-Decode)
|
||
|
||
### FFmpeg V4L2 request API
|
||
- [PATCH v2 0/8 Add V4L2 Request API hwaccels for MPEG2, H.264 and HEVC (ffmpeg-devel, 2024-08)](https://ffmpeg.org/pipermail/ffmpeg-devel/2024-August/332034.html)
|
||
- [Miouyouyou/FFmpeg-V4L2-Request (build script)](https://github.com/Miouyouyou/FFmpeg-V4L2-Request)
|
||
- [ffmpeg v4l2 requests 4.4.3 patchset (artemis.sh, 2023-03)](https://artemis.sh/2023/03/06/ffmpeg-v4l2-requests-4.4.3.html)
|
||
|
||
### GStreamer v4l2codecs
|
||
- [GStreamer v4l2codecs plugin docs](https://gstreamer.freedesktop.org/documentation/v4l2codecs/index.html)
|
||
- [Adding VP9 and MPEG2 stateless support in v4l2codecs for GStreamer (Collabora, 2021)](https://www.collabora.com/news-and-blog/blog/2021/06/23/adding-vp9-and-mpeg2-stateless-support-in-v4l2codecs-for-gstreamer/)
|
||
- [GStreamer 1.26 — improved hardware efficiency (Collabora)](https://www.collabora.com/news-and-blog/news-and-events/gstreamer-126-improved-hardware-efficiency.html)
|
||
|
||
### PineTab2 specific
|
||
- [Hardware accelerated playback on PineTab 2 (clehaxze.tw, 2023-09)](https://clehaxze.tw/gemlog/2023/09-17-hardware-accelerated-playback-on-pinetab2.gmi)
|
||
- [PINE64 Mainline Hardware Decoding wiki](https://wiki.pine64.org/wiki/Mainline_Hardware_Decoding)
|
||
- [dreemurrs-embedded/linux-pinetab2 releases (6.15.2-danctnix2 latest)](https://github.com/dreemurrs-embedded/linux-pinetab2/releases)
|