libva-v4l2-request-fourier/STUDY.md

# libva-v4l2-request — Fourier port study

## Goal

Make this libva backend usable on **multiplanar** V4L2 stateless decoders:
specifically the Rockchip Hantro VPU (RK3566 ohm) and the upcoming RK3588
hantro/VDPU381 path. End deliverable: any VAAPI client (Brave, Firefox via
ffmpeg-vaapi, mpv `--hwdec=vaapi`, vlc, ...) gets HW decode for H.264 + MPEG-2
on the Fourier fleet without going through GStreamer.

## Why this fork exists

Bootlin upstream <https://github.com/bootlin/libva-v4l2-request> went dormant
around 2021 and was written for **single-plane** sunxi-cedrus decoders.
Collabora's strategic replacement is `cros-codecs` (Rust) — it bypasses libva
entirely, targets Chromium/Firefox direct integration, and **is not shipping
soon**. That leaves a hole for VAAPI clients on Rockchip. None of the public
forks (jernejsk, ndufresne, pH5, jc-kynesim, ArtSvetlakov) shipped multiplanar.

Reference: Mozilla bug 1833354 / 1965646 explicitly notes "Rockchip uses
v4l2-request, not v4l2-m2m" — Firefox HW decode on RK3566/RK3588 needs exactly
a working libva-v4l2-request to bridge.

## State today

Initial commits applied to bootlin tip `a3c2476`:

1. `V4L2_PIX_FMT_H264_SLICE_RAW` → `V4L2_PIX_FMT_H264_SLICE` (kernel UAPI rename).
2. `src/h264.c`: missing `#include "utils.h"` for `request_log()` (GCC 14 fatal).
3. HEVC stripped — h265.c/h265.h excluded from `meson.build`, hevc-ctrls.h
   replaced by passthrough to `<linux/v4l2-controls.h>`, four HEVC case blocks
   removed from `picture.c` (kernel `V4L2_CID_MPEG_VIDEO_HEVC_*` was renamed
   to `V4L2_CID_STATELESS_HEVC_*`; ohm has no HW HEVC anyway).
4. `src/config.c`: profile probe falls back to `V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE`
   when single-plane returns no formats. **`vainfo` now lists H.264 + MPEG-2
   profiles on ohm**, and Brave's GPU process picks them up via VAAPI.

Failure mode reached and stuck on:

```
ERROR:media/gpu/vaapi/vaapi_wrapper.cc:2407] vaCreateContext failed,
  VA error: operation failed
```

That's the next-phase boundary — the real port work below.

## Port plan

The seam to flip is the entire kernel-userspace V4L2 boundary. The work is
mostly mechanical and concentrated in three files:

### `src/v4l2.c` — helpers (the bottleneck; all other files call into this)

Add a `v4l2_type_is_mplane()` predicate (one already exists upstream — keep it)
and dual paths through:

- `v4l2_set_format()` — populate either `format.fmt.pix` or `format.fmt.pix_mp`,
  including `plane_fmt[0].sizeimage` for OUTPUT and `num_planes` defaulting to 1
  for raw/NV12 capture.
- `v4l2_create_buffers()` / `v4l2_request_buffers()` — set
  `v4l2_create_buffers.format.type` to the MPLANE variant when source is mplane.
- `v4l2_query_buffer()` / `v4l2_export_buffer()` — switch on type, use
  `v4l2_buffer.m.planes` array (length `num_planes`) instead of `m.userptr` /
  `m.fd`. EXPBUF now needs `plane=0` parameter.
- `v4l2_queue_buffer()` / `v4l2_dequeue_buffer()` — same `m.planes[]` switch.
  The OUTPUT side passes the bitstream slice as `m.planes[0].bytesused`.

Reference: `libavcodec/v4l2_buffers.c` and `libavcodec/v4l2_context.c` in
FFmpeg already do this branching cleanly — it's the closest API match. Crib
`V4L2_TYPE_IS_MULTIPLANAR()` style switching there. GStreamer's
`gstv4l2decoder.c` is the second reference; it covers the request-API +
mplane path explicitly for the same Rockchip hardware we target.

### `src/context.c` — context creation

`RequestCreateContext` calls into `v4l2_set_format()` for the OUTPUT and
CAPTURE queues. Detect the queue capability at context creation (cache the
mplane bit on the context object) and pick the right type for every
subsequent helper call.

### `src/picture.c` — frame submission

The QBUF / DQBUF / EXPBUF paths in `RequestEndPicture()` and friends. Same
pattern — switch on the cached mplane bit and use the multiplanar variants
of the `v4l2.c` helpers. The slice-data submission (`m.planes[0].bytesused`)
is the load-bearing change here.

## Reference implementations (read these side-by-side with our diff)

- **FFmpeg** — `libavcodec/v4l2_request.c`, `v4l2_request_buffer.c`, per-codec
  files like `v4l2_request_h264.c`. Already multiplanar, already works on
  hantro/rkvdec — this is the closest-API canonical example.
  - <https://github.com/FFmpeg/FFmpeg/tree/master/libavcodec>
  - 2024-08 v2 patchset: <https://www.mail-archive.com/ffmpeg-devel@ffmpeg.org/msg169515.html>
  - Active downstream: <https://code.ffmpeg.org/Kwiboo/FFmpeg/> (branch `v4l2-request-n8.1`)
- **GStreamer v4l2codecs** — `gst-plugins-bad/sys/v4l2codecs/`. `gstv4l2decoder.c`
  has the canonical multiplanar S_FMT / REQBUFS / EXPBUF code on the exact
  Rockchip drivers we target. `gstv4l2codecsh264dec.c` shows the request-API
  controls submission.
  - <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/tree/main/subprojects/gst-plugins-bad/sys/v4l2codecs>
- **Chromium** — `media/gpu/v4l2/v4l2_video_decoder_backend_stateless.{h,cc}`
  + `v4l2_queue.cc`. ChromeOS-mature multiplanar code; higher abstraction than
  we need but useful for surface lifecycle / request-fd tracking patterns.
  - <https://chromium.googlesource.com/chromium/src/+/refs/heads/main/media/gpu/v4l2/>

## Test fixtures

- **ohm** — RK3566 PineTab2, kernel `6.19.10-danctnix1-1-pinetab2`. Hantro
  decoder exposes S264 / MG2S / VP8F formats on `/dev/video1` (multiplanar).
  This is the primary dev target. Brave on ohm is the integration test
  endpoint; `vainfo LIBVA_DRIVER_NAME=v4l2_request LIBVA_V4L2_REQUEST_VIDEO_PATH=/dev/video1`
  is the unit test.
- **Test clip**: `/moviedata/fourier-test/bbb_1080p30_h264.mp4` on doppler
  (SHA-16 `dcf8a7170fbd49bb`, 1920×1080 H.264, 24 fps source despite the
  name). Pull via hertz `lxc file pull`.
- **Reference path that already works on the same hardware**: GStreamer
  `gst-launch-1.0 filesrc ! qtdemux ! h264parse ! v4l2slh264dec !
  waylandsink` — 6 % CPU, zero drops. That's the ceiling for what we're
  trying to match through the libva path.

## Out of scope (for the first port milestone)

- HEVC — kernel CIDs renamed, RK3566 has no HW HEVC. Deferred until RK3588
  silicon is on the bench AND a separate HEVC-revival pass.
- VP9, VP8, AV1 — no HW path or out of bootlin's original codec set.
- Userspace bitstream parsing — kernel V4L2 stateless API does the parsing;
  this library only forwards parameters. No need to touch.
- HEVC RFC (reference frame compression) — Rockchip-specific, kernel
  config has it disabled (`CONFIG_VIDEO_HANTRO_HEVC_RFC=n` on ohm).

## Build + install

- Build container: `fermi` (Arch ARM aarch64 LXC on hertz). `meson setup`
  + `ninja` straight off the source tree, no makepkg dance needed for
  development iteration.
- Install path: `/usr/lib/dri/v4l2_request_drv_video.so`.
- Activate: `LIBVA_DRIVER_NAME=v4l2_request` plus the path env vars
  `LIBVA_V4L2_REQUEST_VIDEO_PATH=/dev/video1` and
  `LIBVA_V4L2_REQUEST_MEDIA_PATH=/dev/media0`.
- Once the port works: package as `marfrit/libva-v4l2-request-fourier`
  next to `ffmpeg-v4l2-request-git`, with the same
  `provides=(libva-v4l2-request-git)` shape.

## Ack

Bootlin authored the original library under MIT/LGPL2.1; this fork adds
GPL-2.0-licensed shim files (HEVC strip, multiplanar plumbing) and is meant
to track upstream if upstream ever picks the work back up.