libva-v4l2-request-fourier/STUDY.md

libva-v4l2-request — Fourier port study

Goal

Make this libva backend usable on multiplanar V4L2 stateless decoders: specifically the Rockchip Hantro VPU (RK3566 ohm) and the upcoming RK3588 hantro/VDPU381 path. End deliverable: any VAAPI client (Brave, Firefox via ffmpeg-vaapi, mpv --hwdec=vaapi, vlc, ...) gets HW decode for H.264 + MPEG-2 on the Fourier fleet without going through GStreamer.

Why this fork exists

Bootlin upstream https://github.com/bootlin/libva-v4l2-request went dormant around 2021 and was written for single-plane sunxi-cedrus decoders. Collabora's strategic replacement is cros-codecs (Rust) — it bypasses libva entirely, targets Chromium/Firefox direct integration, and is not shipping soon. That leaves a hole for VAAPI clients on Rockchip. None of the public forks (jernejsk, ndufresne, pH5, jc-kynesim, ArtSvetlakov) shipped multiplanar.

Reference: Mozilla bug 1833354 / 1965646 explicitly notes "Rockchip uses v4l2-request, not v4l2-m2m" — Firefox HW decode on RK3566/RK3588 needs exactly a working libva-v4l2-request to bridge.

State today

Bootlin tip is a3c2476. Stack of WIP commits on top:

Build cleanly against current kernel UAPI

1. V4L2_PIX_FMT_H264_SLICE_RAW → V4L2_PIX_FMT_H264_SLICE rename.
2. src/h264.c: missing #include "utils.h" for request_log().
3. HEVC stripped — h265.c/h265.h excluded from meson.build, hevc-ctrls.h replaced by passthrough to <linux/v4l2-controls.h>, four HEVC case blocks removed from picture.c.
4. include/h264-ctrls.h made into a passthrough shim to <linux/v4l2-controls.h> plus V4L2_CID_MPEG_VIDEO_H264_* → V4L2_CID_STATELESS_H264_* aliases (kernel renamed during upstreaming).
5. src/h264.c shape updates to track the upstreamed struct v4l2_ctrl_h264_slice_params: drop .size, use struct v4l2_h264_reference {fields, index} for ref_pic_list*, move pred_weight_table out to its own V4L2_CID_STATELESS_H264_PRED_WEIGHTS control, drop decode_params.num_slices (kernel infers from queued controls).
6. src/tiled_yuv.S: aarch64 stub of tiled_to_planar — the ARMv7 NEON body is #ifndef __aarch64__-guarded; without a stub the .so had an undefined symbol and dlopen failed.

Library now builds clean (~265 KB .so) and vainfo enumerates H.264 + MPEG-2 profiles.

Probe + control flow fixes

7. src/video.c: add NV12 multi-plane format entry; video_format_find() takes bool mplane so single- and multi-plane NV12 entries don't collide on pixelformat.
8. src/surface.c: probe block tries V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE as a fallback after single-plane probes fail.
9. Eager probe in RequestInit — Chromium's vaapi_video_decoder calls vaCreateContext before vaCreateSurfaces2, so the original lazy "set driver_data->video_format in RequestCreateSurfaces" path was too late. Promoted the probe into video_format_probe() in video.c and call it at init time.
10. src/context.c: same V4L2_PIX_FMT_H264_SLICE_RAW → _SLICE rename that was missed in the first pass.
11. WIP: defer STREAMON calls in RequestCreateContext. The V4L2 stateless protocol on hantro requires OUTPUT format → SPS controls → first slice queued → THEN STREAMON. Deferring lets vaCreateContext succeed; proper sequencing is the next phase.

Diagnostic logging (will revert before final)

12. src/utils.c: request_log() tees to /tmp/libva-fourier.log so sandboxed Chromium GPU processes don't swallow the trace output.

Failure mode reached now (2026-04-26)

vainfo works fully. Brave on ohm with the patched .so reaches deeper:

• Init: detects NV12 multi-plane CAPTURE format ✓
• vaCreateContext: profile=H264Main 1920×1088, S_FMT + CREATE_BUFS pass ✓
• vaCreateContext: STREAMON OUTPUT returned EINVAL ← deferred to unblock
• Now stuck at: failed Initialize()ing the frame pool from vaapi_video_decoder.cc

That's the next-phase boundary. vaCreateSurfaces2 (frame-pool init) is where Chromium discovers buffer geometry for output frames; the library needs to do CAPTURE-side S_FMT + REQBUFS + EXPBUF without single- plane assumptions on a freshly-created context, and the V4L2 stateless protocol's source-change-event dance probably needs implementing too.

Port plan

The seam to flip is the entire kernel-userspace V4L2 boundary. The work is mostly mechanical and concentrated in three files:

src/v4l2.c — helpers (the bottleneck; all other files call into this)

Add a v4l2_type_is_mplane() predicate (one already exists upstream — keep it) and dual paths through:

• v4l2_set_format() — populate either format.fmt.pix or format.fmt.pix_mp, including plane_fmt[0].sizeimage for OUTPUT and num_planes defaulting to 1 for raw/NV12 capture.
• v4l2_create_buffers() / v4l2_request_buffers() — set v4l2_create_buffers.format.type to the MPLANE variant when source is mplane.
• v4l2_query_buffer() / v4l2_export_buffer() — switch on type, use v4l2_buffer.m.planes array (length num_planes) instead of m.userptr / m.fd. EXPBUF now needs plane=0 parameter.
• v4l2_queue_buffer() / v4l2_dequeue_buffer() — same m.planes[] switch. The OUTPUT side passes the bitstream slice as m.planes[0].bytesused.

Reference: libavcodec/v4l2_buffers.c and libavcodec/v4l2_context.c in FFmpeg already do this branching cleanly — it's the closest API match. Crib V4L2_TYPE_IS_MULTIPLANAR() style switching there. GStreamer's gstv4l2decoder.c is the second reference; it covers the request-API + mplane path explicitly for the same Rockchip hardware we target.

src/context.c — context creation

RequestCreateContext calls into v4l2_set_format() for the OUTPUT and CAPTURE queues. Detect the queue capability at context creation (cache the mplane bit on the context object) and pick the right type for every subsequent helper call.

src/picture.c — frame submission

The QBUF / DQBUF / EXPBUF paths in RequestEndPicture() and friends. Same pattern — switch on the cached mplane bit and use the multiplanar variants of the v4l2.c helpers. The slice-data submission (m.planes[0].bytesused) is the load-bearing change here.

Reference implementations (read these side-by-side with our diff)

• FFmpeg — libavcodec/v4l2_request.c, v4l2_request_buffer.c, per-codec files like v4l2_request_h264.c. Already multiplanar, already works on hantro/rkvdec — this is the closest-API canonical example.
  • https://github.com/FFmpeg/FFmpeg/tree/master/libavcodec
  • 2024-08 v2 patchset: https://www.mail-archive.com/ffmpeg-devel@ffmpeg.org/msg169515.html
  • Active downstream: https://code.ffmpeg.org/Kwiboo/FFmpeg/ (branch v4l2-request-n8.1)
• GStreamer v4l2codecs — gst-plugins-bad/sys/v4l2codecs/. gstv4l2decoder.c has the canonical multiplanar S_FMT / REQBUFS / EXPBUF code on the exact Rockchip drivers we target. gstv4l2codecsh264dec.c shows the request-API controls submission.
  • https://gitlab.freedesktop.org/gstreamer/gstreamer/-/tree/main/subprojects/gst-plugins-bad/sys/v4l2codecs
• Chromium — media/gpu/v4l2/v4l2_video_decoder_backend_stateless.{h,cc}
  • v4l2_queue.cc. ChromeOS-mature multiplanar code; higher abstraction than we need but useful for surface lifecycle / request-fd tracking patterns.
  • https://chromium.googlesource.com/chromium/src/+/refs/heads/main/media/gpu/v4l2/

Test fixtures

• ohm — RK3566 PineTab2, kernel 6.19.10-danctnix1-1-pinetab2. Hantro decoder exposes S264 / MG2S / VP8F formats on /dev/video1 (multiplanar). This is the primary dev target. Brave on ohm is the integration test endpoint; vainfo LIBVA_DRIVER_NAME=v4l2_request LIBVA_V4L2_REQUEST_VIDEO_PATH=/dev/video1 is the unit test.
• Test clip: /moviedata/fourier-test/bbb_1080p30_h264.mp4 on doppler (SHA-16 dcf8a7170fbd49bb, 1920×1080 H.264, 24 fps source despite the name). Pull via hertz lxc file pull.
• Reference path that already works on the same hardware: GStreamer gst-launch-1.0 filesrc ! qtdemux ! h264parse ! v4l2slh264dec ! waylandsink — 6 % CPU, zero drops. That's the ceiling for what we're trying to match through the libva path.

Out of scope (for the first port milestone)

• HEVC — kernel CIDs renamed, RK3566 has no HW HEVC. Deferred until RK3588 silicon is on the bench AND a separate HEVC-revival pass.
• VP9, VP8, AV1 — no HW path or out of bootlin's original codec set.
• Userspace bitstream parsing — kernel V4L2 stateless API does the parsing; this library only forwards parameters. No need to touch.
• HEVC RFC (reference frame compression) — Rockchip-specific, kernel config has it disabled (CONFIG_VIDEO_HANTRO_HEVC_RFC=n on ohm).

Build + install

• Build container: fermi (Arch ARM aarch64 LXC on hertz). meson setup
  • ninja straight off the source tree, no makepkg dance needed for development iteration.
• Install path: /usr/lib/dri/v4l2_request_drv_video.so.
• Activate: LIBVA_DRIVER_NAME=v4l2_request plus the path env vars LIBVA_V4L2_REQUEST_VIDEO_PATH=/dev/video1 and LIBVA_V4L2_REQUEST_MEDIA_PATH=/dev/media0.
• Once the port works: package as marfrit/libva-v4l2-request-fourier next to ffmpeg-v4l2-request-git, with the same provides=(libva-v4l2-request-git) shape.

Ack

Bootlin authored the original library under MIT/LGPL2.1; this fork adds GPL-2.0-licensed shim files (HEVC strip, multiplanar plumbing) and is meant to track upstream if upstream ever picks the work back up.