libva-v4l2-request-fourier/phase0_pi5_hevc.md

Phase 0 — Pi 5 / CM5 HEVC chapter

Opened 2026-05-17 evening, after the failed libva-v4l2-stateful-fourier scaffold attempt. Brother-session empirical Phase 0 on higgs invalidated the stateful premise: rpi-hevc-dec is V4L2 stateless, so Pi 5 HEVC belongs in this backend, not a separate sibling.

No code in this chapter yet. This doc is the substrate. Phase 1 picks up from the "Open questions" section.

Substrate

Target host

higgs — Pi CM5 module on Pi CM5 IO board. BCM2712 SoC. VPN-only, often offline; wake via HIS skill recipe (no Fritz!Box plug — runs on power when on). Debian-based. Sole HW video decoder is rpi-hevc-dec at /dev/video19 + /dev/media1.

Backend baseline at chapter open

libva-v4l2-request-fourier master tip cf8cd9d (iter39 + Option B + h265 ref-list cap fix). Multi-device probe (iter38) already opens rkvdec + hantro slots; adding a third decoder slot for rpi-hevc-dec is a natural extension of that architecture.

iter2 (ampere VDPU381 HEVC EXT_SPS) added the GStreamer 1.28.2 H.265 parser vendor + EXT_SPS_ST_RPS / _LT_RPS dynamic-array submission. That plumbing is probe-gated (has_hevc_ext_sps_rps_rkvdec), so it stays dormant on hosts where the controls don't exist.

Empirical higgs probe (brother session)

v4l2-ctl -d /dev/video19 --list-formats-ext --list-ctrls:

Stateless Codec Controls

  hevc_sequence_parameter_set        (compound, V4L2_CID_STATELESS_HEVC_SPS)
  hevc_picture_parameter_set         (compound, V4L2_CID_STATELESS_HEVC_PPS)
  slice_param_array                  (compound dynamic-array dims=[4096])
  hevc_scaling_matrix                (compound)
  hevc_decode_parameters             (compound)
  hevc_decode_mode                   (menu, "Frame-Based")
  hevc_start_code                    (menu, default "No Start Code")

OUTPUT formats:
  S265  V4L2_PIX_FMT_HEVC_SLICE  (parsed slice payload)

CAPTURE formats:
  NC12  V4L2_PIX_FMT_NV12_COL128       (8-bit  SAND 128-column tiled)
  NC30  V4L2_PIX_FMT_NV12_10_COL128    (10-bit SAND 128-column tiled)

Conclusion: this is the standard V4L2_CID_STATELESS_HEVC_* control set exposed under the V4L2-request uAPI, exactly the same family our backend already drives for rkvdec/hantro/cedrus HEVC paths. The novel parts are two pixel formats (NC12, NC30) and one driver-id (rpi-hevc-dec).

What carries forward unchanged

• VAAPI HEVC profile enumeration (config.c)
• h265_set_controls core path (h265.c) — same compound ctrl set
• Synthetic SPS pre-seed pattern (iter25/26) — already runs pre-CAPTURE-alloc
• Multi-device dispatch in RequestCreateConfig (iter38)
• VAAPI slice / picture / IQ matrix buffer parsing
• HEVC h264-style start-code policy (we already DON'T prepend for HEVC)

What needs adding

Item	Location	Sizing
RPI_HEVC_DEC enum in driver_kind_t	request.h	trivial
Multi-device probe extends to /dev/video19 discovery	context.c / request.c init	small — mirror hantro slot
V4L2_PIX_FMT_NV12_COL128 (NC12) video_format entry	video.c	small
V4L2_PIX_FMT_NV12_10_COL128 (NC30) video_format entry	video.c	small
NC12 → NV12 detile primitive	new nv12_col128.c	mid — column tile layout, see kernel docs
NC30 → P010 detile primitive	new nv12_col128.c	mid — 10-bit variant of above
copy_surface_to_image branch for NC12/NC30	image.c	small (mirror NV15→P010 gating)
Per-driver gating for any rpi-specific quirks discovered	various	per per-driver-kludge-gating

Open questions for Phase 1

Lock these before Phase 1 commits to a goal.

1. EXT_SPS controls on rpi-hevc-dec? Brother's --list-ctrls output above shows the standard V4L2_CID_STATELESS_HEVC_* family — NOT the EXT_SPS_ST_RPS / EXT_SPS_LT_RPS extensions that VDPU381 needs. Verify: does slice_param_array[4096] accept st_rps_bits / lt_rps_bits in the per-slice payload, or does rpi-hevc-dec parse RPS itself from the slice header? If the latter, the iter2 EXT_SPS path stays dormant (probe-gated already), and rpi-hevc-dec just needs the picture->st_rps_bits → slice_params->short_term_ref_pic_set_size plumbing that iter31 α-29 already wired. Expectation: works out of the box. Confirm before assuming.

2. hevc_start_code ctrl: "No Start Code" vs Annex B? Brother saw default "No Start Code" — matches our behavior (we don't prepend on HEVC). But the ctrl is configurable. Verify the menu values exposed and confirm "No Start Code" passes our raw slice-NAL payload as-is. If it doesn't, set the ctrl explicitly per unconditional-codec-state gating.

3. NC12 / NC30 SAND tile layout — exact spec. Read Documentation/userspace-api/media/v4l/pixfmt-yuv-planar.rst for the COL128 variants. Confirm: column stride = 128 bytes (Y) / 128 bytes (UV interleaved). Row count = ALIGN(height, 16) or ALIGN(height, 8)? Get the exact alignment and tile-traversal order before writing the detile primitive. Cite from kernel doc, NOT inferred from a hex dump.

4. drm_prime / SAND modifier round-trip. Does ffmpeg-vaapi (and Firefox) accept the NC12 buffer via DRM_PRIME export carrying the DRM_FORMAT_MOD_BROADCOM_SAND128_COL_HEIGHT modifier, allowing zero-copy to a SAND-aware compositor? Or is libva-side detile to a linear NV12 buffer the only viable Firefox path? If detile is required for the consumer, the rockchip-pixel-verify-path rule (DMA-BUF GL preferred over cached mmap) might NOT apply since SAND is Pi-specific and not in the wider Wayland modifier ecosystem.

5. rpi-hevc-dec quirks on first SPS submission. rkvdec needs image_fmt pre-seed before CAPTURE alloc (iter25). Does rpi-hevc-dec have an analogous "must set OUTPUT pix_fmt + SPS before CAPTURE" ordering? Verify with strace early.

6. higgs OS + libva versioning. Brother probed on Debian. We package for Arch ALARM. What's the install path on higgs — Arch / Debian / Raspberry Pi OS? If Debian, the package needs a debian/ tree, not just PKGBUILD. Decide packaging target before Phase 8.

Phase 1 goal sketch (NOT locked)

Firefox HW HEVC playback on higgs at ≥30fps for 1080p Main, byte-exact libva-vs-kdirect for ≥3 reference fixtures (8-bit Main and 10-bit Main10).

Two measurable subgoals follow naturally:

• libva (this backend, NV12 image output) == kdirect (ffmpeg-v4l2request, NV12 image output) byte-exact for the same input.
• Firefox VA-API path engages (verify via chrome://gpu equivalent / log inspection — MOZ_LOG=PlatformDecoderModule:5).

Phase 3 baseline plan

Before any backend code touches rpi-hevc-dec:

• kdirect floor: ffmpeg -hwaccel v4l2request -hwaccel_output_format drm_prime -i bbb_720p10s_hevc.mp4 -vf hwdownload,format=nv12 -frames:v 10 ... and sha256 the YUV.
• SW reference: same ffmpeg without -hwaccel, sha256 the YUV.
• Both runs N=3 per replicate-baseline-first.
• Capture strace -f -e ioctl of the kdirect run — gives the canonical ioctl sequence rpi-hevc-dec expects.

Phase 0 closing

This doc commits the substrate. Phase 1 starts when:

• higgs is up + reachable
• Open questions 1+2 (EXT_SPS + start_code) are answered live, in one short probe session
• Phase 3 baseline floors are captured

No work blocks the close of iter39 / fresnel campaign — those are shipped.

Phase 0 close addendum (2026-05-17 evening, higgs probe session)

Empirical probes on higgs answered Q1, Q2, partial Q3, full Q5, full Q6. Q4 (DRM modifier round-trip) remains open. Phase 0 is closed; Phase 1 opens with what's below.

Q1 — EXT_SPS controls on rpi-hevc-dec: NOT present

v4l2-ctl -d /dev/video19 --list-ctrls confirms ONLY the standard V4L2_CID_STATELESS_HEVC_* set:

• hevc_sequence_parameter_set (0x00a40a90)
• hevc_picture_parameter_set (0x00a40a91)
• slice_param_array (0x00a40a92, dynamic-array dims=[4096])
• hevc_scaling_matrix (0x00a40a93)
• hevc_decode_parameters (0x00a40a94)
• hevc_decode_mode (0x00a40a95, menu min=1 max=1 default=1 = Frame-Based)
• hevc_start_code (0x00a40a96, menu min=0 max=1 default=0 = No Start Code)
• 0x00a40a97 returns EINVAL (no EXT_SPS_*_RPS controls)

ioctl trace confirms ffmpeg's VIDIOC_QUERY_EXT_CTRL for 0xa97 returns EINVAL — same probe pattern our backend uses for has_hevc_ext_sps_rps_rkvdec. The iter2 path stays dormant; the iter31 α-29 slice_params->short_term_ref_pic_set_size plumbing is the correct one for rpi-hevc-dec.

Q2 — hevc_start_code: default 0 (No Start Code), values {0, 1}

Default 0 matches our backend's "don't prepend HEVC start code" stance. Confirm in Phase 1: rpi-hevc-dec accepts our raw NAL slice payload as-is.

Q3 — NC12 / NC30 SAND tile layout: PARTIAL

CAPTURE S_FMT result for 1280×720 NC12:

• sizeimage=1382400 = 1280 × 720 × 1.5 (linear NV12 byte count)
• bytesperline=1080 (NOT 1280)

The bytesperline=1080 for a 1280-wide CAPTURE buffer is suspect — likely encodes SAND column count rather than linear stride. Read drivers/staging/media/rpivid/ (or wherever NC12_COL128 lives in 6.12) kernel source + drm_fourcc.h / nv12_col128.rst (if it exists) for exact tile layout BEFORE writing the detile primitive. Do NOT infer layout from this single observation.

Q4 — DRM modifier round-trip: BLOCKED on hwdownload

ffmpeg -hwaccel drm -hwaccel_output_format drm_prime -vf hwmap=mode=read,format=nv12 returns Failed to map frame: -38 (Function not implemented). hwdownload cannot consume the SAND modifier directly.

ffmpeg's path that DOES work: -hwaccel drm -c:v hevc WITHOUT -hwaccel_output_format drm_prime lets ffmpeg's internal pipeline pull back, detile (presumably via a Pi-specific helper or libdrm transform), and present NV12 to the next filter. Bit-exact vs SW for the test fixture (1280×720 Main 8-bit) — confirms HW engagement.

Phase 1 / Phase 4 will need to decide:

• Detile in the backend (CPU SIMD), exposing NV12 via VAImage; or
• Pass-through DRM_PRIME with SAND modifier and let the consumer (compositor / Firefox) detile. Firefox almost certainly can't, so CPU detile is the safe bet.

Q5 — rpi-hevc-dec submission ordering: empirically locked

strace -e ioctl of the kdirect run shows:

1. MEDIA_IOC_DEVICE_INFO + MEDIA_IOC_G_TOPOLOGY (per media node)
2. VIDIOC_QUERYCAP per video node — driver="rpi-hevc-dec" identifies the right one
3. VIDIOC_ENUM_FMT OUTPUT → S265 only
4. VIDIOC_S_FMT OUTPUT (HEVC_SLICE, placeholder dims)
5. VIDIOC_REQBUFS OUTPUT (DMABUF, count=N) — count=6 in kdirect
6. VIDIOC_S_FMT CAPTURE (NC12, actual dims from SPS parse)
7. VIDIOC_CREATE_BUFS CAPTURE (DMABUF, count=16)
8. VIDIOC_STREAMON both queues
9. VIDIOC_QUERY_EXT_CTRL enumeration
10. VIDIOC_S_EXT_CTRLS (decode_mode + start_code) — global ctrls
11. Per frame: VIDIOC_S_EXT_CTRLS (SPS+PPS+decode_params+slice_array, class=0xf010000 = per-request) + VIDIOC_QBUF CAPTURE + VIDIOC_QBUF OUTPUT (with V4L2_BUF_FLAG_IN_REQUEST | V4L2_BUF_FLAG_REQUEST_FD) + VIDIOC_DQBUF OUTPUT + VIDIOC_DQBUF CAPTURE

Two structural notes for the backend:

• OUTPUT + CAPTURE both use V4L2_MEMORY_DMABUF in kdirect. Our backend currently uses MMAP for CAPTURE on rkvdec/hantro. For Pi 5 we should either follow kdirect (DMABUF, allows zero-copy DRM_PRIME export) or use MMAP and CPU-detile. Phase 4 design decision.
• The order S_FMT OUTPUT → REQBUFS OUTPUT → S_FMT CAPTURE → CREATE_BUFS CAPTURE → STREAMON differs from our iter25 rkvdec pre-seed pattern (where SPS via S_EXT_CTRLS must come BEFORE CAPTURE alloc to resolve the image_fmt). rpi-hevc-dec apparently DOESN'T need that pre-seed — CAPTURE S_FMT just takes the explicit NC12 + caller's dims. Confirm in Phase 1 by trying our existing iter25 pre-seed flow against it.

Q6 — packaging: Debian 13 trixie, NOT Arch

higgs runs Debian 13 trixie (PRETTY_NAME="Debian GNU/Linux 13 (trixie)"), not Arch ALARM. Phase 8 (per the dev-process Phase 8 packaging rule) for the Pi 5 chapter needs a debian/ packaging tree, not just a PKGBUILD.

Decide in Phase 1 whether to:

• Add Debian packaging to marfrit-packages as a second target, OR
• Use distrobox/podman with an Arch ALARM container on higgs for install (test-only, not production), OR
• Pi 5 chapter ships a Debian source pkg via gitea / a personal Debian repo.

Other new findings from the probe session

• ffmpeg 7.1.3 from Debian 13 is built with --enable-v4l2-request — the kdirect path exists. Invocation is ffmpeg -hwaccel drm -c:v hevc (not just -hwaccel drm; the explicit codec flag matters for the negotiation). Engagement log line is Hwaccel V4L2 HEVC stateless V4; devices: /dev/media1,/dev/video19; buffers: src DMABuf, dst DMABuf; swfmt=rpi4_8. Per hw-decode-engagement-check, grep for that line to confirm HW path engaged.
• No libva ICD installed on higgs — only armada-drm_dri.so ships, which doesn't apply. We'd be the first VA-API HW path for HEVC on Pi 5 once installed.
• mpv is apt-installable (mpv 0.40.0-3+deb13u1) — useful as a pixel-readback verifier once the backend works (mpv --vo=image or --vo=drm).
• Firefox 145.0.1 + rpi-firefox-mods 20251016 installed (firefox-esr package status was rc = removed but config remains). The mods package likely contains VA-API plumbing prefs.

What changes for Phase 1

• Goal is now phrasable: HEVC bit-exact libva-vs-kdirect on higgs for the 1280×720 Main 8-bit test fixture (same generator as /tmp/bbb_main.mp4 here). Kdirect engagement signal is the Hwaccel V4L2 HEVC stateless V4 log line.
• Most backend code reuses existing rkvdec/hantro HEVC path: ctrls, per-frame submission, request_fd, multi-device probe pattern.
• New code: NC12 video_format entry + detile primitive (sibling to nv15_unpack_plane_to_p010) + RPI_HEVC_DEC driver_kind.
• Packaging target = Debian, not Arch.