fresnel-fourier/phase2_iter2_situation.md

Iteration 2 — Phase 2 (situation analysis)

Phase 2 source-read of the HEVC path in libva-v4l2-request-fourier post-iter1-close (master tip 229d6d1). Six bugs identified, all in the libva backend. Kernel + driver path was already proven for HEVC in Phase 0 cross-validator sweep.

Caveat on substrate timing: this Phase 2 was conducted against fresnel running linux-eos-arm 6.19.9-99. Phase 3 baselines + Phase 6 implementation + Phase 7 verification will run against an imminent linux-7 rolling upgrade (operator-scheduled). Phase 2 source-read findings are kernel-agnostic (fork code + kernel UAPI + FFmpeg reference + VAAPI buffers); they carry forward across the kernel jump unchanged.

Reset context

$ git -C /home/mfritsche/src/fresnel-fourier status -sb
## master
$ git -C /home/mfritsche/src/fresnel-fourier log --oneline -3
6e8c970 iter2 Phase 0 + Phase 1 lock: HEVC Main on rkvdec
dc69378 iter1 Phase 8 close: 2/5 codecs passing
ec9133a iter1 Phase 7: verification — all 5 criteria GREEN

$ git -C /home/mfritsche/src/libva-multiplanar/libva-v4l2-request-fourier log --oneline -1
229d6d1 fresnel-fourier iter1 Phase 6 commit D: drop missed mpeg2-ctrls.h include from context.c
$ ssh fresnel 'uname -r'
6.19.9-99-eos-arm

Source files inspected:

File	Lines
src/h265.c	407 (currently excluded from build)
src/picture.c	403 (HEVCMain reject at 204-206)
src/config.c	255 (HEVCMain fall-through at 67)
src/surface.h	182 (params.h265 union member)
src/meson.build	98 (h265.c + h265.h commented out)
include/hevc-ctrls.h	9 (thin shim, NOT a duplication header)

Reference checkouts:

• Linux mainline at ~/src/libva-multiplanar/references/linux-mainline/ (7.1.0-rc2). Note: rkvdec_hevc.c does not exist in this checkout (rkvdec only has h264, vp9). The HEVC stateless API is in UAPI but the rkvdec driver-side HEVC handler is out-of-tree on fresnel's 6.19 (Christian Hewitt patch series). May land in stable 7.x.
• FFmpeg Kwiboo v4l2-request-n8.1 branch — libavcodec/v4l2_request_hevc.c (752 lines, modern API).
• VAAPI headers on fresnel — <va/va.h> defines VAPictureParameterBufferHEVC, VASliceParameterBufferHEVC, VAIQMatrixBufferHEVC.
• Kernel UAPI on fresnel — <linux/v4l2-controls.h> lines 2090+ define new HEVC stateless controls.

Bug 1 — RequestCreateConfig HEVCMain fall-through to default

src/config.c:55-71:

case VAProfileMPEG2Simple:
case VAProfileMPEG2Main:
    // fresnel-fourier iter1: MPEG-2 enabled. ...
    break;
case VAProfileHEVCMain:
default:
    return VA_STATUS_ERROR_UNSUPPORTED_PROFILE;

iter1 added the break; for MPEG-2 cases. HEVCMain still falls through to default: and returns VA_STATUS_ERROR_UNSUPPORTED_PROFILE (= 12). Verbatim match for iter1 Bug 1's pattern, applied to a different profile.

Fix shape: 3-line break; addition (analogous to iter1 Commit A pattern). Matches existing H.264 + MPEG-2 case structure. No profile-specific validation logic in RequestCreateConfig (validation happens at vaCreateContext / control submission time).

Bug 2 — picture.c::codec_set_controls explicit HEVCMain reject

src/picture.c:204-206:

case VAProfileHEVCMain:
    /* Fourier-local: HEVC stripped, no HW support on RK3566. */
    return VA_STATUS_ERROR_UNSUPPORTED_PROFILE;

Comment is stale — references RK3566 (PineTab2 / ohm-side context); fresnel-fourier targets RK3399 where rkvdec DOES support HEVC. Phase 0 cross-validator sweep confirmed ffmpeg -hwaccel v4l2request decodes the BBB HEVC fixture exit 0 on fresnel.

Fix shape: replace the explicit reject with a dispatch to h265_set_controls() (mirroring the MPEG-2 dispatch at picture.c:186-191). 5-line change. Comment updated to remove the stale RK3566 reference.

Bug 3 — src/h265.c uses staging-era control IDs

src/h265.c calls v4l2_set_control with three control IDs:

• V4L2_CID_MPEG_VIDEO_HEVC_PPS (line 386)
• V4L2_CID_MPEG_VIDEO_HEVC_SPS (line 393)
• V4L2_CID_MPEG_VIDEO_HEVC_SLICE_PARAMS (line 401)

These are staging-era — they don't exist on fresnel's 6.19 kernel headers. Empirical test:

$ gcc -c /tmp/test_compile_h265.c -I include
/tmp/test_compile_h265.c:3:29: error: 'V4L2_CID_MPEG_VIDEO_HEVC_PPS' undeclared (first use in this function);
                                       did you mean 'V4L2_CID_MPEG_VIDEO_DEC_PTS'?
... (similar for _SPS and _SLICE_PARAMS)

So h265.c won't compile against the modern kernel UAPI. That's why src/meson.build comments it out (sources line 50, headers line 73).

The mainline kernel UAPI defines (<linux/v4l2-controls.h>:2096-2100 and following):

#define V4L2_CID_STATELESS_HEVC_SPS              (V4L2_CID_CODEC_STATELESS_BASE + 400)  /* 0xa40a90 */
#define V4L2_CID_STATELESS_HEVC_PPS              (V4L2_CID_CODEC_STATELESS_BASE + 401)  /* 0xa40a91 */
#define V4L2_CID_STATELESS_HEVC_SLICE_PARAMS     (V4L2_CID_CODEC_STATELESS_BASE + 402)  /* 0xa40a92 */
#define V4L2_CID_STATELESS_HEVC_SCALING_MATRIX   (V4L2_CID_CODEC_STATELESS_BASE + 403)  /* 0xa40a93 */
#define V4L2_CID_STATELESS_HEVC_DECODE_PARAMS    (V4L2_CID_CODEC_STATELESS_BASE + 404)  /* 0xa40a94 */
#define V4L2_CID_STATELESS_HEVC_DECODE_MODE      (V4L2_CID_CODEC_STATELESS_BASE + 405)  /* 0xa40a95 */
#define V4L2_CID_STATELESS_HEVC_START_CODE       (V4L2_CID_CODEC_STATELESS_BASE + 406)  /* 0xa40a96 */
#define V4L2_CID_STATELESS_HEVC_ENTRY_POINT_OFFSETS (V4L2_CID_CODEC_STATELESS_BASE + 407)

Plus EXT_SPS_ST_RPS, EXT_SPS_LT_RPS for HEVC range extensions (likely not needed on fresnel for Main profile).

The fresnel V4L2 inventory (phase0_evidence/2026-05-07/v4l2_inventory.txt) confirms /dev/video<rkvdec> exposes the new IDs 0xa40a90/91/92/93/94/95/96 (matches mpeg_2_* style for MPEG-2 in iter1; here hevc_*).

Fix shape: rewrite h265.c against the new split API. Significantly larger than iter1's mpeg2.c rewrite — see Bug 4-6 for additional structural changes.

Bug 4 — h265.c uses old struct shape (slice_params is now dynamic-array)

The current src/h265.c submits one slice_params per frame (one v4l2_set_control call with &slice_params, sizeof(slice_params)). The new mainline API treats slice_params as a dynamic array — one entry per slice in the frame.

Per Phase 0 inventory, fresnel's rkvdec advertises:

hevc_slice_parameters 0x00a40a92 (hevc-slice-params): elems=1 dims=[600] flags=has-payload, dynamic-array

So the kernel accepts up to 600 slice_params entries per submission. FFmpeg libavcodec/v4l2_request_hevc.c:540-547:

if (ctx->max_slice_params && controls->num_slice_params) {
    control[count++] = (struct v4l2_ext_control) {
        .id = V4L2_CID_STATELESS_HEVC_SLICE_PARAMS,
        .ptr = controls->frame_slice_params,
        .size = sizeof(*controls->frame_slice_params) *
                FFMIN(controls->num_slice_params, ctx->max_slice_params),
    };
}

FFmpeg accumulates slice_params across multiple decode_slice callbacks and submits them all together at end_frame.

Fix shape: h265_set_controls() needs to iterate over all slices in the frame (the libva backend currently sees them as multiple VARenderPicture(VASliceParameterBufferType) calls — codec_store_buffer:115-135 for HEVC currently does memcpy(&surface->params.h265.slice, …) which overwrites previous slice params with the latest one). For multi-slice frames, the libva backend must accumulate slice params into an array, not overwrite.

This is the largest behavioral change vs iter1 — the existing codec_store_buffer doesn't support per-slice accumulation. Phase 4 plan needs to extend surface_object->params.h265 to hold a slice_params array (or use a separate accumulator), and codec_store_buffer needs to append rather than overwrite.

For the iter2 fixture (bbb_720p10s_hevc.mp4, libx265 ultrafast 1280×720), the typical slices-per-frame count is small (probably 1-4 depending on x265's slice configuration). Phase 3 baseline will measure the actual count via cross-validator strace.

Bug 5 — h265.c missing controls (DECODE_PARAMS, SCALING_MATRIX, DECODE_MODE, START_CODE)

Current h265.c submits only PPS, SPS, SLICE_PARAMS. The new API has:

Control	Required	Frequency	Source data
STATELESS_HEVC_SPS	yes	per-frame (or once per stream)	VAPictureParameterBufferHEVC
STATELESS_HEVC_PPS	yes	per-frame	VAPictureParameterBufferHEVC
STATELESS_HEVC_SLICE_PARAMS	yes	per-frame, dynamic-array	VASliceParameterBufferHEVC × N
STATELESS_HEVC_SCALING_MATRIX	conditional on sps_scaling_list_data_present_flag	per-frame	VAIQMatrixBufferHEVC
STATELESS_HEVC_DECODE_PARAMS	yes	per-frame	DPB info + POC, derived from VAPictureParameterBufferHEVC
STATELESS_HEVC_DECODE_MODE	once	device-wide at init	menu (FRAME_BASED on rkvdec)
STATELESS_HEVC_START_CODE	once	device-wide at init	menu (ANNEX_B on rkvdec)

Per FFmpeg v4l2_request_hevc.c:512-565 v4l2_request_hevc_queue_decode, the per-frame batch contains 3 mandatory + 3 conditional = up to 6 controls (no DECODE_MODE/START_CODE — those are device-wide menus set once at init).

The new DECODE_PARAMS control carries:

• dpb[] array of DPB entries (timestamp + flags per ref pic) — was inside slice_params in old API; moved to a per-frame DECODE_PARAMS struct in new API.
• num_active_dpb_entries, num_poc_st_curr_before/after, num_poc_lt_curr — same; moved to DECODE_PARAMS.
• POC info per frame.

Fix shape: h265_fill_decode_params() extracts DPB+POC info from VAAPI's ReferenceFrames[15] — the existing h265_fill_slice_params lines 269-315 already does this work, just into the wrong destination struct. New code populates v4l2_ctrl_hevc_decode_params.dpb[] instead of slice_params.dpb[].

The old dpb[] location in the staging API was inside slice_params; new API moves it to decode_params (per-frame, sensible since DPB is shared across all slices in a frame). Phase 4 plan should preserve the existing extraction logic and route it into the new struct.

SCALING_MATRIX: only needed when sps->scaling_list_enabled_flag is set AND pps->pps_scaling_list_data_present_flag or SPS provides explicit lists. For the BBB-720p10s fixture, most likely default (flat) scaling — iqmatrix_set will be false, and we omit SCALING_MATRIX from the batch (kernel uses spec defaults).

DECODE_MODE + START_CODE: device-wide controls set once at context init, not per frame. Currently src/context.c:142-155 sets H.264-specific device controls; iter2 should NOT add HEVC device init there because HEVC decoding shares the rkvdec device with H.264 (and possibly future VP9). The H.264 device-init currently in context.c works because the device is rkvdec; for HEVC we need to set HEVC's _DECODE_MODE and _START_CODE on the same device. Either extend context.c's device-init block or set the HEVC-specific controls inside h265_set_controls() once-per-context (with a flag to skip on subsequent frames).

Bug 6 — src/meson.build excludes h265.c + h265.h

src/meson.build:50 and :73:

sources = [
    ...
#	'h265.c'
]

headers = [
    ...
#	'h265.h'
]

Both commented out. iter2 fix: uncomment both. Trivial 2-line change in meson.build.

Bug 7 (verify-only) — include/hevc-ctrls.h is a thin shim

Unlike mpeg2-ctrls.h (which iter1 deleted because it was a duplication header that masked kernel UAPI), include/hevc-ctrls.h is already a 9-line shim:

/* Fourier-local override: HEVC controls are upstream since linux-media
 * 6.6+, so defer to the kernel's linux/v4l2-controls.h instead of
 * duplicating the struct definitions (duplication causes redefinition
 * errors on newer linux-api-headers). */
#ifndef _LIBVA_V4L2_REQUEST_HEVC_CTRLS_H
#define _LIBVA_V4L2_REQUEST_HEVC_CTRLS_H
#include <linux/v4l2-controls.h>
#endif

The shim adds zero value — <linux/v4l2-controls.h> is already pulled transitively via <linux/videodev2.h> from any .c file that uses HEVC controls.

Decision (defer to Phase 4): leave the shim in place (lower-risk path; iter1 Phase 5 Nit 6 deferral). Deletion is vestigial-cleanup, not iter2-scope. Verify in Phase 4 plan whether src/h265.c or any other file uniquely depends on <hevc-ctrls.h> being a separate include; if not, deletion is a 1-line follow-up cleanup but not blocking.

Bug 8 (NEW, latent) — picture.c:287 params.h264.matrix_set = false corrupts h265.picture

Per offsetof verification on fresnel via gcc + libva:

h264.matrix_set offset    = 240
h265.picture  range = [0 .. 604)  size=604
h265.slice    range = [604 .. 868)  size=264
h265.iqmatrix range = [868 .. 1884)  size=1016
h265.iqmatrix_set offset = 1884

The unconditional params.h264.matrix_set = false write at picture.c:287 lands at union byte 240, which falls inside h265.picture (range 0..604). Specifically byte 240 of VAPictureParameterBufferHEVC — checking the va.h struct layout, byte 240 is somewhere in the middle (likely inside one of the bitfield sub-structs pic_fields or slice_parsing_fields or near ReferenceFrames[]).

For iter2's binding cells: same masking-by-RenderPicture-overwrite mechanism as iter1's MPEG-2 case. ffmpeg-vaapi sends VAPictureParameterBufferHEVC every frame via vaRenderPicture(VAPictureParameterBufferType), which codec_store_buffer:104-108 copies wholesale into surface->params.h265.picture (overwriting the corrupted byte 240). Net safe for iter2.

Latent bug confirmed for HEVC too: a VAAPI client that reuses a surface in BeginPicture without re-sending VAPictureParameterBufferType (legal VAAPI for surfaces with prior picture-param state) would see byte 240 corrupted. iter2 doesn't fix this; iter2+ Phase 4 cross-cutting backlog item B3 (BeginPicture profile-aware reset) covers it.

Things verified NOT bugs

h265.c field extraction logic from VAAPI is sound

The 407-line h265.c code:

• h265_fill_pps() (lines 48-102): extracts PPS fields from VAAPI picture (VAPictureParameterBufferHEVC) and slice (VASliceParameterBufferHEVC). Field-by-field source reads are correct against VAAPI's struct shape.
• h265_fill_sps() (lines 104-158): extracts SPS from VAAPI picture. Same shape; correct against VAAPI struct.
• h265_fill_slice_params() (lines 160-365): extracts slice params + DPB info + ref index lists + pred weight tables. The DPB extraction (lines 269-315) reads VAPictureHEVC ReferenceFrames[15] correctly.
• The data extraction targets the OLD struct layout (where DPB is in slice_params, where slice_params is single not dynamic-array, etc.). Phase 4 rewrite preserves the extraction logic and re-routes the destination to the new struct layout.

h265.c parses NAL header from bitstream

Lines 184-209: extracts nal_unit_type, nuh_temporal_id_plus1, and data_bit_offset from the slice data buffer. The data_bit_offset calculation searches for the slice-segment-header start-code-prefix bit. This logic is preserved — the new V4L2 API still requires slice_params.bit_size and slice_params.data_bit_offset per slice. Field locations differ; computation is the same.

v4l2.c batching API is in place

src/v4l2.c:475-489 v4l2_set_controls accepts an v4l2_ext_control[] array. iter2's batched 6-control-per-frame submission uses this same API (single VIDIOC_S_EXT_CTRLS call), matching iter1's pattern.

Open questions for Phase 3 baseline

Before Phase 4 plan, Phase 3 should capture (after the linux-7 kernel upgrade to anchor against the shipping kernel):

1. iter1 + T4 references on linux-7 — re-run iter1 Phase 7 criterion 5 (T4 H.264) and criterion 4 (MPEG-2 +02s). If hashes differ, that's a substrate regression to investigate before iter2 proceeds. Criterion 1 (vainfo) regression is also worth checking.
2. fresnel SDDM watchpoint on linux-7 — verify the greeter still works after the kernel jump. If regression fires, run ~/.claude/plans/dynamic-forging-piglet.md qFatal capture.
3. Cross-validator HEVC re-anchor — re-capture ffmpeg -hwaccel v4l2request -i bbb_720p10s_hevc.mp4 -frames:v 5 -f null - strace + ftrace on linux-7. Capture the VERBATIM payload bytes for SPS, PPS, DECODE_PARAMS, SLICE_PARAMS array entries, SCALING_MATRIX (if present). Phase 4 transcription anchor (per memory/feedback_review_empirical_over_theoretical.md).
4. Pre-fix scratch test — apply Bug 1 (config break) + Bug 2 (picture.c reject removal) as a scratch patch on a throwaway branch; rebuild WITHOUT touching h265.c (keep it commented out). Verify mpv --hwdec=vaapi MPEG-2 + H.264 still pass (no collateral). Then attempt to compile h265.c uncommented to confirm the staging-era CID errors. Document the exact compile errors as Phase 3 baseline.
5. Verify the slice-count for bbb_720p10s_hevc.mp4 via FFmpeg's slice count or by counting decode_slice entries — small (1-4 per frame) vs large affects whether iter2 needs to allocate slice_params arrays statically or dynamically.
6. Determine if linux-7 brings rkvdec_hevc.c into mainline — if yes, read the kernel-side HEVC handler for additional contract verification (especially around DECODE_PARAMS and slice_params dynamic-array semantics).

Phase 4 plan inputs

Once Phase 3 baseline confirms the situation on linux-7, Phase 4 plan should specify:

1. Diff scope:

   • src/config.c:64-69 — add break; for VAProfileHEVCMain case (3-line change, mirrors iter1 Bug 1 fix shape).
   • src/picture.c:204-206 — replace explicit reject with dispatch to h265_set_controls().
   • src/h265.c — substantial rewrite (~250-400 lines):
     • Replace V4L2_CID_MPEG_VIDEO_HEVC_* with V4L2_CID_STATELESS_HEVC_*.
     • Restructure for slice_params dynamic-array (per-slice append in codec_store_buffer + flush at frame end).
     • Add DECODE_PARAMS struct fill (split from slice_params).
     • Add SCALING_MATRIX (conditional on iqmatrix_set + sps.scaling_list_enabled).
     • Add device-wide DECODE_MODE + START_CODE at h265_set_controls first-call (or context init).
     • Update flag bitmasks: collapse boolean SPS/PPS fields into u64 flags (analogous to MPEG-2 picture.flags collapse in iter1).
   • src/picture.c::codec_store_buffer — add HEVC slice_params accumulation logic (append-not-overwrite for VASliceParameterBufferType).
   • src/surface.h::params.h265 — add a slice_params array (or pointer + count) to hold accumulated slice params.
   • src/meson.build — uncomment h265.c (line 50) and h265.h (line 73).
   • include/hevc-ctrls.h — keep as-is (not a duplication header; lower-risk path).
   • src/context.c — possibly extend H.264 device-init block to also set HEVC's device-wide DECODE_MODE + START_CODE if h265_set_controls doesn't handle that itself.

2. Contract anchor (per feedback_dev_process.md Phase 6 contract-before-code):

   • Cite verbatim from <linux/v4l2-controls.h>:2090+ — the 7 stateless HEVC control struct definitions and flag constants.
   • Cite verbatim from FFmpeg libavcodec/v4l2_request_hevc.c:512-565 — the queue_decode batched submission shape.
   • Cite Phase 3 Baseline C HEVC anchor's verbatim payload (re-captured on linux-7).
   • If linux-7 has rkvdec_hevc.c in mainline, cite the driver's slice_params handling, DECODE_PARAMS handling.

3. Phase 7 verification harness: re-use iter1's 5-criterion shape with HEVC fixture substituted. Add bonus byte-compare of post-fix VIDIOC_S_EXT_CTRLS payload vs Baseline C HEVC anchor (per memory/feedback_review_empirical_over_theoretical.md — empirical wins).

4. Header deletion completeness check: per memory/feedback_header_deletion_check.md. For iter2: include/hevc-ctrls.h is preserved as the lower-risk path, so this rule mostly doesn't apply. But: any <mpeg2-ctrls.h> reference iter1 might have missed (defensive grep in Phase 6 — even though iter1's Commit D fixed the one we found in context.c, do another sweep).

Predicted iter2 outcome

The bugs are well-bounded but the fix is structurally larger than iter1. Phase 4 plan will produce:

• Trivial diffs for Bugs 1 + 2 + 6 (config.c break, picture.c dispatch, meson uncomment).
• Substantial rewrite of h265.c for Bugs 3-5 (~350 lines, ~3× iter1's mpeg2.c rewrite).
• New per-slice accumulation logic in picture.c::codec_store_buffer for the slice_params dynamic-array semantics.

If Phase 7 misses a check, most likely culprits in priority order:

1. VIDIOC_S_EXT_CTRLS returns EINVAL because of a struct-size mismatch (HEVC structs are large; v4l2_ctrl_hevc_sps is ~70 bytes, _pps ~80 bytes, _decode_params ~100+ bytes, _slice_params ~150 bytes per slice). Mitigation: pahole / sizeof against kernel UAPI; compare to Baseline C verbatim sizes.

2. Slice_params dynamic-array submission shape wrong — kernel may reject if size is not a multiple of sizeof(struct v4l2_ctrl_hevc_slice_params), or if num_slices doesn't match an internal kernel expectation. Phase 6 should test single-slice + multi-slice cases.

3. DECODE_PARAMS DPB ordering — VAAPI's ReferenceFrames[] uses POC-sorted order; kernel's DECODE_PARAMS.dpb[] may want a specific order (long-term first, then short-term-curr-before, etc.). FFmpeg reference shows the canonical population.

4. SCALING_MATRIX presence/absence — if BBB has flat scaling and we omit SCALING_MATRIX from the batch, kernel may default OK or may EINVAL if pps.scaling_list_data_present_flag is set. Phase 6 should test both branches if possible.

5. Pixel hash mismatch HW vs SW — many possible causes (DPB ref ordering, slice_params bit_offset wrong, NAL unit type mis-extracted). Mitigation: byte-compare per-control payload against cross-validator before debugging pixels.

If Phase 7 → Phase 4 loopback fires for any reason, the loopback target is bounded: contract is well-cited, mismatches localize to specific control struct fields.

What this leaves Phase 3 with

Phase 3 baseline measurements should:

1. Wait for linux-7 kernel upgrade (substrate update) and re-verify iter1 + T4 + SDDM.
2. Re-capture the cross-validator HEVC anchor on linux-7 (Baseline C-equivalent for HEVC).
3. Apply Bug 1 + Bug 2 as scratch patches to confirm the next failure mode is the h265.c compile error (not some new surprise).
4. Capture VAAPI VAPicture/VASlice/VAIQMatrix HEVC payload via LIBVA_TRACE for one frame's decode (to confirm field values our backend reads from VAAPI match what we expect to populate into V4L2 controls).

After Phase 3 closes, Phase 4 plan can lock the implementation diff against the captured contract.