fresnel-fourier/phase5b_iter8_review.md

Iteration 8 — Phase 5b Review (α-1 SPS constraint_set_flags fix)

Reviewed 2026-05-13. Reviewer: claude-sonnet-4-6.

Plan under review: /home/mfritsche/src/fresnel-fourier/phase4b_iter8_plan.md Empirical anchor: /home/mfritsche/src/fresnel-fourier/phase7_iter8_verification.md

Sources read (empirical, not theoretical):

• rkvdec-h264.c at /home/mfritsche/src/linux-rfc/drivers/staging/media/rkvdec/rkvdec-h264.c
• v4l2_request_h264.c at /home/mfritsche/src/libva-multiplanar/references/ffmpeg-kwiboo/libavcodec/v4l2_request_h264.c
• h264.c (libva backend) at /home/mfritsche/src/libva-multiplanar/libva-v4l2-request-fourier/src/h264.c
• v4l2.c (libva backend) at /home/mfritsche/src/libva-multiplanar/libva-v4l2-request-fourier/src/v4l2.c
• /usr/include/linux/v4l2-controls.h — V4L2 UAPI struct v4l2_ctrl_h264_sps

---

CRIT findings

CRIT-1: rkvdec does NOT read constraint_set_flags — the hypothesis is empirically falsified

The plan's core claim is that constraint_set_flags may trigger a "decoder-config table dispatch" or "profile-detection hint" inside rkvdec. The driver source falsifies this.

In assemble_hw_pps() (rkvdec-h264.c:633), the function translates the entire v4l2_ctrl_h264_sps struct into the hardware PPS packet. Crucially:

• sps->constraint_set_flags is never read anywhere in rkvdec-h264.c. Zero references. Confirmed by grep -n "constraint_set" /home/mfritsche/src/linux-rfc/drivers/staging/media/rkvdec/rkvdec-h264.c returning no output.
• The hardware PPS packet defines CONSTRAINT_SET3_FLAG at bit offset 12 (line 45), but that slot is hardcoded to 1 unconditionally (line 660: WRITE_PPS(1, CONSTRAINT_SET3_FLAG)). The incoming sps->constraint_set_flags does not feed it.
• PROFILE_IDC in the hardware packet is also hardcoded to 0xFF (line 659). The incoming sps->profile_idc does not feed it.

The kernel's rkvdec_h264_validate_sps() (line 1029–1065) is the only SPS validation that can abort decoding. It checks:

• chroma_format_idc <= 1
• bit_depth_luma_minus8 == bit_depth_chroma_minus8
• bit_depth_luma_minus8 == 0
• Frame dimensions within coded_fmt

constraint_set_flags is not checked. The field is fully ignored by the rkvdec driver — it does not branch on it, does not write it to hardware, and does not validate it.

Verdict: The α-1 fix will compile and land cleanly, but cannot unblock decoding. constraint_set_flags = 0x02 versus 0x00 makes zero difference to rkvdec's hardware dispatch or validation path. R-1 in the plan ("constraint_set_flags hypothesis is wrong — probability: low-to-medium") should be rated near-certain based on source evidence.

This is a plan-killer in the strict sense: α-1 will not fix Bug 4. It is low-risk (no regression hazard), but it will consume a build/install/test cycle without advancing the criterion-1 hash match.

Recommendation: Either (a) implement α-1 as a correctness hygiene patch knowing it will not fix Bug 4 and move immediately to the next hypothesis, or (b) skip α-1 and triage the next candidate directly. If α-1 is implemented, the Phase 7b γ dump will show the same 16×32 partial fill, providing one more confirmed-excluded hypothesis, which has diagnostic value.

---

CRIT-2: The ConstrainedBaseline mapping 0x42 is wrong — bit 6 does not exist

The plan's mapping table assigns VAProfileH264ConstrainedBaseline → 0x42. The plan's own SPS bit table lists:

• bit 0: constraint_set0_flag
• bit 1: constraint_set1_flag
• bit 2: constraint_set2_flag
• bit 3: constraint_set3_flag
• bit 4: constraint_set4_flag
• bit 5: constraint_set5_flag
• bits 6-7: reserved

0x42 = 0b01000010 = bit 1 + bit 6. But bit 6 is a reserved bit — there is no constraint_set6_flag. The v4l2_ctrl_h264_sps UAPI struct confirms bits 6-7 are unspecified/reserved.

Per H.264 §A.2.1.1 (Constrained Baseline profile, Amendment 2 and later), Constrained Baseline is signalled as profile_idc=66, constraint_set1_flag=1, constraint_set4_flag=1. That yields bit 1 | bit 4 = 0x02 | 0x10 = 0x12, not 0x42.

The value 0x40 that the plan mentions as an alternative ("or 0x40") is bit 6 only — also reserved, also wrong.

Since CRIT-1 already establishes that rkvdec ignores constraint_set_flags entirely, a wrong value in the mapping table causes no runtime damage. However, if α-1 is landed as a hygiene patch, the ConstrainedBaseline mapping is incorrect and would produce a non-conformant SPS for any ConstrainedBaseline stream sent to a future driver that does read this field.

Amendment required: Change VAProfileH264ConstrainedBaseline → 0x12 (bit 1 + bit 4).

---

IMP findings

IMP-1: DECODE_PARAMS DPB flags diff is the strongest remaining candidate — plan under-invests in it

Phase 3 established a second diff beyond constraint_set_flags:

libva  frame-2 DPB flags: 0x00000003 (VALID | ACTIVE)
kdirect frame-2 DPB flags: 0x00030001 (contains bit 16+17 beyond the defined flags)

The plan classifies this as "alternative α-2" and defers it. But after CRIT-1 confirms constraint_set_flags is inert, the DPB flags diff is now the leading candidate. The plan should be amended to make α-2 the primary Phase 6b/7b investigation, not a fallback after α-1 fails.

The rkvdec driver reads dpb[i].flags in config_registers() at lines 965-973:

if (dpb[i].flags & V4L2_H264_DPB_ENTRY_FLAG_ACTIVE)
    refer_addr |= RKVDEC_COLMV_USED_FLAG_REF;
if (dpb[i].flags & V4L2_H264_DPB_ENTRY_FLAG_FIELD)
    refer_addr |= RKVDEC_FIELD_REF;

And dpb[i].fields at lines 970-973:

if (dpb[i].fields & V4L2_H264_TOP_FIELD_REF)
    refer_addr |= RKVDEC_TOPFIELD_USED_REF;
if (dpb[i].fields & V4L2_H264_BOTTOM_FIELD_REF)
    refer_addr |= RKVDEC_BOTFIELD_USED_REF;

The driver also reads DPB flags in lookup_ref_buf_idx() at line 747: if (dpb[i].flags & V4L2_H264_DPB_ENTRY_FLAG_ACTIVE). If ACTIVE is not set correctly, run->ref_buf[i] remains NULL, and the driver substitutes the destination buffer as reference — producing corrupted motion compensation without emitting FLAG_ERROR.

The Phase 3 diff for frame 2 (first P-frame) showed libva DPB flags=0x3 while kdirect had flags=0x00030001. The undefined upper bits in kdirect's value are suspicious but may be a strace read artifact (the reference_ts u64 and flags u32 are adjacent in the struct; a misaligned read could merge them). Before building on this finding, it needs re-examination. The Phase 3 note already flagged this uncertainty (line 107: "may be a kernel-aligned padding artifact").

Required for Phase 6b plan: Trace the libva backend's fill_dpb_entry analog — in h264.c, the DPB entry construction — and compare it field-by-field against the ffmpeg fill_dpb_entry (v4l2_request_h264.c:71). Specifically: does libva correctly set V4L2_H264_DPB_ENTRY_FLAG_ACTIVE for short-ref frames? Does it set top_field_order_cnt / bottom_field_order_cnt?

IMP-2: The plan does not address PPS diff

Phase 3 noted PPS first bytes were identical. But Phase 3 only compared the first 12 bytes. The PPS struct has 28+ fields; a diff in num_ref_idx_l0_default_active_minus1 or pic_init_qp_minus26 or CABAC flag would also cause silent decode errors. This was acknowledged as "PPS identical" but with a shallow check.

Before the Phase 6b plan finalises its α-2 candidate, a full PPS byte comparison between libva and kdirect should be on the checklist. This is a strace re-run (no code change needed), not a code audit. Low effort, eliminates a variable.

IMP-3: The success/partial-close criteria need tightening

The plan states: "If α-1 doesn't fix → iter8 = PARTIAL — Bug 4 narrowed; iter9 candidate."

With CRIT-1 established before any build, we already know α-1 will not fix Bug 4. The partial-close framing should be revised:

• iter8 is not partial because α-1 failed — iter8 is partial because Bug 4's root cause is not yet identified. The empirical narrowing achieved (kernel-side partial decode, constraint_set_flags inert, rkvdec validate_sps not triggering) is meaningful progress and should be committed to the memory log regardless of whether α-1 is implemented.
• The Phase 8 close should enumerate what is definitively excluded: (a) libva readback bug — excluded by γ. (b) slot binding / CAPTURE buffer lifecycle — excluded by IMP-1 memset. (c) constraint_set_flags hypothesis — excluded by source read. (d) SPS validate_sps rejection — excluded by source read (none of the checked fields differ between libva and kdirect).

Naming what's excluded is more useful than noting α-1 failed, because it constrains iter9's search space.

---

MIN findings

MIN-1: The plan conflates "informational per spec" with "not inspected by drivers"

The plan argues that because H.264 §7.4.2.1.1 marks constraint_set_flags as "informational," the kernel may still use it. This reasoning is correct in principle — drivers often rely on informational SPS fields for hardware quirk routing. However, the plan then asserts that "kernel decoders have been observed to use them as profile-detection hints" without citing a specific driver. This is generic background, not evidence. Having now confirmed rkvdec does not use it, the language in the plan should not be reused as a template for future hypotheses without grounding.

MIN-2: default: fallthrough returns 0x00 for unrecognised profiles

The helper in the plan code:

default:
    return 0x00;

This is correct fallback behaviour — unknown profiles get no constraint bits. No change needed; note it for hygiene.

MIN-3: The h264_derive_level_idc parallel in the plan is valid motivation

The plan correctly observes that level_idc was previously derived from frame dimensions because VAAPI doesn't forward it, making constraint_set_flags derivation the same class of problem. This framing is accurate and worth keeping as code documentation if α-1 lands as a hygiene patch.

---

Amendments to Phase 6b

1. Drop α-1 as the primary fix hypothesis. The kernel driver does not read constraint_set_flags. α-1 may be landed as a conformance/hygiene one-liner if desired, but Phase 6b should not treat it as the iter8 criterion-1 candidate.

2. Fix the ConstrainedBaseline mapping before landing α-1. Change the value from 0x42 to 0x12 (constraint_set1_flag | constraint_set4_flag). Value: correct conformance for any future driver that reads this field.

3. Elevate α-2 (DECODE_PARAMS / DPB diff) to primary Phase 6b candidate. The next empirical step is a full field-by-field comparison of the libva DPB construction against the ffmpeg DPB construction for frame 2 (first P-frame). Specifically inspect:

   • top_field_order_cnt, bottom_field_order_cnt (libva may be zero-ing these for progressive frames where ffmpeg sends real values).
   • flags field — does libva set VALID|ACTIVE correctly for short-ref frames?
   • fields field — does libva set V4L2_H264_TOP_FIELD_REF for progressive frame references?

4. Add a PPS full-byte comparison to the Phase 7b checklist (strace re-run, no build required) to exclude PPS as a diff source.

5. Revise the iter8 close criteria. Phase 8 should log the definitively excluded hypotheses regardless of whether a criterion-1 PASS is achieved.

---

Summary table

ID	Severity	Finding	Impact
CRIT-1	CRIT	rkvdec ignores constraint_set_flags entirely — driver source confirmed	α-1 will not fix Bug 4
CRIT-2	CRIT	ConstrainedBaseline mapping 0x42 is wrong (bit 6 = reserved); correct value is 0x12	Wrong SPS if α-1 landed as-is
IMP-1	IMP	DPB flags diff is now primary candidate; Phase 6b plan should triage it before building	Redirection required
IMP-2	IMP	PPS was only shallowly compared in Phase 3	Add full-byte PPS check to Phase 7b
IMP-3	IMP	Partial-close framing incomplete; excluded hypotheses not enumerated	Phase 8 close quality
MIN-1	MIN	"Drivers have been observed" claim lacks citation	Language quality
MIN-2	MIN	default: return 0x00 is correct; no change needed	Hygiene confirmation
MIN-3	MIN	level_idc derivation parallel is valid motivation for code comment	Code documentation