libva-multiplanar/phase0_findings_iter6.md

# Iteration 6 — Phase 0 (substrate / motivation / inventory)

Opens 2026-05-05 immediately after iter5 close (`phase8_iteration5_close.md`, fork commit `c8b6ede`, campaign close `8e6d9e6`).

## Predecessor close-out summary (iteration 5 → iteration 6)

iter5 was a four-track iteration: A (DEBUG sweep) + G (PGO-disabled Firefox rebuild) + B (mpv libplacebo segfault) + E (multi-context safety). All four closed GREEN. Driver source clean (1 v4l2-request log line per 2000-frame stress, was ~30+ lines/frame pre-iter5). Process-global mutable state eliminated. Firefox-fourier 150.0.1-1.1 deployed (169 MB libxul, 21× smaller than iter3 PGO-instrumented).

The campaign's original substrate question — "make multi-planar libva work end-to-end on Rockchip hantro for production VAAPI consumers" — is empirically achieved at the libva-side decode layer. iter6 is about closing the remaining quality + verification gates and (gated on operator instruction) the upstream prep work.

## Iteration 6 candidate research questions

### A. Cap_pool resolution-change race (carried from iter5 Phase 5 sonnet C4)

> Fix the latent REQBUFS-EBUSY race in `CreateSurfaces2`. When a libva consumer probes with `vaCreateSurfaces(N, M)` then re-allocates with different dimensions while CAPTURE STREAMON is active, the cap_pool drain doesn't fully complete before REQBUFs(0) on OUTPUT — kernel returns EBUSY, driver pushes ahead with garbage `sizes[1]` (uninitialized memory shape), consumer hits SIGSEGV or falls back to SW.

**Empirical signature** (iter3 substrate, iter5 Phase 7B re-test): `Unable to request buffers: Device or resource busy` at init, then `cap_pool_init: query_buffer failed for slot N`, then ffmpeg `AVHWFramesContext: Failed to create surface: 2 (resource allocation failed)`. mpv recovers via SW fallback (no SIGSEGV in iter5 — the consumer happens to handle it). Other consumers (chromium-style: probe + real-alloc + STREAMON in tight succession) might not.

**Fix shape**: in `surface.c::CreateSurfaces2` resolution-change branch, ensure ordering is `STREAMOFF on CAPTURE → REQBUFS(0) on CAPTURE → STREAMOFF on OUTPUT → REQBUFS(0) on OUTPUT → S_FMT on OUTPUT → REQBUFS(N) on OUTPUT`. ~30 lines. The current code calls `cap_pool_destroy` (issues REQBUFS(0) on CAPTURE) then `v4l2_request_buffers(output, 0)` but doesn't STREAMOFF first — kernel rejects REQBUFS(0) on a streaming queue.

**Why first**: highest-leverage closure of a known-broken edge case. Required for cleaner upstream submission. Small scope.

### B. msync removal pixel-correctness verification (carried from iter5 Phase 5 sonnet C3)

> Add a frame-hash spot check to confirm the iter5 sweep's msync(MS_SYNC|MS_INVALIDATE) removal in `surface.c::RequestSyncSurface` doesn't silently corrupt CAPTURE buffer reads on this CMA-backed hantro setup.

**Plan**: extend mpv vaapi-copy stress test to checksum the first-frame Y-plane and compare against a known-good baseline captured pre-removal. If they match, msync removal is verified. If they diverge, restore the msync + investigate.

**Fast**: ~1 hour total (capture baseline, write checksum harness, run side-by-side).

### C. Verify chromium-fourier + iter5 driver compatibility (NEW from iter5)

> Run the existing chromium-fourier 149 (`~/src/chromium-fourier/` if it exists, or rebuild from chromium-fourier patch series) against the iter5-end libva-v4l2-request-fourier driver. Determine whether iter4+iter5 libva-side fixes obviate any of chromium-fourier's Step-1 patches.

**Why this question exists**: stock Brave on PineTab2 doesn't engage VAAPI at all (its GPU process dies at GL bindings init — `InitializeStaticGLBindingsOneOff failed`, `GLES3 is unsupported`). chromium-fourier's value is partially the GL-stack fix that gets Chromium's GPU process running, separate from any libva-side patches it carries. With iter4 (DPB FFmpeg-semantics, fresh request_fd) and iter5 (debug-clean, multi-context safe) landed, some chromium-fourier libva-side patches may now be redundant.

**Tested in iter5**: stock Brave with three GL flags (`default`, `--use-gl=egl`, `--use-gl=desktop`) all fail at GL bindings before reaching VAAPI. Different failure mode for each but never reaches the decoder.

**Diagnostic plan**:
1. Confirm chromium-fourier 149 still builds and runs on ohm
2. Point it at /home/mfritsche/fourier-test/bbb_1080p30_h264.mp4 with iter5 driver
3. Capture stderr for v4l2-request: + vaapi: traces
4. If decode works: which chromium-fourier libva-side patches are still load-bearing? (revert + retest each)
5. Surface a "minimal chromium-fourier" patch set if some patches are obsoleted

**Output**: an updated chromium-fourier patch matrix (which patches are still needed; which iter4/iter5 obsoletes).

### D. Bootlin / Mozilla upstreaming prep (carried from iter3 candidate G + iter4 + iter5)

> File the Mozilla Bugzilla bug for `/dev/media*` + V4L2-stateless RDD sandbox with the firefox-fourier patch. File a bootlin issue on `bootlin/libva-v4l2-request` with iter1+2+3+4+5 patches as a cohesive working set.

**Why now**: the iter5 sweep + multi-context fix shifts the fork toward upstream-readiness. Patches are clean. Mozilla bug doesn't exist yet (per iter3 Phase 0 Sonnet research). bootlin upstream is dormant since 2021 but the fork is now substantially ahead with empirically-verified fixes.

**Stance**: per `feedback_no_upstream.md`, gated on explicit operator instruction. Listed for completeness.

### E. Performance binding cell (carried from iter1+2+3+4+5)

> Establish a measurement protocol: drop counts, effective FPS, browser CPU%, scanout-plane residency for {mpv vaapi DMA-BUF, mpv vaapi-copy, Firefox-fourier HW (with iter5 G's PGO-disabled binary), SW baseline}. Anchor in `phaseN_evidence/`.

**Why**: anchors all iter1–iter5 claims to numbers. Carried five iterations. iter5 G's PGO-disabled Firefox is now deployed — first iteration that has both a clean driver AND a release-quality Firefox to measure with.

### F. Multi-codec audit (carried from iter1 lock backlog)

> Verify MPEG-2 decode path. iter1 lock said "H.264 first; MPEG-2 next." iter1+2+3+4+5 all H.264-only.

**Plan**: find an MPEG-2 test fixture, decode via mpv vaapi-copy + vainfo. Verify hantro G1's MPEG-2 path through libva-v4l2-request-fourier. Surface any codec-specific bugs (the iter4 DPB+request_fd fixes were H.264-specific; MPEG-2 has different control flow).

### I. Firefox VIDIOC_QBUF EINVAL on first frame (NEW from iter5 amendment 2026-05-05)

> The iter5 amendment (Utility seccomp fix) closes Track F: sandbox no longer blocks the V4L2 request API. With sandbox open, Firefox loads the v4l2_request driver, `cap_pool_init: 24 slots ready` succeeds, then a single `Unable to queue buffer: Invalid argument` (VIDIOC_QBUF EINVAL) on what looks like the first frame, after which Firefox falls back to SW.

**Why this is a new candidate, not a Track A regression**: mpv `--hwdec=vaapi-copy` decoded 2000 frames clean on the same iter5-end driver build (sha `4bed52ec5d44b389…`). Only Firefox triggers the EINVAL. So the bug lives in the Firefox-specific consumer path through libva, not in the per-frame request_fd/DPB logic that iter4 closed.

**Failure modes observed (iter5-amend telemetry, 2026-05-05)**:
```
bbb_1080p30_h264.mp4 (single decode attempt):
  v4l2-request: cap_pool_init: 24 slots ready
  v4l2-request: Unable to queue buffer: Invalid argument

YouTube avc1 (Enhancer for YouTube forcing h264, multiple decode attempts on same tab):
  v4l2-request: cap_pool_init: 24 slots ready  (×4)
  v4l2-request: Unable to set control(s): Invalid argument
  v4l2-request: Unable to queue buffer: Invalid argument
```

The S_EXT_CTRLS EINVAL is the more diagnostic signal — one of the compound H.264 controls (SPS/PPS/SLICE_PARAMS/DECODE_PARAMS/SCALING_MATRIX) is being rejected by the kernel. iter5 sweep removed the iter4 per-control TRY isolation (`d3a299b`); reinstating it temporarily (or strace) is iter6 step one.

**Diagnostic plan**:
1. Reinstate per-control VIDIOC_TRY_EXT_CTRLS isolation in v4l2.c (the iter4 diagnostic that pinpointed which compound control fails). Do not commit; use only for diagnosis.
2. Strace the Firefox Utility process during decode init. Capture exact `v4l2_ext_control` payloads. Cross-reference with FFmpeg's `v4l2_request_h264.c` (cached at `references/ffmpeg-kwiboo/`, the empirical authority per memory).
3. Once the offending control is identified, look at how Firefox's libva path constructs it differently from mpv-vaapi-copy. Likely: a header field Firefox doesn't populate, an order-of-operations difference, or a VAImage / VAExportSurfaceHandle side effect that perturbs driver state.

**Reproducibility**: 100% on either `file:///home/mfritsche/fourier-test/bbb_1080p30_h264.mp4` OR YouTube avc1 (with enhanced-h264ify or Enhancer for YouTube forcing h264) + `LIBVA_DRIVER_NAME=v4l2_request` env vars + iter5-amend Firefox 150 + sandbox enabled.

**Risk**: medium. Could be a 5-line fix in v4l2.c QBUF prep, or could surface a fundamental Firefox-vs-mpv divergence in libva surface management.

### G. V4L2_MEMORY_DMABUF (carried from iter2+3+4+5)

> Replace V4L2_MEMORY_MMAP with userspace dma-buf allocation (iter2 Fix 3 was statistical / LRU mitigation; this is architectural).

**Why**: race window mathematically eliminated. Significant kernel-side test surface (does hantro on this kernel actually accept DMABUF type? GStreamer's v4l2slh264dec uses MMAP, so DMABUF on hantro may not be tested upstream).

**Risk**: highest unknown of any candidate. Possibly requires kernel work.

### H. fourier-fresnel campaign (carried from iter5 followon-campaigns memory)

> Open the `fourier-fresnel` campaign — port libva-v4l2-request-fourier from ohm RK3568 to fresnel RK3399 (Pinebook Pro). Validates generality of iter1+2+3+4+5 fixes on a second hardware target.

**Stance**: this is a **separate top-level campaign**, not an iter6 candidate. Listed here for completeness because operator may pick it as the next thing to open (sequenced ahead of `panvk-bifrost`). Charter at `~/src/fourier-fresnel/` once opened.

### Recommended pairings

- **A + B** (cap_pool race fix + msync verify) — small, surgical, both close iter5 carryovers, both prerequisites for clean upstream submission. Tightest scope.
- **A + C** (cap_pool fix + chromium-fourier verify) — closes the cap_pool race AND validates the libva backend on a second consumer family. Mid scope.
- **D alone** (upstream prep) — gated on operator instruction. Small if everything else is already clean (which iter5 mostly achieved).
- **E alone** (perf binding cell) — anchors campaign-wide claims to numbers. Carried five iterations.
- **F alone** (MPEG-2) — validates beyond H.264 scope.
- **G alone** (DMABUF) — high-risk architectural.
- **I alone** (Firefox QBUF EINVAL) — narrow, deterministic repro, gates the only known consumer-with-iter5-amendment hard-failure. Strong candidate for the iter6 lock.

## State that carries (re-verified iter5 close)

- **Hardware**: ohm RK3568 hantro G1/G2, kernel 6.19.10. Access: `ohm` (LAN) or `ohm.vpn`.
- **Userspace**: firefox 150.0.1-1.1 (iter5 G PGO-disabled fourier rebuild), libva 2.23.0, mesa 26.0.5, libdrm 2.4.131, mpv 0.41.0-3.
- **Driver installed**: `/usr/lib/dri/v4l2_request_drv_video.so` sha256 `4bed52ec5d44b389...` (iter5-end, post-cleanup).
- **Test fixture**: bbb_1080p30_h264.mp4 sha256 `dcf8a7170fbd...`. (For Track F MPEG-2: need new fixture.)
- **Build container**: firefox-fourier LXD on boltzmann, persistent.

## State that does NOT carry

- iter5 mpv stress logs are tmpfs-volatile.
- chromium-fourier 149 — uncertain whether it still builds against current Chromium upstream / mesa / etc. Track C's Phase 2 has to confirm or rebuild.

## Tooling and measurement-instrument inventory

Same as iter5. Plus for new candidates:
- For A (cap_pool race fix): write a probe-pattern test program (`vaCreateSurfaces` 16x16 then 1920x1080 in tight succession) to deterministically trigger the race. Quicker repro than waiting for mpv libplacebo to randomly hit it.
- For B (msync verify): need a frame-hash baseline pre-iter5-sweep. Could use git checkout to a pre-c8b6ede commit + capture, then compare to iter5-end.
- For C (chromium-fourier): need to find/rebuild chromium-fourier. Operator's `~/src/chromium-fourier/` likely has the build.
- For E (perf): `pidstat -u` for CPU%, Mali-G52 freq via `/sys/class/devfreq/fde60000.gpu`.
- For F (MPEG-2): need an MPEG-2 fixture (`mpv --dump-stream` from a public DVD or transcode bbb to MPEG-2).

## In-scope (LOCKING DEFERRED — Phase 1 user input)

To be locked at Phase 1 from candidates A..G above. H is a separate top-level campaign decision, not an iter6 candidate.

## Out-of-scope (LOCKED 2026-05-05 for iteration 6)

- iter5-completed work (Track A sweep, Track G PGO rebuild, Track E multi-context, Track B mpv libplacebo) — done.
- iter4-completed work (Track A frame-11 EINVAL fix) — done.
- iter3-completed work (Track F sandbox patch) — done in firefox-fourier; upstream filing is iter6 candidate D.
- New codecs OUTSIDE H.264 / MPEG-2 (VP8/VP9/AV1/HEVC out per iter1 lock).
- New target hardware (fresnel, ampere) — separate campaign (H above).

## Phase 1 success criterion (will lock after user picks candidate)

Pre-lock template:
- For candidate A: "`vaCreateSurfaces` race-probe test program runs cleanly with no REQBUFS EBUSY events; mpv libplacebo --vo=gpu test still GREEN; iter5-end smoke test still 0 EINVAL."
- For candidate B: "vaapi-copy 100-frame run produces frame-hash matching pre-iter5-sweep baseline; OR if divergence, msync restored and verified."
- For candidate C: "chromium-fourier 149 builds + runs against iter5 driver. v4l2-request: traces present in stderr. Decode works at ≥24 fps. Patch matrix updated indicating which chromium-fourier libva-side patches are obsolete vs still required."
- For candidate D: "Mozilla Bugzilla bug filed with firefox-fourier patch attached; bootlin issue filed against libva-v4l2-request with iter1-iter5 patch series."
- For candidate E: "Anchored perf table for {mpv vaapi DMA-BUF, mpv vaapi-copy, Firefox-fourier HW, SW baseline} across drop count + CPU% + frame timing on bbb_1080p30. Reproducible from documented script."
- For candidate F: "MPEG-2 fixture decodes through hantro G1 via libva-v4l2-request-fourier without crashes / EINVAL."
- For candidate G: "vaapi-copy + vaapi --vo=null still produce real frames with V4L2_MEMORY_DMABUF-backed CAPTURE buffers; race window architecturally closed."

## Stop point

**Phase 1 lock requires user input** — pick from A..G (and any pairing).

Recommended primary: **A + B** (cap_pool race fix + msync verify) — both are iter5 sonnet review caveats, both small scope, both prerequisites for clean upstream submission. If the operator wants to make iter6 the upstream-prep iteration, **A + B + D** (the pair plus upstream filings) is the natural next step.

Alternative leans:
- **C alone** if "do we still need chromium-fourier?" is the more pressing question
- **E alone** if perf measurement matters more than upstream prep
- **F alone** if multi-codec coverage is a higher priority than refinement

After lock, iter6 phases 2..8 proceed autonomously per "Stop only if user is needed."