libva-multiplanar/phase0_findings_iter5.md

Iteration 5 — Phase 0 (substrate / motivation / inventory)

Opens 2026-05-05 immediately after iteration 4 close (phase8_iteration4_close.md, fork commit b81ce69, campaign close 67494ae).

Predecessor close-out summary (iteration 4 → iteration 5)

iter4 was the first iteration that closed Track A — the iter1+iter2+iter3 frame-11 EINVAL carryover. Three correctness fixes landed in fork:

• 74d8dd1 — DPB fields = V4L2_H264_FRAME_REF + skip stale entries (FFmpeg-semantics match)
• 385dee1 — fresh request_fd per frame (THE load-bearing fix)
• b81ce69 — B-slice L1 reflist .fields copy-paste

Plus diagnostic instrumentation (a12d299, 4892656, f21bdf0) accumulated during the diagnostic journey.

iter4 verified Track A via mpv direct stress test on ohm: 2130 BeginPictures over 90s with 0 EINVAL of any kind — real-time HW decode through libva-v4l2-request-fourier without MOZ_DISABLE_RDD_SANDBOX=1. Track F (sandbox patch) from iter3 stays GREEN; the campaign now has a working H.264 decode pipeline through libva on hantro.

The campaign's original substrate question — "make multi-planar libva work on Rockchip hantro for production VAAPI consumers" — is empirically achieved at the libva-side decode layer.

Iteration 5 candidate research questions

A. DEBUG instrumentation sweep (carried from iter1+iter2+iter3+iter4)

Remove all accumulated diagnostic instrumentation commit-by-commit, building cleanly between each removal. End state: zero request_log() calls in non-error paths, no patch-0011 sentinel write in EndPicture, no msync workaround (or document why it stays). Driver source builds clean and vaapi-copy + vaapi smoke tests still green.

Inventory of instrumentation to remove (or keep, as decided per item):

• iter1 ENTER traces in surface entry points (CreateBuffer, BeginPicture, etc.)
• iter1 patch-0011 sentinel write in EndPicture
• iter1 patch-0010 CAPTURE/OUTPUT hex-dumps in SyncSurface
• iter1 msync(MS_SYNC|MS_INVALIDATE) workaround in SyncSurface (probably keep — was load-bearing for cache coherency)
• iter1 POC sentinel strip (KEEP — load-bearing for ffmpeg-vaapi consumers)
• iter1 patch-0014 EACCES retry-skip in v4l2_get_controls (KEEP — load-bearing reflective behavior)
• iter1 slice_header bit-precise parser + dec_ref_pic_marking_bit_size etc. (KEEP — fixes hantro hw decode)
• iter3 Y2 v1 in v4l2_ioctl_controls (REMOVE — superseded by iter4 Y2 v3)
• iter4 Y2 v3 with TRY_EXT_CTRLS retry (REMOVE — fault no longer reproduces)
• iter4 DPB census + per-entry dump (REMOVE — fault no longer reproduces)
• iter4 per-control TRY isolation (REMOVE — fault no longer reproduces)

Why first: required prerequisite for any upstream snapshot (iter5 candidate F). Was deferred at iter1+iter2+iter3+iter4. Smaller scope than C or F, fits in any iteration's slack.

Risk: removing instrumentation that's actually load-bearing. Each removal verified by re-running mpv + Firefox + vainfo smoke tests.

B. mpv libplacebo --vo=gpu segfault (carried from iter3 substrate, never iter3+iter4 scope)

Resolve the segfault on LIBVA_DRIVER_NAME=v4l2_request mpv --hwdec=vaapi --vo=gpu after 4 frames on bbb_1080p30 when Vulkan init fails (VK_ERROR_INITIALIZATION_FAILED).

Symptom (captured iter3 substrate): Vulkan init fails, mpv falls through to GPU non-vulkan path, decode runs for 4 frames cleanly, then Unable to request buffers: Device or resource busy (REQBUFS EBUSY mid-stream), then bizarre CreateSurfaces2: surf_width=16 surf_height=16 sizes[1]=1050626 (uninitialized memory shape), then SIGSEGV.

Hypothesis (iter3-era): cap_pool resolution-change path doesn't fully drain CAPTURE before REQBUFs → kernel returns EBUSY → driver pushes ahead with garbage → mmap or pool-init crashes. Could be a Mesa update side effect.

iter4 evidence point: mpv + --vo=null works for 2130 frames. So the issue is consumer-side compositor path, not libva-side decode. Diagnosis path: --vo=null (works) vs --vo=gpu (segfault) → bisect by mpv flags.

Risk: may surface a Mesa or libplacebo bug we can't fix from the libva side.

C. Performance binding cell (deferred from iter1+iter2+iter3+iter4)

Establish a measurement protocol for HW vs SW decode on this rig: drop counts, effective FPS, browser CPU%, scanout-plane residency for {mpv vaapi DMA-BUF, mpv vaapi-copy, Firefox-fourier HW (sandbox-on), SW baseline}. Anchor in phaseN_evidence/.

Why: anchors all iter1+iter2+iter3+iter4 claims to numbers. Carried four iterations. iter4's mpv stress test is a partial perf measurement (2130 frames clean, but no CPU%/drop count anchor).

Pairing potential: A (DEBUG sweep) before C — perf measurements should be on a clean instrumentation-free build. Or, run a baseline-vs-iter4 comparison BEFORE the sweep to capture the value of each instrumentation point.

D. V4L2_MEMORY_DMABUF (carried from iter2+iter3+iter4)

Replace V4L2_MEMORY_MMAP with userspace dma-buf allocation. iter2 Fix 3 was statistical (LRU mitigation); Option B is architectural (userspace owns the buffer).

Why: the cap_pool LRU is empirically working but doesn't formally close the DMA-BUF lifecycle race window. Option B closes it.

Risk: highest unknown. Possibly requires kernel work. Hantro on this kernel may not support V4L2_MEMORY_DMABUF at all; gstreamer's v4l2slh264dec uses MMAP only. Worth a probe before commit.

E. Multi-context libva safety (Sonnet review 9.6 from iter1, carried iter2/3/4)

Make the backend safe for two concurrent libva contexts in the same process (e.g. Firefox tab playing one video while another tab plays a different resolution). LAST_OUTPUT_WIDTH/HEIGHT is a process-global static; cap_pool is per-driver_data but the V4L2 device is shared.

Why: four iterations carried this. Real consumers (Firefox multi-tab, mpv-while-Firefox) would surface it. With Track A fixed, this becomes the next architectural correctness piece.

Risk: moderate. The fix shape is similar to iter2 Fix 1 (per-context state instead of process-global) but applied to more state.

F. Bootlin / Mozilla upstreaming (combined from iter3 candidate G + iter4 carryover)

File the Mozilla Bugzilla bug for /dev/media* + V4L2-stateless RDD sandbox with the iter3 firefox-fourier patch. File a bootlin issue on bootlin/libva-v4l2-request with iter1+iter2+iter3+iter4 patches as a cohesive working set.

Why: with Track A fixed, the libva-v4l2-request-fourier fork has empirical proof of working H.264 decode on hantro for any libva consumer. The patches are upstreamable in shape, just need the DEBUG sweep (A) cleanup first.

Stance: per feedback_no_upstream.md, no PR/MR/bug-file happens without explicit operator instruction. F is gated on operator decision.

G. PGO-disabled Firefox rebuild

Rebuild firefox-fourier without --enable-profile-generate=cross to get a release-quality binary suitable for performance measurement and Firefox-side stress testing.

Why: iter3's PGO-instrumented binary is 3.6 GB libxul.so and decodes at ~0.23x realtime under sandbox. iter4 verified Track A via mpv direct because the PGO Firefox couldn't reach 720+ frames in 90s. A clean Firefox-fourier build would let iter5 do Firefox-side stress testing.

Risk: ~2h rebuild on boltzmann. The infrastructure is in place (firefox-fourier LXD container persists). Edit the PKGBUILD to skip PGO, rebuild, redeploy.

Pairing potential: G + C (rebuild + perf measurement) is natural. G + B (rebuild + libplacebo investigation through Firefox-side path) is also possible.

H. New codec / hardware (deferred from iter1+ scope)

Extend to MPEG-2 (next codec per iter1 lock) or to fresnel RK3399 / ampere RK3588 hardware (next platforms).

Why: the campaign's original locked scope was H.264-first then MPEG-2; ohm RK3568 first then fresnel and ampere/boltzmann. With ohm+H.264 working, the natural extensions become possible.

Risk: new hardware iterations are their own can-of-worms. Probably one-codec-OR-one-hardware per iteration.

Recommended pairings

• A + F (DEBUG sweep + upstream prep). Most natural sequence — sweep makes the patches mailing-list-ready. Smallest combined scope.
• A + C (sweep + perf). Sweep first to get clean measurements, then C anchors the campaign-wide claims.
• B alone (libplacebo) — separate consumer-side investigation, doesn't share authoring with anything else.
• E alone (multi-context safety) — architectural correctness piece, requires focused attention.
• G + C or G + B (PGO-disabled rebuild + perf or libplacebo) — Firefox-side validation matrix.

State that carries (re-verified 2026-05-05 close)

• Hardware: ohm RK3568 hantro G1/G2, kernel 6.19.10. ohm.vpn access path. Plasma 6 Wayland session interactive.
• Userspace: firefox 150.0.1 stock + firefox-fourier 150.0.1-1.1 (PGO-instrumented) at /opt/firefox-fourier/, libva 2.23.0, mesa 26.0.5, libdrm 2.4.131, mpv 0.41.0-3.
• Test fixture: /home/mfritsche/fourier-test/bbb_1080p30_h264.mp4 sha256 dcf8a7170fbd....
• Driver installed: /usr/lib/dri/v4l2_request_drv_video.so post-iter4 (sha256 to recompute on iter5 start; rebuild on ohm via meson+ninja in /tmp/libva-src to redeploy with iter4 commits).
• Build container: firefox-fourier LXD on boltzmann, ssh -J boltzmann builder@firefox-fourier. Persistent. Source still extracted at /build/aur/firefox-fourier/src/firefox-150.0.1/ with iter3 patches applied — incremental rebuilds via ./mach build.
• Phase 7 scripts: /home/mfritsche/iter3_phase7_evidence.sh + /tmp/run_phase7_v2.sh on ohm.vpn.
• mpv stress test command: LIBVA_DRIVER_NAME=v4l2_request mpv --hwdec=vaapi-copy --vo=null --no-audio bbb_1080p30_h264.mp4 — proven Track A verifier.
• References cache: references/ffmpeg-kwiboo/ (FFmpeg V4L2-request reference), references/linux-mainline/ (kernel hantro source), references/firefox-master/ (Mozilla sandbox source).

State that does NOT carry

• Performance numbers. Same caveat as iter1+iter2+iter3+iter4. Candidate C is the natural anchor.
• iter4 driver build state on ohm /tmp/libva-src is tmpfs-volatile; rsync+rebuild from rpi at iter5 start.

Tooling and measurement-instrument inventory

Carried from iter4:

• strace -f -e trace=openat,close,ioctl for libva-side V4L2 ioctl tracing
• sudo ftrace events/v4l2/* events/vb2/* events/dma_fence/* for kernel-side V4L2/vb2 lifecycle
• sudo dmesg -w for kernel-side warnings
• mpv --frames=N --vo=null with stderr capture for libva stress
• mpv --frames=N --vo=gpu with stderr capture for full-pipeline (will surface candidate B's segfault)
• Firefox MOZ_LOG=PlatformDecoderModule:5,VideoBridge:5 (under firefox-fourier, no MOZ_DISABLE_RDD_SANDBOX needed)
• Operator visual inspection on real screen (load-bearing for "frames reach screen" claims)
• iter3 Y2 v1 + iter4 Y2 v3 + iter4 DPB census + iter4 per-control TRY iso (ALL up for removal in candidate A)

Likely needed for specific iter5 candidates:

• For A (sweep): per-removal smoke test recipe (vainfo + mpv vaapi-copy + Firefox-fourier 30s).
• For B (libplacebo): mpv --vo=gpu minimal repro, possibly Mesa bisect or rollback.
• For C (perf): pidstat -u -p $(pidof ...) for CPU%, Mali-G52 freq via /sys/class/devfreq/fde60000.gpu, scanout-plane query (Wayland ext-output-management is hard — may need ftrace).
• For D (DMABUF): gbm_bo_create test program + VIDIOC_QBUF type=V4L2_MEMORY_DMABUF exploratory.
• For G (PGO-disabled rebuild): edit firefox-fourier PKGBUILD to skip --enable-profile-generate=cross, ./mach build incremental, redeploy via 600 MB tarball.

In-scope (LOCKED 2026-05-05 for iteration 5) — A + G + B + E

Heavyweight four-track iteration. Natural execution sequence:

1. Track A first (DEBUG sweep) — clean the codebase before other changes ripple. Per-removal commits with smoke verification. Estimated ~30-60 min for the removal commits + smoke runs.
2. Track G in parallel (PGO-disabled Firefox rebuild) — kick off on boltzmann firefox-fourier container while local A work proceeds. ~2h rebuild + redeploy to ohm; mostly background time. Edit PKGBUILD to remove --enable-profile-generate=cross, ./mach build from existing source tree (already extracted with iter3 patches).
3. Track E (multi-context libva safety) — architectural change after the codebase is clean from A. Move LAST_OUTPUT_WIDTH/HEIGHT from process-global static to per-driver-data (or per-context). Audit other process-global state. Verify via two simultaneous mpv processes on different fixtures.
4. Track B last (mpv libplacebo --vo=gpu segfault) — consumer-side investigation on the post-A post-E driver. Bisect mpv flags between --vo=null (works) and --vo=gpu (segfaults). May surface a Mesa/libplacebo upstream issue.

iter5 is large. Phase 4 will subdivide into 4A / 4G / 4E / 4B sub-phases. Phase 5 sonnet review + Phase 7 verify span all four tracks.

Out-of-scope (LOCKED 2026-05-05 for iteration 5)

• Candidates C (perf binding cell), D (V4L2_MEMORY_DMABUF), F (upstreaming), H (new codec/hardware) — deferred to iter6+.
• Same standing OOS: bootlin/Mozilla upstreaming PR/MR/bug-file unless explicitly tasked, new codecs outside H.264/MPEG-2 scope, new platforms beyond ohm RK3568.

Phase 1 success criterion (LOCKED 2026-05-05)

Track A: Driver source builds clean with zero request_log() calls in non-error paths. iter1 patch-0010/0011/0014 DEBUG, iter3 Y2 v1, iter4 Y2 v3 / DPB census / per-control TRY iso all removed (or explicitly kept with justification). mpv stress test still 2000+ BeginPictures clean post-sweep.

Track G: Firefox-fourier rebuilt without --enable-profile-generate=cross (PGO-instrumented), redeployed to ohm. firefox --version reports Mozilla Firefox 150.0.1. Resulting libxul.so is materially smaller than the 3.6 GB instrumented build.

Track E: Two concurrent mpv processes on different bbb fixtures decode independently with no cross-context state corruption. LAST_OUTPUT_WIDTH/HEIGHT (and any other audited process-global) moved to per-driver-data or per-context.

Track B: mpv --hwdec=vaapi --vo=gpu plays bbb_1080p30 for ≥30s without segfault — OR root cause documented as Mesa/libplacebo upstream issue with operator-actionable workaround captured in iter5 close.

Joint success: all four tracks independently verifiable on the same iter5-end driver build. Phase 7 verifies each.

Stop point

Phase 1 LOCKED. iter5 proceeds to Phase 2 (situation analysis covering all four tracks: instrumentation inventory, PKGBUILD PGO edit, libva-side process-global state audit, libplacebo segfault repro), Phase 3 (baseline anchor), Phase 4 (split into 4A/4G/4E/4B sub-phases per Track), Phase 5 (sonnet review of all four), Phase 6 (deploy + smoke), Phase 7 (verify each track), Phase 8 (close). Stop only if user input is needed.