panvk-bifrost/mesa-panvk-bifrost-video/phase0_findings.md

Phase 0 — substrate / motivation / inventory for panvk-bifrost-video

Research question (one sentence)

Can mesa-panvk-bifrost-video expose VK_KHR_video_decode_h264 (plus its supporting extensions VK_KHR_video_queue, VK_KHR_video_decode_queue, VK_KHR_video_maintenance1) backed by the RK3566 hantro V4L2 stateless VPU, such that Khronos vk-video-samples decodes a 1080p H.264 BBB clip on ohm end-to-end with hantro engagement provable via fuser /dev/video1?

Operator-supplied mechanism (load-bearing claim — verbatim from session)

"brave is closed source and walled off from v4l2-request (checks for CHROME_OS at build time) and walled off from vaapi (expects a Vulkan output device I think). This is the exact reason I want the Vulkan driver - so brave does not just use vulkan to draw buttons, but to actively use the features to offload, create buffers that kwin can understand, yadda yadda younameit."

The structural insight: the unmodifiable consumer (Brave) speaks Vulkan natively as its compositor + GPU process buffer broker. If Vulkan grows a decode capability, Brave's existing dispatch hits it without changes. The bridge to actual decoder hardware (V4L2-stateless hantro) lives on the driver side of the boundary.

The structural claim has three parts that the campaign relies on:

1. H.264 spec parameters map across protocols. Both VkVideoDecodeH264PictureInfoKHR / VkVideoDecodeH264DpbSlotInfoKHR (Vulkan side) and the V4L2 stateless H264 controls (V4L2_CID_STATELESS_H264_SPS|PPS|DECODE_PARAMS|SLICE_PARAMS| PRED_WEIGHTS|SCALING_MATRIX) carry the same underlying H.264 spec fields. The mapping is a tedious but mechanical translation, not a semantic gap.

2. Buffers can move across protocols zero-copy. Both Vulkan (VkBuffer / VkImage with VK_EXTERNAL_MEMORY_HANDLE_TYPE_DMA_BUF_BIT_EXT) and V4L2 (V4L2_MEMORY_DMABUF) speak dmabuf. The compressed bitstream buffer (Vulkan side) → V4L2 OUTPUT queue, and the V4L2 CAPTURE queue → decoded NV12 VkImage, can both route through dmabuf fd handoffs.

3. No GPU-side computation is required for the actual decode. The hantro is autonomous; once parameters and buffers are queued via V4L2 ioctls, the VPU executes asynchronously. panvk's role is protocol translation, not GPU shader execution.

Predecessor carry-over (panvk-bifrost campaign close)

State carried forward:

• mesa-panvk-bifrost r4 installed on ohm: /usr/lib/panvk-bifrost/libvulkan_panfrost.so (md5 7810235db2a8379323acf8d2d521be9a)
• ICD JSON at /usr/lib/panvk-bifrost/icd.json
• VK_ICD_FILENAMES opt-in pattern (via brave-vulkan launcher or direct env)
• PAN_I_WANT_A_BROKEN_VULKAN_DRIVER=1 requirement (still in force — panvk's self-non-conformance gate)
• libva-v4l2-request-fourier installed on ohm (proves V4L2 H.264 decode path on hantro)
• Source pointers: NIR pass + sysval pattern in ~/src/panvk-bifrost/iter17/applied_state/panvk_vX_xfb_lower.c; PKGBUILD shape in ~/src/marfrit-packages/arch/mesa-panvk-bifrost/

Data NOT carried forward (reference history only):

• iter15's 75.7% CTS pass — wrong metric for this campaign.
• iter17's 91.7% post-XFB-decomp — wrong metric.
• libva-v4l2-request-fourier 1.16× realtime — wrong protocol layer.

This campaign measures VK_KHR_video_decode engagement + decode throughput + frame correctness in its own metric space. Phase 1 hypothesis goes here, Phase 3 measures fresh.

Tooling and measurement-instrument inventory

What's installed on ohm right now (live verification, not paper)

• mesa-panvk-bifrost r4-1 — Vulkan ICD substrate
• vulkan-headers (presumably — to be live-checked)
• libva-v4l2-request-fourier — currently holding /dev/video1 while running. Coexistence policy needed.
• ffmpeg-v4l2-request-fourier — uses libva path, same device contention
• mpv-fourier, kwin-fourier, qt6-base-fourier — display stack
• Kernel: linux-fresnel-fourier — provides hantro v4l2 stateless driver and the dma_resv patches

What needs verification (Phase 0 open items)

• Does vulkaninfo on ohm enumerate ANY video queue family today? Likely no, but baseline the no.
• Is the Vulkan loader on ohm new enough to support the VK_KHR_video_* extension surface negotiation? (Vulkan headers 1.3.221+ minimum.)
• Are vk-video-samples buildable on aarch64 today? Khronos repo KhronosGroup/Vulkan-Samples and nvpro-samples/vk_video_samples. Build deps + cmake config.
• Does Mesa ship src/vulkan/runtime/vk_video.c helpers in 26.0.6, and are they usable from a video-queue-bearing driver?
• What's the device-ownership policy between libva-v4l2-request-fourier (currently using /dev/video1) and panvk-bifrost-video if both want decode access? V4L2 m2m allows only one process at a time.

Reference implementations to read (not copy)

• Mesa NVK — src/nouveau/vulkan/nvk_video.c and surrounding. Most recent Mesa VK_KHR_video implementation. Uses NVIDIA's NVDEC via class methods. Read for: extension advertisement shape, queue family registration, session/command lifecycle, DPB management state machine.
• Mesa Anv — src/intel/vulkan/anv_video.c. Intel VCN. Mature. Read for: parameter object handling, multi-decoder DPB tracking.
• Mesa RADV — src/amd/vulkan/radv_video.c. AMD UVD/VCN. Read for: a third reference point on the abstractions Mesa's vk_video.c runtime helper expects from a driver.

Crucial: do NOT copy these. Each driver dispatches into the GPU's video engine via a tightly bound submit path. Our submit path is ioctl(VIDIOC_QBUF) to /dev/video1, a fundamentally different shape. Read the high-level structure (extension surface, queue family bring-up, session object lifecycle), then implement against the V4L2 backend ourselves.

Reference for the V4L2 side (proven-working)

• libva-v4l2-request-fourier on github → marfrit fork on packages.reauktion.de. Specifically:
  • H.264 frame-based path (single CTRL_REQ, full frame in one slice)
  • DECODE_PARAMS / SPS / PPS / SLICE_PARAMS / PRED_WEIGHTS / SCALING_MATRIX control marshalling
  • dmabuf import/export for CAPTURE queue
• Kernel v4l2-request docs: Documentation/userspace-api/media/v4l/ ext-ctrls-codec-stateless.rst — authoritative H.264 control reference.
• hantro_h264.c in the kernel — read assemble_scaling_list, reference_picture_list builder for the actual per-decode hardware ops, gives a sense of what V4L2 will accept.

In-session baseline anchor (per Phase 0 dev_process rule)

Predecessor's reference floors that must replicate at N=3 before binding cells anchor to them:

1. mesa-panvk-bifrost r4 enumerates a Vulkan device and probe_winding passes 3/3 topologies. → Verified earlier this session at 14:30 UTC with packaged r4-1; sufficient as session anchor.
2. libva-v4l2-request-fourier decodes BBB H.264 via hantro. → Verified 2026-05-21: ffmpeg -hwaccel vaapi + libva = 1.56× realtime on the same BBB file used in this session's brave instrumentation run. ffmpeg -hwaccel v4l2request (direct, bypassing libva) = 1.73× realtime. Both paths green at N=1 each; N=3 anchor still pending but the single-rep result reproduces the iter14 measurement at same magnitude so likely-stable.
3. vulkaninfo reports advertised extensions and queue families. → Measured 2026-05-21 with VK_ICD_FILENAMES=/usr/lib/panvk-bifrost/icd.json PAN_I_WANT_A_BROKEN_VULKAN_DRIVER=1: Vulkan 1.4.350 loader; 19 instance extensions; zero VK_KHR_video_* extensions; single queue family, queueCount=1; no VK_QUEUE_VIDEO_DECODE_BIT anywhere. Clean baseline — campaign deliverable is 0→1 queue family + extensions on panvk-bifrost.

If (2) or (3) fail to anchor, loop back: investigate the rig before moving to Phase 1.

Open questions for Phase 1 to resolve

These are known unknowns — they don't block Phase 0 close, but Phase 1's metric choice depends on the answer.

Q1 — Device ownership: how do libva and panvk-bifrost-video coexist?

/dev/video1 (hantro m2m) accepts one process at a time. Options:

• A. Mutually exclusive use: only one runtime holds the device at a time; user picks via env var (LIBVA_DRIVER_NAME=null → Vulkan path, etc.).
• B. Shared-device daemon: a small userspace daemon owns /dev/video1 and arbitrates V4L2 requests from multiple clients via a custom IPC protocol. Complex. Not for Phase 1.
• C. Drop libva entirely for the consumers we care about: brave uses Vulkan; firefox-fourier already uses V4L2-direct, not libva; mpv-fourier uses ffmpeg-v4l2-request-fourier. If libva-v4l2-request isn't the path for any consumer in scope, drop it from the running set for video tasks.

Recommendation for Phase 1: lock A. Document the env-toggle. Defer B to later iteration if real workloads need it.

Q2 — Does Brave even probe VK_KHR_video_decode_h264 today? — ANSWERED 2026-05-21

No, and won't engage even if we offer it. strings /opt/brave-bin/* returns zero hits for VK_KHR_video / VulkanVideoDecoder. Chromium's VulkanVideoDecoder is a Khronos design draft (Dec 2025, 13-week implementation plan, not merged) — see Vulkan Video Integration into Chromium. Beyond probing, brave-bin on PineTab2 is structurally unable to engage HW video decode at all due to the chromeos-pipeline ImageProcessor wall — see fourier:brave_arm64_vaapi_wall on DokuWiki or ~/src/brave-vaapi-fourier/DEFINITIVE_FINDING.md (measured 2026-05-21).

Implication for this campaign: Brave is NOT a Phase 1 consumer. The immediate consumer story:

• mpv with --hwdec=vulkan — enumerated today on ohm for h264 / hevc / vp9 / av1 (mpv hwdec=help confirms). Uses libavcodec's hwcontext_vulkan.c path. Once panvk-bifrost-video exposes VK_KHR_video_decode_h264, mpv-fourier becomes an immediate consumer.
• ffmpeg with -hwaccel vulkan — first-class hwaccel method, confirmed in ffmpeg -hwaccels on ohm.
• gstreamer 1.28.3 vulkan plugin — gst-plugins-bad ships vulkan{h264,h265,av1}dec (per-codec presence on this build TBD).
• Future Brave: gets it free when chromium upstream lands VulkanVideoDecoder (months/year-ish).

Phase 1 milestone stays vk-video-samples as the test client (isolates driver work from consumer-side bugs). Phase 8 close-criteria will add "mpv-fourier --hwdec=vulkan decodes BBB H.264 on ohm with fuser showing /dev/video1 engagement" — the real-world consumer proof.

Q3 — Vulkan ↔ V4L2 DPB management mismatch

Vulkan API thinks of DPB (Decoded Picture Buffer) as an array of VkImage slots owned by the driver, with the application telling the driver which slot is the output frame, which slots are references for the current decode, and when a slot can be reused.

V4L2 stateless H.264 thinks of DPB as a runtime data structure encoded in V4L2_CID_STATELESS_H264_DECODE_PARAMS (the dpb[16] array of v4l2_h264_dpb_entry), pointing at indices of frames in the CAPTURE queue.

The mapping is doable but not trivial. The Mesa NVK/Anv/RADV implementations have abstractions around this in src/vulkan/runtime/vk_video.c. Phase 0 close: read that file end to end, decide whether it's a usable harness for our V4L2 backend or whether we need a parallel set of helpers.

Q4 — Vulkan video queue family expectations vs panvk's job manager

panvk on Bifrost is JM-class (Job Manager). Job Manager has graphics

• compute + fragment ringbuffers; it has no concept of a separate video ring. The Vulkan API expects a queue family with VK_QUEUE_VIDEO_DECODE_BIT_KHR and an associated VkQueue instance you submit decode commands to.

Our submit path won't actually go to the JM at all — it'll go to V4L2. So the panvk video queue is "fake" from the GPU's perspective: it's a userspace queue that translates command-buffer-recorded video ops into V4L2 ioctls. This is fine architecturally but needs the queue infrastructure (synchronization, timeline semaphores between graphics and video families) to be wired up correctly. NVK probably has the cleanest reference for this since NVDEC is also architecturally separate from the graphics scheduler on Nvidia.

Q5 — Hantro per-stream device contention vs concurrent decodes

VK_KHR_video allows multiple VkVideoSessionKHR instances per device. If two of them concurrently want to decode different streams, the hantro m2m driver serializes them via the V4L2 queueing model, but performance contention is a real issue. Phase 1's target is a single decode session; multi-session concurrency is a Phase >>1 problem.

Predecessor inheritance summary

Inherited	Source	How used
Vulkan ICD substrate (libvulkan_panfrost.so r4)	panvk-bifrost campaign	The library we extend with video
PKGBUILD pattern for mesa-panvk-bifrost-* packages	marfrit-packages/arch/mesa-panvk-bifrost	Template for new mesa-panvk-bifrost-video
V4L2 stateless H.264 control marshalling	libva-v4l2-request-fourier	Reference; not linked into panvk
Kernel dma_resv patches	linux-fresnel-fourier	Buffer fence correctness on V4L2 producers
Build/CI on Gitea Actions aarch64 runner	marfrit-packages	Same pipeline, new package
Dev process 9(+1)-phase loop	feedback_dev_process.md	This campaign follows

Phase 0 close criteria (when this loop step is done)

• Research question + mechanism locked
• Predecessor state vs data categorized
• Live verification on ohm — vulkaninfo baselined, libva-v4l2-request re-anchored via ffmpeg side-by-side (1.56×/1.73× realtime confirmed)
• Open questions tabled — Q1 (device ownership, lock A: mutex with env), Q2 (Brave probe — ANSWERED: no, won't engage, see DokuWiki finding), Q3 (DPB mapping — Phase 1 reads Mesa NVK reference), Q4 (video queue family on JM — Phase 1 design item), Q5 (multi-session concurrency — Phase >>1 scope, lock single-session for now)
• vk-video-samples build attempt on aarch64 — PENDING, last gating item for Phase 0 close
• Phase 0 evidence dir populated with anchored measurements (phase0_evidence/) — PENDING packaging the raw measurements

What Phase 1 will lock against

After Phase 0 closes, Phase 1 will state the success metric in measurable terms. Tentative: "vk-video-samples (or equivalent Khronos Vulkan video test client, version locked) decodes a 1080p H.264 sample to NV12 frames on ohm using mesa-panvk-bifrost-video, with fuser /dev/video1 confirming hantro engagement, with no software fallback in chrome://media-internals-equivalent diagnostics, at no worse than 1.0× realtime."

The 1.0× threshold is conservative; libva-v4l2-request-fourier already does 1.16× via the same V4L2 path. The driver-bridge cost should be a few percent at worst. Anything below 0.7× indicates a buffer-copy regression to investigate.

— claude-noether, 2026-05-21