Files

T

marfrit f04d7000f8 Phase 8.13: byte-exact end-to-end via libva (consumer target hit)

The project's consumer-side goal landed: a real VAAPI consumer
(ffmpeg with -hwaccel vaapi) drives our libva backend → V4L2
driver → daemon → byte-exact NV12 output back to ffmpeg.

  ffmpeg -hwaccel vaapi -hwaccel_device /dev/dri/renderD128 \
         -hwaccel_output_format nv12 -i vp9_small.ivf \
         -f rawvideo -y /tmp/vp9_via_libva.nv12
  cmp /tmp/vp9_via_libva.nv12 /tmp/vp9_ref_for_libva.nv12  → match

18432-byte NV12 byte-for-byte identical to plain ffmpeg
-pix_fmt nv12 software decode. The project_consumer_target
memory's deliverable shape — "V4L2 stateless node consumed by
a real VAAPI client" — is achieved.

Two related kernel changes:

1. v4l2_ctrl_handler_setup(&ctx->hdl) after registration —
   matches rkvdec/cedrus/hantro. Brings each registered
   compound control out of "uninitialised" state via
   std_init_compound defaults.

2. Per-request control completion in the decode path —
   the real fix for "Timeout when waiting for media request".
   vb2-core's vb2_buffer_done unbinds the BUFFER's req_obj
   on normal decode completion, but the per-request CONTROL
   object stays bound. buf_request_complete fires only from
   queue-cancel paths (vb2-core line 2284), NOT from normal
   buf_done. The driver must call
   v4l2_ctrl_request_complete(req, hdl) explicitly from the
   completion path.

   struct daedalus_inflight gained a `struct media_request
   *req` field, captured from src_buf->vb2_buf.req_obj.req
   in device_run. daedalus_complete_resp_frame then calls
   v4l2_ctrl_request_complete before
   v4l2_m2m_buf_done_and_job_finish — triggers
   MEDIA_REQUEST_STATE_COMPLETE and wakes the request fd
   poll.

   For non-request flows (test_m2m_stream direct QBUF)
   inf->req is NULL; the conditional skips the call.
   Both consumer styles work concurrently.

Diagnostic clarification (was Phase 8.13a):

strace traced three S_EXT_CTRLS calls per frame:
  1. H264_PROFILE + H264_LEVEL → EINVAL  (we don't register)
  2. HEVC_PROFILE + HEVC_LEVEL → EINVAL  (we don't register)
  3. VP9_FRAME + VP9_COMPRESSED_HDR → SUCCESS

The first two are harmless: libva probes whether we support
H264/HEVC integer profile/level controls during config
negotiation; we don't (we expose them as stateless), so EINVAL
just falls through. The actual VP9 stateless controls (#3)
succeeded all along — the libva-side "Unable to set control(s)"
log was misleading us into thinking the control path was the
bug.

Verification on hertz (Pi 5, 6.12.75+rpt-rpi-2712):

  daemon log:
    REQ_DECODE cookie=1 codec=1 bitstream=1566 bytes capture=128x96 1 planes
    decoder: opened vp9 context
    decoder: OK 128x96 fmt=0 (yuv420p) fnv1a=0x1eb34bfe ...

  ffmpeg side:
    no Timeout, no Decoding error
    /tmp/vp9_via_libva.nv12: 18432 bytes

  cmp vs reference: byte-for-byte identical.

Roadmap update:
- 8.10/8.11, 8.12, 8.13 marked closed with closure docs.
- 8.14 = multi-frame VP9 via libva, AV1 + H.264, mpv/Firefox
  higher-level consumers.

Per correctness-before-speed:
- strace + kernel-source-reading found the actual root cause
  rather than guessing.
- Conditional v4l2_ctrl_request_complete preserves the existing
  test_m2m_stream non-request path — both consumer styles work
  concurrently without per-flow branching elsewhere.
- Byte-exact pixel comparison, not "frame size matches."

Phase 8.14 next: multi-frame stream + multi-codec via libva +
mpv/Firefox.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

2026-05-18 18:14:34 +00:00

7.4 KiB

Raw Permalink Blame History

daedalus-v4l2 — roadmap

Sub-phases

Phase 8.1 — kernel module skeleton

Out-of-tree kernel module that:

Registers /dev/videoNN with VFL_TYPE_VIDEO + a no-op V4L2 stateless dispatch table.
Accepts open/close, S_FMT, REQBUFS ioctls without doing anything (yet).
Builds against /lib/modules/$(uname -r)/build.

Deliverable: modprobe daedalus_v4l2 works, v4l2-ctl --list-devices shows the new device.

Phase 8.2 — kernel ↔ daemon chardev bridge

Kernel module creates /dev/daedalus-v4l2 chardev.
Defines a simple req/resp protocol in include/daedalus_v4l2_proto.h.
Daemon connects, exchanges echo requests.

Deliverable: ping-pong test passes.

Phase 8.3 — daemon FFmpeg dlopen + parse

Daemon links libdaedalus_core.a from sibling.
Daemon dlopens FFmpeg.
Test program: feed a VP9 IVF file to FFmpeg parsers, extract block-level metadata, validate against expected.

Deliverable: daemon can parse a VP9 frame and walk the block-level info.

Phase 8.4 — daemon ↔ kernel decode round-trip (closed 2026-05-18)

Shipped as a debugfs-triggered chardev round-trip rather than the original V4L2-ioctl plan (which moved to Phase 8.5).

REQ_DECODE / RESP_FRAME wire protocol
Daemon decodes VP9 via FFmpeg dlopen, returns FNV-1a digest
Verified content-dependent + deterministic; structured error handling for bad bitstreams

See docs/phase_8_4_closure.md.

Phase 8.5 — full V4L2 m2m driver (closed 2026-05-18)

Real V4L2 m2m driver — userspace clients drive S_FMT/REQBUFS/QBUF/DQBUF the standard way. Bitstream flows kernel→daemon as inline REQ_DECODE payload; decoded NV12 pixels flow daemon→kernel as inline RESP_FRAME payload. Works end-to-end for small frames (≤ ~64 KiB NV12).

Deliverable hit: kernel m2m driver passes most v4l2-compliance checks; tools/test_m2m_decode produces a NV12 frame that's byte-for-byte identical to ffmpeg -pix_fmt nv12 reference.

See docs/phase_8_5_closure.md.

Phase 8.6 — dmabuf + AV1 + H.264 + stateless controls (closed 2026-05-18)

CAPTURE (and OUTPUT) on vb2_dma_contig_memops.
New DAEDALUS_IOC_GET_DMABUF chardev ioctl — daemon mmaps the in-flight CAPTURE buffer, decodes pixels in place, sends RESP_FRAME metadata-only.
64 KiB frame-size cap removed. 1080p VP9 + 128×96 AV1
- 128×96 H.264 all byte-exact against reference FFmpeg decode.
V4L2 stateless controls registered for VP9 / AV1 / H.264 (11 controls visible to userspace).
Colorspace round-trip fix (TRY_FMT preserve, S_FMT OUTPUT→CAPTURE propagation).
Cookie unified across V4L2 + debugfs paths.
v4l2-compliance: 47/48 (only DECODER_CMD remains, needs media controller — moved to 8.7).

See docs/phase_8_6_closure.md.

Phase 8.7 — media controller + multi-frame streaming (closed 2026-05-18)

Media controller bound via v4l2_m2m_register_media_controller + media_device_register; /dev/mediaN published.
tools/test_m2m_stream parses IVF and pushes frames sequentially through a 4-deep buffer ring; daemon AVCodecContext preserves reference frames across calls.
30-frame VP9 320×240 stream byte-exact (3.46 MB across 1 keyframe + 29 P-frames).
10-frame VP9 1080p stream byte-exact (31 MB across 10 frames at full HD).
v4l2-compliance: 49/49 passing (was 47/48 in 8.6; media controller added a 49th test and closed DECODER_CMD).

See docs/phase_8_7_closure.md.

Phase 8.8 — throughput baseline + multi-codec streams + HDR (closed 2026-05-18)

Per-frame µs timing in test_m2m_stream; multi-codec baseline:
- VP9 1080p: 83.1 fps
- AV1 1080p: 65.0 fps
- H.264 1080p: 88.3 fps All byte-exact vs ffmpeg reference; all 2-3× over the 30fps-floor-is-fine criterion.
QPU dispatch substitution explicitly not needed — measurement shows the FFmpeg software path already clears the target on Pi 5's Cortex-A76. Substitution moves to the optimisation roadmap.
Annex-B H.264 access-unit splitter in the test harness (NALs grouped by VCL boundary).
HDR / 10-bit: V4L2_PIX_FMT_P010 added as CAPTURE format; daemon pack_p010_to_plane handles YUV420P10LE → P010 with MSB-aligned 10-bit data. 10-bit 1080p byte-exact at 48.8 fps.

See docs/phase_8_8_closure.md.

Phase 8.9 — long-form stress + multi-codec HDR + libva scoping (closed 2026-05-18)

libva-v4l2-request investigation: upstream supports only MPEG-2 / H.264 / HEVC (no VP9 or AV1) and expects the older V4L2_PIX_FMT_H264_SLICE_RAW fourcc. Real integration requires adding VP9 + AV1 support to the library itself — pushed to Phase 8.10.
Long-form stress: 1800-frame VP9 1080p (60s @ 30fps), 120.9 fps sustained, p99 17.3 ms/frame, no errors, no leaks, daemon alive after 3620 cookies across two runs.
HDR multi-codec byte-exact: VP9-10bit (48.8 fps, from 8.8), AV1-10bit (17.1 fps), H.264-10bit (26.9 fps). 10-bit is intrinsically more expensive — AV1 falls short of 30fps but acceptable for the user-facing goal (mostly SDR YouTube).

See docs/phase_8_9_closure.md.

Phase 8.10 + 8.11 — libva consumer integration scaffold (closed 2026-05-18)

Forked bootlin/libva-v4l2-request to marfrit/libva-v4l2- request-fourier (gitea); discovered the existing fourier fork already had VP9/AV1/HEVC support on Rockchip.
Added daedalus_v4l2 to known_decoder_drivers + meson build option.
Added V4L2_PIX_FMT_NV12 single-plane + request API media ops + stateless control registration to our kernel.
vainfo enumerates 7 VAProfile entries via our driver.

See docs/phase_8_10_11_closure.md.

Phase 8.12 — first VP9 frame decoded via libva (closed 2026-05-18)

vb2_queue supports_requests; vb2_ops buf_out_validate + buf_request_complete; v4l2_ctrl_new_custom for stateless ctrl registration.
Daemon decoded byte-exact VP9 keyframe via the full libva path (FNV-1a 0x1eb34bfe matches standalone).
ffmpeg still timed out waiting for media_request completion (request bind state).

See docs/phase_8_12_closure.md.

Phase 8.13 — byte-exact end-to-end via libva (closed 2026-05-18)

Traced the misleading "Unable to set control(s): Invalid argument" — actually libva probing H264/HEVC PROFILE/LEVEL we don't expose (harmless); the real VP9 stateless control SET succeeds.
Diagnosed the "Timeout when waiting for media request" root cause: per-request control object stays bound because vb2's normal buf_done path doesn't fire buf_request_complete (only queue-cancel does).
Fix: capture media_request from src_buf in inflight entry, call v4l2_ctrl_request_complete from the completion path before buf_done_and_job_finish.
Byte-exact end-to-end: ffmpeg -hwaccel vaapi → libva-v4l2-request-fourier → /dev/video0 → daedalus_v4l2 → daemon → 18432-byte NV12 byte-for-byte identical to ffmpeg software reference.
Project consumer target hit: V4L2 stateless node consumed by a real VAAPI client.

See docs/phase_8_13_closure.md.

Phase 8.14 — multi-frame + AV1/H.264 + higher-level consumers

Multi-frame VP9 stream via libva (vp9_60s.ivf from 8.9 stress test) — P-frame references across requests.
AV1 + H.264 single-frame via libva.
mpv --hwdec=vaapi end-to-end.
Firefox / WebRTC if motivated.

Optimisation work (QPU dispatch, 4K, HDR-in-libva) ships when there's a concrete user-facing need.

Effort estimate

Each phase: ~1 week of focused work (~40 hours). Total: 7 weeks for v1.

Could be split across multiple sessions / contributors.

7.4 KiB Raw Permalink Blame History Unescape Escape