fresnel-fourier/phase2_iter8_situation.md

Iteration 8 — Phase 2 (situation analysis)

H.264 backend source-read on iter7 fork tip 6df2159. Maps the per-frame decode pipeline, enumerates code paths the kernel reads, and refines the hypothesis surface ahead of Phase 3 empirical strace-diff.

The per-frame pipeline (libva → V4L2)

For each frame in ffmpeg-vaapi-copy's 3-frame H.264 sweep:

1. RequestBeginPicture (picture.c:286)

For surface S:

• If S had a current_slot from a prior decode → surface_unbind_slot releases it.
• cap_pool_acquire(&capture_pool, surface_id) → fresh CAPTURE slot (LRU). Bind via surface_bind_slot → S's destination_index, destination_data, etc.
• request_pool_acquire(&output_pool) → fresh OUTPUT slot. Bind S's source_index, source_data (mmap pointer into the slot), source_size, and request_fd = slot->request_fd (the slot's permanent fd, REINIT'd between cycles).
• Reset slices_size = 0, slices_count = 0, params.h264.matrix_set = false.
• Set status VASurfaceRendering, record render_surface_id.

2. RequestRenderPicture (picture.c:369)

For each VAAPI buffer ID:

• VASliceDataBufferType → if context->h264_start_code, prepend 00 00 01 start code, then memcpy slice bytes into source_data + slices_size. Increment slices_size and slices_count.
• VAPictureParameterBufferH264 → memcpy into surface->params.h264.picture (the VAAPI struct, not a V4L2 control yet).
• VASliceParameterBufferH264 → memcpy into surface->params.h264.slice (LAST slice's params wins; multi-slice frames lose all but the last).
• VAIQMatrixBufferH264 → memcpy into surface->params.h264.matrix; set matrix_set = true.

3. RequestEndPicture (picture.c:408)

Synchronous decode cycle:

• gettimeofday(&surface->timestamp, NULL) — stamps the OUTPUT QBUF's timestamp = the tag the kernel will associate with the decoded CAPTURE buffer.
• request_fd = surface->request_fd (from BeginPicture's pool-slot binding).
• codec_set_controls → h264_set_controls (see §h264_set_controls below). This builds all V4L2 controls and does S_EXT_CTRLS(video_fd, request_fd=request_fd, controls, num).
• v4l2_queue_buffer CAPTURE — QBUF CAPTURE buffer (destination_index), no request.
• v4l2_queue_buffer OUTPUT with request_fd (QBUF flags |= REQUEST_FD), slices_size bytes payload.
• RequestSyncSurface (surface.c:278):
  • media_request_queue(request_fd) → MEDIA_REQUEST_IOC_QUEUE — atomically commits S_EXT_CTRLS + OUTPUT QBUF as one decode request.
  • media_request_wait_completion — POLL until decode finishes.
  • media_request_reinit — reset request fd for next cycle.
  • v4l2_dequeue_buffer OUTPUT.
  • v4l2_dequeue_buffer CAPTURE.
  • cap_pool_mark_decoded → slot state IN_DECODE → DECODED.

4. h264_set_controls (h264.c:797)

DPB mutation order:

1. output = dpb_lookup(context, CurrPic, NULL, NULL) — find existing entry for current frame's picture_id (returns NULL on first decode of a fresh surface).
2. If NULL → output = dpb_find_entry(context) = first invalid-and-unreserved entry, or oldest unused entry.
3. dpb_clear_entry(output, reserved=true) — memset 0, mark reserved=true (excludes from dpb_find_invalid_entry).
4. dpb_update(context, &surface->params.h264.picture):
   • Increment dpb.age.
   • Clear used on every entry.
   • For each ReferenceFrame in VAPicture->ReferenceFrames[i] (i < num_ref_frames):
     • If VA_PICTURE_H264_INVALID, skip.
     • dpb_lookup for that picture_id; if found, age=current, used=true. Else dpb_insert (inserts the missing reference into a free entry).

Then construct V4L2 controls: 5. h264_va_picture_to_v4l2 (h264.c:346) — fills v4l2_ctrl_h264_decode_params, pps, sps. Includes:

• Bit-parses the slice_header from the OUTPUT buffer bytes (skipping start_code if h264_start_code) for idr_pic_id, pic_order_cnt_lsb, delta_pic_order_cnt*, pic_order_cnt_bit_size, dec_ref_pic_marking_bit_size.
• Calls h264_fill_dpb which iterates DPB entries with valid && used and fills v4l2 decode->dpb[i] with reference_ts = v4l2_timeval_to_ns(ref_surface->timestamp), frame_num, pic_num, top/bottom_field_order_cnt, flags = VALID|ACTIVE(|LONG_TERM), fields = V4L2_H264_FRAME_REF.
• Sets decode->flags |= IDR_PIC if nal_unit_type == 5. Sets FIELD_PIC, BOTTOM_FIELD flags.
• Sets pps, sps flags (entropy_coding_mode, weighted_pred, transform_8x8, etc.).

6. Scaling matrix (always submitted): from VAAPI's IQMatrix if matrix_set, else flat 16 default.
7. h264_va_slice_to_v4l2 — fills slice_params + pred_weights (used only in SLICE_BASED mode; here unused).
8. pps.flags |= SCALING_MATRIX_PRESENT.
9. pps.num_ref_idx_l0/l1_default_active_minus1 = slice.num_ref_idx_l0/l1_active_minus1 (slice-derived since VAAPI doesn't forward PPS values).
10. Switch on slice_type % 5 → decode.flags |= PFRAME or |= BFRAME.
11. sps.profile_idc = h264_profile_to_idc(profile), sps.level_idc = derive_level_idc(width_mbs, height_mbs).
12. const bool slice_based = false; hardcoded. Only SPS + PPS + DECODE_PARAMS + SCALING_MATRIX controls submitted. SLICE_PARAMS and PRED_WEIGHTS skipped (per UAPI doc, must not submit in FRAME_BASED).
13. v4l2_set_controls(video_fd, surface->request_fd, controls, 4) — S_EXT_CTRLS with WHICH_REQUEST_VAL = request_fd.
14. dpb_insert(context, &CurrPic, output=entry) — current frame's CurrPic stored into the DPB entry we reserved at step 3. entry.pic = CurrPic, valid=true, reserved=false, used=true.

Per-frame state walkthrough on bbb_1080p30_h264.mp4 3-frame

Frame 1 (IDR keyframe)

• DPB empty before this frame.
• dpb_lookup(CurrPic_1) → NULL.
• dpb_find_entry → dpb_find_invalid_entry → entry[0] (all entries are !valid && !reserved). Returns entry[0].
• dpb_clear_entry(entry[0], reserved=true) → memset 0, reserved=1.
• dpb_update: clear all used. ReferenceFrames[] for IDR is empty. No further mutation.
• h264_fill_dpb: iterate; no entry with valid && used. v4l2 decode->dpb[] all-zero.
• decode.flags |= IDR_PIC (nal_unit_type==5).
• Submit S_EXT_CTRLS, QBUF CAPTURE (slot S_C0), QBUF OUTPUT (slot S_O0, timestamp tt_1).
• MEDIA_REQUEST_IOC_QUEUE, wait, REINIT, DQBUF OUTPUT, DQBUF CAPTURE.
• dpb_insert(CurrPic_1, entry[0]) → entry[0].pic = CurrPic_1, valid=1, used=1, age=1.

Frame 2 (P-frame, refs frame 1)

• DPB has entry[0] = frame_1 (valid, used, age=1).
• dpb_lookup(CurrPic_2) → NULL (CurrPic_2 has new picture_id).
• dpb_find_entry → dpb_find_invalid_entry skips entry[0] (valid=true). Returns entry[1].
• dpb_clear_entry(entry[1], reserved=true) → memset 0, reserved=1.
• dpb_update:
  • age=2.
  • Clear used on all entries (entry[0].used=0 temporarily).
  • For ReferenceFrames[0] = frame_1: dpb_lookup finds entry[0]. entry[0].age=2, entry[0].used=1.
• h264_fill_dpb:
  • entry[0] valid && used: fill v4l2 dpb[0]:
    • reference_ts = v4l2_timeval_to_ns(SURFACE(driver_data, entry[0].pic.picture_id)->timestamp) = tt_1.
    • frame_num = entry[0].pic.frame_idx (frame 1's frame_num, typically 0).
    • pic_num = FrameNumWrap(0, 1) = 0.
    • top/bottom_field_order_cnt = strip_sentinel(entry[0].pic.TopFieldOrderCnt, .flags).
    • flags = VALID|ACTIVE.
    • fields = V4L2_H264_FRAME_REF.
  • entry[1]: clear (reserved, !valid). Skip.
  • entries[2..15]: not valid. Skip.
• decode.flags |= PFRAME, frame_num=1.
• Submit S_EXT_CTRLS, QBUF CAPTURE (slot S_C1 ≠ S_C0), QBUF OUTPUT (slot S_O1 ≠ S_O0, timestamp tt_2).
• MEDIA_REQUEST_IOC_QUEUE → kernel: reads dpb[0].reference_ts=tt_1, looks up CAPTURE buffer with that ts in vb2 queue, uses as reference; decodes P MB partition; CAPTURE buffer S_C1 receives decoded pixels.
• DQBUF OUTPUT, DQBUF CAPTURE.
• dpb_insert(CurrPic_2, entry[1]) → entry[1] gets frame_2.

Frame 3 (next inter)

Same shape. Both frame_1 and frame_2 are in DPB; ReferenceFrames[] for B/P may include both. The bug signature says frame 3 is also all-zero.

Empirical bug signature recap

• Frame 1: 99.99% zero, but first 16 bytes = 81 81 80 80 80 7f 7f 7f 7f 7f 7f 80 80 80 81 81 (real chroma DC near gray). Suggests decode partially executed for a small region of frame 1 (or those 16 bytes leaked from kernel into the cap_pool buffer position; the cap_pool init pattern is constant 0x4c green; these bytes are not the init pattern). Hash 71ac099b….
• Frame 2: 100% zero (or matches the cap_pool init pattern). No decoded content.
• Frame 3: same as frame 2.

The "real first 16 bytes of frame 1" is the strongest cue: decode started writing but stopped after a few bytes. Three possibilities:

• A — Decode aborted after a partial write due to an error mid-decode.
• B — Decode completed correctly but only wrote a few bytes due to a stride / pitch / partial-DMA truncation.
• C — Memory was initialized with garbage (not 0x4c green) just before the read.

(A) and (B) are kernel-side. (C) is a libva-side allocation bug. iter5b-β changed cap_pool initialization patterns; need to verify what frame 1 actually shows vs what frame 2/3 show byte-by-byte to choose between these.

Refined hypothesis surface (Phase 0 H-A through H-I narrowed)

From source-read, I can now narrow:

• H-A (DPB stale): Backend logic is sound per the per-frame walkthrough. Eliminated unless kernel reads a control we don't fill.
• H-B (POC sentinel mis-strip): h264_strip_ffmpeg_poc_sentinel strips bit 16 if set, else returns as-is. Frame 2 POC values are computed by ffmpeg-vaapi from H.264 spec. If sentinel detection misfires on a legit POC ≥ 65536, it would subtract 65536 silently. For 30-frame BBB content, POC values are tiny — eliminate.
• H-C (slice flags missing): SLICE_PARAMS not submitted in FRAME_BASED. Eliminate.
• H-D (per-frame request_fd lifecycle): iter6 model is "one fd per slot, REINIT between cycles." Each frame gets a possibly-different request_fd (depending on which OUTPUT pool slot it borrows). The fd is REINIT'd cleanly after DQBUF. Eliminate.
• H-E (CAPTURE slot rotation): Each frame gets a different cap_pool slot. Frame 1's CAPTURE buffer (S_C0) stays in DECODED state after DQBUF. The kernel must find it by reference_ts = tt_1 to use as reference for frame 2. Live hypothesis: kernel's reference-resolution may require frame 1's CAPTURE buffer to be still QBUF'd, not DQBUF'd. The kernel-direct path may keep CAPTURE buffers QBUF'd for longer (asynchronous decode pipeline) while libva synchronously DQBUFs each frame's CAPTURE before starting the next.
• H-F (pred-weight): Not submitted in FRAME_BASED. Eliminate.
• H-G (scaling matrix not re-uploaded): Submitted every frame. Eliminate.
• H-H (start_code prefix): h264_start_code is set from MEDIA_IOC_G_EXT_CTRLS of START_CODE control in h264_get_controls. iter6 added profile gating via feedback_unconditional_codec_state.md. Need to verify on fresnel: does rkvdec advertise V4L2_STATELESS_H264_START_CODE_ANNEX_B or _NONE? If misaligned, slice data is misinterpreted. Live hypothesis — Phase 3 strace check.
• H-I (kernel UAPI contract drift): linux-fresnel-fourier 7.0-1 is v6.16-rc4-base; check for new mandatory controls added since v5.13. Specifically: V4L2_CID_STATELESS_H264_DECODE_MODE was probed at context init; the answer is FRAME_BASED on rkvdec. Other controls — none known to be required.
• NEW H-J (reference_ts derivation): v4l2_timeval_to_ns converts (s64 sec * 1e9 + ns) from surface->timestamp.tv_sec/tv_usec. The kernel-side vb2_buffer.vb2_v4l2_buffer.vb2_buffer.timestamp is also nanoseconds. Check: does surface->timestamp get set to the same value on the OUTPUT QBUF and on reference_ts? Yes — line 440 sets it, line 461 reads it. Eliminate.
• NEW H-K (CAPTURE buffer count): How many CAPTURE buffers does the backend REQBUFS? If only 1 or 2, frame 2's CAPTURE may collide with frame 1's reference. iter5b-β surface_fill_format_uniform and the cap_pool size — need to check.
• NEW H-L (slice_data_bit_offset misalignment in FRAME_BASED): In FRAME_BASED mode rkvdec parses the slice header on its own from OUTPUT bytes. If the OUTPUT bytes have the wrong start code presence vs what rkvdec expects, slice header parse misaligns. The start_code prefix is gated by context->h264_start_code which mirrors the START_CODE control's value. Live hypothesis.
• NEW H-M (frame_num field on rkvdec frame_based): rkvdec FRAME_BASED parses its own slice header but may still cross-check against decode->frame_num. If frame 2's decode->frame_num=1 and the bitstream says frame_num=1, ok. If they mismatch, kernel rejects. Probably OK; verify in strace.

Phase 3 must answer

Empirical questions for Phase 3 strace-diff:

1. Does rkvdec accept the decode? Look for DQBUF FLAG_ERROR per frame in libva-h264 vs kdirect-h264. If libva's frame 2 DQBUF carries FLAG_ERROR, the kernel rejected; this is similar to Bug 5 (HEVC).
2. What does the libva ioctl stream look like per frame vs kdirect's? Diff VIDIOC_S_EXT_CTRLS payload by frame. Look for:
   • control set differences (kdirect includes a control libva omits)
   • DECODE_PARAMS field differences (frame_num, reference_ts, flags)
   • DPB array differences (kdirect filled vs libva filled)
3. What's cap_pool's buffer count on fresnel? REQBUFS count for CAPTURE on the rkvdec OUTPUT/CAPTURE queues.
4. What's h264_start_code set to on rkvdec? Inspect MEDIA_IOC_G_EXT_CTRLS at init. Verify START_CODE = ANNEX_B (i.e., h264_start_code = true, slices prefixed with 00 00 01).
5. Byte-level frame examination: is frame 1's 16-byte real content at frame start (Y plane top-left) or somewhere in the middle? Is frame 2 truly all-zero, or is it the cap_pool init pattern 0x4c?
6. strace REQUEST_IOC_QUEUE ordering: does the request_fd carry the controls correctly? Look at kernel logs (dmesg) for any rkvdec rejection messages.

Scope confirmation

In scope (this iteration): backend code only.

• src/h264.c, src/h264_slice_header.c, src/picture.c H.264 paths, src/surface.c::RequestSyncSurface, src/cap_pool.c, src/request_pool.c request_fd lifecycle.

Out of scope: kernel patches, VP9/VP8/HEVC/MPEG-2 (read-only for regression-verify), front-end libva, performance.

Phase 3 plan

Run on fresnel:

1. Capture strace of libva-vaapi H.264 decode: strace -f -e trace=ioctl -o libva_h264_strace.log ffmpeg -hwaccel vaapi -i bbb_1080p30_h264.mp4 -frames:v 3 -vf hwdownload,format=nv12,format=yuv420p -f rawvideo libva_h264.yuv.
2. Capture strace of kdirect: same fixture via Kwiboo's ffmpeg-v4l2request: strace -f -e trace=ioctl -o kdirect_h264_strace.log ffmpeg-v4l2request -i bbb_1080p30_h264.mp4 ... > kdirect_h264.yuv.
3. Compare strace logs per-frame. Extract S_EXT_CTRLS payloads; decode struct v4l2_ext_controls + per-control payloads (sps, pps, decode_params, scaling_matrix, dpb[]) using a small python parser if scale demands.
4. Compare raw byte content of libva_h264.yuv (frame 1, 2, 3) and kdirect_h264.yuv to characterize what's actually written and where the diverence starts.
5. Check dmesg on fresnel during the run for rkvdec messages.
6. Capture findings in phase3_iter8_findings.md.

Predicted Phase 3 runtime: 1 hour wallclock contingent on fresnel uptime.

Phase 4 plan-shape predicate

Phase 4 plan is contingent on Phase 3 narrowing. Three likely outcomes:

• Outcome A — Phase 3 finds a specific control or DECODE_PARAMS field divergence (e.g., libva omits a field kdirect always fills). Fix is mechanical, 5-30 LOC in h264.c::h264_va_picture_to_v4l2 or h264_set_controls.
• Outcome B — Phase 3 finds DQBUF FLAG_ERROR on frame 2 inter. Then this is similar to Bug 5: kernel-side rejection. Iter8 either fixes whatever the wire-protocol gap is (backend), or downgrades to "narrowed but unfixed" close (deferred to a kernel-touching iteration).
• Outcome C — Phase 3 finds the kernel accepts the decode but writes to a wrong CAPTURE buffer / partial fill. Cap-pool / surface_bind_slot bug; 10-30 LOC fix in cap_pool.c or surface.c.

LOC estimate at this stage: 5-50 LOC in 1-3 files. One commit, possibly two.