codec_store_buffer: VASliceDataBufferType memcpy has no bounds check vs surface_object->source_data capacity — SEGV on resolution upshift #13

New Issue

2026-05-21T10:10:39Z

marfrit commented

2026-05-21 10:10:39 +00:00

Symptom

mpv crashes immediately at the first frame of a 1920×1088 H.264 source when the libva session was initialized at a smaller resolution. Backtrace:

#0  libc.so.6 + 0xa3188            (memcpy)
#1  RequestRenderPicture           (v4l2_request_drv_video.so + 0x6864)
#2  vaRenderPicture                (libva.so.2 + 0xe7b8)
#3  ff_vaapi_decode_issue          (libavcodec.so.62 + 0x8d1ab8)
#4  vaapi_h264_end_frame           (libavcodec.so.62 + 0x8dc43c)

Verified on higgs (Pi CM5, daedalus path) with LIBVA_DRIVER_NAME=v4l2_request mpv --hwdec=vaapi-copy bbb_1080p30_h264.mp4 against a libva session whose CAPTURE buffers were sized for an earlier 1280×720 cap_pool_init.

Location

src/picture.c:64-118 — codec_store_buffer, VASliceDataBufferType branch.

case VASliceDataBufferType:
    if (context->h264_start_code) {
        static const char start_code[3] = { 0x00, 0x00, 0x01 };
        memcpy(surface_object->source_data +
               surface_object->slices_size,
               start_code, sizeof(start_code));
        surface_object->slices_size += sizeof(start_code);
    }
    if (profile == VAProfileVP8Version0_3 &&
        surface_object->params.vp8.iqmatrix_set) {
        unsigned int header_size = ...;
        memset(surface_object->source_data +
               surface_object->slices_size,
               0, header_size);
        surface_object->slices_size += header_size;
    }
    memcpy(surface_object->source_data +
               surface_object->slices_size,
           buffer_object->data,
           buffer_object->size * buffer_object->count);  // <-- UNBOUNDED
    surface_object->slices_size +=
        buffer_object->size * buffer_object->count;
    surface_object->slices_count++;
    break;

No surface_object->source_data_capacity (or equivalent) is consulted. source_data was allocated at surface-creation time for the resolution / level the libva client originally probed. When a stream uses larger slices than the buffer was sized for — e.g. a stream-level resolution upshift, or a single I-frame whose slice happens to be unusually large — the memcpy walks past the allocation.

Fatal in mpv's --hwdec=vaapi-copy because the libavcodec VAAPI driver pre-allocates a single VA surface per FFmpeg frame; when the stream's first SPS lands at 1080p and the surface had been sized for 720p, the very first slice copy overflows.

In Firefox the bug is latent but doesn't crash visibly — VAAPI surfaces happen to be allocated with enough slack for typical YouTube resolution swings, and any small overrun lands in adjacent heap that isn't immediately dereferenced. Still a hazard.

Proposed fix

Track the source_data buffer capacity on object_surface (or wherever source_data is allocated; likely surface.c vaCreateSurfaces2 / object_surface_data_setup) and consult it before each memcpy/memset:

size_t needed = surface_object->slices_size + sizeof(start_code);
if (needed > surface_object->source_data_capacity) {
    request_log("codec_store_buffer: slice payload %zu exceeds source_data "
                "capacity %zu (resolution upshift mid-stream?)\n",
                needed, surface_object->source_data_capacity);
    return VA_STATUS_ERROR_ALLOCATION_FAILED;
}
memcpy(surface_object->source_data + surface_object->slices_size, ...);

Applied at all three append sites (Annex-B start code, VP8 frame header padding, the slice payload itself).

Better fix

Grow source_data on demand when capacity is exceeded (realloc if heap-backed, or a configurable cap level chosen at surface creation that scales with the stream's max slice size). The current design pre-sizes for the resolution at surface creation, which is brittle against any stream-level resolution change.

For a Phase 8 daedalus pipeline specifically, the bigger root cause is that surfaces should be recreated on resolution change rather than reused — but that's a larger refactor. Bounds-check first.

Severity

Medium-high — SIGSEGV inside a sandboxed process is recoverable (mpv crashes, user blinks, restarts), but the same overrun in Firefox's RDD process could in principle land in security-sensitive heap. Not exploitable as observed (memcpy is well-defined writes), but no reason to leave it unguarded.

Repro

Pi CM5 with daedalus_v4l2 stack alive, libva-v4l2-request-fourier loaded.
LIBVA_DRIVER_NAME=v4l2_request mpv --hwdec=vaapi-copy <a 1080p H.264 mp4> where the daemon's startup default capture resolution is 720p.
mpv prints 3-4 lines of h264_set_controls then SIGSEGV.

Graceful workaround for testing: pre-scale the source to match the daemon's startup capture default (1280×720 today), or restart the daemon with the larger resolution as default.

## Symptom mpv crashes immediately at the first frame of a 1920×1088 H.264 source when the libva session was initialized at a smaller resolution. Backtrace: ``` #0 libc.so.6 + 0xa3188 (memcpy) #1 RequestRenderPicture (v4l2_request_drv_video.so + 0x6864) #2 vaRenderPicture (libva.so.2 + 0xe7b8) #3 ff_vaapi_decode_issue (libavcodec.so.62 + 0x8d1ab8) #4 vaapi_h264_end_frame (libavcodec.so.62 + 0x8dc43c) ``` Verified on higgs (Pi CM5, daedalus path) with `LIBVA_DRIVER_NAME=v4l2_request mpv --hwdec=vaapi-copy bbb_1080p30_h264.mp4` against a libva session whose CAPTURE buffers were sized for an earlier 1280×720 cap_pool_init. ## Location `src/picture.c:64-118` — `codec_store_buffer`, `VASliceDataBufferType` branch. ```c case VASliceDataBufferType: if (context->h264_start_code) { static const char start_code[3] = { 0x00, 0x00, 0x01 }; memcpy(surface_object->source_data + surface_object->slices_size, start_code, sizeof(start_code)); surface_object->slices_size += sizeof(start_code); } if (profile == VAProfileVP8Version0_3 && surface_object->params.vp8.iqmatrix_set) { unsigned int header_size = ...; memset(surface_object->source_data + surface_object->slices_size, 0, header_size); surface_object->slices_size += header_size; } memcpy(surface_object->source_data + surface_object->slices_size, buffer_object->data, buffer_object->size * buffer_object->count); // <-- UNBOUNDED surface_object->slices_size += buffer_object->size * buffer_object->count; surface_object->slices_count++; break; ``` No `surface_object->source_data_capacity` (or equivalent) is consulted. `source_data` was allocated at surface-creation time for the resolution / level the libva client originally probed. When a stream uses larger slices than the buffer was sized for — e.g. a stream-level resolution upshift, or a single I-frame whose slice happens to be unusually large — the memcpy walks past the allocation. Fatal in mpv's --hwdec=vaapi-copy because the libavcodec VAAPI driver pre-allocates a single VA surface per FFmpeg frame; when the stream's first SPS lands at 1080p and the surface had been sized for 720p, the very first slice copy overflows. In Firefox the bug is latent but doesn't crash visibly — VAAPI surfaces happen to be allocated with enough slack for typical YouTube resolution swings, and any small overrun lands in adjacent heap that isn't immediately dereferenced. Still a hazard. ## Proposed fix Track the source_data buffer capacity on `object_surface` (or wherever `source_data` is allocated; likely `surface.c` `vaCreateSurfaces2` / `object_surface_data_setup`) and consult it before each `memcpy`/`memset`: ```c size_t needed = surface_object->slices_size + sizeof(start_code); if (needed > surface_object->source_data_capacity) { request_log("codec_store_buffer: slice payload %zu exceeds source_data " "capacity %zu (resolution upshift mid-stream?)\n", needed, surface_object->source_data_capacity); return VA_STATUS_ERROR_ALLOCATION_FAILED; } memcpy(surface_object->source_data + surface_object->slices_size, ...); ``` Applied at all three append sites (Annex-B start code, VP8 frame header padding, the slice payload itself). ## Better fix Grow `source_data` on demand when capacity is exceeded (`realloc` if heap-backed, or a configurable cap level chosen at surface creation that scales with the stream's max slice size). The current design pre-sizes for the resolution at surface creation, which is brittle against any stream-level resolution change. For a Phase 8 daedalus pipeline specifically, the bigger root cause is that surfaces *should* be recreated on resolution change rather than reused — but that's a larger refactor. Bounds-check first. ## Severity Medium-high — SIGSEGV inside a sandboxed process is recoverable (mpv crashes, user blinks, restarts), but the same overrun in Firefox's RDD process could in principle land in security-sensitive heap. Not exploitable as observed (memcpy is well-defined writes), but no reason to leave it unguarded. ## Repro 1. Pi CM5 with daedalus_v4l2 stack alive, libva-v4l2-request-fourier loaded. 2. `LIBVA_DRIVER_NAME=v4l2_request mpv --hwdec=vaapi-copy <a 1080p H.264 mp4>` where the daemon's startup default capture resolution is 720p. 3. mpv prints 3-4 lines of `h264_set_controls` then SIGSEGV. Graceful workaround for testing: pre-scale the source to match the daemon's startup capture default (1280×720 today), or restart the daemon with the larger resolution as default.

claude-noether referenced this issue from a commit

2026-05-21 10:14:55 +00:00

picture: bounds-check codec_store_buffer slice writes against source_size

marfrit referenced this issue

2026-05-21 10:16:37 +00:00

picture: bounds-check codec_store_buffer slice writes against source_size (#13) #14

marfrit closed this issue

2026-05-21 10:17:17 +00:00

marfrit referenced this issue from a commit

2026-05-21 10:17:18 +00:00

Merge pull request 'picture: bounds-check codec_store_buffer slice writes against source_size (#13)' (#14) from claude-noether/libva-v4l2-request-fourier:noether/codec-store-buffer-bounds-check-13 into master

marfrit referenced this issue