bohr runner: sid sources leak transitively into trixie-target builds — root cause of #78 #81

New Issue

2026-05-22T06:32:57Z

claude-noether commented

2026-05-22 06:32:57 +00:00

Summary

PR #78 (ffmpeg-v4l2-request-fourier: drop --enable-libxml2) was a symptom-mitigation, not a root-cause fix. The actual root cause: bohr (debian-aarch64 runner) has sid sources configured and a transitive dependency from a -t sid apt-get install pulled libxml2-dev from 2.12 (trixie, SONAME 2) up to 2.15.2 (sid, SONAME 16) silently. Subsequent ffmpeg builds on bohr then linked libavformat.so.62 against libxml2.so.16, which trixie consumers (higgs, hertz) cannot load — daedalus-v4l2 daemon dlopen fails, Firefox VAAPI playback breaks.

Filing this so the diagnosis is on the record and the fix history doesn't get lost in conversation.

Forensic timeline

2026-05-18 22:32-22:37 (bohr apt history):

Commandline: apt-get install -y -t sid libnss3-dev rustc cargo cbindgen
Install: libxml2-16:arm64 (2.15.2+dfsg-0.1, automatic),
         libllvm21:arm64 (1:21.1.8-7+b1, automatic),
         libnss3-dev:arm64 (2:3.123.1-1),
         libstd-rust-1.94:arm64 (1.94.1+dfsg1-1, automatic)
Upgrade: libxml2-dev:arm64 (2.12.7+dfsg+really2.9.14-2.1+deb13u2 -> 2.15.2+dfsg-0.1)

The -t sid was needed to bootstrap rustc/cargo/cbindgen for Firefox 150 builds (which require rust ≥ 1.90). Transitive: sid rustc -> sid libllvm21 -> sid libxml2-16 -> sid libxml2-dev upgrade. Trixie pin file /etc/apt/preferences.d/no-sid-binaries.pref (Priority: 100) was bypassed by the explicit -t sid target-release flag.

2026-05-2x: ffmpeg-v4l2-request-fourier build pkgrel=5 ran with the bumped libxml2-dev on bohr. ./configure --enable-libxml2 linked against the sid header, embedding NEEDED libxml2.so.16 in the produced libavformat.so.62. The runtime libxml2 on bohr was still 2.12 (SONAME 2), so the binary couldn't even load on its build host — but dpkg-buildpackage doesn't run binary-runtime tests, so the broken .deb shipped.
2026-05-21: higgs upgraded to the broken pkgrel=5; daedalus-v4l2 daemon dlopen libavformat.so.62 failed with libxml2.so.16: cannot open shared object file; Firefox VAAPI silently fell back to software; user reverted higgs to cached pkgrel=4. PR #78 landed pkgrel=6 dropping --enable-libxml2 to side-step.
2026-05-22: root cause investigated. bohr's apt-cache policy libxml2-dev showed *** 2.15.2 100 (installed sid) vs 2.12.7 500 (candidate trixie) — confirmed installed version was sid.

Mitigation applied (2026-05-22)

Per-package pin file added on bohr at /etc/apt/preferences.d/libxml2-trixie-pin.pref:

Package: libxml2 libxml2-dev libxml2-utils libxml2-16
Pin: release n=trixie
Pin-Priority: 1001

Plus explicit downgrade:

apt-get install --allow-downgrades \
  libxml2-dev=2.12.7+dfsg+really2.9.14-2.1+deb13u2 \
  libxml2=2.12.7+dfsg+really2.9.14-2.1+deb13u2 \
  libxml2-16-

Side effect: the sid rustc/cargo/libllvm21/libstd-rust-* were removed (they had libxml2-16 as a hard dep). Firefox-fourier CI builds bootstrap rustc via curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh + cargo install --locked cbindgen anyway (per the EC2 build script pattern in use since 2026-05-20), so this is not a regression.

Status after mitigation

bohr libxml2-dev: 2.12.7-deb13u2 (trixie, SONAME 2), pinned at priority 1001
bohr libxml2: 2.12.7-deb13u2 (trixie)
libxml2-16: removed
sid sources still configured but no sid-versioned libxml2* can install

Belt-and-suspenders status

PR #78's --enable-libxml2 removal is now redundant for libxml2 specifically — bohr can no longer produce a sid-linked binary. But the patch is harmless to keep (FFmpeg's libxml2-backed DASH demuxer is genuinely unused on our fleet per the PR body), so leave it. Revert only if a future consumer actually needs DASH demux through libavformat.

Open: this is one of many

Other sid packages currently installed on bohr (via the same -t sid mechanism — see apt history line):

libnss3 / libnss3-dev 3.123.1 (sid)
libnspr4 / libnspr4-dev 4.38.2 (sid)
ffmpeg-v4l2-request-fourier=2:8.1+rfourier+gb57fbbe-4 (our own package — separate concern)

Each of these is a potential future #78. If any consumer-targeted package's ./configure decides to link against one of those headers, we hit the same wall.

Robust long-term options (not done in this issue):

Per-package pin all consumer-bound libraries to trixie — extend the pref file pattern. Tedious; needs auditing what's "consumer-bound".
Substrate separation — second LXD container bohr-pure with no sid sources, route firefox-fourier-only builds to bohr and everything else to bohr-pure. Cleanest but ~bohr-clone effort.
Status quo + ad-hoc reaction — fix #78-class issues as they appear. Cheap until the next one bites.

Closing this issue once a decision on (1)-(3) is made, or leaving it open as the umbrella for #78-class regressions.

## Summary PR #78 (`ffmpeg-v4l2-request-fourier`: drop `--enable-libxml2`) was a symptom-mitigation, not a root-cause fix. The actual root cause: **bohr (debian-aarch64 runner) has sid sources configured and a transitive dependency from a `-t sid` apt-get install pulled `libxml2-dev` from 2.12 (trixie, SONAME 2) up to 2.15.2 (sid, SONAME 16) silently**. Subsequent ffmpeg builds on bohr then linked `libavformat.so.62` against `libxml2.so.16`, which trixie consumers (higgs, hertz) cannot load — daedalus-v4l2 daemon `dlopen` fails, Firefox VAAPI playback breaks. Filing this so the diagnosis is on the record and the fix history doesn't get lost in conversation. ## Forensic timeline - **2026-05-18 22:32-22:37** (bohr apt history): ``` Commandline: apt-get install -y -t sid libnss3-dev rustc cargo cbindgen Install: libxml2-16:arm64 (2.15.2+dfsg-0.1, automatic), libllvm21:arm64 (1:21.1.8-7+b1, automatic), libnss3-dev:arm64 (2:3.123.1-1), libstd-rust-1.94:arm64 (1.94.1+dfsg1-1, automatic) Upgrade: libxml2-dev:arm64 (2.12.7+dfsg+really2.9.14-2.1+deb13u2 -> 2.15.2+dfsg-0.1) ``` The `-t sid` was needed to bootstrap rustc/cargo/cbindgen for Firefox 150 builds (which require rust ≥ 1.90). Transitive: sid `rustc` -> sid `libllvm21` -> sid `libxml2-16` -> sid `libxml2-dev` upgrade. Trixie pin file `/etc/apt/preferences.d/no-sid-binaries.pref` (Priority: 100) was bypassed by the explicit `-t sid` target-release flag. - **2026-05-2x**: ffmpeg-v4l2-request-fourier build pkgrel=5 ran with the bumped `libxml2-dev` on bohr. `./configure --enable-libxml2` linked against the sid header, embedding `NEEDED libxml2.so.16` in the produced `libavformat.so.62`. The runtime libxml2 on bohr was *still* 2.12 (SONAME 2), so the binary couldn't even load on its build host — but `dpkg-buildpackage` doesn't run binary-runtime tests, so the broken .deb shipped. - **2026-05-21**: higgs upgraded to the broken pkgrel=5; daedalus-v4l2 daemon `dlopen libavformat.so.62` failed with `libxml2.so.16: cannot open shared object file`; Firefox VAAPI silently fell back to software; user reverted higgs to cached pkgrel=4. PR #78 landed pkgrel=6 dropping `--enable-libxml2` to side-step. - **2026-05-22**: root cause investigated. bohr's `apt-cache policy libxml2-dev` showed `*** 2.15.2 100` (installed sid) vs `2.12.7 500` (candidate trixie) — confirmed installed version was sid. ## Mitigation applied (2026-05-22) Per-package pin file added on bohr at `/etc/apt/preferences.d/libxml2-trixie-pin.pref`: ``` Package: libxml2 libxml2-dev libxml2-utils libxml2-16 Pin: release n=trixie Pin-Priority: 1001 ``` Plus explicit downgrade: ``` apt-get install --allow-downgrades \ libxml2-dev=2.12.7+dfsg+really2.9.14-2.1+deb13u2 \ libxml2=2.12.7+dfsg+really2.9.14-2.1+deb13u2 \ libxml2-16- ``` Side effect: the sid `rustc`/`cargo`/`libllvm21`/`libstd-rust-*` were removed (they had `libxml2-16` as a hard dep). Firefox-fourier CI builds bootstrap rustc via `curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh` + `cargo install --locked cbindgen` anyway (per the EC2 build script pattern in use since 2026-05-20), so this is not a regression. ## Status after mitigation - bohr `libxml2-dev`: 2.12.7-deb13u2 (trixie, SONAME 2), pinned at priority 1001 - bohr `libxml2`: 2.12.7-deb13u2 (trixie) - `libxml2-16`: removed - sid sources still configured but no sid-versioned `libxml2*` can install ## Belt-and-suspenders status PR #78's `--enable-libxml2` removal is now redundant for libxml2 specifically — bohr can no longer produce a sid-linked binary. But the patch is harmless to keep (FFmpeg's libxml2-backed DASH demuxer is genuinely unused on our fleet per the PR body), so leave it. Revert only if a future consumer actually needs DASH demux through libavformat. ## Open: this is one of many Other sid packages currently installed on bohr (via the same `-t sid` mechanism — see apt history line): - `libnss3` / `libnss3-dev` 3.123.1 (sid) - `libnspr4` / `libnspr4-dev` 4.38.2 (sid) - `ffmpeg-v4l2-request-fourier=2:8.1+rfourier+gb57fbbe-4` (our own package — separate concern) Each of these is a potential future #78. If any consumer-targeted package's `./configure` decides to link against one of those headers, we hit the same wall. Robust long-term options (not done in this issue): 1. **Per-package pin all consumer-bound libraries to trixie** — extend the pref file pattern. Tedious; needs auditing what's "consumer-bound". 2. **Substrate separation** — second LXD container `bohr-pure` with no sid sources, route firefox-fourier-only builds to bohr and everything else to bohr-pure. Cleanest but ~bohr-clone effort. 3. **Status quo + ad-hoc reaction** — fix #78-class issues as they appear. Cheap until the next one bites. Closing this issue once a decision on (1)-(3) is made, or leaving it open as the umbrella for #78-class regressions. ## Related - PR #78 (the symptom-fix): https://git.reauktion.de/marfrit/marfrit-packages/pulls/78 - PR #76 / #77 (preceding ffmpeg PRs): #76, #77 - PR #62 (libva runner ABI skew precedent — also a symptom-fix, same root-cause class)

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: marfrit/marfrit-packages#81