dcw2: agent-container storage is a USB flash drive that dropped off the bus (btrfs went read-only, host rebooted) #8

New Issue

2026-07-22T13:39:00Z

williams commented

2026-07-22 13:39:00 +00:00

What happened

On 2026-07-22 ~09:23 dcw2 rebooted after its btrfs storage pool failed. Kernel log from the previous boot:

BTRFS error (device sda state EA): bdev /dev/sda errs: wr 132, rd 8125, flush 1, corrupt 0, gen 0
readlink: attempt to access beyond end of device
         sda: rw=0, sector=41958744, nr_sectors = 48 limit=0

Diagnosis

limit=0 — the kernel saw /dev/sda as zero sectors. The device did not corrupt, it fell off the USB bus while mounted. Every subsequent read is "beyond end of device" because there is no device.
corrupt 0, gen 0 — no checksum or generation errors. Data was fine; the link died. Confirmed on the next boot by a clean start tree-log replay.
state EA — btrfs had already flipped to error + aborted-transaction, i.e. the filesystem was unusable, which took down the incus storage pool and the containers on it.
Uptime at failure was ~14 days, so this is not a boot-time issue.

The hardware

/dev/sda is 154b:1007 — a PNY USB 3.2 flash drive (250 GB), on the uas driver, backing the incus storage pool usb and /mnt/usb-noether.

This drive hosts noether — the primary Claude Code agent container, including the canonical agent memory and CLAUDE.md.

Ruled out:

Power — vcgencmd get_throttled = 0x0, no undervoltage, not even a historical flag.
Data corruption — see corrupt 0, gen 0 above.
Capacity — pool is 27% used, 168 GiB unallocated.

Remaining suspect: cheap flash controller + UAS + sustained container write load. A well-known Pi 4 failure mode.

Options

Replace with a real USB3 SSD (preferred). A flash drive is not a wear-levelled sustained-random-write device and is a poor backing store for container storage, let alone the canonical agent memory.
Mitigate in place: usb-storage.quirks=154b:1007:u in /boot/cmdline.txt to disable UAS and fall back to BOT. Slower, far more tolerant of flaky controllers. Requires a reboot of dcw2, which drops noether/pica/pipi/orca/dcc2.
Independently: verify the backup situation for noether's memory.

Related fix already applied

dcw2 had no persistent journal (Raspberry Pi OS ships /usr/lib/systemd/journald.conf.d/40-rpi-volatile-storage.conf forcing Storage=volatile) and no rsyslog, so boot -1 was unrecoverable — the log above came from a human clipboard, not from disk. Persistent journald is now configured (99-persistent.conf, Storage=persistent, SystemMaxUse=500M, MaxRetentionSec=2month) and verified writing to /var/log/journal/. A recurrence will now be diagnosable.

## What happened On 2026-07-22 ~09:23 dcw2 rebooted after its btrfs storage pool failed. Kernel log from the previous boot: ``` BTRFS error (device sda state EA): bdev /dev/sda errs: wr 132, rd 8125, flush 1, corrupt 0, gen 0 readlink: attempt to access beyond end of device sda: rw=0, sector=41958744, nr_sectors = 48 limit=0 ``` ## Diagnosis - **`limit=0`** — the kernel saw `/dev/sda` as **zero sectors**. The device did not corrupt, it **fell off the USB bus while mounted**. Every subsequent read is "beyond end of device" because there is no device. - **`corrupt 0, gen 0`** — no checksum or generation errors. Data was fine; the *link* died. Confirmed on the next boot by a clean `start tree-log replay`. - **`state EA`** — btrfs had already flipped to error + aborted-transaction, i.e. the filesystem was unusable, which took down the incus storage pool and the containers on it. - Uptime at failure was ~14 days, so this is not a boot-time issue. ## The hardware `/dev/sda` is `154b:1007` — a **PNY USB 3.2 flash drive** (250 GB), on the `uas` driver, backing the incus storage pool `usb` and `/mnt/usb-noether`. **This drive hosts `noether`** — the primary Claude Code agent container, including the canonical agent memory and CLAUDE.md. Ruled out: - **Power** — `vcgencmd get_throttled` = `0x0`, no undervoltage, not even a historical flag. - **Data corruption** — see `corrupt 0, gen 0` above. - **Capacity** — pool is 27% used, 168 GiB unallocated. Remaining suspect: cheap flash controller + UAS + sustained container write load. A well-known Pi 4 failure mode. ## Options 1. **Replace with a real USB3 SSD** (preferred). A flash drive is not a wear-levelled sustained-random-write device and is a poor backing store for container storage, let alone the canonical agent memory. 2. **Mitigate in place**: `usb-storage.quirks=154b:1007:u` in `/boot/cmdline.txt` to disable UAS and fall back to BOT. Slower, far more tolerant of flaky controllers. **Requires a reboot of dcw2**, which drops noether/pica/pipi/orca/dcc2. 3. Independently: verify the backup situation for noether's memory. ## Related fix already applied dcw2 had **no persistent journal** (Raspberry Pi OS ships `/usr/lib/systemd/journald.conf.d/40-rpi-volatile-storage.conf` forcing `Storage=volatile`) and no rsyslog, so boot -1 was unrecoverable — the log above came from a human clipboard, not from disk. Persistent journald is now configured (`99-persistent.conf`, `Storage=persistent`, `SystemMaxUse=500M`, `MaxRetentionSec=2month`) and verified writing to `/var/log/journal/`. A recurrence will now be diagnosable.

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: marfrit/claude-his-agent#8