besser/notes/phase1-bug5-2026-05-07.md

Bug #5 campaign — Phase 1 metric + Phase 2 situation

Date assembled: 2026-05-07 Module under test (baseline): bes2600.ko srcversion 1B3B3ED096AAD7217FEDE11 (cleanups + Patch A + Patch B)

Phase 0 anchor at N=3 reps (10 min each, 4 MB/s pv-cap on boltzmann → ohm 2.4GHz 5b:32) reproduces the throughput regression and traces it to lock- cycling cost in the bes2600 BH path. See notes/observed-bugs.md Bug #5 for the original report. This document locks Phase 1 and prepares Phase 4 candidates.

---

Phase 1 — measurable target (locked)

Reduce the bes2600 BH path's spin-unlock-irqrestore cost so that under sustained 4 MB/s sender pressure on a healthy 2.4GHz link (signal -55 to -65 dBm), ohm sustains ≥ 2 MB/s observed RX throughput (vs 663–725 KB/s baseline at N=3) AND the link survives ≥ 30 min continuous load without cascading into beacon-loss disconnect (vs rep3's failure at ~9 min).

Three measurable outcomes, single sentence:

• (a) Throughput floor: ≥ 2 MB/s sustained RX at ohm
• (b) Lock-cycle ceiling: % CPU in _raw_spin_unlock_irqrestore from bes2600_bh.isra.0 callstack drops to < 10 % (currently ~10 % rep1, ~16 % rep3)
• (c) Cascade prevention: no link death under continuous 30 min @ 4 MB/s

---

Phase 0 anchor — receipts

Reproduction protocol (same units as Phase 7 will use)

1. boltzmann: pv -L 4M -q < /dev/zero | nc ohm.fritz.box 12345
2. ohm: sudo $RUN/rep-trace.sh 600 (10 min capture window)
3. Rep dirs: bug5/rep-<ts>/{mmc.log, perf.data, rx_bytes.tsv, start.txt, end.txt}
4. N=3 reps with 60 s cooldowns

Observed (2026-05-07 15:36–16:08)

rep	duration	avg KB/s	near-zero ticks	end state
1	600 s	725	1/119	associated, MCS 6
2	600 s	663	5/119	associated, MCS 4
3	600 s	75	53/119	passive (link died at sample ~82, ~9 min in)

mmc transaction rate: rep1 = 5793/s sustained, rep3 = 6000/s for first ~10s then collapse to <100/s.

Hot-symbol receipts (perf record on bes_sdio | bes2600_bh kworkers)

symbol	rep 1 (healthy)	rep 3 (cascade)
_raw_spin_unlock_irqrestore (sum across kworker variants)	~19 %	~21 %
handle_softirqs	5.4 %	4.3 %
__tasklet_schedule	2.4 %	2.0 %
dw_mci_start_command (SDIO host)	1.5 %	< 0.6 %
bes2600_sw_retry_requeue	0.79 %	0.70 %

Top callchain leading to _raw_spin_unlock_irqrestore:

process_one_work → worker_thread → wsm_configuration → wsm_cmd_send → bes2600_bh.isra.0

---

Phase 2 — situation analysis

Relevant source pins

• bes2600/wsm.c:98 — wsm_cmd_send(), the function in the hot callstack. Body:
  • holds &hw_priv->wsm_cmd.lock (spin) only briefly to fill the cmd struct (lines ~145–152)
  • calls bes2600_bh_wakeup() then wait_event_interruptible_timeout for the response
  • outer lock: down(&hw_priv->wsm_cmd_sema) from callers (wsm_cmd_lock() at wsm.c:105)
• bes2600/bh.c:435,559,847 — bes2600_bh_work() takes &hw_priv->vif_list_lock 3× per pass through the dispatcher
• bes2600/bh.c:172,195,487,581,861,1361,1427 — multiple wait_event* calls; the BH thread cycles through wait/wake/dispatch

What the lock-cycling cost is buying

Each WSM command from the host (mac80211, NM, kernel scan etc.) goes through the same path:

1. caller acquires wsm_cmd_sema (outer)
2. wsm_cmd_send() acquires wsm_cmd.lock, fills the struct, releases the lock
3. bes2600_bh_wakeup() schedules the BH
4. BH dispatches the command, takes vif_list_lock to walk vifs
5. BH talks to chip via SDIO
6. response arrives, BH wakes the waiter
7. caller releases wsm_cmd_sema

Under sustained TCP RX, mac80211 issues lots of small WSM commands (TX-scheduling hints, rate updates, etc.) — every one cycles through this path. The spin-unlock-irqrestore cost is the floor on this cycle rate.

What's been ruled out

• AP-side bug (AVM Fritz!Box, reliable per Markus's testimony — see feedback_phase7_stress_ramp.md reasoning in the campaign-so-far prior).
• Patch A / Patch B (target different triggers; would not address lock cost).
• Decrypt-failure storm (Patch A handles; this regression occurs in rep1 with zero decrypt-fails).
• mac80211 scan-fail / scan-comeback (cosmetic; doesn't account for the throughput floor).

---

Phase 4 — candidate plans (preliminary, not locked)

Three candidates surfaced from the perf data. Listed cheapest to most-invasive.

B5-1 — Reduce wsm_cmd_send lock scope

The spin_lock around the cmd-struct fill (wsm.c:145–152) can probably be replaced with WRITE_ONCE of a single sentinel field, since the BH thread reads these fields cooperatively (BH only reads after bh_wakeup schedules it, and only writes back via the response path). Eliminates the per-command spin-lock cycle for the host side.

Risk: race with BH if the protocol isn't strictly happens-before. Need to read bh.c:486-500 (where BH reads wsm_cmd.ptr) carefully.

Predicted delta: small but measurable. Maybe 2–3 % CPU off the lock floor.

B5-2 — Coalesce vif_list_lock in BH dispatcher

bh.c takes vif_list_lock 3× per dispatcher pass. If these 3 critical sections are within a single iteration, they should be merged into one acquire/release.

Risk: vif teardown might depend on releasing the lock between iterations to allow concurrent vif removal. Needs careful audit.

Predicted delta: significant under multi-vif (we're single-vif STA today, so probably small immediate gain).

B5-3 — Move WSM send out of process context, use ringbuffer

Replace the wsm_cmd_sema + wsm_cmd struct mechanism with an SPSC ringbuffer between caller and BH. Caller writes to ring, no lock needed (one producer); BH reads from ring, no lock needed (one consumer).

Risk: significant rework. cw1200 ancestor doesn't have this; we'd be inventing it.

Predicted delta: large — could halve the lock cost. But cost-to-implement is also large.

---

Open question for the reviewer

Which Phase 4 candidate to lock? My ranking by ROI:

1. B5-1 (smallest, fastest, cleanest source pin) — try first
2. B5-2 (medium, conditional on multi-vif applicability)
3. B5-3 (largest rework, biggest potential)

Or: instrument deeper before committing to a fix (e.g., add tracepoints around wsm_cmd_send enter/exit to measure lock holdtime distribution, not just CPU%).