A pi-agent stop did not reach models served through the proxy: the forward loop
only noticed the client was gone when it next tried to WRITE, so during the
blocking non-streaming getresponse() (full generation) the backend ran to
completion holding the single-slot admission slot — a cancelled request kept
burning the NPU/DGX. Add a client-liveness watchdog: while a backend request is
in flight, select() on the client socket; on EOF close the backend conn so it
aborts and the slot frees. New _try_backend return "aborted" (no failover;
client is gone). _stream_response read loop hardened to stop cleanly when the
watchdog closes the conn mid-read. Live-verified: non-streaming request cut at
3s -> CLIENT-CANCELLED logged, slot freed, followup 200 in 4s.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EWpfhDgYNA21tETDP9ueBE
ThreadingHTTPServer fanned unbounded concurrent forwards into single-slot llama.cpp
-> pile-up (client-disconnect never cancels upstream, so contention cascades into a
hang). Add a BoundedSemaphore per local backend, slots from local-backends.json
(qwen3-4b=1, bosch-dspark=32), in-system cap = slots*4 -> fast 503 backend_busy on
saturation, ~90s bounded wait otherwise. Gates local backends only. Verified: 6
concurrent -> 4 served serially + 2 fast-503, no pile-up. Deployed on hossenfelder.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
williams