Compare commits

2 Commits

Author SHA1 Message Date
williams 6cbf352851 llm-proxy: propagate client cancellation to the backend (free the slot)
A pi-agent stop did not reach models served through the proxy: the forward loop
only noticed the client was gone when it next tried to WRITE, so during the
blocking non-streaming getresponse() (full generation) the backend ran to
completion holding the single-slot admission slot — a cancelled request kept
burning the NPU/DGX. Add a client-liveness watchdog: while a backend request is
in flight, select() on the client socket; on EOF close the backend conn so it
aborts and the slot frees. New _try_backend return "aborted" (no failover;
client is gone). _stream_response read loop hardened to stop cleanly when the
watchdog closes the conn mid-read. Live-verified: non-streaming request cut at
3s -> CLIENT-CANCELLED logged, slot freed, followup 200 in 4s.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EWpfhDgYNA21tETDP9ueBE
2026-07-20 22:14:40 +00:00
claude-noether 06f20ae752 llm-proxy: per-backend admission gate (semaphore, bounded queue, 503 on saturation)
ThreadingHTTPServer fanned unbounded concurrent forwards into single-slot llama.cpp
-> pile-up (client-disconnect never cancels upstream, so contention cascades into a
hang). Add a BoundedSemaphore per local backend, slots from local-backends.json
(qwen3-4b=1, bosch-dspark=32), in-system cap = slots*4 -> fast 503 backend_busy on
saturation, ~90s bounded wait otherwise. Gates local backends only. Verified: 6
concurrent -> 4 served serially + 2 fast-503, no pile-up. Deployed on hossenfelder.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-20 13:50:18 +02:00
3 changed files with 1047 additions and 351 deletions
+1043 -324
View File
File diff suppressed because it is too large Load Diff
+4
View File
@@ -0,0 +1,4 @@
[
{"name":"qwen3-4b","host":"boltzmann.fritz.box","port":8091,"slots":1},
{"name":"bosch-dspark","host":"bosch.fritz.box","port":8888,"slots":32}
]
-27
View File
@@ -1,27 +0,0 @@
[Unit]
Description=rk-llama.cpp NPU backend: Qwen3 4B Q8_0 on RK3588 NPU (port 8091)
After=network-online.target
Wants=network-online.target
[Service]
Type=simple
# Kernel gate: only run under an NPU-capable kernel (rkopnu present; matches *npu*).
ExecCondition=/bin/sh -c 'case "$(uname -r)" in *npu*) exit 0;; *) exit 1;; esac'
LimitNOFILE=65536
WorkingDirectory=/home/mfritsche/npu
ExecStart=/home/mfritsche/src/rk-llama.cpp/build/bin/llama-server\
-m /home/mfritsche/models/Qwen3-4B-Instruct-2507-Q8_0.gguf\
--alias qwen3-4b-2507-npu\
-c 65536 -t 4 --cache-type-k q8_0 --cache-type-v q8_0\
--host 0.0.0.0 --port 8091
Restart=on-failure
RestartSec=5
TimeoutStartSec=180
[Install]
WantedBy=default.target