williams/llmproxy
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Compare commits

merge into: williams/llmproxy:master
Branches Tags
williams/llmproxy:main williams/llmproxy:master
...
pull from: williams/llmproxy:main
Branches Tags
williams/llmproxy:main williams/llmproxy:master

2 Commits
master ... main

Author SHA1 Message Date
williams 6cbf352851 llm-proxy: propagate client cancellation to the backend (free the slot) 
A pi-agent stop did not reach models served through the proxy: the forward loop
only noticed the client was gone when it next tried to WRITE, so during the
blocking non-streaming getresponse() (full generation) the backend ran to
completion holding the single-slot admission slot — a cancelled request kept
burning the NPU/DGX. Add a client-liveness watchdog: while a backend request is
in flight, select() on the client socket; on EOF close the backend conn so it
aborts and the slot frees. New _try_backend return "aborted" (no failover;
client is gone). _stream_response read loop hardened to stop cleanly when the
watchdog closes the conn mid-read. Live-verified: non-streaming request cut at
3s -> CLIENT-CANCELLED logged, slot freed, followup 200 in 4s.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EWpfhDgYNA21tETDP9ueBE
 2026-07-20 22:14:40 +00:00
claude-noether 06f20ae752 llm-proxy: per-backend admission gate (semaphore, bounded queue, 503 on saturation) 
ThreadingHTTPServer fanned unbounded concurrent forwards into single-slot llama.cpp
-> pile-up (client-disconnect never cancels upstream, so contention cascades into a
hang). Add a BoundedSemaphore per local backend, slots from local-backends.json
(qwen3-4b=1, bosch-dspark=32), in-system cap = slots*4 -> fast 503 backend_busy on
saturation, ~90s bounded wait otherwise. Gates local backends only. Verified: 6
concurrent -> 4 served serially + 2 fast-503, no pile-up. Deployed on hossenfelder.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
 2026-07-20 13:50:18 +02:00
2 changed files with 1047 additions and 324 deletions
Expand all files Collapse all files
llm-proxy.py
+1028 -309
View File
File diff suppressed because it is too large Load Diff
local-backends.json
+4
View File
@@ -0,0 +1,4 @@
[
{"name":"qwen3-4b","host":"boltzmann.fritz.box","port":8091,"slots":1},
{"name":"bosch-dspark","host":"bosch.fritz.box","port":8888,"slots":32}
]