llmproxy

williams/llmproxy

Fork 0

...

Author	SHA1	Message	Date
williams	6cbf352851	llm-proxy: propagate client cancellation to the backend (free the slot) A pi-agent stop did not reach models served through the proxy: the forward loop only noticed the client was gone when it next tried to WRITE, so during the blocking non-streaming getresponse() (full generation) the backend ran to completion holding the single-slot admission slot — a cancelled request kept burning the NPU/DGX. Add a client-liveness watchdog: while a backend request is in flight, select() on the client socket; on EOF close the backend conn so it aborts and the slot frees. New _try_backend return "aborted" (no failover; client is gone). _stream_response read loop hardened to stop cleanly when the watchdog closes the conn mid-read. Live-verified: non-streaming request cut at 3s -> CLIENT-CANCELLED logged, slot freed, followup 200 in 4s. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EWpfhDgYNA21tETDP9ueBE	2026-07-20 22:14:40 +00:00
claude-noether	06f20ae752	llm-proxy: per-backend admission gate (semaphore, bounded queue, 503 on saturation) ThreadingHTTPServer fanned unbounded concurrent forwards into single-slot llama.cpp -> pile-up (client-disconnect never cancels upstream, so contention cascades into a hang). Add a BoundedSemaphore per local backend, slots from local-backends.json (qwen3-4b=1, bosch-dspark=32), in-system cap = slots*4 -> fast 503 backend_busy on saturation, ~90s bounded wait otherwise. Gates local backends only. Verified: 6 concurrent -> 4 served serially + 2 fast-503, no pile-up. Deployed on hossenfelder. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-07-20 13:50:18 +02:00

Author

SHA1

Message

Date

williams

6cbf352851

llm-proxy: propagate client cancellation to the backend (free the slot)

A pi-agent stop did not reach models served through the proxy: the forward loop
only noticed the client was gone when it next tried to WRITE, so during the
blocking non-streaming getresponse() (full generation) the backend ran to
completion holding the single-slot admission slot — a cancelled request kept
burning the NPU/DGX. Add a client-liveness watchdog: while a backend request is
in flight, select() on the client socket; on EOF close the backend conn so it
aborts and the slot frees. New _try_backend return "aborted" (no failover;
client is gone). _stream_response read loop hardened to stop cleanly when the
watchdog closes the conn mid-read. Live-verified: non-streaming request cut at
3s -> CLIENT-CANCELLED logged, slot freed, followup 200 in 4s.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EWpfhDgYNA21tETDP9ueBE

2026-07-20 22:14:40 +00:00

claude-noether

06f20ae752

llm-proxy: per-backend admission gate (semaphore, bounded queue, 503 on saturation)

ThreadingHTTPServer fanned unbounded concurrent forwards into single-slot llama.cpp
-> pile-up (client-disconnect never cancels upstream, so contention cascades into a
hang). Add a BoundedSemaphore per local backend, slots from local-backends.json
(qwen3-4b=1, bosch-dspark=32), in-system cap = slots*4 -> fast 503 backend_busy on
saturation, ~90s bounded wait otherwise. Gates local backends only. Verified: 6
concurrent -> 4 served serially + 2 fast-503, no pile-up. Deployed on hossenfelder.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

2026-07-20 13:50:18 +02:00

2 changed files with 1047 additions and 324 deletions

llm-proxy.py

+1043 -324

View File

File diff suppressed because it is too large Load Diff

									
										local-backends.json
									
		+4
		
												View File
												
				@@ -0,0 +1,4 @@

				[

				  {"name":"qwen3-4b","host":"boltzmann.fritz.box","port":8091,"slots":1},

				  {"name":"bosch-dspark","host":"bosch.fritz.box","port":8888,"slots":32}

				]

Compare commits

2 Commits

master ... main

Compare commits

2 Commits master ... main

2 Commits

master ... main