llm-proxy: per-backend admission gate (semaphore, bounded queue, 503 on saturation)

ThreadingHTTPServer fanned unbounded concurrent forwards into single-slot llama.cpp
-> pile-up (client-disconnect never cancels upstream, so contention cascades into a
hang). Add a BoundedSemaphore per local backend, slots from local-backends.json
(qwen3-4b=1, bosch-dspark=32), in-system cap = slots*4 -> fast 503 backend_busy on
saturation, ~90s bounded wait otherwise. Gates local backends only. Verified: 6
concurrent -> 4 served serially + 2 fast-503, no pile-up. Deployed on hossenfelder.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

This commit is contained in:

claude-noether

2026-07-20 13:50:18 +02:00

parent c121428c58

commit 06f20ae752

2 changed files with 996 additions and 327 deletions

llm-proxy.py

+992 -327

View File

File diff suppressed because it is too large Load Diff

									
										local-backends.json
									
		+4
		
												View File
												
				@@ -0,0 +1,4 @@

				[

				  {"name":"qwen3-4b","host":"boltzmann.fritz.box","port":8091,"slots":1},

				  {"name":"bosch-dspark","host":"bosch.fritz.box","port":8888,"slots":32}

				]