llmproxy

T

claude-noether 06f20ae752 llm-proxy: per-backend admission gate (semaphore, bounded queue, 503 on saturation)

ThreadingHTTPServer fanned unbounded concurrent forwards into single-slot llama.cpp
-> pile-up (client-disconnect never cancels upstream, so contention cascades into a
hang). Add a BoundedSemaphore per local backend, slots from local-backends.json
(qwen3-4b=1, bosch-dspark=32), in-system cap = slots*4 -> fast 503 backend_busy on
saturation, ~90s bounded wait otherwise. Gates local backends only. Verified: 6
concurrent -> 4 served serially + 2 fast-503, no pile-up. Deployed on hossenfelder.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

2026-07-20 13:50:18 +02:00

.gitignore

Initial commit: Boltzmann LLM proxy with compression + classifier middleware

2026-06-15 15:12:25 +02:00

classifier-server.py

Initial commit: Boltzmann LLM proxy with compression + classifier middleware

2026-06-15 15:12:25 +02:00

compress-server.py

Initial commit: Boltzmann LLM proxy with compression + classifier middleware

2026-06-15 15:12:25 +02:00

download-models.sh

Initial commit: Boltzmann LLM proxy with compression + classifier middleware