2 Commits

Author	SHA1	Message	Date
claude-noether	06f20ae752	llm-proxy: per-backend admission gate (semaphore, bounded queue, 503 on saturation) ... ThreadingHTTPServer fanned unbounded concurrent forwards into single-slot llama.cpp -> pile-up (client-disconnect never cancels upstream, so contention cascades into a hang). Add a BoundedSemaphore per local backend, slots from local-backends.json (qwen3-4b=1, bosch-dspark=32), in-system cap = slots*4 -> fast 503 backend_busy on saturation, ~90s bounded wait otherwise. Gates local backends only. Verified: 6 concurrent -> 4 served serially + 2 fast-503, no pile-up. Deployed on hossenfelder. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-07-20 13:50:18 +02:00
Markus Fritsche	c121428c58	Initial commit: Boltzmann LLM proxy with compression + classifier middleware ... - llm-proxy.py v4: aggregator with classify/compress endpoints, gemma4 backend - classifier-server.py: Llama 3.2 1B query complexity classifier - compress-server.py: token-budget compression middleware - start-proxy.sh / start.sh: launcher scripts	2026-06-15 15:12:25 +02:00