llmproxy

T

williams 6cbf352851 llm-proxy: propagate client cancellation to the backend (free the slot)

A pi-agent stop did not reach models served through the proxy: the forward loop
only noticed the client was gone when it next tried to WRITE, so during the
blocking non-streaming getresponse() (full generation) the backend ran to
completion holding the single-slot admission slot — a cancelled request kept
burning the NPU/DGX. Add a client-liveness watchdog: while a backend request is
in flight, select() on the client socket; on EOF close the backend conn so it
aborts and the slot frees. New _try_backend return "aborted" (no failover;
client is gone). _stream_response read loop hardened to stop cleanly when the
watchdog closes the conn mid-read. Live-verified: non-streaming request cut at
3s -> CLIENT-CANCELLED logged, slot freed, followup 200 in 4s.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EWpfhDgYNA21tETDP9ueBE

2026-07-20 22:14:40 +00:00

.gitignore

Initial commit: Boltzmann LLM proxy with compression + classifier middleware

2026-06-15 15:12:25 +02:00

classifier-server.py

Initial commit: Boltzmann LLM proxy with compression + classifier middleware

2026-06-15 15:12:25 +02:00

compress-server.py

Initial commit: Boltzmann LLM proxy with compression + classifier middleware

2026-06-15 15:12:25 +02:00

download-models.sh

Initial commit: Boltzmann LLM proxy with compression + classifier middleware