stream fix: read1() not read(65536) — flush SSE as it arrives

_stream_response did resp.read(65536), which blocks until 64KB or EOF. A whole chat
response is only a few KB, so it never filled the buffer and only released at EOF —
reasoning + content arrived in one burst at the end instead of streaming. Upstream
(bosch sglang) streams fine (reasoning 0.2->6.5s); the proxy was the buffer. read1()
returns on first available data. Verified reasoning now streams through :8082. Also
fixes streaming for OWUI.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
Markus Fritsche
2026-07-24 15:35:03 +02:00
parent 484296b277
commit cf4d6ca610
+5 -1
View File
@@ -659,7 +659,11 @@ class Handler(http.server.BaseHTTPRequestHandler):
bytes_out = 0 bytes_out = 0
try: try:
while True: while True:
chunk = resp.read(65536) # read1(): return as soon as ANY data is available (one underlying read),
# instead of read()'s "block until 64KB or EOF" -- otherwise a small
# streaming response (a few KB of SSE) only flushes at EOF, so reasoning
# + content land in one burst at the end instead of streaming through.
chunk = resp.read1(65536)
if not chunk: if not chunk:
break break
self.wfile.write(chunk) self.wfile.write(chunk)