stream fix: read1() not read(65536) — flush SSE as it arrives
_stream_response did resp.read(65536), which blocks until 64KB or EOF. A whole chat response is only a few KB, so it never filled the buffer and only released at EOF — reasoning + content arrived in one burst at the end instead of streaming. Upstream (bosch sglang) streams fine (reasoning 0.2->6.5s); the proxy was the buffer. read1() returns on first available data. Verified reasoning now streams through :8082. Also fixes streaming for OWUI. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
+5
-1
@@ -659,7 +659,11 @@ class Handler(http.server.BaseHTTPRequestHandler):
|
||||
bytes_out = 0
|
||||
try:
|
||||
while True:
|
||||
chunk = resp.read(65536)
|
||||
# read1(): return as soon as ANY data is available (one underlying read),
|
||||
# instead of read()'s "block until 64KB or EOF" -- otherwise a small
|
||||
# streaming response (a few KB of SSE) only flushes at EOF, so reasoning
|
||||
# + content land in one burst at the end instead of streaming through.
|
||||
chunk = resp.read1(65536)
|
||||
if not chunk:
|
||||
break
|
||||
self.wfile.write(chunk)
|
||||
|
||||
Reference in New Issue
Block a user