stream fix: read1() not read(65536) — flush SSE as it arrives
_stream_response did resp.read(65536), which blocks until 64KB or EOF. A whole chat response is only a few KB, so it never filled the buffer and only released at EOF — reasoning + content arrived in one burst at the end instead of streaming. Upstream (bosch sglang) streams fine (reasoning 0.2->6.5s); the proxy was the buffer. read1() returns on first available data. Verified reasoning now streams through :8082. Also fixes streaming for OWUI. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
+5
-1
@@ -659,7 +659,11 @@ class Handler(http.server.BaseHTTPRequestHandler):
|
|||||||
bytes_out = 0
|
bytes_out = 0
|
||||||
try:
|
try:
|
||||||
while True:
|
while True:
|
||||||
chunk = resp.read(65536)
|
# read1(): return as soon as ANY data is available (one underlying read),
|
||||||
|
# instead of read()'s "block until 64KB or EOF" -- otherwise a small
|
||||||
|
# streaming response (a few KB of SSE) only flushes at EOF, so reasoning
|
||||||
|
# + content land in one burst at the end instead of streaming through.
|
||||||
|
chunk = resp.read1(65536)
|
||||||
if not chunk:
|
if not chunk:
|
||||||
break
|
break
|
||||||
self.wfile.write(chunk)
|
self.wfile.write(chunk)
|
||||||
|
|||||||
Reference in New Issue
Block a user