Fixing SSE Streaming That Freezes Behind Nginx for LLM Apps

The shop's parts-lookup assistant streams its answer token by token � you watch it type, the same as any chat app. That's how it behaved when I hit the Node port directly. The moment it went behind nginx on the VPS, the "typing" vanished: spinner for forty seconds, then the entire answer appearing in one lump. Nothing errored, nothing timed out. Search "nginx sse streaming not working" and you'll find a hundred people describing exactly this non-failure.

The stream isn't broken. It's being buffered � nginx is helpfully holding the bytes so it can forward them efficiently, and for server-sent events that helpfulness is the bug. The fix is one location block, scoped so the rest of the site keeps its compression and caching. (The Node/Express side of streaming � compression middleware, flush headers � is a separate post; this one is strictly the nginx layer.)

Confirm it's buffering, not breakage

Two curls tell the whole story:

# direct to the app � tokens trickle line by line:
curl -N http://127.0.0.1:3000/api/chat

# through nginx � silence... then the entire answer at once:
curl -N https://parts.example.com/api/chat

The -N disables curl's own buffering, so what you see is what the wire delivers. Same app, same request � the difference appears exactly at the proxy. If the direct curl trickles and the proxied one dumps, you have your diagnosis: nginx is standing between your users and the tokens, collecting them first.

The one location block

location /api/chat {
    proxy_pass http://127.0.0.1:3000;

    proxy_buffering off;
    gzip off;
    proxy_cache off;

    proxy_http_version 1.1;
    proxy_set_header Connection "";
    proxy_set_header Host $host;

    proxy_read_timeout 1h;
}

Line by line, because each one kills a different way this freezes:

• proxy_buffering off � the main fix. By default nginx reads the upstream response into buffers and forwards it in efficient chunks. For a page, invisible; for a stream, it's a dam. Off means each event leaves the moment it arrives.
• gzip off � gzip cannot compress what it hasn't collected, so a gzipping proxy wants batches of bytes. This disables gzip for this route only. That's the part most copy-paste configs miss: you want to disable gzip for the SSE location in nginx, not for the whole site � everywhere else keeps compression.
• proxy_cache off � if a proxy_cache_path is in scope, this location must not try to store a never-ending response. Even if you think no cache applies here, an inherited zone from the surrounding server block is a classic surprise.
• proxy_http_version 1.1 + Connection "" � nginx talks to upstreams over HTTP/1.0 by default; 1.1 with a cleaned connection header is what a persistent event stream expects.
• proxy_read_timeout 1h � the quiet one. The default is 60 seconds between reads. A model that thinks before its first token, or pauses mid-generation, crosses that gap and gets cut mid-stream � the "nginx server-sent events timeout" people chase for days. Raise it for this location; an idle stream is not a stuck request here.

If you can't touch nginx: the response-header escape hatch

Docs sometimes mention X-Accel-Buffering: no � the app sends that header and nginx skips buffering for that response. It works, but it lives in the application, which is exactly the layer this post isn't about. Worth knowing it exists, because your framework's SSE helper may already send it � and if tokens still freeze with it set, the cause is usually gzip or the read timeout above.

Verify it the way you diagnosed it

sudo nginx -t && sudo systemctl reload nginx
curl -N https://parts.example.com/api/chat

nginx -t first, always � a broken config reloaded at 9 PM is a worse evening than a frozen stream. Then the same proxied curl -N from before: tokens should now trickle exactly like the direct one did. In the browser, the typing animation just comes back.

For the record, the failure-to-knob mapping:

One sentence to keep: nginx buffers because it's trying to be a good proxy � for one streaming route, a scoped location block that turns that off is the whole fix, and the rest of the site never notices.