How to stream LLM responses with server-sent events

An LLM can take several seconds to finish an answer. If you wait for the whole thing, the page looks frozen. If you stream it, the user starts reading while the model is still writing.

The model doesn’t get any faster, but the page feels faster because text shows up as soon as it exists.

Here is what we’ll build:

sequenceDiagram
  participant B as Browser
  participant S as Your server
  participant P as Model provider
  B->>S: prompt
  S->>P: prompt + API key
  P-->>S: SSE stream
  S-->>B: SSE stream

The API key stays on the server. The browser only sees the events we forward.

If SSE itself is new to you, read Server-Sent Events first. The free HTTP course covers response headers, bodies, caching, and connection behavior.

What is being streamed

Models generate text token by token. But a provider event does not hold exactly one token. It can hold half a word, one word, or several tokens.

Network chunks are yet another thing. One reader.read() can give you half an SSE event, or five events glued together.

So we have three layers, and they don’t line up:

model tokens != provider events != network chunks

All the UI has to do is append the deltas in order. Don’t look for word or sentence boundaries inside a delta. There aren’t any you can count on.

The SSE wire format

An SSE response uses the text/event-stream content type.

Each event is made of fields and ends with a blank line:

event: delta
data: {"text":"Hello"}

event: delta
data: {"text":" there"}

event: done
data: {}

The event field names the event. The data field carries its payload.

A full SSE parser also handles comments, id, retry, multiple data lines, and both LF and CRLF line endings. Ours needs less: named events and JSON data. It still has to deal with events that arrive cut in half.

Keep the provider key on the server

Never call the model API from the browser with your secret key. Anyone who opens DevTools has it.

The browser sends the prompt to your server instead:

await fetch('/api/chat', {
  method: 'POST',
  headers: {
    'content-type': 'application/json'
  },
  body: JSON.stringify({
    prompt: 'Explain DNS in plain English'
  })
})

The server checks the input, adds the key, and opens the stream to the provider.

Proxy the provider stream

Here is a handler built on the Web Request and Response types, calling OpenAI’s Responses API. It works in any server runtime that has fetch and ReadableStream.

export async function POST(request) {
  const body = await request.json().catch(() => null)
  const prompt = body?.prompt

  if (
    typeof prompt !== 'string' ||
    prompt.trim().length === 0 ||
    prompt.length > 4000
  ) {
    return Response.json(
      { error: 'Prompt must contain 1 to 4000 characters' },
      { status: 400 }
    )
  }

  const upstream = await fetch('https://api.openai.com/v1/responses', {
    method: 'POST',
    headers: {
      authorization: `Bearer ${process.env.OPENAI_API_KEY}`,
      'content-type': 'application/json'
    },
    body: JSON.stringify({
      model: 'gpt-5.4',
      input: prompt,
      stream: true
    }),
    signal: request.signal
  })

  if (!upstream.ok || !upstream.body) {
    const detail = await upstream.text()

    console.error('Model request failed', {
      status: upstream.status,
      detail
    })

    return Response.json(
      { error: 'The model request failed' },
      { status: 502 }
    )
  }

  return new Response(upstream.body, {
    headers: {
      'content-type': 'text/event-stream; charset=utf-8',
      'cache-control': 'no-cache, no-transform',
      'x-content-type-options': 'nosniff'
    }
  })
}

The server never holds the full answer. It passes the provider’s ReadableStream straight through as its own response body.

request.signal is there so that when the browser disconnects, the request to the provider gets cancelled too. That only works if your runtime propagates the signal. Test it where you deploy.

no-transform tells proxies and CDNs to leave the response alone. Otherwise compression or buffering can hold back the small chunks.

The full provider error goes to the logs, and the browser gets a generic message. Provider errors can echo request details you don’t want to leak.

Read a POST stream in the browser

EventSource is the built-in browser API for SSE, but it only does GET. Our prompt goes in a POST body. So we use fetch() and read the stream ourselves:

<form id="chat-form">
  <label for="prompt">Message</label>
  <textarea id="prompt" required></textarea>
  <button>Send</button>
</form>

<button id="stop" type="button" disabled>Stop</button>
<pre id="output"></pre>

Create the request when the form is submitted:

const form = document.querySelector('#chat-form')
const prompt = document.querySelector('#prompt')
const output = document.querySelector('#output')
const stop = document.querySelector('#stop')
let activeRequest

form.addEventListener('submit', async event => {
  event.preventDefault()

  activeRequest?.abort()
  activeRequest = new AbortController()
  output.textContent = ''
  stop.disabled = false

  try {
    await streamAnswer(prompt.value, output, activeRequest.signal)
  } catch (error) {
    if (error.name !== 'AbortError') {
      output.textContent += '\nThe answer could not be completed.'
      console.error(error)
    }
  } finally {
    stop.disabled = true
    activeRequest = undefined
  }
})

stop.addEventListener('click', () => {
  activeRequest?.abort()
})

Now implement streamAnswer():

async function streamAnswer(prompt, output, signal) {
  const response = await fetch('/api/chat', {
    method: 'POST',
    headers: {
      'content-type': 'application/json'
    },
    body: JSON.stringify({ prompt }),
    signal
  })

  if (!response.ok || !response.body) {
    throw new Error(`Chat returned HTTP ${response.status}`)
  }

  const reader = response.body.getReader()
  const decoder = new TextDecoder()
  let buffer = ''
  let pendingCarriageReturn = false

  while (true) {
    const { value, done } = await reader.read()
    let text = decoder.decode(value, { stream: !done })

    if (pendingCarriageReturn) {
      text = `\r${text}`
      pendingCarriageReturn = false
    }

    if (!done && text.endsWith('\r')) {
      text = text.slice(0, -1)
      pendingCarriageReturn = true
    }

    buffer += text.replace(/\r\n/g, '\n').replace(/\r/g, '\n')

    if (done && pendingCarriageReturn) {
      buffer += '\n'
      pendingCarriageReturn = false
    }

    const events = buffer.split('\n\n')
    buffer = events.pop() ?? ''

    for (const block of events) {
      const event = parseSseEvent(block)

      if (!event.data) continue

      const data = JSON.parse(event.data)

      if (event.name === 'response.output_text.delta') {
        output.textContent += data.delta
      }

      if (event.name === 'response.failed') {
        throw new Error('The model failed during generation')
      }
    }

    if (done) break
  }
}

The Responses API sends named events. response.output_text.delta carries a piece of text in delta. That’s the one we append. Everything else we ignore, except response.failed.

Here’s parseSseEvent():

function parseSseEvent(block) {
  let name = 'message'
  const data = []

  for (const line of block.split('\n')) {
    if (line.startsWith(':')) continue

    const separator = line.indexOf(':')
    const field = separator === -1
      ? line
      : line.slice(0, separator)
    let value = separator === -1
      ? ''
      : line.slice(separator + 1)

    if (value.startsWith(' ')) value = value.slice(1)

    if (field === 'event') name = value
    if (field === 'data') data.push(value)
  }

  return {
    name,
    data: data.join('\n')
  }
}

Multiple data lines belong to the same event, so the parser joins them with newlines.

Use a tested parser in production

Our parser handles a lot already: UTF-8 split across chunks, incomplete events, LF and CRLF, a carriage return cut in half, comments, named events, multiple data lines.

It does not handle id or retry. If you need the full protocol, use a tested SSE parser package instead of adding edge cases to this one.

Whatever you use, don’t do this on a raw chunk:

JSON.parse(decoder.decode(value))

A network chunk can stop in the middle of an event, so the JSON inside it can be cut anywhere.

Rendering without creating an XSS bug

Use textContent for model text:

output.textContent += data.delta

Never use innerHTML for model output. It’s untrusted, even if you told the model to only return safe HTML.

If your chat supports Markdown, accumulate the text, parse it with a maintained Markdown library, and sanitize the generated HTML before inserting it.

Markdown while streaming is also half-finished most of the time. A code fence opens in one delta and closes ten deltas later. Re-render the whole accumulated message on each update instead of parsing deltas one by one.

Handle errors before and after streaming starts

Before sending response headers, the server can return a normal HTTP error such as 400, 401, 429, or 502.

After the stream starts, the status is already 200. A later provider failure must travel inside the stream or appear as an unexpected disconnect.

This gives you two error channels:

before first byte -> HTTP status and JSON body
after first byte  -> SSE error event or interrupted stream

When a stream fails, keep the partial text on screen and mark the answer as incomplete. Don’t wipe it. The user may still find it useful, and you’ll want it when debugging.

Cancellation is part of the feature

The Stop button saves money. Every token the model generates after the user stopped reading is a token you pay for.

When the user clicks it, the browser aborts its fetch. The server has to notice the closed connection and cancel the request to the provider. Some runtimes do this through request.signal, others need you to clean up the stream yourself.

Test it with a long prompt and check that all three layers stop:

1. The browser reader rejects with AbortError.
2. The server request closes.
3. The provider generation is cancelled or disconnected.

If only the first one happens, you’re paying for text nobody sees.

Backpressure and slow clients

Backpressure is what happens when the reader is slower than the writer. Piping the provider’s ReadableStream straight into the response lets the runtime handle that buffering for you.

Don’t collect the provider’s chunks into an array and return them at the end. You’d lose the streaming and hold the whole answer in memory.

And set a deadline. A client that opened a stream and stopped reading should not keep a server connection open forever.

Production checks

If chat is private, authenticate the route before it calls the model.

Limit the request body size, the prompt length, the number of messages, and the total size of the conversation. These control your cost as much as they validate input.

Set the provider’s output-token limit. A public endpoint with no cap on output can generate very expensive answers.

Rate-limit per user or per API key. Limiting by IP alone punishes people on shared networks and is easy to get around.

Use two timeouts: one for opening the connection, one for the whole generation. A stream that sent one byte and went silent should not live forever.

Test the deployed path, not only localhost. Reverse proxies and CDNs can buffer small writes. Load the page in production and check that the first delta shows up right away.

Log request IDs, duration, provider status, whether the stream completed, and token usage. Don’t log prompts by default, because they can contain private data.

When not to stream

Streaming costs you a parser, cancellation logic, a UI that handles half-finished answers, and a second error channel. Only pay that cost when someone is watching the answer arrive.

Return plain JSON when the output is short or has to be validated as one object. Classification, moderation, routing a request, extracting fields from a document: none of these gain anything from streaming.

Chat, long explanations, and code generation are worth it. A yes/no answer or a background job nobody is looking at is not.

How I would build it

I’d ship the non-streaming version first. Same route, same validation, same auth, same error handling, but stream: false and a JSON response. That proves everything except the streaming.

Then I’d flip stream to true and pass the body through. The rest of the contract stays the same. I’d add the Stop button before I even thought about Markdown rendering.

Last, I’d test through the real proxy in production: one slow answer, one provider error, one cancellation.

Want me to talk about your product? You can sponsor this site.