5. Streaming Chat UX and Conversation State

Build responsive streaming interfaces, manage cancellation and history, and handle errors and concurrent turns.

Why Stream Model Output?

Streaming delivers partial output before the full response is ready, improving perceived responsiveness for long generations. A stream may include text deltas, tool-call events, usage metadata, and terminal status depending on the provider. Do not assume every event is a user-visible string; parse event types explicitly.

Streaming complicates error handling: a request can fail after some text has already appeared. The UI should indicate incomplete responses, allow retry, and avoid presenting partial output as a completed answer. Define cancellation behavior and clean up readers or connections when the user navigates away.

Read a streaming response conceptually

const response = await fetch("/api/ai/stream", {
  method: "POST",
  headers: { "Content-Type": "application/json" },
  body: JSON.stringify({ prompt })
});

if (!response.ok || !response.body) {
  throw new Error("Streaming request failed");
}
const reader = response.body.getReader();
// Decode and process chunks according to your server's wire protocol.

💡 Fetch exposes a ReadableStream response body. The application must implement the server's protocol correctly, including chunk boundaries, encoding, terminal events, and cancellation.

Key Takeaways

Streaming improves perceived latency but needs explicit protocol handling.
Partial output must be distinguished from a completed answer.
Support cancellation, retries, and terminal error states.

Knowledge Check

Why must a streaming client parse event types rather than append every event blindly?

Chunk Parsing, AbortController, and Backpressure

Network chunks do not necessarily align with logical events or UTF-8 characters. A robust reader decodes incrementally and buffers incomplete protocol frames until a delimiter is found. If using Server-Sent Events, parse event boundaries according to the SSE format rather than assuming one fetch chunk equals one event.

Use AbortController to cancel a request when the user stops generation or the component unmounts. Propagate cancellation to the server and provider where supported, and ensure background work does not continue consuming resources unnecessarily. Backpressure matters when a consumer cannot process data as fast as it arrives.

Cancel a request

const controller = new AbortController();

try {
  const response = await fetch("/api/ai/stream", {
    method: "POST",
    body: JSON.stringify({ prompt }),
    headers: { "Content-Type": "application/json" },
    signal: controller.signal
  });
  // Read the stream...
} catch (error) {
  if (error.name === "AbortError") {
    console.log("Generation cancelled");
  } else {
    throw error;
  }
}

// Call controller.abort() from a Stop button.

💡 AbortController cancels the client-side fetch and can signal cancellation to downstream work if the server propagates it. The server should still clean up provider requests and release resources.

Key Takeaways

Protocol frames can span multiple network chunks.
Use AbortController for cancellation.
Propagate disconnects and release resources on the server.

Knowledge Check

What is a common reason to use AbortController for AI streaming?

Conversation State, Persistence, and Turn Ordering

Conversation state may include messages, tool calls and results, attachments, summaries, and safety metadata. Persist only what is needed and define tenant/user ownership on every read and write. Concurrent turns can race: two requests may read the same history and append responses in the wrong order. Use per-conversation locking, optimistic versions, or a turn sequence number when ordering matters.

A client-provided conversation ID is not proof of access. Load the conversation through an authorization-scoped query. For long conversations, maintain a compact summary plus recent turns and retrieve older facts only when needed.

Versioned conversation update

async function appendTurn({ conversationId, userId, expectedVersion, message }) {
  return db.conversations.updateOne(
    { id: conversationId, userId, version: expectedVersion },
    {
      $push: { messages: message },
      $inc: { version: 1 }
    }
  );
}

💡 This MongoDB-style pseudocode uses ownership and a version check to prevent a stale write. Check the update result and handle version conflicts explicitly; the exact syntax depends on your data layer.

Key Takeaways

Persist conversations with ownership checks.
Handle concurrent turn ordering.
Use summaries and retrieval to manage long history.

Knowledge Check

Why is a conversation ID not sufficient authorization by itself?

Why Stream Model Output?

Chunk Parsing, AbortController, and Backpressure

Conversation State, Persistence, and Turn Ordering

STAY UPDATED WITH BYTES!