Why Stream Model Output?
Streaming delivers partial output before the full response is ready, improving perceived responsiveness for long generations. A stream may include text deltas, tool-call events, usage metadata, and terminal status depending on the provider. Do not assume every event is a user-visible string; parse event types explicitly.
Streaming complicates error handling: a request can fail after some text has already appeared. The UI should indicate incomplete responses, allow retry, and avoid presenting partial output as a completed answer. Define cancellation behavior and clean up readers or connections when the user navigates away.
Read a streaming response conceptually
const response = await fetch("/api/ai/stream", {
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify({ prompt })
});
if (!response.ok || !response.body) {
throw new Error("Streaming request failed");
}
const reader = response.body.getReader();
// Decode and process chunks according to your server's wire protocol.💡 Fetch exposes a ReadableStream response body. The application must implement the server's protocol correctly, including chunk boundaries, encoding, terminal events, and cancellation.
Key Takeaways
- Streaming improves perceived latency but needs explicit protocol handling.
- Partial output must be distinguished from a completed answer.
- Support cancellation, retries, and terminal error states.
Knowledge Check
Why must a streaming client parse event types rather than append every event blindly?