Messages, Roles, Tokens, and Context Windows
A model request typically contains instructions, conversation messages, optional tool definitions, generation settings, and output constraints. Provider APIs differ in terminology and semantics, so treat a provider's current API reference as authoritative. Tokenization is not the same as word count: punctuation, code, and multilingual text may use different token counts.
The context window limits how much input and output can be processed in one request. Long history increases cost and can reduce relevance. Use concise conversation summaries, retrieve only relevant documents, cap output length, and avoid sending secrets or unrelated personal data.
Diagram
Provider-neutral request shape
const request = {
instructions: "Answer using only the supplied policy context.",
input: [
{ role: "user", content: "Can I return an opened item?" }
],
metadata: { feature: "returns-help" }
};💡 This is an illustrative provider-neutral shape, not a direct SDK call. Map it to the current provider API you choose, and document any semantic differences in roles and instructions.
Key Takeaways
- Token counts are not equivalent to word counts.
- Context limits apply to input and output.
- Use provider documentation for exact request semantics.
Knowledge Check
Why should an application avoid sending the entire conversation history on every request?