11. Conversational RAG and Memory

Resolve follow-ups, separate memory from authoritative knowledge, and reapply permissions every turn.

Resolve Follow-Up Questions

Follow-ups like “Does it apply to contractors?” need conversational context. A query rewriter can produce a standalone query while preserving the original wording. Keep a compact summary of topic, confirmed constraints, entities, and open questions rather than sending all history. If multiple interpretations remain plausible, ask for clarification.

Evaluate pronouns, topic switches, corrections, and ambiguous references separately.

Implementation example

Prior: Compare Basic and Pro plans.
Follow-up: What about annual billing?
Rewrite: How does annual billing work for Basic and Pro?

💡 The rewrite resolves context without inventing an answer.

Key Takeaways

Inspect the relevant pipeline stage instead of guessing.
Preserve source provenance, versions, and access scope.
Measure quality on representative queries.

Knowledge Check

What should a rewriter do if the referent remains ambiguous?

Conversation Memory vs. Knowledge Base

Conversation memory stores turn history or user preferences; a knowledge base stores source documents. A model-generated summary is not authoritative policy. Use conversation context to resolve references, but retrieve current approved policies and live transactional facts from their systems of record.

Keep memory scoped, minimize retention, preserve provenance, and define deletion behavior for both conversation records and indexed copies.

Implementation example

Conversation store: topic, constraints, turn history
Knowledge index: approved documents and citations
System of record: live account / order / transaction state

💡 Different fact types require different authoritative sources.

Key Takeaways

Inspect the relevant pipeline stage instead of guessing.
Preserve source provenance, versions, and access scope.
Measure quality on representative queries.

Knowledge Check

Where should a current company policy answer come from?

Multi-Turn Permissions

Permissions can change during a conversation. Reapply authorization on every retrieval, source fetch, and cache access. A conversation ID is not proof of ownership. If a user's access is revoked, summaries or cached answers may also require redaction or invalidation. Test attempts to carry restricted facts across tenants or identities.

Implementation example

async function loadSource(sourceId, auth) {
 return db.documents.findOne({
   sourceId, tenantId:auth.tenantId, allowedUserIds:auth.userId
 });
}

💡 The exact ACL schema varies, but access must be checked at request time.

Key Takeaways

Inspect the relevant pipeline stage instead of guessing.
Preserve source provenance, versions, and access scope.
Measure quality on representative queries.

Knowledge Check

Why re-check permissions on every turn?

Evaluate Conversational Retrieval

Test follow-ups, pronouns, topic switches, corrections, multiple active documents, and earlier citations. Measure rewrite fidelity, retrieval quality, and answer quality separately. Fluent rewrites can still change intent. Do not rewrite self-contained questions unnecessarily.

Implementation example

{
 "priorContext":"Question about Pro plan storage",
 "followUp":"Does it include backups?",
 "expectedQuery":"Does the Pro plan include backups?"
}

💡 A full test set should include ambiguous cases where clarification is correct.

Key Takeaways

Inspect the relevant pipeline stage instead of guessing.
Preserve source provenance, versions, and access scope.
Measure quality on representative queries.

Knowledge Check

What should rewrite evaluation measure?

Resolve Follow-Up Questions

Conversation Memory vs. Knowledge Base

Multi-Turn Permissions

Evaluate Conversational Retrieval

STAY UPDATED WITH BYTES!