2. Data Sources, Ingestion, and Governance

Inventory sources, design connectors, normalize content, and preserve freshness and access-control metadata.

Source Inventory and Authority

List each source, owner, format, authority, access scope, update frequency, freshness target, and deletion policy. Approved policy should outrank an old draft or informal ticket. Store publication status, effective dates, and version so conflicting sources can be resolved transparently.

Source governance comes before embeddings. It reveals missing permissions, unsupported formats, stale content, and unclear ownership. If no authoritative source exists, the assistant should say so rather than synthesizing certainty from weak evidence.

Implementation example

{
  "sourceId":"returns-policy",
  "authority":"approved",
  "owner":"support-operations",
  "accessScope":"tenant",
  "freshnessTargetHours":2
}

💡 Metadata supports authority, filtering, freshness, and lifecycle management.

Key Takeaways

Inspect the relevant pipeline stage instead of guessing.
Preserve source provenance, versions, and access scope.
Measure quality on representative queries.

Knowledge Check

Which metadata helps resolve conflicting policy documents?

Connectors, Crawlers, and Incremental Sync

Connectors emit create, update, delete, and access-change events. Use stable IDs, content hashes, pagination, throttling, retries, and durable checkpoints. Idempotent ingestion avoids duplicate chunks when a job repeats. Deletions and permission revocations must propagate to every index and cache.

Respect source permissions and applicable crawling rules. Do not index data merely because it is technically accessible. Keep sync jobs resumable and ensure stale jobs cannot overwrite a newer document version.

Implementation example

async function handleEvent(event) {
  if (event.type === "deleted") {
    await index.deleteBySourceId(event.sourceId);
    return;
  }
  const hash = sha256(event.normalizedContent);
  if (await index.hasVersion(event.sourceId, hash)) return;
  await index.upsert({ ...event, contentHash: hash });
}

💡 This pseudocode illustrates idempotency and deletion propagation; production code needs version-aware writes and transactional guarantees.

Key Takeaways

Inspect the relevant pipeline stage instead of guessing.
Preserve source provenance, versions, and access scope.
Measure quality on representative queries.

Knowledge Check

Why propagate source deletion to the index?

Normalization and Deduplication

Normalize whitespace and encoding while preserving headings, lists, tables, code, units, and legal qualifications. Exact hashes catch identical content; normalized hashes catch formatting-only differences. Near-duplicate detection should preserve meaningful historical versions.

Version normalization logic. A parser or cleaner change may require reprocessing. Store raw source references and normalized output so a retrieval failure can be traced back to the original.

Implementation example

function normalizeText(text) {
  return text.replace(/\r\n/g, "\n")
    .replace(/[\t ]+/g, " ")
    .replace(/\n{3,}/g, "\n\n")
    .trim();
}

💡 This is a conservative baseline; format-aware parsers must preserve semantic structure.

Key Takeaways

Inspect the relevant pipeline stage instead of guessing.
Preserve source provenance, versions, and access scope.
Measure quality on representative queries.

Knowledge Check

What is a risk of aggressive normalization?

Freshness, Versioning, and ACL Metadata

Store stable document ID, version, content hash, effective date, ingestion time, tenant/group ACL, and parser/chunker/embedding versions. Access filters must be derived from trusted identity and applied before chunks reach the model. Test revocation as carefully as granting access.

Define freshness SLOs, such as the target time from source change to searchable update. Crawl time alone does not prove a policy is current; effective date and supersession status are different concepts.

Implementation example

const filter = {
  tenantId: authenticatedUser.tenantId,
  status: "published",
  deleted: false
};

💡 Build filters from server-authenticated identity, never from model-generated or untrusted tenant values.

Key Takeaways

Inspect the relevant pipeline stage instead of guessing.
Preserve source provenance, versions, and access scope.
Measure quality on representative queries.

Knowledge Check

Where should tenant filtering be enforced?

Source Inventory and Authority

Connectors, Crawlers, and Incremental Sync

Normalization and Deduplication

Freshness, Versioning, and ACL Metadata

STAY UPDATED WITH BYTES!