Connectors, Crawlers, and Incremental Sync
Connectors emit create, update, delete, and access-change events. Use stable IDs, content hashes, pagination, throttling, retries, and durable checkpoints. Idempotent ingestion avoids duplicate chunks when a job repeats. Deletions and permission revocations must propagate to every index and cache.
Respect source permissions and applicable crawling rules. Do not index data merely because it is technically accessible. Keep sync jobs resumable and ensure stale jobs cannot overwrite a newer document version.
Implementation example
async function handleEvent(event) {
if (event.type === "deleted") {
await index.deleteBySourceId(event.sourceId);
return;
}
const hash = sha256(event.normalizedContent);
if (await index.hasVersion(event.sourceId, hash)) return;
await index.upsert({ ...event, contentHash: hash });
}💡 This pseudocode illustrates idempotency and deletion propagation; production code needs version-aware writes and transactional guarantees.
Key Takeaways
- Inspect the relevant pipeline stage instead of guessing.
- Preserve source provenance, versions, and access scope.
- Measure quality on representative queries.
Knowledge Check
Why propagate source deletion to the index?