Database Migrations, Change Data Capture, and Recovery
Safely evolve schemas, backfill large datasets, synchronize derived stores, and prove backup recovery.
Expand-and-Contract Migrations and Backfills
Zero-downtime migrations must tolerate old and new application versions running simultaneously. First add a backward-compatible schema, deploy code that supports both shapes, backfill old records, switch reads and writes, validate results, then remove the obsolete shape after all old versions are gone.
Backfills should be chunked, idempotent, checkpointed, and rate-limited. Monitor database saturation, lock waits, replication lag, and production latency. A migration that succeeds on a small staging dataset may overload production when indexes and concurrent writes are involved.
A code rollback is not necessarily a data rollback. Once new writes depend on a new schema, reversing the migration may lose information. Define validation queries, a forward-fix plan, and the point after which destructive cleanup cannot safely be reversed.
- Mixed-version compatibility is essential for rolling deployment.
- Backfills need checkpoints, idempotency, and load limits.
- Schema rollback and application rollback are different operations.
CDC, Backup Integrity, and Point-in-Time Recovery
Change Data Capture (CDC) publishes database changes from logs or change streams to downstream systems such as search indexes, warehouses, and caches. Consumers must handle duplicates, lag, schema evolution, initial snapshots, and recovery when a log position expires. A storage-level row change is not automatically a meaningful domain event.
A consistent CDC bootstrap must establish a snapshot and a corresponding stream position without leaving a gap. Monitor connector offsets, lag age, schema errors, and sink failures. Make derived projections rebuildable where practical.
Point-in-time recovery combines a base backup with a complete sequence of archived changes up to a target time. Backups require encryption, access controls, retention, isolation from accidental deletion, and restore tests. Replication is not a substitute: corrupted or deleted data can replicate to every replica. Validate restored records with checksums, row counts, and business invariants.
- CDC needs snapshot/stream coordination and idempotent consumers.
- Backups are unproven until restore drills succeed.
- Replication does not protect against every deletion or corruption event.