8. Distributed Systems and Consistency
Consistency models
- Strong consistency: reads observe writes according to the system's defined ordering guarantees.
- Eventual consistency: replicas may temporarily disagree but converge if updates stop and propagation succeeds.
- Read-your-writes: a client can observe its own completed writes.
- Monotonic reads: a client does not move backward to older observed values.
- Causal consistency: causally related operations preserve their ordering.
CAP theorem When a network partition occurs, a distributed data system cannot guarantee both linearizable consistency and availability for every request in the formal CAP sense. It must define which operations are rejected, delayed, or served with weaker guarantees during a partition. CAP is not a general rule that a system simply chooses any two qualities at all times.
Replication concepts
- Leader-based replication routes writes through a leader or leader group.
- Multi-leader designs allow writes in multiple locations but need conflict handling.
- Leaderless designs may use quorums and reconciliation.
- Quorum arithmetic alone does not guarantee linearizability; the protocol and failure model matter.
Idempotency An operation is idempotent if repeating it has the same intended effect as executing it once. Use idempotency keys, deduplication records, unique constraints, or naturally idempotent state transitions to control duplicate effects.
Delivery semantics
- At-most-once: messages may be lost, but are not redelivered under the defined semantics.
- At-least-once: messages may be redelivered, so consumers must tolerate duplicates.
- Exactly-once claims are usually scoped to a specific system boundary. External side effects often still require idempotency or transactional coordination.
Clocks and ordering Wall clocks can drift and do not reliably establish causal order across machines. Use sequence numbers, versions, logical clocks, or protocol-specific ordering when correctness depends on ordering.
- Choose consistency guarantees per operation and user expectation.
- Design retries and duplicate delivery into distributed workflows.
- Do not assume timestamps provide a perfect global order.
- Explain behavior during network partitions and partial failures.