Advanced Database Internals: Indexes, MVCC, and Storage Engines
Understand database internals that influence read/write amplification, transaction correctness, concurrency, and performance.
B-Trees, LSM Trees, and Write-Ahead Logs
B-tree-family indexes keep keys ordered and support point lookups and range scans with bounded tree depth. Page splits, index size, random I/O, and write amplification influence performance. An index helps only when its ordering and selectivity match the query.
LSM-tree storage buffers writes, flushes sorted files, and compacts them over time. This can improve write throughput but introduces read amplification and background compaction I/O. Bloom filters and block caches reduce unnecessary reads. Write amplification measures physical bytes written relative to logical writes; compaction can become a latency bottleneck during sustained ingestion.
A write-ahead log records changes before modified data pages are considered durable, supporting crash recovery. Actual durability still depends on flush settings, storage guarantees, and replication acknowledgment policy. Benchmark realistic mixed workloads rather than selecting a database based on a single throughput number.
Ordered access and range scansBuffered writes and sorted immutable filesMVCC, Isolation Anomalies, Locking, and Deadlocks
Multi-Version Concurrency Control (MVCC) allows transactions to read versions of records according to a snapshot while writers create newer versions. This reduces some read/write blocking, but old versions need cleanup. Long-running transactions may prevent cleanup and increase storage pressure.
Isolation levels describe which anomalies may occur. Read committed prevents dirty reads but may allow non-repeatable reads. Snapshot isolation provides repeatable snapshots but can still permit write skew in some systems. Serializable isolation aims for behavior equivalent to serial execution, often using locks or conflict detection. Exact semantics differ between database engines, so consult the implementation's guarantees.
Deadlocks occur when transactions hold resources while waiting for one another. Reduce risk by keeping transactions short, acquiring resources in consistent order, and indexing predicates. Retry an aborted transaction only when its logic is safe to repeat. Avoid holding locks while waiting on remote services.
- Isolation level names do not mean identical behavior across databases.
- Long-running snapshots can delay cleanup of old versions.
- Keep transactions short and make retries safe.