Sharding, Partition Keys, and Hotspots
Sharding divides a dataset across partitions so storage and throughput can grow beyond one node. A partition key determines where data lives. A good key distributes traffic and storage while supporting common queries; a poor key creates hotspots or forces expensive scatter-gather queries.
Range partitioning supports range scans but can overload a current-time or monotonically increasing key range. Hash partitioning distributes keys more evenly but makes range queries harder. Composite keys can balance query needs and distribution.
Resharding changes partition ownership and requires data movement, routing updates, and careful handling of concurrent writes. Design for partition-key evolution and monitor skew. Averages hide hot partitions: one shard at 95% utilization can be the bottleneck while the cluster average appears comfortable.
- Partition keys are architectural decisions with long-term consequences.
- Monitor skew and hot partitions, not only cluster averages.
- Resharding is a data migration, not just a configuration toggle.