{
  "id": "sharding",
  "title": "Sharding",
  "lang": "en",
  "source_lang": "en",
  "summary": "Sharding divides data or computational state into independently managed partitions to distribute storage, processing, and workload across multiple resources.",
  "category": "computer-science",
  "subcategory": "software",
  "type": "concept",
  "aliases": [
    "Database sharding",
    "horizontal sharding"
  ],
  "generated_at": "2026-11-05",
  "model": "pro/gpt-6.1-sol",
  "url": "https://aiwiki.page/en/sharding/",
  "translations": {
    "zh-Hans": {
      "url": "https://aiwiki.page/zh/sharding/",
      "json": "https://aiwiki.page/zh/sharding.json"
    }
  },
  "sections": [
    {
      "heading": "",
      "text": "Sharding is a technique for dividing a dataset or computational state into smaller units, called shards, that can be stored and processed separately. In its most common use, a database distributes different subsets of its records across multiple servers while presenting applications with a single logical service. Each shard is responsible for part of the data rather than for a complete copy.\n\nSharding enables systems to grow beyond the storage capacity, write throughput, or processing power of one machine. It is a central technique in distributed computing, but it introduces coordination problems: requests must reach the correct shard, operations spanning shards must preserve their intended semantics, and data must sometimes move while the system remains available. Its effectiveness depends less on the number of servers than on how well data placement matches the workload.\n\nThe term also appears in search systems, distributed caches, blockchains, and machine-learning infrastructure. These applications share the principle of dividing responsibility, although they differ in what is partitioned and in the guarantees required when partitions interact."
    },
    {
      "heading": "Foundations and historical development",
      "text": "Database sharding is usually a form of horizontal partitioning: different rows or records belong to different partitions, while their logical structure remains the same. A customer table might place customers numbered 1–1,000,000 on one shard and customers numbered 1,000,001–2,000,000 on another. Vertical partitioning instead separates attributes, such as placing profile information and large document contents in different storage structures. Dividing an application into separate customer, inventory, and billing services is functional decomposition; each service may subsequently shard its own data.\n\nPartitioning and sharding are related but not identical terms. A database management system can partition a table within one server to simplify maintenance or improve query execution. Sharding generally implies that partitions can be placed on independently provisioned resources, usually different machines. Product terminology varies: a shard may be a logical partition, a physical storage unit, or a group of replicas. Consequently, the number of shards does not necessarily equal the number of servers.\n\nSharding is also distinct from database replication. Replication maintains multiple copies of the same information; sharding assigns different information to different partitions. A system can combine both techniques by dividing its data into 100 shards and maintaining three replicas of each shard. Replication can improve availability and read capacity, while sharding can expand total storage and distribute writes. Neither automatically provides the benefits of the other: adding replicas does not necessarily relieve a single write leader, and distributing unreplicated data creates additional components whose loss can make some records unavailable.\n\nThe underlying ideas predate widespread use of the word sharding. Distributed relational databases and shared-nothing parallel databases explored data partitioning during the 1970s and 1980s. The Gamma database machine project, begun at the University of Wisconsin–Madison in 1984, was an important example of partitioned parallel query processing. Internet-scale systems later made partitioning a routine operational concern. Google’s Bigtable paper in 2006 described tables divided into tablets, and Amazon’s Dynamo paper in 2007 described decentralized key placement and replication.\n\nThese developments did not establish one universal architecture. Some systems placed partitioning inside the database engine; others implemented it in application code or middleware. Modern distributed relational databases often automate placement, replication, and transaction coordination, whereas application-managed sharding leaves more of these responsibilities to developers. The common architectural objective is to distribute the dominant resource demands without introducing a larger coordination bottleneck."
    },
    {
      "heading": "Partitioning strategies and shard-key selection",
      "text": "A shard key determines the partition responsible for a record. It may be a primary key, a tenant identifier, a timestamp, or a compound value. Its selection influences data distribution, query locality, transaction boundaries, and the difficulty of later expansion. A key that distributes stored bytes evenly may still distribute requests badly: one customer can generate most of the traffic even when every shard contains approximately the same number of customers.\n\nRange partitioning assigns contiguous key intervals to shards. For example, one shard might contain timestamps before a boundary and another the following month’s records. This preserves ordering and supports efficient range scans because related keys occupy a small number of adjacent partitions. Its main difficulty is skew. A monotonically increasing key can direct nearly every new write to the last range. Splitting historical ranges does not remove that moving write hotspot unless the design also changes how current writes are distributed.\n\nHash partitioning applies a hash function to a key and maps the result to a partition. A simple scheme with \\(n\\) shards is\n\n\\[s(k)=h(k)\\bmod n.\\]\n\nWith a suitable hash function and many distinct keys, this tends to distribute records more evenly than partitioning directly by an uneven business attribute. However, nearby original keys no longer remain nearby, so a query over an interval of customer numbers or timestamps may require every shard. Changing \\(n\\) in this simple formula also changes the destination of many existing records, making it unsuitable for inexpensive incremental expansion without another level of indirection.\n\nConsistent hashing reduces reassignment when participating nodes change. In the classic ring model, keys and node positions occupy the same circular hash space, and each key is assigned to the next node position. Virtual nodes give each physical machine multiple positions, improving distribution and accommodating unequal capacity. Under idealized uniform conditions, adding one node to \\(n\\) equivalent nodes moves approximately \\(1/(n+1)\\) of the keys. Other systems use fixed logical buckets and a mutable bucket-to-server map rather than a ring.\n\nDirectory-based and hierarchical schemes offer additional flexibility. A directory can explicitly map each tenant to a shard, allowing a large tenant to receive dedicated resources. A compound design might first select a geographic region and then hash a customer identifier within that region. Such arrangements trade simpler placement formulas for more metadata and control-plane responsibility. High key cardinality, workload balance, locality, and stable ownership are desirable properties, but they can conflict; shard-key selection is therefore a workload-specific design decision rather than a purely statistical exercise."
    },
    {
      "heading": "Routing, metadata, and logical ownership",
      "text": "A sharded system needs a routing mechanism that converts a logical request into communication with the correct partition. In client-side routing, a library obtains a partition map and sends requests directly to storage servers. This avoids a mandatory proxy hop but distributes routing logic across applications. In proxy-based routing, applications connect to a gateway that forwards requests. Database-integrated routing can make distribution largely transparent to applications, although transparency does not eliminate the underlying network and coordination costs.\n\nRouting metadata normally distinguishes logical ownership from physical location. A logical shard identifier can remain stable while its replicas move between machines. A directory may record key-range boundaries, replica addresses, the current write leader, and an ownership version. Keeping these concepts separate allows capacity changes without altering application keys. It also permits many small logical shards to share a smaller number of physical servers, so that load balancing can move selected units rather than entire machines’ workloads.\n\nClients and routers commonly cache placement information to avoid consulting a central directory for every request. Cached information becomes stale after migration or leader changes. Systems handle this through redirects, explicit stale-version errors, refreshed metadata, or forwarding from the previous owner. Version numbers or epochs distinguish successive ownership configurations. A server receiving a request for an obsolete configuration must not accept a conflicting write merely because an old router still believes it is authoritative.\n\nThe placement directory is therefore part of the system’s correctness boundary. If competing controllers assign the same writable range to different owners, the result can be divergent state. Many systems protect configuration changes using distributed consensus, leases, or a combination of mechanisms. A lease grants authority for a bounded period under specified timing assumptions; an epoch can serve as a fencing token that causes downstream components to reject operations from superseded owners. These mechanisms require careful integration with the storage layer rather than reliance on routing conventions alone.\n\nRouting also interacts with application interfaces. A request containing a tenant identifier and record identifier may identify its destination immediately, whereas a request containing only an email address may require a secondary lookup. Some systems encode routing information in identifiers; others maintain a global directory. Both approaches impose costs when records change ownership. Stable external identifiers and mutable internal placement can be separated through indirection, but that extra lookup becomes another dependency to cache, replicate, and maintain."
    },
    {
      "heading": "Query execution, indexes, and data locality",
      "text": "Queries are cheapest when the routing layer can identify a small set of relevant shards. Partition pruning uses predicates and partition metadata to exclude shards that cannot contain matching records. If an orders table is sharded by customer identifier, a query for one customer can usually reach one shard. A query for all orders above a price threshold may need every shard unless another access path exists. These two queries can have similar-looking SQL statements but very different distributed execution costs.\n\nA scatter–gather query sends work to multiple shards and combines their responses. Simple aggregation often decomposes well: each shard can compute a count and partial sum, and a coordinator can combine them. An average must be reconstructed from sums and counts, not by taking an unweighted average of shard averages. Exact distinct counts, percentiles, and some statistical operations require more state or more communication. Approximate methods can reduce that cost when their error guarantees suit the application.\n\nJoins are particularly sensitive to placement. If customers and orders are partitioned using the same customer key and their corresponding partitions are colocated, many joins can execute locally. If related rows reside on unrelated shards, the system may redistribute one or both inputs across the network. Another option is to broadcast a sufficiently small table to all participating workers. The query optimizer must compare data volumes, available indexes, network costs, and expected selectivity, often using imperfect statistics about distributed data.\n\nA database index can be local or global. A local index describes only one shard’s records and is easier to maintain alongside local writes. It cannot, by itself, identify the shard containing an arbitrary indexed value. A global index spans the entire logical dataset, often with its own independent partitioning scheme. It can accelerate lookups on non-shard keys, but an update may now involve the base-record shard and an index shard. Enforcing global uniqueness has the same underlying difficulty: every competing insertion must converge on an authority that can detect conflicts.\n\nResult assembly introduces further constraints. For a global top-\\(k\\) query, each shard can return its local top \\(k\\), after which a coordinator merges the candidates, provided ordering semantics are consistent. Deep pagination based on offsets can force repeated scans and large intermediate results. Keyset pagination can reduce this overhead but requires a stable ordering and continuation state. Even when each shard responds quickly, the slowest participant can dominate overall latency, making fan-out width an important part of query planning rather than merely an implementation detail."
    },
    {
      "heading": "Transactions and cross-shard consistency",
      "text": "A database transaction confined to one shard can often use the same concurrency-control and recovery mechanisms as a conventional database. Once a transaction updates multiple shards, the system must coordinate their outcomes. An account transfer illustrates the issue: subtracting funds on one shard and adding them on another should not leave only one change durable after a failure. Sharding changes the physical boundary within which atomic execution is straightforward.\n\nTwo-phase commit is a standard protocol for atomic commitment across participants. In the prepare phase, each participant determines whether it can commit and durably records the necessary state. If all vote to commit, the coordinator records and distributes the commit decision; otherwise, the transaction aborts. Participants may hold locks or retain versions while awaiting the outcome. If the coordinator fails at a critical point, prepared participants can be unable to decide independently until the decision becomes available. Replication of coordinator and participant state can improve fault tolerance, but atomic commitment still requires communication and recovery rules.\n\nAtomic commitment is not the same as transaction isolation. A system may ensure that all writes commit together while still allowing anomalies caused by concurrent transactions. Serializable execution requires a concurrency-control strategy, such as distributed locking, validation, or timestamp-based ordering, that covers relevant read and write dependencies. A consistent cross-shard snapshot likewise needs a way to ensure that all reads correspond to a compatible logical point. Timestamp assignment alone does not solve this unless visibility and transaction dependencies are handled correctly.\n\nThe ACID properties therefore cannot be inferred merely from whether a database is sharded. Some sharded systems provide distributed serializable transactions; others support transactions only within one partition or offer weaker guarantees across partitions. The CAP theorem concerns the inability to guarantee both its specified consistency and availability properties during a network partition. It does not state that sharded databases must always choose between all useful forms of consistency and normal-operation availability, nor does it directly classify transaction isolation levels.\n\nApplications sometimes replace a single distributed transaction with a saga: a sequence of committed local transactions accompanied by compensating actions. A reservation workflow might reserve inventory, authorize payment, and then create a shipment. If a later step fails, earlier steps may require cancellation or refund operations. Compensation is a business action, not necessarily a perfect reversal of history. Such designs need explicit intermediate states, retry behavior, and reconciliation. They shift part of consistency management into the application rather than making cross-shard dependencies disappear."
    },
    {
      "heading": "Replication, failures, and availability boundaries",
      "text": "Production systems commonly replicate each shard to avoid making one machine the sole holder of its data. A replicated shard may use a leader that orders writes and followers that copy its log, or a protocol in which several replicas participate in accepting operations. Raft is one example of a consensus protocol used to maintain replicated state. Many distributed databases run a separate replication group for each range or shard, so one machine may host leaders for some groups and followers for others.\n\nReplication policies determine which failures a shard can tolerate. In a majority-based consensus group of three voting replicas, progress normally requires two available voters; such a group can tolerate one unavailable voter. This statement depends on correct protocol operation and on the absence of failures outside the model. Placing all three replicas on one machine defeats the intended machine-failure protection. Placing them in one facility may protect against individual server failures but not the loss of that entire facility.\n\nSharding creates multiple availability boundaries. If one shard is unavailable, requests directed exclusively to healthy shards may continue. A cross-shard transaction or query can nevertheless fail when any required participant is unavailable. Under the simplified assumption that each of \\(m\\) required shards is independently available with probability \\(a\\), the probability that all are available is \\(a^m\\). Real failures are often correlated, so this expression is not a general reliability forecast, but it illustrates why broad fan-out increases exposure to unavailable components.\n\nFailure detection and recovery can themselves cause load spikes. Electing new leaders, rebuilding replicas, replaying logs, and warming caches consume resources that would otherwise serve requests. If a failed server hosted replicas from many shards, its replacement can trigger substantial copying across the cluster. Recovery traffic must coexist with foreground traffic, and replica placement affects whether one failure overloads a small subset of surviving machines. Sufficient spare capacity is therefore part of fault tolerance, not merely a provisioning convenience.\n\nReplication also differs from backup. An erroneous deletion or corrupt application update can be propagated to every live replica. Recovering earlier state requires retained snapshots, logs, or other recovery material. For a sharded database, restoring each shard independently to an arbitrary moment may produce a combination of states that never existed logically, especially when transactions crossed shard boundaries. Coordinated snapshots, transaction-aware log replay, and documented recovery points are important components of disaster recovery for distributed data."
    },
    {
      "heading": "Splitting, migration, and online resharding",
      "text": "Sharding is rarely a one-time division. Data volumes grow, tenants change behavior, machines are replaced, and previously adequate shard keys become unbalanced. Resharding changes how records are partitioned or assigned. It can mean moving an existing shard to another server, splitting a large range into smaller ranges, merging small partitions, or repartitioning the dataset by a different key. These operations differ substantially in cost and in their effects on routing.\n\nMoving an unchanged shard is simpler than changing the partitioning function because record ownership remains logically similar. A typical online migration first copies a snapshot from the source to the destination, then transfers changes that occurred during copying. A write-ahead log or change data capture stream can supply those changes. Once the destination has caught up sufficiently, the system changes ownership and routes new operations there. The old copy is retained temporarily or removed after the protocol establishes that it is no longer needed.\n\nThe cutover is the most correctness-sensitive stage. Writes must not be silently lost between the last copied update and the new owner’s first accepted operation. Common techniques include a short write pause, a coordinated ownership barrier, log-position handoff, or forwarding under versioned ownership. Requests already in flight need defined handling. Retrying a timed-out request can otherwise apply an operation twice, so request identifiers and deduplication may be necessary in addition to ordinary routing updates.\n\nChanging a shard key is more demanding because every record may acquire a different destination. Applications may also depend on the previous locality: transactions that were local can become distributed, and queries that reached one shard can become scatter–gather operations. Transitional designs sometimes read from both old and new layouts or maintain both temporarily. Uncoordinated dual writes are unsafe because one destination can accept an update while the other fails. Reliable migration requires a durable source of truth, ordered change propagation where necessary, and validation that detects missing or inconsistent records.\n\nMigration cost depends on both dataset size and ongoing mutation. If a destination receives copied data at a rate only slightly above the source’s change rate, catching up may take a long time. Copying also competes for disk bandwidth, network capacity, and cache space. Automated systems therefore throttle migrations, limit concurrent moves, and account for replica placement. Range splitting can often be cheap at the metadata level when storage structures permit it, but redistributing the resulting ranges still requires moving data or changing physical access to it."
    },
    {
      "heading": "Capacity planning, hotspots, and operational control",
      "text": "The idealized attraction of sharding is linear growth: twice as many comparable servers support twice as much data or work. Actual throughput is bounded by uneven load, shared dependencies, and coordination. If fraction \\(f\\) of a workload remains inherently serialized while the rest parallelizes perfectly across \\(p\\) workers, Amdahl’s law gives the upper-bound speedup\n\n\\[S(p)=\\frac{1}{f+(1-f)/p}.\\]\n\nA global identifier service, transaction coordinator, metadata store, or heavily contended record can act as the serialized component. Network communication and additional coordination can reduce performance further.\n\nShard balance must be measured in more than bytes or record counts. Important dimensions include request rate, write amplification, processor time, memory occupancy, disk operations, and replication traffic. One shard containing small but frequently updated records may be more expensive than another containing large, rarely accessed objects. Capacity planning also distinguishes average utilization from short-lived peaks. A cluster with comfortable average utilization can still exhibit severe latency when one shard saturates or when many background tasks start together.\n\nHot keys are a special limitation. If all operations on one logical key must be serialized, moving that key to a larger or less busy server can help only up to a point. Splitting the surrounding key range does not divide the key itself. Some workloads allow a logical counter to be distributed across subkeys and combined during reads; others require strict ordering that prevents such decomposition. Read replication and caching can reduce repeated reads, but they do not automatically remove write contention or guarantee immediate visibility of updates.\n\nOperational controls include admission limits, per-tenant quotas, bounded request queues, migration throttles, and load shedding. These are especially important in shared services, where one tenant’s burst can otherwise impair unrelated users. Observability must connect logical queries with their distributed execution: per-shard latency, queue depth, leader changes, replication lag, and fan-out are more informative than a single cluster-wide average. Tail latency matters because a user request may wait for the slowest of many shard responses.\n\nSecurity and placement policies impose additional constraints. Geographic restrictions may limit which servers can hold particular records, while access control must remain correct as ownership changes. Shard boundaries alone do not constitute a security boundary; shared processes, credentials, backups, and administrative tools can cross them. Similarly, data privacy obligations cannot be met solely by assigning a regional shard key if replicas, logs, or exported data are stored elsewhere. Operational placement is an end-to-end property of the data lifecycle."
    },
    {
      "heading": "Uses beyond transactional databases",
      "text": "Search engines commonly divide their document collections into shards, each maintaining a local inverted index. A query is distributed to relevant shards, which return candidate documents and scores for merging. Replicas can serve additional queries or replace failed copies. Relevance scoring can depend on collection-wide statistics, such as document frequencies, so shard-local statistics and globally coordinated statistics may produce different rankings. Increasing shard count can improve parallelism but also increases coordination, per-shard overhead, and the number of responses needed for a complete result.\n\nDistributed caches and key–value stores partition keys to expand memory capacity and request throughput. Redis Cluster, for example, maps keys to 16,384 hash slots, with slots assigned to cluster nodes. Hash tags allow selected keys to map to the same slot, which supports operations that require colocation. The slot layer separates the number of logical placement units from the number of servers. This illustrates a general pattern: a fixed or independently managed partition space can simplify movement without exposing physical server identities in every application key.\n\nIn stream processing, partitioning often determines which worker owns the state for a key. Events for the same account or device can be routed to one partition so that a worker maintains local aggregates and processes them in order within the system’s ordering guarantees. Adding workers may require transferring both ownership and accumulated state. The number and distribution of input partitions can constrain parallelism, while highly active keys can create the same hotspots encountered in databases. Coordinated recovery must relate state snapshots to positions in the input log.\n\nBlockchain sharding applies partitioning to data storage, transaction execution, or validation responsibilities. Its central difficulty is not merely routing: participants must obtain sufficient assurance that other shards’ data are available and their state transitions are valid. Cross-shard transfers require protocols that prevent inconsistent ownership or duplicate spending under the relevant adversarial model. Some designs partition execution; others focus on distributing data availability. These are distinct uses of the term, and a network described as sharded does not necessarily execute independent smart-contract state on separate shards.\n\nIn deep learning, sharding can distribute model parameters, gradients, and optimizer state across accelerators. The Zero Redundancy Optimizer, introduced in 2019, described stages that progressively partition optimizer state, gradients, and parameters rather than retaining every category in full on every data-parallel worker. Fully sharded training uses communication to reconstruct the parameter portions needed for computation and redistribute updates. This differs from dataset sharding, in which workers receive different training examples, and from tensor parallelism, in which individual tensor operations are divided across devices. The scarce resource is often accelerator memory, and the performance trade-off is between reduced redundant storage and additional communication."
    }
  ],
  "links": [
    "database",
    "distributed-computing",
    "database-management-system",
    "database-replication",
    "primary-key",
    "hash-function",
    "metadata",
    "load-balancing",
    "distributed-consensus",
    "sql",
    "query-optimizer",
    "database-index",
    "database-transaction",
    "acid-properties",
    "cap-theorem",
    "raft-consensus-algorithm",
    "data-backup",
    "disaster-recovery",
    "write-ahead-logging",
    "change-data-capture",
    "amdahls-law",
    "access-control",
    "data-privacy",
    "inverted-index",
    "redis",
    "stream-processing",
    "deep-learning"
  ],
  "sources": [],
  "license": "https://creativecommons.org/licenses/by/4.0/"
}