{
  "id": "log-structured-merge-tree",
  "title": "Log-Structured Merge Tree",
  "lang": "en",
  "source_lang": "en",
  "summary": "A write-optimized indexing structure that buffers updates and merges sorted data across storage levels, trading cheaper ingestion for additional reading and rewriting.",
  "category": "computer-science",
  "subcategory": "algorithms-theory",
  "type": "concept",
  "aliases": [
    "LSM tree",
    "LSM-tree",
    "log-structured merge-tree"
  ],
  "generated_at": "2026-11-05",
  "model": "pro/gpt-6.1-sol",
  "url": "https://aiwiki.page/en/log-structured-merge-tree/",
  "translations": {
    "zh-Hans": {
      "url": "https://aiwiki.page/zh/log-structured-merge-tree/",
      "json": "https://aiwiki.page/zh/log-structured-merge-tree.json"
    }
  },
  "sections": [
    {
      "heading": "",
      "text": "A log-structured merge tree, usually abbreviated LSM tree, is a data structure for maintaining an ordered collection of keys and associated values under frequent insertion, update, and deletion. Instead of immediately modifying the final storage location of each record, it accumulates changes in memory, writes them to storage in batches, and progressively merges the resulting sorted components. This organization converts many small, potentially random writes into larger transfers and distributes maintenance work across many updates.\n\nLSM trees are widely used in database management systems, persistent key–value stores, and distributed storage services. Their central trade-off is that an inexpensive foreground write creates deferred work: readers may have to consult several components, and background merging repeatedly rewrites data. Performance therefore depends not just on the tree’s logical organization, but also on merge policy, memory allocation, durability requirements, workload, and storage hardware.\n\nThe term describes a family of related designs rather than one standardized file format or algorithm. The original proposal used a small memory-resident tree and larger disk-resident components; many contemporary implementations instead use mutable memory buffers and immutable sorted files arranged into levels or size groups."
    },
    {
      "heading": "Origins and design rationale",
      "text": "The LSM tree was introduced by Patrick O’Neil, Edward Cheng, Dieter Gawlick, and Elizabeth O’Neil in the 1996 paper The Log-Structured Merge-Tree (LSM-Tree), published in Acta Informatica. Its motivating problem was the cost of maintaining a database index under a high rate of record insertions. On magnetic disks, a small update could require a mechanical seek and rotational delay even when the actual record occupied only a small fraction of a disk block. Accumulating many changes before transferring them to disk offered a way to amortize these costs.\n\nThe original presentation described a memory-resident component, conventionally called \\(C_0\\), and a larger disk-resident component, \\(C_1\\), with extensions to additional components. Entries migrated through a rolling merge rather than being installed individually into their eventual disk locations. The important property was not that every update avoided random access under every circumstance, but that substantial portions of index maintenance could be performed through ordered, batched processing. Amortized analysis makes this distinction explicit: one insertion may trigger considerable work, while the average cost across many insertions remains favorable.\n\nThe principal comparison was with the B-tree, whose pages are updated as records change. A B-tree offers a relatively direct search path and generally keeps one current logical position for each indexed entry. An LSM tree allows recent changes and older representations to coexist in different components, then reconciles them later. Neither organization is universally superior. A buffer pool can combine B-tree updates effectively, while an LSM tree may spend substantial bandwidth rewriting the same surviving data.\n\n“Log-structured” refers to the strategy of accumulating changes and processing them in large transfers. It does not mean that an LSM tree is simply an append-only log, nor that it has the same organization as a log-structured file system. Sorted order remains fundamental: it supports efficient merging, point lookup within files, and ordered iteration. A separate recovery log is common, but is not itself the ordered index.\n\nThe design became especially influential as large storage systems demanded sustained ingestion across many machines. Google’s 2006 description of Bigtable documented an architecture using memory buffers, sorted persistent tables, and compaction. Later open-source implementations made similar mechanisms available as embedded storage engines. These developments broadened the term “LSM tree” to cover practical designs whose files, merge scheduling, and concurrency mechanisms differ substantially from the original rolling-merge proposal."
    },
    {
      "heading": "Write buffering and persistent layout",
      "text": "A typical contemporary LSM write first enters a write-ahead log and an in-memory ordered buffer called a memtable. The log supplies a recoverable record of changes that have not yet reached durable sorted files; the memtable supplies efficient access to those changes. The exact acknowledgement point depends on the durability contract. Writing into an operating-system buffer is not equivalent to forcing the log to persistent storage, and some applications explicitly choose weaker durability for higher throughput.\n\nMemtables commonly use a skip list, a balanced search tree, or another structure that supports updates and ordered iteration. An update generally inserts a new internal record containing a user key, a version or sequence identifier, an operation type, and, when applicable, a value. A deletion is usually represented by a deletion marker rather than by an immediate search for every older copy. Consequently, the memory representation can contain more than one version of the same user key.\n\nWhen a memtable reaches a size threshold, the engine normally makes it immutable and creates a new mutable memtable. A background flush writes the immutable contents into one or more sorted string table files, often called SSTables. This separation allows new writes to continue while the previous buffer is being persisted. If flushing cannot keep pace, immutable buffers accumulate; engines limit their number or total memory consumption and eventually slow or stop foreground writes.\n\nAn SSTable is usually an immutable sequence of records sorted by an internal comparator. Its physical layout often includes data blocks, index blocks, filter information, checksums, and a footer identifying important offsets. Block-level compression is common because readers can decompress only the blocks they need. An index maps key ranges to blocks, while a file’s smallest and largest keys allow the engine to reject irrelevant files without reading their contents. The exact meaning and encoding of “SSTable” varies between systems.\n\nThe active database state is more than its collection of data files. Persistent metadata identifies which files belong to the current version of the database, their levels and key ranges, and the relevant recovery-log state. New files can be created before becoming visible, then installed through a metadata change. Old files remain available until readers using earlier versions no longer need them. Thus, immutability simplifies sharing, but safe publication and reclamation remain essential parts of the storage engine."
    },
    {
      "heading": "Point lookups, filters, and ordered scans",
      "text": "A point lookup must determine which visible record for a key is authoritative. It ordinarily searches the mutable memtable, any immutable memtables, and relevant persistent components. In an unversioned interface, the newest applicable value wins; a newest deletion marker means the key is absent. Under snapshot reads, “newest” instead means the newest version permitted by the snapshot’s visibility rules. Search order and file metadata can often avoid examining components that cannot contain a more relevant result.\n\nPersistent files are commonly equipped with a Bloom filter, a compact probabilistic structure that can rule out membership without loading a data block. A conventional Bloom filter has no false negatives when correctly constructed for the queried key representation, but may report that an absent key is possibly present. If it uses \\(m\\) bits for \\(n\\) distinct keys and \\(k\\) hash-derived bit positions per key, its approximate false-positive probability is\n\n\\[p \\approx \\left(1-e^{-kn/m}\\right)^k.\\]\n\nThe approximation assumes suitable hashing and the usual independence model. The approximate optimum is \\(k=(m/n)\\ln 2\\); at ten bits per key, a well-sized filter has a false-positive rate around one percent.\n\nFilters are especially useful for unsuccessful point lookups, which otherwise may require checking many files before establishing absence. They do not reveal the stored value, select the latest version, or eliminate the need for an index inside a matching file. They also consume memory or storage bandwidth. Engines may filter at file or block granularity, partition filters, or allocate different filter sizes to different levels. A filter that is expensive to retrieve can save fewer resources than its nominal false-positive rate suggests.\n\nOrdered range scans have a different access pattern. Each relevant component supplies an iterator positioned near the lower bound, and the engine combines these streams in key order. A priority queue is one common mechanism for selecting the next candidate record. For each user key, the scan must suppress hidden versions and respect deletion markers before returning a visible value. Merge-sort techniques make this combination efficient, but overlapping components still add iterator state, comparisons, block reads, and decompression.\n\nCaches further separate logical search work from physical device access. File indexes, filters, and data blocks may already be in an engine-managed block cache or the operating system page cache. A lookup that examines several components therefore need not issue the same number of storage reads. Conversely, a scan over a large working set can displace useful cached blocks. Meaningful performance descriptions distinguish candidate files, filter probes, cache misses, physical reads, and returned records rather than treating all of them as one “read cost.”"
    },
    {
      "heading": "Compaction strategies and their consequences",
      "text": "Compaction merges sorted components, resolves redundant records when permitted, and creates replacement components. Its algorithmic core is an ordered merge, but its policy determines which files participate, when work occurs, and how much overlap remains afterward. Compaction is not merely optional housekeeping: without it, a continually updated LSM tree would accumulate files, obsolete versions, and deletion markers until lookup and storage costs became unacceptable. Nevertheless, compaction can be deferred or reorganized, and different policies make markedly different trade-offs.\n\nLeveled compaction organizes most persistent data into levels with geometrically increasing target capacities. Files within an ordinary nonzero level usually cover nonoverlapping user-key ranges, so a point lookup needs at most one candidate file from that level, subject to implementation details. A newly flushed level, often called level 0, commonly permits overlapping files. To move data downward, the engine merges selected source files with files in the next level whose key ranges overlap. This limits the number of persistent runs a reader must consider, but can repeatedly rewrite existing data.\n\nTiered compaction retains several similarly sized sorted runs and merges them when a count or size threshold is reached. Because incoming data need not immediately be merged into one consolidated run at each size scale, a record can experience less rewriting. The cost is more overlapping runs to search and, often, greater temporary or obsolete-data space. “Size-tiered,” “universal,” and related names describe particular implementations rather than perfectly interchangeable algorithms; their triggering rules and file grouping can differ.\n\nA simple example illustrates the distinction. Suppose four newly created runs cover the same broad key range. A tiered policy can keep all four until merging them together becomes worthwhile. A leveled policy may merge each batch into existing lower-level data sooner, maintaining fewer read candidates but rewriting some older records several times. If the same hot keys are overwritten repeatedly, delayed merging may discard several intermediate versions at once. If readers frequently scan those ranges, earlier consolidation may instead be advantageous.\n\nCompaction can also be selected by time window, key range, deletion density, or workload-specific priorities. Time-window grouping is useful for data whose expiration and access patterns closely follow event time, although out-of-order writes complicate the design. Output files are usually split to bounded sizes so that later maintenance can operate on smaller key ranges. Parallel compactions are possible when their inputs and output ranges satisfy the engine’s safety rules. Scheduling must balance immediate read needs against long-term maintenance, because a policy with low total work can still produce disruptive bursts."
    },
    {
      "heading": "Amplification and analytical cost models",
      "text": "LSM performance is often expressed through write amplification, read amplification, and space amplification. Write amplification compares bytes written by the storage engine with bytes of logical application updates. Read amplification describes extra work needed to answer a query, but its unit must be specified: candidate runs, block reads, bytes transferred, or records examined. Space amplification compares physical storage consumption with a defined logical baseline, such as the size of currently live records. Different definitions can produce different numerical results for the same database.\n\nLet \\(N\\) be the persistent data volume, \\(M\\) the effective memory-buffer capacity in the same units, and \\(T>1\\) the target size ratio between successive levels. An idealized geometric organization has approximately\n\n\\[L \\approx \\left\\lceil \\log_T\\!\\left(\\frac{N}{M}\\right)\\right\\rceil\\]\n\npersistent size scales, when \\(N>M\\). This estimate suppresses partially filled levels, metadata, multiple memory buffers, compression, and reserved capacity. It is useful for reasoning about scale, not for predicting an exact file count or latency.\n\nUnder common simplified models, tiering permits on the order of \\(T\\) runs per size scale, yielding roughly \\(O(TL)\\) candidate runs for an unfiltered point lookup, while each surviving record is rewritten on the order of \\(L\\) times. A leveled arrangement reduces candidate runs toward \\(O(L)\\), but overlap with larger levels can produce write work on the order of \\(TL\\). These are broad tendencies under specific assumptions about uniformity, merge selection, and occupancy, not universal bounds for all implementations or workloads. Filters and caching can substantially reduce physical read costs without changing the underlying run count.\n\nThe denominator of an amplification measurement matters. An application that repeatedly changes a small value under a long key can produce a different ratio depending on whether logical bytes include the key, operation metadata, and transaction log. Engine write amplification may exclude the recovery log, whereas device-level amplification includes additional rewriting inside a storage device. Similarly, a database with long-lived snapshots may retain old versions intentionally; calling that storage “obsolete” would misrepresent its semantics.\n\nA practical throughput constraint follows from available maintenance bandwidth. If an engine must write approximately \\(a\\) storage bytes per ingested logical byte and has \\(D\\) bytes per second of sustainable write bandwidth available, its steady-state ingestion rate cannot exceed approximately \\(D/a\\) on that constraint alone. Compaction also reads data and consumes processor time, so this is not a complete throughput model. Short benchmarks can exceed the sustainable rate by accumulating pending work, making compaction debt and final storage state important parts of any measurement."
    },
    {
      "heading": "Recovery, snapshots, and deletion safety",
      "text": "An LSM tree’s deferred writes require a recovery protocol that connects the log, memory state, and persistent file set. A common sequence is to record updates in a recovery log, apply them to a memtable, flush that memtable to new sorted files, durably install those files in database metadata, and only then retire log segments no longer needed. Recovery reconstructs the current file set and replays necessary log records. Checksums and record framing help identify incomplete or corrupted log records, while correct handling of persistence ordering remains essential.\n\nCompaction uses a related publication protocol. Replacement files are written before a metadata operation replaces the old inputs with the new outputs. A crash before publication can leave unused output files; a crash after publication can leave old files awaiting deletion. In either case, the durable metadata must identify a coherent logical state. File creation, rename, metadata logging, and synchronization guarantees depend on the underlying platform. Immutability does not remove the need to reason carefully about failures between these operations.\n\nMany LSM engines use multiversion concurrency control or related sequence-based visibility. Internal keys can combine a user key with a sequence number and record type, allowing multiple historical versions to coexist in sorted order. A snapshot fixes a visibility boundary so that later writes do not change its results. Compaction may discard a superseded version only when no relevant snapshot or other retention rule requires it. A long-lived snapshot can therefore increase disk use and reduce the amount of data compaction is allowed to reclaim.\n\nDeletion markers, often called tombstones, prevent older values from reappearing. Removing a tombstone merely because its local file has been compacted is unsafe: an older value may still exist in another component. Safe removal requires proving that no relevant older version can survive outside the compaction’s coverage and that no snapshot needs the deleted history. Range tombstones add another dimension by deleting an interval of keys; scans and compactions must account for overlapping intervals and their visibility boundaries.\n\nLocal storage correctness is distinct from database replication correctness. In a distributed database, a replica that missed a deletion may later reintroduce an older value unless repair, version reconciliation, and deletion-retention rules prevent it. Likewise, an LSM tree does not by itself provide ACID properties or multi-key transactions. Transactional systems add mechanisms such as locking, conflict detection, transaction records, and commit protocols. The LSM engine supplies durable ordered storage on which those mechanisms can be built."
    },
    {
      "heading": "Storage hardware and value placement",
      "text": "The original appeal of LSM trees was closely tied to magnetic-disk behavior. Sequential transfers amortize seeks and rotational delay over many records, whereas unrelated small updates can be dominated by positioning time. Batched merges exploit this asymmetry. However, “sequential” is not an absolute property of a whole engine: simultaneous flushes, compactions, log writes, and foreground reads can interleave at the device. File fragmentation and scheduling influence whether logically ordered activity becomes efficient physical traffic.\n\nA solid-state drive changes, but does not eliminate, the trade-offs. Flash memory supports relatively fast random reads, while writes and erasures obey page and erase-block constraints. A flash translation layer remaps logical addresses and performs its own garbage collection. Repeated LSM compaction can therefore combine with device-internal rewriting to increase physical write traffic. The engine’s write amplification and the device’s internal amplification are separate quantities; both affect bandwidth consumption and endurance.\n\nWith fast storage, processor and memory costs become increasingly visible. Comparing keys, computing filter hashes, decompressing blocks, maintaining iterators, and copying data can dominate latency or consume substantial CPU capacity. Large values intensify compaction traffic, while long shared key prefixes increase the usefulness of prefix compression. Memory allocation between memtables, filters, indexes, and data caches also matters: enlarging the write buffer may reduce flush frequency but leave less space for frequently accessed read blocks.\n\nOne response to large-value workloads is key–value separation. The sorted structure stores keys and references, while values reside in a separate log or storage area. WiscKey, presented in 2016, is a prominent research example. Because compaction moves comparatively small index entries rather than full values, this design can reduce rewriting. It also introduces value-log garbage collection, reference management, and additional indirection. Range scans may lose the physical locality they would have when neighboring keys and values are stored together.\n\nDisaggregated storage and object storage introduce further constraints. Immutable sorted files are convenient objects to upload, share, cache, and reference through metadata, but request latency, transfer granularity, and network cost can make small random reads expensive. Prefetching, local caches, larger read units, and separate compaction workers are possible responses. Device-specific designs may also exploit placement controls or sequential-write zones. These adaptations retain the central buffering-and-merging principle while changing where data movement occurs and which resource limits performance."
    },
    {
      "heading": "Major implementations and database integration",
      "text": "Bigtable, described by Google in 2006, is an influential example of LSM-style storage within a distributed data service. Its published design used a commit log, an in-memory memtable, and immutable SSTables, with compaction combining stored data and removing eligible deleted information. The system partitioned its ordered row space into tablets. This illustrates an important separation: the LSM-like structure manages local persistent representations, while tablet assignment, failure handling, and distributed coordination operate at another layer.\n\nLevelDB, released as open-source software by Google in 2011, provided an embedded ordered key–value library with memtables, a log, SSTables, and leveled compaction. Its small interface helped make the architecture accessible beyond large distributed services. It supplies storage inside an application process rather than acting as a complete distributed database server. Applications using such a library must still decide how to provide networking, authorization, transaction semantics beyond the library’s capabilities, and operational management.\n\nRocksDB, developed at Facebook from LevelDB and released in 2013, extended this lineage with a strong emphasis on production storage workloads and fast devices. Its facilities include multiple compaction styles, configurable memory structures and caches, column families, and mechanisms for controlling background work. A column family provides a separately configurable logical key space within a database, but can share resources such as a write-ahead log. Such features make the actual performance of an LSM engine highly configuration-dependent rather than a consequence of the data-structure label alone.\n\nApache Cassandra and Apache HBase use LSM-style storage in distributed database architectures, though their data models, consistency mechanisms, and operational designs differ. Cassandra combines commit logging, memtables, SSTables, and configurable compaction strategies. HBase, associated with the Hadoop ecosystem, uses write-ahead logging, memstores, and persistent HFiles. Their distributed behavior cannot be inferred solely from these local storage similarities: partitioning, repair, replica coordination, and failure recovery are separate design choices.\n\nLSM storage is not restricted to systems marketed as NoSQL. A relational database can place primary records and secondary indexes in an LSM-backed engine, and a distributed SQL system can use such an engine beneath a transactional key–value layer. The relational model and SQL describe logical organization and access, not a mandatory physical index format. Secondary indexes can multiply write work because one logical row change may generate several independently stored key updates. Transaction processing and query execution must account for that physical cost."
    },
    {
      "heading": "Operational behavior and continuing research",
      "text": "A defining operational concern is compaction debt: pending maintenance that must eventually be performed to maintain the intended storage organization. If incoming writes exceed the rate at which flushes and compactions can process them, upper-level files and immutable memory buffers accumulate. Engines commonly respond with write slowdowns or stalls. These mechanisms protect bounded memory and read performance, but expose deferred maintenance as foreground latency. Sustained throughput therefore depends on balancing ingestion with background capacity, not merely making the initial write path fast.\n\nWorkload distribution strongly affects that balance. Sequential inserts into new key ranges can have different overlap patterns from uniformly random updates. A small hot set may generate many obsolete versions that can be eliminated together, while long-lived snapshots prevent such elimination. Large batches can amortize logging and synchronization, but may also create bursts of memory and compaction work. Time-to-live expiration makes records logically invisible before their bytes are necessarily reclaimed; when records are grouped suitably, entire files may sometimes become reclaimable without rewriting their contents.\n\nCompaction scheduling is a multi-objective problem. An engine may prioritize files contributing most to read latency, data approaching capacity limits, or ranges with many obsolete records. Background concurrency can improve device utilization until it competes too strongly with foreground reads for bandwidth, processor time, or cache capacity. Rate limiting and resource isolation seek to control these interactions. Average operation throughput alone can conceal severe high-percentile latency, so evaluations commonly need latency distributions, workload duration, cache conditions, and the amount of pending maintenance.\n\nResearch has investigated jointly optimizing merge policies and filter memory rather than tuning each independently. Monkey, presented in 2017, showed the importance of distributing Bloom-filter memory across levels according to their contribution to lookup cost rather than assigning an identical number of bits per key everywhere. Dostoevsky, presented in 2018, explored lazy leveling and a broader design space between conventional tiering and leveling. These approaches illustrate that the read–write trade-off depends on the entire configuration, including memory budgets and the relative frequencies of lookups, scans, and updates.\n\nOther research directions include reducing value movement, parallelizing compaction, improving range-deletion handling, using newer filter structures, and separating storage from compute. Automated tuning is difficult because the best configuration can change with key distribution, value size, device behavior, and retention requirements. Evaluation must also distinguish a database that is temporarily absorbing writes into spare capacity from one in steady state. For delete-heavy or overwrite-heavy workloads in particular, reporting the final live-data size, physical footprint, and remaining compaction work is necessary to interpret apparently high ingestion rates."
    }
  ],
  "links": [
    "data-structure",
    "database-management-system",
    "database-index",
    "amortized-analysis",
    "b-tree",
    "file-system",
    "write-ahead-logging",
    "data-compression",
    "metadata",
    "priority-queue",
    "merge-sort",
    "operating-system",
    "database-replication",
    "acid-properties",
    "database-transaction",
    "flash-memory",
    "central-processing-unit",
    "nosql",
    "relational-database",
    "sql"
  ],
  "sources": [],
  "license": "https://creativecommons.org/licenses/by/4.0/"
}