How LSM Trees Actually Work

Many engineers first meet LSM trees through a symptom rather than a design document.

A write-heavy service works beautifully at first. Inserts are fast. Disk throughput looks almost sequential. Then a few weeks later the graphs change shape. Read latency gets bumpier. Background I/O rises. SSD wear looks worse than expected. Deleting a lot of keys does not immediately give space back. Somebody says "compaction is behind" and half the room nods as if that phrase explains anything.

It does explain something, but only if you already know what machine is hiding behind it.

An LSM tree, short for log-structured merge tree, is a storage-engine design that turns small random updates into sequential writes by buffering them in memory, flushing them as immutable sorted files, and continuously merging those files in the background.

That sentence is the whole story in miniature. It is also not enough. You need to know what is in memory, what is on disk, what a lookup has to search, why deletes are not immediate, why compaction both saves and hurts you, and why one implementation's "LSM tree" can behave quite differently from another's.

This article stays close to the mechanism. We will follow one write from the WAL to the memtable to an SSTable on disk. We will walk the read path, including filters and index blocks. We will look at compaction in its real forms, especially leveled and tiered layouts. We will look at tombstones, snapshots, and why old data can survive longer than you expect. Then we will connect all of that to the operational tradeoffs people actually feel in RocksDB, LevelDB, Pebble, Cassandra, ScyllaDB, and similar systems.

LSM Trees Are About Changing Write Shape, Not About Growing A Fancy Tree On Disk

The name misleads people in two ways.

First, the important part is not the word "tree". Most of the time you are reasoning about a stack of sorted runs, not about a pointer-rich disk tree like a B+ tree. The memtable in RAM might be implemented as a skip list, a balanced tree, an array structure, or something similar. The files on disk are immutable sorted tables. The background merger is what ties those pieces into one logical keyspace.

Second, the important property is not that the engine stores data in sorted order. Plenty of systems do that. The distinctive move is where the sorting happens and when disk rewrite work is paid.

A traditional B+ tree engine updates pages in place, or at least behaves as if it does. One row change means walking to a leaf page, changing that page, possibly splitting it, and writing dirty pages back later. That design is excellent for point lookups and range scans because the on-disk structure is already one globally organised search tree. But random writes can become random I/O, random page rewrites, and awkward amplification once the working set is larger than memory.

An LSM tree takes the opposite bet.

Instead of rewriting the final on-disk place for each small update, it says:

1. append the change to a log for crash recovery
2. insert the change into an in-memory sorted structure
3. when memory fills, dump that sorted content to disk as a new immutable file
4. later, merge old files into larger, cleaner files

That design swaps one kind of pain for another.

You usually get:

• excellent ingest throughput
• sequential flushes and merge I/O
• good space utilisation on modern SSDs and cloud block devices
• easy batched writes

But you also inherit:

• more moving parts on the read path
• background compaction as a permanent cost centre
• delayed cleanup of old versions and deletes
• a constant balancing act between read amplification, write amplification, and space amplification

That last line is the core tradeoff. Most serious LSM tuning is really about deciding which amplification you are willing to pay more of.

Why Storage Engines Keep Choosing This Design For Write-Heavy Workloads

Suppose you run a metrics pipeline in Amsterdam ingesting millions of sensor updates per second. Most keys are tiny. Each update changes one small value. The workload is dominated by writes, and the values are usually written once or updated a few times before they age out.

A page-oriented engine can do that job, but it is fighting the shape of the workload. Each tiny update wants to become a page lookup, a page latch, a page rewrite, and later a checkpoint or page flush. If the working set is much larger than RAM, those writes bounce around the disk layout.

An LSM tree is better aligned with the physics.

Small updates arrive in a stream. The engine can batch them in RAM, log them sequentially, then flush whole sorted chunks. SSDs love long writes. Cloud volumes usually prefer them too. Object-backed local disks and EBS-like devices are much happier with predictable merge traffic than with scattered synchronous page rewrites.

There is another reason this design keeps winning: modern systems often write more often than they read each individual key.

Think about:

• telemetry
• logs and traces
• message offsets
• user activity feeds
• counters and state checkpoints
• caches with persistence
• event-sourced or append-heavy systems
• coordination metadata with frequent churn

In these workloads, the expensive thing is not finding the right place for one key. The expensive thing is absorbing an endless stream of tiny mutations without thrashing the storage device.

That is why the LSM family shows up in systems such as:

• RocksDB and LevelDB
• Pebble in CockroachDB's storage layer
• Cassandra and ScyllaDB, where SSTables are foundational
• HBase and Bigtable style systems
• many embedded key-value stores used inside larger products

The design is especially attractive when you can say, honestly, that write throughput matters more than having the simplest possible read path.

The Write Path Is WAL, Memtable, Flush, Then Manifest Update

A correct mental model starts with one write.

Imagine a key meter:ath-17 whose value is changing from 3412 to 3413.

In a typical LSM engine, the write does something close to this:

client put("meter:ath-17", "3413")
  -> append record to WAL
  -> insert key/version/value into current memtable
  -> acknowledge when durability contract is satisfied

The exact acknowledgement point depends on the implementation and options. Some systems acknowledge after the WAL append reaches the kernel page cache. Some wait for fsync. Some batch several clients into one WAL sync. But the structural order is stable: log first, then publish into memory.

The WAL exists so the memtable can be fast and disposable

The memtable is an in-memory sorted structure. It is fast, mutable, and not durable by itself. If the process crashes, RAM is gone. So the engine first writes a recovery record to a write-ahead log.

A simplified WAL record might look like this:

sequence=880132
op=put
key=meter:ath-17
value=3413
crc=0x7e14b19c

Real records usually include length prefixes, checksums, batch framing, sequence numbers, and sometimes column-family or transaction metadata. The details vary. The important point is that the WAL is append-only. That makes it cheap to write and straightforward to replay.

On restart, the engine replays surviving WAL segments and rebuilds any memtables that had not yet been flushed into SSTables.

The memtable is the newest sorted view

After logging, the engine inserts the write into the active memtable.

The memtable is usually sorted by internal key, not just by user key. The internal key often includes:

• the user key
• a sequence number or timestamp
• an operation type such as put or delete

That ordering matters because the same user key may appear multiple times with different versions. The storage engine has to decide which one is visible to which reader.

A simplified memtable view might look like this:

meter:ath-16 @ seq 880130 -> 9921
meter:ath-17 @ seq 880132 -> 3413
meter:ath-18 @ seq 880131 -> 7740

The values are sorted by key, and versions are sorted so the engine can find the newest visible record quickly.

When the memtable fills, it stops being mutable

The current memtable cannot grow forever. Once it reaches a configured size, several things happen:

1. the current memtable is frozen and becomes immutable
2. a new active memtable is created for fresh writes
3. a background flush task writes the immutable memtable to disk as an SSTable

That split is a big part of the trick. Writes do not stop just because one chunk is being flushed. They move to the next memtable while the old one drains in the background.

This is also why write stalls can happen if background work cannot keep up. If too many immutable memtables accumulate, the engine eventually has nowhere safe to buffer more data and must slow or stop writers. We will come back to that.

Flush produces an SSTable, not a heap file

The flush output is a new SSTable, historically "sorted string table". The name comes from old terminology where values were called strings even when they were arbitrary byte arrays.

A flush walks the immutable memtable in sorted order and serialises it into an immutable disk file. The file typically contains:

• ordered key-value entries or key-value-version entries
• data blocks
• per-block restart points or compression boundaries
• a block index
• optional Bloom filter data
• metadata blocks
• a footer pointing to important structures

Once written and made visible through metadata, that SSTable becomes part of the persistent keyspace.

Metadata has to say which files now exist

Creating the file bytes is not enough. The engine also needs durable metadata describing the set of live SSTables.

In LevelDB and RocksDB style engines, this is done through a manifest or version-edit log. A flush effectively says something like:

add file 000321.sst to level 0
smallest key = meter:ath-01
largest key = meter:ath-99
sequence range = 880001..880132

Why does this matter? Because a crash halfway through file creation must not make a partially written SSTable look authoritative. Recovery needs a consistent source of truth for which files belong to the database version.

So the durable write path is not just WAL plus data file. It is WAL, memtable state, file creation, and metadata publication.

SSTables Are Immutable Sorted Runs With Their Own Mini Indexes

An SSTable is the basic disk unit of an LSM tree. If you do not understand SSTables, the rest of the design feels magical and vague.

The right way to think about one is: an immutable sorted run plus enough side structures to avoid reading unnecessary bytes.

A simplified file layout might look like this:

[data block 0]
[data block 1]
[data block 2]
...
[filter block]
[index block]
[properties block]
[footer]

Data blocks hold ordered records

The data section is usually divided into blocks, often on the order of 4 KiB to 64 KiB depending on implementation and compression settings.

Inside each block, entries are sorted by internal key. To reduce size, engines often use prefix compression or restart intervals. That means not every key is stored in full. If adjacent keys share a long prefix such as user:rotterdam: or sensor:tram:athens:, the file can store one full key and then shorter suffix-based encodings for neighbours.

That helps a lot because many keyspaces are lexicographically structured. Prefix compression reduces disk bytes, cache pressure, and sometimes I/O count.

Index blocks map key ranges to block locations

You do not want to scan a whole SSTable just to find one key. So the file includes an index mapping key separators to data-block offsets.

A rough idea looks like this:

index:
  meter:ath-20 -> block 0 @ offset 0
  meter:ath-40 -> block 1 @ offset 8192
  meter:ath-60 -> block 2 @ offset 16384

For a point lookup, the engine binary-searches the index, jumps to the candidate block, decompresses or reads that block, and then searches inside it.

This means one SSTable is not just a bag of rows. It already contains the structure needed for efficient local search.

Bloom filters answer one question only

A Bloom filter on an SSTable does not tell you where a key is. It tells you whether the key is definitely absent from that SSTable or might be present.

That negative test is hugely valuable.

Suppose a lookup for meter:ath-17 has to consider ten SSTables. If nine of them can say "definitely not here" using an in-memory Bloom filter, the engine avoids nine data-block reads.

This is one reason LSM trees can remain practical for point lookups even when several files exist. Without filters, the read path would degrade much faster.

But Bloom filters do not solve everything.

• They can return false positives.
• They do nothing for range scans because a range query usually needs actual key order, not just membership.
• They do not eliminate the cost of overlapping files at newer levels such as L0.

So when somebody says "the filter handles it", ask which access pattern they mean.

Sequence numbers and tombstones live in SSTables too

SSTables do not only store current values. They also store historical versions and delete markers until compaction proves older state is no longer needed.

A delete usually becomes a tombstone, not immediate physical erasure.

For example:

meter:ath-17 @ seq 880132 -> 3413
meter:ath-17 @ seq 880145 -> TOMBSTONE

That tombstone says: for readers at sequence 880145 or later, this key is deleted. Older snapshots may still need the previous value. Older files may still contain that value. Until compaction reconciles the versions safely, both can coexist.

That is one of the most important LSM facts to internalise. Deletion is usually a new record, not immediate removal of old bytes.

The Read Path Starts With The Newest Structures And Works Backward

If the write path is about postponing expensive rewrite work, the read path is about recovering a coherent answer from several places that may all contain the same key.

A point lookup usually checks structures in freshness order.

1. active memtable
2. immutable memtables waiting to flush
3. newest disk files, often level 0 first
4. older, more compacted levels

The engine stops when it finds the newest visible version for the reader's snapshot.

Newest state wins

Suppose these records exist across the system:

memtable:    meter:ath-17 @ seq 880150 -> 3414
L0 file A:   meter:ath-17 @ seq 880145 -> TOMBSTONE
L1 file X:   meter:ath-17 @ seq 880132 -> 3413
L2 file Z:   meter:ath-17 @ seq 870500 -> 3391

A normal latest-value read should return 3414, because the memtable has the newest visible version.

A snapshot pinned at sequence 880146 should see the tombstone and treat the key as deleted.

A snapshot pinned at 880140 should still see 3413.

That means the read path is not only searching for a key. It is also evaluating visibility by version.

Level 0 is special because files can overlap

When memtables flush, they become new SSTables, usually in level 0. Those files are individually sorted, but they are not globally non-overlapping with each other.

Why not? Because each flush reflects whatever key range happened to be hot during that memtable's lifetime. One flush might contain meter:ath-01 through meter:ath-99. The next might contain some of the same range plus many unrelated keys.

So for a point lookup, L0 is the messy frontier. Several L0 files might all plausibly contain the key. The engine often checks them from newest to oldest, using filters and key-range metadata to skip what it can.

This is why a buildup of too many L0 files hurts read latency quickly. Even if every file is small, overlap means extra filter probes, extra index lookups, and sometimes extra block reads.

Lower levels are usually cleaner

In a leveled LSM layout, levels below L0 are usually organised so that files within the same level do not overlap in key range. That gives the engine a powerful property.

For a point lookup in level 2, level 3, or level 4, the engine can often say: only one file in this level could possibly contain the key.

That keeps point reads manageable even when the total database is large.

Tiered or universal layouts relax that rule in exchange for lower write cost, so multiple runs per level or tier may need checking. Again, this is the same three-way tradeoff showing up in a different place.

Range scans are where LSM tradeoffs become more obvious

A point lookup can rely heavily on Bloom filters. A range scan cannot.

If you ask for keys from meter:ath-10 to meter:ath-90, the engine has to merge iterators from multiple live structures:

• memtables
• overlapping L0 files
• candidate files from lower levels

That merge is not conceptually hard, but it means a range scan may touch many files and reconcile many versions. If compaction is lagging or tombstones are abundant, the scan can do more work than a reader expects from the surface API.

This is one reason LSM engines shine brightest when the dominant reads are point lookups or short bounded ranges rather than long analytic scans across a churn-heavy keyspace.

Compaction Is The Real Engine, Not Background Housekeeping

People often talk about compaction as if it were janitorial work that happens after the interesting part. In an LSM tree, compaction is the interesting part.

Without compaction, flushes would keep creating more immutable files forever. Reads would need to check an ever-growing pile of overlapping runs. Tombstones would never clean up older values. Space amplification would drift upward. Eventually the engine would collapse under its own history.

Compaction is the mechanism that pays back the debt created by cheap writes.

What compaction actually does

At a high level, compaction:

1. selects one or more input SSTables
2. reads them in sorted order
3. merges records by key and version
4. drops obsolete versions or expired tombstones when safe
5. writes new output SSTables
6. atomically swaps metadata so outputs become live and inputs become obsolete

That is a read-rewrite-delete cycle on storage bytes. It consumes bandwidth, CPU, cache, and SSD endurance. But it also restores order.

Minor flush versus major compaction

Implementations use different terminology, but it helps to separate two operations.

• Flush: immutable memtable to a new SSTable, usually into L0
• Compaction: merge existing SSTables into new SSTables, often between lower levels

Flush makes data persistent as a new run.

Compaction reduces the future cost of having too many runs.

They are related but not identical. A healthy system needs both pipelines to keep up.

Leveled compaction aims for low read amplification

Leveled compaction keeps each lower level much larger than the previous one, often by a ratio around 10, and tries to maintain non-overlapping files within a level.

A cartoon layout might look like this:

L0:  6 files, overlapping
L1:  200 MiB total, non-overlapping
L2:  2 GiB total, non-overlapping
L3: 20 GiB total, non-overlapping

When an L0 file overlaps part of L1, compaction reads that L0 file plus the overlapping L1 files, merges them, and writes new L1 output files. Later, similar work pushes data from L1 to L2, then L2 to L3.

The benefit is clear:

• point lookups in lower levels touch at most one file per level
• range reads are more predictable
• old versions and tombstones are cleaned steadily

The cost is also clear:

• the same bytes may be rewritten several times on their journey down the levels
• write amplification can be substantial
• compaction bandwidth must be provisioned continuously

Leveled compaction is common when read latency matters a lot and write amplification is acceptable.

Tiered or universal compaction aims for lower rewrite cost

Tiered designs keep multiple runs around longer and merge them in larger batches less aggressively. Universal compaction in RocksDB is one prominent example in this family.

The benefit is that a byte may be rewritten fewer times, especially for data that is overwritten or deleted before it ever reaches the deepest levels.

The costs are:

• more overlapping files remain live
• point lookups may have to check more runs
• space amplification can be higher
• long-range reads may become less tidy

A simple comparison helps.

There is no universally correct winner. The right choice depends on whether your pain budget is on reads, writes, or space.

Compaction picks victims based on debt, not on beauty

Real systems do not compact files purely because the layout looks ugly. They maintain heuristics and scores.

Typical triggers include:

• too many L0 files
• a level exceeding its target size
• tombstone-heavy files worth cleaning
• expired TTL windows
• manual or forced compaction requests

A useful operational principle follows from that: compaction is policy. Two engines with the same high-level LSM idea can behave very differently because their file-picking heuristics, level sizing, subcompaction support, and throttling policies differ.

That is why copying tuning advice from Cassandra into RocksDB, or from Pebble into ScyllaDB, is usually a mistake. The family resemblance is real. The details are not interchangeable.

Deletes, Updates, And Snapshots Are Where The Model Stops Being Toy-Simple

The beginner version of an LSM tree says writes go to memtables, then to files, then files merge. That is fine until you ask one of the questions that production systems actually care about.

• What does delete mean?
• When can old versions disappear?
• How do snapshots work?
• Why is space not reclaimed immediately after heavy deletion?

The answer to all four is version visibility.

Deletes are tombstones first

A delete usually inserts a tombstone record with a newer sequence number.

Suppose user:rot-48192 currently exists in L2 with value premium=true. A delete today does not seek out that old byte range and erase it. It appends something like:

user:rot-48192 @ seq 910220 -> TOMBSTONE

Now any normal read at sequence 910220 or later treats the key as deleted.

But the old value file still exists. The tombstone and the old value may sit in different levels. Until compaction merges the relevant key range and proves that no visible snapshot needs the older version, the old bytes remain on disk.

That is why a delete-heavy workload can create surprising space pressure in LSM engines. Logical deletion is immediate. Physical reclamation is delayed and conditional.

Updates are really new versions

Most updates are not in-place modifications. They are new versions of the same key.

account:berlin-884 @ seq 50101 -> balance=1200
account:berlin-884 @ seq 50177 -> balance=1180
account:berlin-884 @ seq 50202 -> balance=1230

The engine keeps enough version history to satisfy its snapshot and compaction rules. Older versions become droppable only when they are both:

1. shadowed by a newer visible version
2. not needed by any snapshot or replication mechanism that depends on the older sequence window

Snapshots pin history

A snapshot gives a reader a consistent view of the database at a particular sequence number. That is useful for iterators, transactions, backups, replication, and many internal tasks.

But snapshots also keep old versions alive.

If a long-running snapshot is pinned at sequence 50200, then a later version at 50202 cannot simply erase everything older on the next compaction. The snapshot still needs the version visible at 50200.

This matters more than people expect. A stuck snapshot, a slow backup, or a replica that depends on old sequence windows can turn compaction cleanup from a quick background task into a long-lived storage burden.

You see the same pattern in larger distributed systems. Cassandra tombstones kept for repair safety are one famous example. Local embedded stores see the same issue in smaller form when snapshots or iterators live too long.

Space reclamation depends on overlap and age

When can a tombstone finally drop the old value?

Only when compaction can prove that no older surviving version is needed and no older file below it can still surface that value for a live reader.

In a leveled layout, this often means waiting until the tombstone has compacted down far enough to meet the older copies in overlapping key ranges.

So a delete in L0 might not reclaim much space immediately. It might take several compaction waves before the old value and the tombstone meet in the right merge.

This is why operators talk about tombstone debt. The system may look logically clean from the API while physically carrying a large amount of dead history.

The Three Amplifications Explain Almost Every LSM Tuning Argument

If you want one compact vocabulary for LSM tradeoffs, use these three phrases.

• read amplification
• write amplification
• space amplification

Every meaningful tuning decision pushes on at least one of them.

Read amplification

Read amplification is the extra work required to answer a read because data is split across several structures.

For a point lookup, this can mean:

• checking the memtable
• probing several Bloom filters in L0
• reading one candidate file per lower level
• consulting multiple blocks or cache entries

For a range scan, it can mean merging iterators from many live runs and skipping shadowed versions and tombstones.

High read amplification shows up as higher CPU, more cache misses, more small I/O, and more variable latency.

Leveled compaction is mainly a strategy for buying this down.

Write amplification

Write amplification is the ratio between bytes the user asked to write and bytes the engine actually wrote to storage over time.

A useful simplified formula is:

effective write amplification =
  (WAL bytes + flush bytes + compaction rewrite bytes)
  / user bytes written

If a 1 MiB user write is appended to the WAL, flushed once, then rewritten across several compaction levels, the final physical write cost can be many megabytes.

This matters for:

• sustained throughput
• SSD endurance
• cloud storage bills
• tail latency during compaction bursts

Tiered layouts and larger memtables often try to reduce this at the expense of other costs.

Space amplification

Space amplification is the amount of physical storage consumed beyond the live logical dataset.

Sources include:

• overlapping runs holding several versions of the same key
• tombstones waiting to meet older values
• snapshots pinning history
• partially obsolete files that compaction has not rewritten yet
• metadata and filter structures

A delete-heavy service in Lisbon can halve its logical row count and still show almost the same disk footprint for hours or days if compaction has not caught up or snapshots keep history pinned.

That is not corruption. It is the design doing exactly what it promised.

You cannot minimise all three at once

This is the unavoidable point.

• If you compact aggressively, reads get cheaper and space gets cleaner, but writes get more expensive.
• If you compact lazily, writes get cheaper, but reads and space usually get worse.
• If you keep lots of memory and cache, some read pain is masked, but flush and restart behaviour change.

So when a storage-engine team debates configuration, they are usually not arguing about correctness. They are arguing about which amplification hurts their workload least.

What Real Operational Failures Look Like In LSM Systems

The neat textbook diagrams hide the fact that an LSM engine is always managing backlog. The writes arriving now create future merge work. If the future work cannot keep up, the present eventually notices.

Too many L0 files means reads and writes are both in danger

A rising count of L0 files is usually the first obvious warning.

Why it hurts reads is easy to see: overlap means more file checks.

Why it hurts writes is just as important: L0 is where new flush output lands. If the system is not compacting L0 down fast enough, new flushes have nowhere safe to go. Engines then start slowing writes, delaying new memtable creation, or stalling entirely.

In RocksDB-style systems you often watch metrics such as:

rocksdb.num-files-at-level0
rocksdb.mem-table-flush-pending
rocksdb.compaction-pending
rocksdb.stall-micros

The exact names differ elsewhere, but the principle is universal. Flush backlog plus compaction backlog eventually becomes writer-visible pressure.

Write stalls are not bugs in the front door

When a client thread suddenly blocks on put(), the cause is often nowhere near the front-door API.

The real chain may be:

1. flushes created too many L0 files
2. compaction could not clear them fast enough
3. immutable memtables accumulated
4. the engine hit a hard threshold and started write stalls

If you debug only the writer thread, you miss the actual bottleneck. You need the background pipeline state.

Tombstone storms can make reads weirdly expensive

Delete-heavy workloads create a distinctive problem. Even after the application has logically removed a large slice of data, readers may still need to step over many tombstones and obsolete versions until compaction cleans them.

This shows up as:

• slower scans
• surprising cache churn
• lower useful compression ratio
• lingering disk use after large deletes

Systems with TTL expiration feel this a lot. Expiration is convenient, but mass expiry still has to be reconciled by compaction.

Compression helps and hurts in different places

Most SSTables are compressed, often per block. Compression reduces disk bytes and can improve cache density. But it also means compaction and reads spend CPU on encode and decode.

So a CPU-bound compaction pipeline can bottleneck even on fast storage. Conversely, disabling compression may relieve CPU pressure but increase I/O and cache miss rates.

This is another example of the design being multi-dimensional. There is rarely one global knob labelled "faster".

Storage latency still matters, even though the design is log-structured

People sometimes hear "sequential writes" and imagine that the storage device almost stops mattering. Not so.

The WAL still has durability cost. Flushes still write large files. Compaction still streams input and output bytes continuously. If the SSD in your Frankfurt node has poor sustained write behaviour or thermal throttles under merge load, the LSM engine will surface that weakness quickly.

The whole point of the design is to turn random mutation into more storage-friendly traffic, not to make storage irrelevant.

Backups, replication, and ingestion bursts can interact badly with compaction

A backup that scans huge key ranges, a replica catch-up task, and a burst of fresh writes all compete for the same underlying reality:

• block cache
• disk bandwidth
• compaction threads
• file descriptors
• page cache or mmap pressure, depending on implementation

This is why healthy LSM operation usually involves explicit rate limiting and scheduling for background work. Left alone, every useful maintenance job tends to behave as if it were the only important consumer on the machine.

The Best Way To Judge An LSM Tree Is To Ask Which Costs It Moves Where

At this point the core mechanism should be visible.

A write becomes:

• WAL append
• memtable insert
• flush to SSTable
• later compaction through deeper levels

A read becomes:

• search newest structures first
• use filters and indexes to skip impossible files
• reconcile versions and tombstones

A delete becomes:

• tombstone now
• cleanup later when compaction can prove it is safe

Everything else is engineering around that core.

Once you see it that way, a lot of practical decisions get clearer.

Why LSM trees are excellent

They are excellent when you need to absorb many small writes cheaply and continuously.

They are excellent when batching, background merge work, and sequential file creation match your hardware and your workload.

They are excellent when the database can afford a more complicated read path in exchange for much better ingest behaviour.

They are often excellent for:

• time-series data
• logs and events
• counters and metadata
• high-churn key-value traffic
• durable caches
• systems where writes dominate and point reads are common

Why LSM trees are awkward

They are awkward when you need the simplest possible point-read latency under all conditions.

They are awkward when long scans dominate and background rewrite work competes with them.

They are awkward when delete-heavy behaviour, long snapshots, or replication safety keep history pinned for too long.

They are awkward when the team operating them treats compaction as a detail instead of a first-class capacity consumer.

They are often a poor fit for workloads that want:

• minimal write amplification on durable storage
• consistently tiny read fan-out
• heavy ad hoc analytical scans over mutable keyspaces
• immediate physical reclamation after deletes

The design is not one product feature

Finally, remember that "LSM tree" names a family, not a single implementation contract.

RocksDB, Pebble, Cassandra, ScyllaDB, HBase, and Bigtable-inspired systems all share the same broad shape, but they differ in:

• compaction heuristics
• file formats
• Bloom-filter strategies
• cache architecture
• transaction support
• snapshot handling
• replication model
• memory management
• rate limiting and stall policies

So the useful question is not "does it use an LSM tree?"

The useful question is: how does this engine implement the LSM tradeoff, and where will my workload pay for it?

That is what decides whether the design feels brilliant or painful in production.

The Mental Model To Keep

Keep this one.

An LSM tree is a machine that delays expensive random disk updates by collecting writes in memory, persisting them as immutable sorted files, and paying back the resulting read and space debt through continuous background merges.

The WAL makes the in-memory buffer safe enough to use.

The memtable makes small writes cheap.

The SSTable makes flushed data searchable without rewriting it in place.

Bloom filters and per-file indexes keep point reads from turning into blind scans.

Compaction stops the pile of immutable files from becoming unsearchable, while also deciding when older versions and tombstones can finally die.

And the whole design lives inside a three-way negotiation between read amplification, write amplification, and space amplification.

Once you internalise that, the usual production symptoms stop looking mysterious.

A write stall is future merge debt arriving in the present.

A slow point read is often overlap, version churn, or a cache miss through too many candidate files.

A delete that does not free space yet is waiting for a safe compaction boundary.

A storage bill or SSD wear problem is often the cost of rewriting bytes several times in exchange for fast ingest.

That is how LSM trees actually work. Not as a magic black box for "fast writes", but as a careful trade: cheap arrival now, organised cleanup later, and a read path smart enough to make the whole bargain usable.