How etcd Actually Works

Most explanations of etcd stop at two claims: it is a distributed key value store, and Kubernetes depends on it. Both are true. Neither tells you why etcd behaves the way it does under load, during elections, or when a control plane starts falling behind.

etcd is not interesting because it stores keys and values. Redis stores keys and values. A local SQLite file stores keys and values if you squint a bit. etcd is interesting because every successful write becomes part of one strictly ordered history replicated across a small cluster, exposed through monotonically increasing revisions, and turned into watches, leases, elections, and transactional compare-and-swap operations that higher-level systems can build real control planes on top of.

That design gives etcd a very specific personality. It is excellent at saying which update happened first, whether a value is durably committed, and how far a follower has progressed through the same history. It is much less good at being large, cheap, or forgiving. If you treat it like a generic data store, it eventually teaches you about quorum loss, space alarms, compaction windows, and why a single replicated log is not a free substitute for a sharded database.

This article walks through etcd from the inside. We will look at why the cluster is really a revision machine backed by Raft, what a write does before the client gets a response, how linearizable reads differ from serializable reads and watch streams, how MVCC revisions sit on top of the bbolt backend, why leases and locks need fencing discipline, how snapshots and compaction keep the system alive, and which operational mistakes turn a healthy control plane into a long day.

etcd Is A Replicated Revision Machine, Not Just A Key Value Store

The easiest way to misunderstand etcd is to picture a map of keys that happens to be copied onto three servers.

That picture misses the main invariant. etcd does not primarily care that three nodes contain similar bytes. It cares that the cluster agrees on one ordered sequence of committed mutations. Every mutating operation that succeeds gets a single increasing revision. That revision becomes the logical clock for the whole keyspace.

The official API guarantees say this plainly: KV operations are durable and strictly serializable, and each modifying operation is assigned one increasing revision. If a transaction touches multiple keys, those keys share the same revision because they became visible as one atomic state transition.

That is what makes etcd useful for control planes. A scheduler, controller, or leader election service usually does not need ad hoc joins or terabytes of storage. It needs a trustworthy answer to questions like these:

• did this update happen before or after that one?
• if I resume watching from revision 481922, will I miss anything still inside the history window?
• when I compare a key's current revision to an expected value, am I guarding against a stale writer?
• if the client got a success response, was the result only in RAM or is it committed history?

Even a tiny etcdctl session shows the revision-first model:

etcdctl put /clusters/berlin-api/leader node-2
etcdctl get /clusters/berlin-api/leader -w=json

A trimmed JSON response looks roughly like this:

{
  "kvs": [
    {
      "key": "L2NsdXN0ZXJzL2Jlcmxpbi1hcGkvbGVhZGVy",
      "value": "bm9kZS0y",
      "create_revision": 812,
      "mod_revision": 481922,
      "version": 7,
      "lease": 90114
    }
  ]
}

The value matters, but the metadata matters more than many teams realise.

• create_revision tells you when this key first came into existence.
• mod_revision tells you when its current value was last written.
• version counts how many times the key was updated.
• lease ties liveness or expiry to a lease, if there is one.

That metadata is how higher layers reason safely.

A Kubernetes controller, for example, does not need to guess whether an object changed after it was read. It can compare resourceVersion semantics at the API layer, which ultimately rest on etcd's revisioned history. A custom coordination service can refuse to overwrite a key unless its mod_revision still matches the value seen earlier. A broken watch can resume from a known revision boundary rather than starting from scratch and hoping nothing happened during the gap.

This is why etcd fits metadata and coordination work so well. The cluster is one small, strongly ordered world. Everyone who writes to it has to go through the same history.

One Leader Owns The Write Order Because Raft Is The Spine Of The System

etcd gets that ordered history from Raft.

A production etcd cluster is usually three or five members. Only one is the Raft leader for the current term. Clients can talk to several endpoints, but writes are ultimately funnelled through the leader because the leader is the node allowed to propose new log entries for the replication group.

That is not a convenience choice. It is how the cluster avoids split history.

Imagine three members in Amsterdam, Frankfurt, and Warsaw. If Amsterdam and Frankfurt were both allowed to accept authoritative writes at the same time without coordination, the cluster would need another mechanism to merge conflicting histories. etcd does not try to solve that problem. It solves a narrower and more valuable one: maintain one authoritative sequence of log entries for one small metadata store.

Each Raft log entry carries at least two pieces of identity that matter here:

• a term, which identifies the leadership epoch
• an index, which identifies the entry's position in the log

A simplified entry might look like this:

term=91 index=1842203
command=Txn(compare=/leases/alpha mod_revision==481910, put=/leases/alpha value=node-7)

The rules are strict.

1. The leader appends a proposal to its log.
2. Followers receive AppendEntries messages containing the new log tail.
3. Followers accept the entry only if their previous log matches where the leader says this new entry belongs.
4. Once a quorum has persisted the entry, the leader can mark it committed.
5. Committed entries are applied to the state machine in log order.

That quorum point is the whole safety contract.

In a three member cluster, quorum means two. In a five member cluster, quorum means three. The cluster can lose one node out of three or two nodes out of five and still keep operating, but only if the surviving members still contain a majority of the replication group.

This has several direct consequences.

First, write latency is tied to quorum latency, not just local disk latency. A leader on fast local storage is still waiting for enough followers to persist the same entry.

Second, a minority partition does not get to keep accepting writes. That is the availability sacrifice etcd deliberately makes in order to avoid split brain. If the Berlin side of a partition has one node and the Prague side has two, the lone Berlin node may still have a full data directory, but it is not allowed to invent new committed history.

Third, member count is not the same thing as geographical wisdom. Stretching a five member cluster across too many far-apart sites can make every quorum round trip slow. etcd is usually happiest when the cluster stays in one region or one metropolitan latency envelope and the application treats cross-region disaster recovery as a separate problem.

Raft also explains why etcd is intentionally not horizontally sharded. There is one replication group for the whole keyspace. That means every write is globally ordered without an extra cross-shard ordering protocol. It also means the total system size is bounded by what one strongly consistent replication group can realistically handle.

That trade-off is not a weakness accidentally left in the design. It is the point.

What A Successful Write Actually Does Before The Client Sees OK

A completed etcd write is more expensive than PUT key value makes it look.

The API guarantee document defines an operation as complete when it has been committed through consensus and permanently stored by the storage engine. That definition is why etcd responses are meaningful. The server is not saying "I queued your idea". It is saying "this mutation is committed cluster history and durable state".

Walk a normal write path step by step.

1. The leader receives the request and checks the transaction conditions

Suppose a controller issues this transaction:

etcdctl txn <<'EOF'
cmp("/leaders/search", "Mod", "=", "481910")
put /leaders/search node-7
EOF

The compare clause matters. etcd is not a dumb blind write sink. Transaction compares let the client say "only perform this write if the key still has the version, creation revision, mod revision, or value I expect".

That turns race conditions into explicit success or failure instead of hidden corruption.

2. The leader creates a new Raft proposal

If the transaction is admissible, the leader packages it into a new Raft log entry. At this moment the change is proposed, not committed. It has a log position, but it is not yet part of durable cluster history.

3. The leader persists the entry to its own write-ahead log

Before the leader can safely talk about replication, it has to make the new entry durable locally. etcd maintains a WAL for Raft state. If the leader crashes before its own log write is durable, the proposal may vanish with the process.

This is one reason local storage quality matters so much. Slow fsync calls do not merely make one node sad. They slow the point where every control plane write can advance.

4. The leader replicates the new log entry to followers

The leader sends the entry to the followers using AppendEntries. Each follower checks whether the leader's view of the previous log tail matches its own. If not, the follower rejects that append, the leader backs up, and log repair begins from an earlier matching point.

This is how a recovering follower can discard divergent uncommitted entries from an old term and converge back onto the leader's authoritative history.

5. A quorum acknowledges durable receipt

Once enough members have persisted the entry, the leader can advance the commit index. In a three member cluster, leader plus one follower is enough. In a five member cluster, leader plus two followers is enough.

A useful mental timeline is:

client -> leader: Put /registry/pods/... 
leader: append entry locally
leader: fsync WAL
leader -> followers: AppendEntries(term=91, index=1842203)
follower A: fsync WAL, ack
follower B: fsync WAL, ack
leader: quorum reached, commit index = 1842203

Only now does the cluster know this entry belongs to durable consensus history.

6. The committed entry is applied to the MVCC state machine

Raft commitment alone is not enough for client-visible state. etcd still has to apply the committed operation to its MVCC keyspace and backend storage. That is when the global revision is assigned to the mutation and the key metadata becomes visible through normal reads and watches.

For a transaction that modifies several keys, all of those key updates share one revision because they were one state transition.

7. The response goes back to the client

A successful response can carry the new revision in the response header. That revision is not cosmetic. It tells the caller exactly where the mutation landed in global history.

A trimmed response shape looks like this:

{
  "header": {
    "revision": 481922,
    "raft_term": 91
  }
}

That revision can immediately feed the next control-plane step.

• A watch can resume from 481923.
• A client can treat 481922 as a fencing point.
• A reader can demand a linearizable read that is at least as fresh as that committed state.

This full path is why etcd is not a cheap data sink. Every write pays for local persistence, quorum replication, ordered apply, and durable metadata bookkeeping. If you keep payloads small and write rates sane, that price is worth paying. If you try to pour bulky, high-churn application data through the same path, you are making the most expensive part of your control plane compete with itself.

Reads Come In Different Consistency Modes, And They Are Not Interchangeable

A lot of etcd confusion starts when people say "read" as if there were one read path.

There are at least three distinct ways applications observe state:

Linearizable reads

By default, etcd promises linearizable reads for normal KV access. The practical meaning is simple: if one client completed a write, and another client performs a later linearizable read, that read will not return older state.

The important nuance is how etcd gets that guarantee.

A linearizable read is not "read a local map and hope the leader is still leader". The server has to ensure the read is consistent with the current committed prefix of the Raft log. In modern etcd this usually goes through the Raft read path rather than appending a dummy write entry for every read, but the read still depends on live consensus knowledge.

That means a linearizable read pays a real price. If the leader cannot confirm its authority through the quorum path, it should not bluff.

Serializable reads

A serializable read can be served from a member's local state without the same live quorum coordination cost. That makes it faster and cheaper. It also means the answer may be stale relative to the latest committed cluster head.

This is fine in some situations.

• a metrics scraper reading non-critical metadata
• a UI page where a small amount of staleness is acceptable
• a local cache warmer that only needs approximate recency

It is not fine when the read decides ownership, safety, or the next write precondition.

If one service instance in Dublin decides whether it owns a lease by doing a stale local read, and another instance in Vienna already took over, you have just saved a millisecond and bought a correctness bug.

Watch streams

Watches are their own thing.

The API guarantees document is explicit about their properties. Watch events are ordered by revision, unique, reliable within the available history window, atomic at revision boundaries, resumable from a known point, and bookmarkable through progress notifications. What watches do not guarantee is linearizability in the same sense as a fresh quorum read.

That difference matters. A watch client must still reason in terms of revisions.

If you need a common control-plane pattern, it looks like this:

1. perform a list or range request
2. note the returned revision
3. start a watch from the next revision
4. process ordered updates from there

That is the standard list-then-watch handoff because it creates one continuous history boundary. You are not saying "I hope nothing changed while I was subscribing". You are saying "I know exactly where my snapshot ended, so now give me every event after that revision".

This is one of the reasons etcd works so well as the event backbone for controller loops. A controller does not need to poll every object repeatedly. It can take one consistent baseline and then follow the ordered change stream.

MVCC Revisions Sit Above bbolt, And That Split Explains Compaction And Defrag

etcd's public model is a revisioned MVCC keyspace. Its persistent local backend is bbolt, a B+ tree based embedded store. Those are different layers, and a lot of operational behaviour only makes sense if you keep them separate in your head.

The MVCC layer

At the API level, etcd keeps multiple versions of keys over time.

If /services/payments/leader changed at revisions 481910, 481922, and 481930, etcd can reason about those versions as a history, not just one latest value. That is what makes historical reads and resumable watches possible inside the retention window.

This is also why compaction exists. If old revisions stayed available forever, a busy cluster would accumulate an ever-growing historical index for the same hot keys. That hurts storage, memory pressure, and watch catch-up behaviour.

The official maintenance guide describes history compaction very directly: compaction drops keyspace history before a chosen revision so the system does not degrade or eventually exhaust space.

A manual example from the docs is:

etcdctl compact 3

After that, a request for an older revision can fail like this:

Error: rpc error: code = 11 desc = etcdserver: mvcc: required revision has been compacted

That error is not a nuisance detail. It is the system telling you the requested history no longer exists in the accessible MVCC window.

The bbolt backend

Underneath the MVCC model, etcd stores its backend data in bbolt on each member. bbolt is local, embedded, and page based. That makes it a good fit for a small strongly consistent metadata store because it is simple, battle tested, and gives etcd a durable backend without an external dependency.

But bbolt page reuse leads to a practical distinction between logical history removal and filesystem space reclamation.

Compaction removes old MVCC history from accessible use. It does not necessarily shrink the backend file on disk. The maintenance guide calls out this internal fragmentation explicitly: after compaction, free space can exist inside the backend database while the host filesystem still sees a large file.

That is why etcd also has defragmentation.

etcdctl defrag --cluster

Defrag rebuilds backend state so the freed internal space can be returned to the filesystem. The same guide also warns that defragmenting a live member blocks that member's reads and writes while it rebuilds state. This is not a background cosmetic cleanup. It is an operational event that needs planning.

Space quotas and the NOSPACE alarm

etcd includes a backend quota because a full metadata store is worse than an inconvenient one. If the backend exceeds the configured quota, the cluster raises a NOSPACE alarm and enters a limited mode where writes are rejected.

A minimal example is:

etcd --quota-backend-bytes=$((16*1024*1024))

and when the backend fills, the system can start returning errors such as:

rpc error: code = 8 desc = etcdserver: mvcc: database space exceeded

The operational lesson is simple. In etcd, deleting application objects is not enough. You also need a retention strategy, compaction policy, and defrag plan. Otherwise the cluster can remain physically bloated even after logical cleanup.

This MVCC-plus-backend split is also why etcd is comfortable with small, frequently changing metadata and uncomfortable with large blobs. You are not only storing current bytes. You are maintaining ordered historical versions, watch replayability, and a compactable backend for the whole replication group.

Watches Are The Real Distribution Engine For Control Planes

If the KV API is how state is written, the watch API is how that state becomes a living control plane.

A watch lets a client subscribe to changes on one key or a prefix and receive ordered events as revisions advance. That turns the store into an event source for controllers, schedulers, sidecars, admission systems, and any other component that needs to react to state change rather than rediscover it by polling.

A realistic control flow looks like this:

1. list /registry/pods/ with revision 481922
2. build local cache from that snapshot
3. watch /registry/pods/ starting at revision 481923
4. apply ADD, UPDATE, DELETE events in order
5. reconnect from the last processed revision if the stream breaks

That pattern is attractive because it is efficient. The expensive part of consistency is paid once by the write path. Readers then consume the ordered result stream.

The guarantee set matters here.

• Ordered means events arrive in revision order.
• Unique means an event is not delivered twice on the same watch stream.
• Reliable within history means a subsequence is not silently dropped while the relevant revisions still exist.
• Atomic means all updates from one revision arrive together.
• Resumable means a broken client can reconnect starting after its last seen revision, as long as that revision has not been compacted away.
• Bookmarkable means progress notifications can tell the client it has definitely seen all events up to a revision.

This is enough to build serious controller caches, but only if the client behaves properly.

Slow consumers are a real failure mode

Watches are not magical infinite hoses. They are downstream consumers attached to a moving history window. If a watcher falls too far behind and the cluster compacts the revisions it still needs, the watch cannot continue from that point.

That usually means the client must relist, rebuild state, and start a new watch from a fresh revision.

In other words, compaction policy and watch lag are linked. Keep too little history and slow controllers or disconnected clients will constantly fall off the back of the stream. Keep too much history and the backend swells.

Watches do not make stale application logic safe

A watch tells you what changed in order. It does not automatically make your business logic race free.

If two controllers observe the same event and both decide to update an external system, you still need compare conditions, ownership checks, or fencing at the write boundary. Watches distribute facts. They do not remove concurrency from the world.

Watches are why etcd can stay small but still drive big systems

A control plane with ten thousand objects does not need ten thousand clients polling full snapshots every second. It needs a strongly ordered source of truth and a cheap way to follow the delta stream after a known revision. etcd gives you exactly that.

That is probably the most important mental model on the read side. etcd is not only a place where values live. It is a place where change history becomes a stream other components can build consistent caches from.

Leases, Locks, And Elections Work Because Revisions Back Them Up

Leases are one of etcd's most useful features and one of its easiest to oversimplify.

A lease is a server-side object with a TTL. Keys can be attached to the lease. The client keeps the lease alive with periodic keepalive messages, and if the lease expires, etcd removes the attached keys.

That makes leases useful for liveness driven metadata.

• service presence records
• ephemeral leader markers
• session-like ownership keys
• lock records that should disappear when the holder dies

A simple sequence is:

etcdctl lease grant 15
etcdctl put /services/riga-api/instances/node-4 ready --lease=90114
etcdctl lease keep-alive 90114

If the process or network path disappears long enough for the lease to expire, etcd can clean up the key automatically.

That sounds like a perfect distributed lock already. It is not.

The etcd documentation is careful about this point. Lease expiry is based on physical time, and client and server clocks are not the same thing. A client may believe it still owns the lease while the cluster has already revoked it. For coordination over external resources, a bare lease is not enough to guarantee mutual exclusion.

This is where revisions and compare operations come back in.

A correct lock needs a fencing story

Suppose worker A held a lease-backed key representing ownership of a queue shard. A GC pause, network stall, or scheduler freeze stops keepalives long enough for the lease to expire. Worker B then acquires the lock and starts processing.

If worker A wakes up and still believes "my lease key used to exist, so I am the owner", you have two writers.

The fix is not mystical. The fix is to make every critical write validate a fencing token or current revision.

In etcd terms, that often means one of these patterns:

• compare the current key revision before updating shared state
• include the ownership revision as a fencing token in downstream writes
• reject any external action whose token is older than the latest known owner

A safe ownership handoff is therefore not "I once got a lease". It is "my operation was accepted under the current revision and any stale owner will now fail its compare".

That is why etcd locks are really built from a combination of leases, revisions, and transactional compare logic.

Elections are the same idea with nicer semantics

An election service built on etcd does not need to invent a new consensus protocol. It uses etcd's ordered history to decide who registered first under the rules of the election namespace and then uses watches plus leases to detect turnover.

Again, the winning property is not just that a value exists. It is that the cluster can prove an ordering boundary for who acquired which revision, and can clean up or supersede that ownership when the lease holder disappears.

This is the theme that keeps repeating through etcd. High-level coordination features are not add-ons hanging from a dumb map. They work because every mutation already lives in one ordered, revisioned, quorum-committed history.

Snapshots, Log Retention, And Recovery Keep Followers And Operators Out Of Trouble

A healthy etcd cluster cannot keep every old Raft entry in memory forever, and it cannot depend on all followers staying perfectly current all the time.

That is why snapshots exist at more than one layer.

Raft snapshots for follower catch-up

The maintenance guide documents --snapshot-count, which controls how many applied Raft entries etcd keeps before persisting a snapshot and truncating older log entries. The default in modern releases is much higher than early etcd versions because the trade-off is real.

• keep too few entries and slow followers need snapshots frequently
• keep too many entries and memory pressure and Go GC overhead rise

When a lagging follower asks for log entries older than the leader still retains, the leader sends a snapshot instead of replaying the missing prefix entry by entry. The follower then overwrites its state from that snapshot and resumes from the newer log tail.

That mechanism is what lets a follower recover after being slow, restarted, or briefly partitioned without forcing the entire cluster to preserve ancient log segments forever.

Backend snapshots for backup and disaster recovery

The operational backup path is different from internal Raft snapshotting. The maintenance guide recommends regular snapshot backups of the member state, for example:

etcdctl snapshot save backup.db
etcdutl --write-out=table snapshot status backup.db

The status output includes a hash, revision, key count, and size so the operator can verify what was captured.

That snapshot is the thing you want for deliberate backup and restore workflows. It gives you a known durable point-in-time state.

Compaction is not backup

People sometimes collapse compaction, defrag, and snapshot into one fuzzy maintenance blob. They solve different problems.

• Compaction removes old MVCC history from the accessible logical window.
• Defrag returns fragmented backend space to the filesystem on a member.
• Snapshot backup preserves a durable point in time for restore.
• Raft snapshotting helps the replication layer truncate old applied log history and catch up slow followers efficiently.

If you keep those separate, etcd maintenance is understandable. If you blur them together, on-call reasoning gets messy fast.

Member replacement is about restoring quorum truth, not cloning files casually

A failed member is not normally recovered by casually copying another node's live data directory in the middle of operations. The cluster state includes member identity, log position, and snapshot relationships that need to remain coherent. Proper recovery uses the documented snapshot and membership flows so the replacement node rejoins the existing truth rather than inventing a confusing near-copy.

The broader point is that etcd recovery is still consensus recovery. A member is not authoritative just because it has a lot of bytes on disk. It is authoritative if it belongs to the surviving quorum history.

The Clusters That Break etcd Usually Break It For Predictable Reasons

etcd failures often look dramatic, but the underlying causes are usually boring and mechanistic.

Quorum loss

This is the most important one.

If a three member cluster loses two members, or a five member cluster loses three, there is no majority. No amount of optimism, retries, or application pressure changes that. The cluster cannot safely commit new writes because it no longer has enough voters to prove a majority history.

That is not etcd being fragile. That is etcd keeping its main promise.

Slow or bursty storage

Every committed write depends on WAL durability and follower acknowledgement. If the leader's fsync latency spikes, or one follower is so slow that quorum effectively depends on one remaining healthy follower, control-plane latency rises immediately.

Teams often watch CPU first and miss the real bottleneck. etcd can look lightly loaded while storage latency quietly stretches every control-plane transaction.

Oversized values and write-heavy prefixes

etcd is designed for metadata. Large objects, giant lists, and extremely high churn under the same prefix multiply pressure on the replicated write path, backend compaction, watch fan-out, and snapshot size.

You can store a lot of things in etcd that you should not.

• large binary blobs
• logs
• chatty per-request state
• rapidly rewritten cache entries
• anything better suited to an object store, queue, or sharded database

The cost of misuse is paid in the most sensitive part of the system.

Watch lag plus aggressive compaction

If consumers are slow and retention is short, watches fall behind and hit compacted revisions. Then clients relist, rebuild, and create more load exactly when the cluster is already stressed.

This is a classic control-plane feedback loop. The fix is usually a combination of better client behaviour, saner compaction windows, and reducing unnecessary event churn.

Space quota alarms without a maintenance habit

The NOSPACE alarm does not appear out of nowhere. It is usually the result of a cluster that kept too much history, skipped compaction or defrag, or stored too much data for etcd's intended role.

Once the quota alarm trips, the write side of the control plane is already in a bad place. Operators then have to compact, defrag, and clear alarms under pressure instead of doing those things routinely.

WAN-spanning design that ignores quorum latency

A three member etcd cluster across Lisbon, Frankfurt, and Bucharest looks geographically sophisticated on a slide. It also bakes long round trips into every committed write.

The common pattern in real systems is to keep the etcd quorum local enough for healthy latency and handle region-level recovery above or around it. etcd is about one good control plane, not globally scattered optimism.

A compact troubleshooting table captures the pattern:

None of these are mysterious if you keep the underlying machinery in view.

etcd Works Brilliantly For Control Planes Because It Refuses To Be A General Database

The most useful thing about etcd is also the reason people misuse it.

Because etcd feels simple, teams are tempted to treat it as a general shared persistence layer. It has keys, values, transactions, watches, authentication, and a network API. Why not store everything there?

Because every feature is shaped around one small consensus group maintaining one globally ordered metadata history.

That gives you excellent properties for control-plane work:

• strict serializability for KV operations
• revision numbers that act as a logical clock
• efficient list-then-watch cache construction
• leases for liveness-driven records
• transactional compare-and-swap for ownership and coordination
• one obvious answer to "which write happened first?"

It also imposes hard limits:

• all writes contend on one replicated history
• quorum and storage latency matter to every commit
• unbounded history retention is impossible
• payload size and event fan-out are not free
• there is no sharding layer waiting to save you later

This is why etcd is such a good substrate for systems like Kubernetes and such a poor place to dump ordinary application data. Kubernetes mostly stores small, structured, high-value metadata: pods, nodes, leases, config, endpoints, secrets, and controller intent. It needs exact ordering, durable commits, and watchable change streams more than it needs massive dataset scale.

That is etcd's home ground.

If your workload is mostly bulk records, ad hoc queries, large documents, or write throughput that should scale horizontally by partitioning, choose a system built for that shape instead.

The Short Version

etcd works because it reduces distributed state to one disciplined idea: all meaningful mutations must enter one quorum-committed history, and the rest of the system should consume that history through revisions, reads, watches, and compare conditions rather than guessing.

The Raft layer decides order and commitment. The MVCC layer turns commitment into revisions and historical visibility. The watch layer distributes ordered changes. Leases and transactions turn those revisions into coordination primitives. Compaction, snapshots, and defrag stop the store from collapsing under its own history.

Once you see etcd that way, a lot of operational behaviour stops feeling surprising. Slow disks hurt because every write is a quorum durability event. Stale reads happen because not every observation path is linearizable. Locks need fencing because leases alone do not defeat stale owners. Space alarms happen because old history and fragmented backend pages are real physical cost, not bookkeeping fiction.

That is the actual machine Kubernetes and many other control planes lean on. Not a magical config database. A small, strict, revisioned consensus system that is very good at ordering metadata and very unforgiving when asked to be something else.