Metadata

Every change to cluster state is a command: register a node, create a stream, open it at a new epoch, commit an object, record a transfer, delete a key. Commands are small binary values with a versioned encoding. Nothing writes cluster state any other way, so the command set is the complete list of things that can happen to a cluster.

The log

Commands are stored in one SQL table, an ordered log of (idx, payload) rows. Appending is optimistic: a writer reads the last index it knows, inserts at the next one, and retries at a higher index if another writer took the slot. The database's uniqueness guarantee on idx is the only coordination primitive in the system.

Proposals from one process are group committed. A flusher task drains the propose queue, packs up to 256 commands into a single row, and appends them together. Under load this collapses many proposals into one insert per round trip.

A proposal returns only after the local tailer has applied its row. The view is published before the result is delivered, so a caller that gets an answer is guaranteed to see its own write in the next view load.

Applying commands

A tailer task on every node fetches rows past its applied index and applies each command to an in-memory state. Apply is a pure function: the same log always produces the same state on every node, with no clocks and no node-local input.

Apply is also where the rules are enforced. Each command validates against the current state and either mutates it or returns an error to the proposer. Duplicate work is answered with a distinct redundant result, which makes retries safe. Commands that include a node identity are checked against the node's registered epoch, so a restarted node at a higher epoch invalidates everything still in flight from its predecessor.

The state is built from persistent maps. Cloning it for a published view is cheap structural sharing, not a copy, so publishing a new view per applied row costs little even with large state.

Views

Readers never query SQL. Each node holds one immutable view, swapped atomically by the tailer, holding the state and the applied index. Anything answering requests loads the current view without locks. Code that needs to observe a future write waits for the applied index to reach a target instead of polling the database.

This is what makes a slow metadata database tolerable. Reads keep their latency regardless of the database, and only new commands wait.

Snapshots

The log would otherwise grow without bound, so every 1024 applied rows a node encodes the whole state into a single snapshot row and truncates the log below it. There is one snapshot row per cluster, not an archive. Any node may run the cycle, whichever crosses the interval first.

A starting node decodes the snapshot and replays the tail, so recovery time is bounded by the interval rather than the cluster's history. A node that lags so far behind that its next row was truncated detects the gap and reinstalls from the newer snapshot instead of continuing from a fork.

A log row that fails to decode is treated as unrecoverable. Skipping it would silently fork that node from every other reader, so the sink stops, fails all waiting proposals, and the node stops accepting metadata writes.