Skip to content

Write-ahead log ​

An append is acknowledged when its record is in the WAL. The WAL is the durable part of the write path, a sequence of small objects that each node writes to the object store under its own key prefix. Caches, committed objects, and compaction sit behind it.

Contract ​

The engine drives the WAL through four operations.

OperationWhenContract
AppendEvery writeStore a framed batch, acknowledge in submission order
ReadRecoveryReturn records from an offset forward
TrimAfter a commit to data objectsRelease everything below the committed offset
ResetAfter recovery is uploadedStart a new session from an empty log

A batch is framed with a header carrying its offset, its length, and a CRC32 over the body, then the records.

Layout ​

Unit of writeOne object per bulk, pipelined, acknowledged in order, see Writes
Key{prefix}{epoch}/wal/{start}-{end}, under the node's prefix and the epoch it writes at
Reservationreservation/_kafka_{cluster}/{node}, a small object naming the node and epoch that may write
TrimThe covered objects are deleted
Append latencyTens of ms on S3, single digits on S3 Express
framed batchheader, crc, recordsobject storePUT{prefix}{epoch}/wal/{start}-{end}reservationread back before each ackacknowledged in order, deleted on trim

Fencing ​

  • Starting the WAL writes the reservation for the node's epoch.
  • After each completed bulk, and before acknowledging it, the writer reads the reservation back. If it names anyone else, or cannot be read, the writer is fenced and every later append fails.
  • A takeover rewrites the reservation for the dead node before reading anything, so the dead writer can never acknowledge again.

Commit path ​

ParameterDefaultEffect
batchInterval250 msHow long records accumulate before a bulk is written
maxBytesInBatch8 MiBSize that closes a bulk early
maxUnflushedBytes1 GiBBytes acknowledged but not yet uploaded to data objects before appends wait
maxInflightUploadCount50Bulks in flight at once

Parameters go on the --wal URI, see Configuration and Tuning.

Takeover ​

Each node registers where its WAL lives. When a node is fenced, the lease holder opens that WAL as the dead node, in recovery mode, see Failover. The takeover claim fences the dead node's writer before anything is read.

The dead node's WAL is inThe lease holder can take it over if
An object store bucketIt can reach the bucket and sees the dead node's reservation at that epoch. The bucket may differ from its own
file://It runs on the same host, which makes file:// a single host setup

A takeover that is refused is retried on each failover tick, and the dead node's open streams stay where they are until it succeeds or the node restarts.

Recovery ​

A node recovers its own WAL when it starts, and the lease holder recovers a fenced node's WAL by taking it over. Recovery lists the node's WAL objects past the last committed offset, checks the CRC of every record, uploads the records to data objects, and closes the streams they belong to. Acknowledged records are recovered because acknowledgement required the write to complete. Records in flight but never acknowledged may be absent.

Durability ​

EventOutcome
Node crash or restartNothing lost
Node host lostNothing lost, the WAL is in the bucket
Zone lostDepends on the bucket's storage class