Writes

An append is durable when the record is in object storage. There is no local disk in the write path, so a node that dies loses nothing that was acknowledged. The cost of that guarantee is object store latency, and the write path is shaped around paying it once for many records instead of once per record.

Note

A local disk backend for the WAL may be considered in the future for low latency use cases. That would only change where the WAL is written, the rest of the write path stays the same.

The append path

An append arrives at the stream's owning node as a POST. The node checks the producer's epoch and sequence against the registry entry, assigns the next offsets, and submits the record to the storage engine. Submission happens under a per-stream lock so the order of offsets is fixed, but the caller waits for durability outside the lock. Requests pipelined on the same stream stack up in the WAL and share the same flush.

Inside the engine the record goes to two places. It enters the log cache, which serves tail reads, and it enters the WAL, which makes it durable.

The WAL

The WAL is a sequence of small objects in the object store, written by one node under its own key prefix. Incoming records accumulate into a bulk, and each bulk becomes one PUT. Uploads are pipelined, so several bulks can be in flight, but acknowledgements are delivered in submission order. A record is acknowledged only when its bulk and every bulk before it are stored.

Group commit is what makes this affordable. One upload of a few hundred kilobytes contains every record that arrived while the previous upload was in flight, so per-record cost drops as concurrency rises. A single append on an idle stream pays one object store round trip.

Records are framed with a checksum and the WAL objects include the node's epoch. A node that lost its registration cannot extend its WAL past a takeover, which is part of the fencing described in Streams.

From WAL to committed objects

WAL objects are a staging area, not the long-term layout. A background upload task drains sealed log cache blocks into read-optimized objects and commits them through the metadata log.

A block holding records from many streams becomes a stream-set object, with large per-stream runs split out into their own stream objects. Object ids are reserved through the metadata log before the upload, so a crashed upload leaves only an unreferenced id that expires. The commit itself is a single command that registers the objects, advances the end offset of every stream involved, and does so atomically. Readers either see none of the commit or all of it.

Once the commit is applied, the WAL objects it covers hold no unique data and are deleted. The WAL stays short, which bounds recovery time.

Recovery

When a stream is opened after a crash, the engine lists the WAL objects under the previous session's prefix and replays the records past the last committed offset. Acknowledged records are recovered because acknowledgement required the WAL upload to complete. Records that were in flight but never acknowledged may be absent, which is the standard contract: an append without an acknowledgement was never promised.