Garbage collection

Records land in object storage and stay there until something says otherwise, so every path that makes data unreachable needs a matching path that reclaims the space. PicoMQ separates the two: making data unreachable is a metadata change, reclaiming space is a background job. Nothing in the request path ever waits on a delete.

Where garbage comes from

Trimming a stream advances its start offset. Objects that fall entirely below the new start hold nothing readable and are marked for destruction. An object that straddles the boundary is kept, since it still holds live records past the trim point.

Deleting a stream marks all of its objects. Compaction rewrites small objects into larger ones and marks the originals. An object id that was reserved for an upload that never committed expires after its TTL and is marked as well. WAL objects are the one exception: they are deleted directly by their owning node once a commit covers them, as described in Writes.

Two-phase deletion

Marking and deleting are separate steps with the metadata log between them.

The mark is part of whatever command created the garbage. Applying a trim, a delete, or a compaction commit atomically removes the objects from the live indexes and appends them to a destruction queue in the replicated state. From that moment no reader can reach them, but the bytes still exist.

The clean runs on the lease holder, described in Leases. Each pass peeks a batch from the queue, deletes those objects from storage, and proposes a command that removes them from the queue and the catalog. Objects whose mark says their data is shared by a newer composite object are dropped from the catalog without touching storage.

The ordering matters. Bytes are deleted from storage before the catalog entry goes away, so a crash between the two steps leaves an entry pointing at a deleted object, and the next pass simply cleans it again. The reverse order could leak bytes forever. Every step is idempotent, so retries and duplicate passes are harmless.

Compaction

Left alone, a busy node produces a steady stream of small objects, and reading history through thousands of them costs a request each. Compaction runs on each node for the streams it owns, merging small committed objects into fewer, larger, stream-major ones. The rewrite is committed through the metadata log like any other object change: new objects in, old objects marked, one atomic step.

Compaction also completes what trim starts. The straddling object a trim left behind is rewritten without its dead prefix on the next compaction round, which is when those bytes are actually reclaimed.