Overview

A PicoMQ deployment has three components. Nodes serve HTTP and hold only caches. An object store holds every record. A SQL database holds the metadata log that coordinates the nodes.

Anatomy of a node

Each node runs the same stack. A protocol listener speaks either the Pico protocol or Durable Streams. An admin listener serves the admin API and the dashboard. Behind them, an ownership router decides whether this node serves a stream or redirects, the stream service manages the registry of names and per-stream state, and the s3stream engine moves records to and from object storage.

The engine is the only writer of record data. It appends to a write-ahead log on object storage for durability, batches records into larger data objects in the background, and serves reads from caches when it can.

The metadata log

All cluster state changes are commands: register a node, create a stream, open it, commit an object, transfer ownership. A node proposes a command to the SQL database, where an ordered log table assigns it a position. Every node tails that table and applies each command to an in-memory state, so all nodes converge on the same view without talking to each other.

The database provides the ordering, which is the property a consensus protocol would otherwise supply. Applying a command is deterministic, so replaying the log always produces the same state. Every 1024 applied commands a node writes a snapshot row and the log below it can be truncated, which keeps replay on startup short. The position of the last applied command is the applied index seen in the admin API.

The applied state answers every routing and placement question: which node owns a stream, which objects exist, which transfers are in flight. Nodes read it from an immutable in-memory view, never from SQL directly, so reads cost nothing and a slow database only delays new commands.

Request routing

Stream names look like URL paths and any node accepts any request. The receiving node checks the view. If the stream is open on another node the client gets a 307 redirect to its advertised address. If it is unowned or local the node serves it directly. Epoch fencing backs this up: a node that lost its registration cannot commit anything, so a stale redirect can waste a hop but never split a stream.

Background work

A few loops run outside the request path. Every node tails the metadata log and watches for stream transfers involving it. One node at a time holds a SQL lease and runs maintenance: expiring abandoned object uploads and deleting destroyed objects from storage. Any node crossing a snapshot interval writes the next snapshot. None of these tasks are special roles, whichever node holds the lease does the work and losing it just moves the work elsewhere.