Introduction

PicoMQ is a durable stream server. Clients create named streams, append records, and read them back over HTTP or the Kafka protocol. Records are stored on S3-compatible object storage and cluster coordination goes through a SQL database. A node is a single binary with no local state worth backing up.

Vision

PicoMQ treats a stream as a small, disposable unit. Streams are named like URL paths, created with one request, and cost nothing while idle. A deployment can hold ten streams or millions, one per order, per session, per device, or per job.

Object storage is what makes that granularity economical. Every record goes there, including the write-ahead log, so durability never depends on a node and an idle stream is a registry entry plus its objects. Coordination goes through a SQL database. There is no consensus protocol and no broker disks.

The structure follows from that. A node can be stopped and replaced at any time because it holds no unique state. Adding capacity is starting another process. Losing a node causes a few seconds of rerouting, not a data rebalance.

Features

• Zero-disk nodes. Records are stored on S3-compatible storage, including the write-ahead log. A node keeps caches, nothing more.
• SQL as the control plane. Cluster metadata is an ordered command log in Postgres, or SQLite for a single node. Nodes tail it and rebuild the same state.
• Three wire protocols. The native Pico protocol, the Durable Streams open protocol, and the Kafka wire protocol for standard Kafka clients. Same engine underneath.
• HTTP or Kafka, your choice. Create with PUT, append with POST, read with GET, tail with long polling or SSE. Or point any Kafka producer or consumer at the node. Both see the same streams.
• Live stream transfer. Ownership of a stream moves between nodes without losing writes, with seconds of handoff.
• Fencing everywhere. Node epochs and stream epochs keep zombie processes from corrupting anything.
• One binary. pico is the server, the client, the admin CLI, and the benchmark tool. The admin dashboard is embedded in it.

Use cases

PicoMQ suits anything modeled as many ordered, resumable streams: a stream per user session or chat, per device, per workflow run, or per agent conversation. Readers resume from any position, so it also serves audit trails, per-entity event history, and real-time delivery to many concurrent readers.

It is not built for single-digit millisecond appends. Durability comes from object storage, so an append costs one round trip there, typically tens of milliseconds.