Why not Kafka?

PicoMQ serves the Kafka protocol. This page explains why Pico's own data model is not Kafka's. Log workloads fall into two types with different storage requirements.

Type	Description	Examples
Funneling	Many producers append to a small number of shared logs. Consumers read each log in full	Telemetry, clickstream ingestion, warehouse loads
Routing	Each entity has its own ordered log. Consumers read individual logs by name	Messaging, feeds, per-entity state, agent sessions

Both need a durable log. Kafka supports funneling only. PicoMQ supports both.

Why Kafka is an excellent funnel

Kafka was built in 2011 to move telemetry from LinkedIn's servers into Hadoop. Early messages carried no keys, and the original implementation discarded the partitioning key after computing a partition. The data model reflects that origin: a small number of wide topics, each split into partitions for parallelism, with consumers reading every partition in order.

Storage systems can be measured by read, write, and space amplification, and improving all three requires restricting the access pattern. Kafka restricts in favor of the funnel. Records land in append-only, immutable segment files that are deleted wholesale when they age out of retention, so write amplification is about 1x. The only read API is a sequential scan in write order, so a full scan also costs about 1x read amplification. For funneling this is close to optimal.

Why Kafka is a bad router

Routing inverts the access pattern. Consumers read a specific subset of the data, usually one entity: one user, one session, one workflow run. The underlying storage needs hundreds of thousands or millions of small ordered logs rather than one large one.

Partitions are too heavy to give each entity its own, so entities are interleaved into shared partitions. The partition key controls placement, not access, and there is no read path that returns one key's records. Reconstructing one entity requires scanning the partition and discarding the rest, so read amplification is the ratio of the partition size to the records requested:

             |partition|
  α_read = ─────────────
              |record|

This is close to the worst case, and the same layout causes problems beyond reads. Offsets are tracked per partition, not per entity, so one bad record blocks every entity behind it. Changing partition counts moves data and disrupts every consumer. A hot key produces a hot partition that cannot be split in place.

The industry's answers

Most Kafka alternatives compete on the funnel: cheaper brokers, tiered storage, object-native backends. The primitive is unchanged.

Some newer systems address routing by changing the storage layout. Records are kept in an LSM tree keyed by (key, sequence), so one key's log is found by binary search instead of a partition scan, and millions of keyed logs fit on one node. This fixes read amplification, but the log remains an entry inside a larger structure. It cannot be listed, sized, placed, or deleted on its own, and continuous compaction is required to keep each key's records collocated.

Pico: the stream is the primitive

Pico makes the per-entity log a first class object. Every entity gets its own stream, named like a path and backed by its own log. Idle streams have no cost and a deployment holds millions of them, so unrelated entities never share a log and no compaction is needed to keep an entity's records together.

Producers write to a named stream, so there is no partitioner. Reading one entity means tailing one stream, which costs 1x read amplification. Reading the aggregate means listing a prefix and fanning in across its streams, scoped to one entity, one customer's subtree, or everything.

	Partitions (Kafka)	Streams (Pico)
Point reads	Finding one entity's records requires scanning the whole partition	The entity is the stream. It is addressed by name and only its records are read
Isolation	Offsets are per partition, so one bad record blocks every entity behind it	Offsets and failures are scoped to one stream
Rescaling	Changing partition counts moves data and disrupts all consumers. Hot partitions cannot be split in place	The stream is the unit of placement. A hot stream moves to another node on its own and offsets do not change
Lifecycle	Retention applies per topic. One entity's data cannot be trimmed or deleted independently	Retention, trimming, and deletion are per stream operations

Funneling works the same way on PicoMQ. A topic with N partitions is N streams under one prefix. Producers hash to a stream, consumers fan in across the prefix. Reads are 1x sequential and retention deletes whole segments, as in Kafka. Kafka clients connect through the Kafka protocol.