Skip to content

Kafka protocol ​

Every node serves the Kafka wire protocol on --kafka-listen, next to its HTTP listeners. Standard Kafka clients, the Java client, librdkafka and everything built on it, connect with a bootstrap address. There is no PicoMQ client for Kafka because none is needed.

bash
pico serve \
    --kafka-listen 0.0.0.0:9092 --kafka-advertise node1.internal:9092 \
    --auth required --auth-bootstrap-token-file /run/secrets/pico-root \
    --meta-url postgres://user:pass@pg:5432/picomq \
    --storage=-2@s3://bucket?region=us-east-1

Topics are shared across protocols. A topic written over HTTP is readable by Kafka clients, and a topic a Kafka client created is readable over HTTP.

bash
# Any Kafka tooling works against the listener.
kcat -b 127.0.0.1:9092 -t orders -P   # produce
kcat -b 127.0.0.1:9092 -t orders -C   # consume

# The same records over HTTP.
curl http://127.0.0.1:4437/orders

Topics ​

A Kafka topic is a PicoMQ topic, with the same partitions. The HTTP name /orders/eu is the Kafka topic orders.eu, and a topic whose name has characters Kafka does not allow is reachable only over HTTP, see Topics and streams. A name starting with _ is refused with INVALID_TOPIC_EXCEPTION, so tools that create _schemas or _confluent-* topics by default need another name.

  • Partitions. CreateTopics takes any partition count, and CreatePartitions adds more later. Explicit replica assignments place each partition on the named node.
  • Replication. Each partition has one replica, its owning node, and durability comes from object storage. The replication factor must be 1 or -1.
  • Leaders. Metadata names each partition's owner as its leader. A request sent to another node answers NOT_LEADER_OR_FOLLOWER, and the client refreshes its metadata.
  • Offsets. Kafka offsets are the partition's record offsets, so earliest is the trim watermark and latest is the high watermark.
  • Auto creation. A metadata request creates a missing topic with one partition when the client asks for it.

Produce and fetch are a copy of the stored RecordBatch v2 bytes.

  • Records. Keys, headers in their order, and null values for tombstones arrive as they were sent. HTTP reads show them in the record shape.
  • Base offsets. The broker patches each produced batch's base offset, the one rewrite a Kafka broker also performs and one the batch CRC excludes.

Everything below the frontend, durability, ownership, transfers, and failover, behaves as the design docs describe.

Supported APIs ​

APIVersionsNotes
ApiVersions0 to 4
Metadata0 to 12
Produce3 to 10acks 0, 1, and all behave alike: an acknowledged record is durable in the WAL
Fetch4 to 16Long polling with fetch.min.bytes and fetch.max.wait.ms
ListOffsets1 to 7Earliest, latest, by timestamp, and max timestamp
OffsetForLeaderEpoch2 to 4The leader epoch is the partition's stream epoch
InitProducerId0 to 4Idempotent producers with epoch fencing and sequence checks
CreateTopics2 to 7
DeleteTopics1 to 6
CreatePartitions0 to 3Partitions only grow
DeleteRecords0 to 2Trims a partition. Waits for buffered records to reach data objects, up to the request timeout
DescribeConfigs1 to 4See Configs
IncrementalAlterConfigs0 to 1See Configs
DescribeCluster0 to 2
FindCoordinator0 to 3Group coordinators only
JoinGroup, SyncGroup, Heartbeat, LeaveGroup0 to 7, 0 to 5, 0 to 4, 0 to 5The classic group protocol, see Consumer groups
OffsetCommit, OffsetFetch2 to 8, 1 to 7A commit without a member is accepted only while the group is empty, as in Kafka
DescribeGroups, ListGroups0 to 5ListGroups answers per broker, as in Kafka
DeleteGroups0 to 2Only groups without members
OffsetDelete0Not for topics the group is subscribed to
SaslHandshake, SaslAuthenticate0 to 1, 0 to 2Advertised with --auth required, see Authentication

Clients negotiate through ApiVersions and stay inside the advertised ranges.

Not supported: transactions and control batches, which are rejected, and the KIP-848 consumer protocol.

Configs ​

ConfigValueChangeable
cleanup.policydeleteNo
retention.msThe topic's limit, -1 when unlimitedYes
retention.bytesThe topic's limit, -1 when unlimitedYes

Both retention configs can be set on CreateTopics too, and -1 removes a limit. Retention goes by when PicoMQ wrote the data, not by record timestamps, so it matches Kafka's LogAppendTime topics, see Retention. Changing any other config fails with INVALID_CONFIG.

PicoMQ does not compact topics, so CreateTopics refuses cleanup.policy=compact with INVALID_CONFIG instead of creating a topic that would never compact. This rules out Kafka Connect and Kafka Streams state stores, which need compacted topics. Broker resources describe no configs.

Errors ​

Kafka error codes for each failure are listed in Protocols. A node that is stopping, or that cannot reach the metadata store, answers NOT_LEADER_OR_FOLLOWER, so clients refresh their metadata and retry on a live node.

Authentication ​

With --auth required the listener authenticates every connection with SASL PLAIN, and the credential is the same token the HTTP listeners take as a bearer. The token model, issuing, and revocation are covered in Authentication.

Client settingValue
security.protocolSASL_PLAINTEXT, or SASL_SSL when the node runs TLS
sasl.mechanismPLAIN
sasl.usernameThe token id, for example svc/ingest
sasl.passwordThe token in wire form, BASE64URL(id).BASE64URL(secret)
ssl.ca.locationWith SASL_SSL and a private CA, the CA's PEM file

The token needs the kafka audience. A token that has pico and kafka works over both protocols with one scope, and a token with only kafka is a Kafka-only credential.

bash
kcat -b node1.internal:9092 -t orders -P \
    -X security.protocol=SASL_PLAINTEXT -X sasl.mechanisms=PLAIN \
    -X sasl.username=svc/ingest -X sasl.password="$TOKEN"
properties
# Java clients and the kafka-*.sh tools, through --command-config
security.protocol=SASL_PLAINTEXT
sasl.mechanism=PLAIN
sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username="svc/ingest" password="<token>";

Until a connection authenticates, the listener holds it to the same limits a Kafka broker does. Breaking any of them closes the connection.

Before authenticationLimit
RequestsApiVersions, SaslHandshake, and SaslAuthenticate only
Frame size512 KiB, as Kafka's sasl.server.max.receive.size
Idle time10 minutes, as Kafka's connections.max.idle.ms
CredentialA wrong one answers SASL_AUTHENTICATION_FAILED

What the scope allows ​

Each request needs an operation from the token's scope on the topics it names, and a group request also needs one on its group. The operations are defined under Scopes. A topic is matched under its HTTP name, so a scope with the prefix /orders/ covers the Kafka topics orders.eu and orders.us.

  • A group.id means group requests. librdkafka's consumer needs a group.id even to assign partitions itself, and with one set it sends FindCoordinator for that group. A consumer that tracks its own offsets therefore still needs describe_group on its group.id. Without it, records arrive but the client reports GroupAuthorizationFailed.
  • Reading with no group needs only read, through a low-level consumer such as kcat -C.
RequestOn the groupOn the topics
Producenoneappend
Fetchnoneread
Metadata, ListOffsets, OffsetForLeaderEpoch, DescribeConfigsnonehead, which any stream right on the topic grants
CreateTopics, CreatePartitions, IncrementalAlterConfigs, auto creationnonecreate
DeleteTopicsnonedelete
DeleteRecordsnonetrim
InitProducerIdnoneappend on some topic
FindCoordinator, DescribeGroupsdescribe_groupnone
JoinGroupconsumeread on every topic a consumer subscribes to
SyncGroup, Heartbeat, LeaveGroupconsumenone
OffsetCommitconsumeread
OffsetFetchdescribe_groupread on each named topic, and a fetch of every offset leaves out unreadable topics
OffsetDeletedelete_groupread
DeleteGroupsdelete_groupnone
ListGroupsLists only the groups the token may describenone

A refused group answers GROUP_AUTHORIZATION_FAILED for that group, whatever the request's topics, as on a Kafka broker. A refused topic answers TOPIC_AUTHORIZATION_FAILED, and a refused InitProducerId answers CLUSTER_AUTHORIZATION_FAILED, so clients surface the same errors they would from a Kafka broker with ACLs. A Metadata request for all topics lists only the topics the token may describe. A token that stops being valid closes the connection on its next request, see Revocation.

Differences from Kafka ​

  • DescribeConfigs needs head, where Kafka needs a separate DescribeConfigs right. Configs hold retention settings, nothing secret.
  • OffsetDelete looks the group up before checking delete_group. A token that may describe a group but not delete it gets GROUP_ID_NOT_FOUND for a group that does not exist, which describing the group would show anyway. Kafka requires Delete on the group for OffsetDelete.
  • SASL PLAIN is the only mechanism. SCRAM is listed under Deliberately excluded.
  • Tokens with autoPrefix are refused at authentication rather than half applied, since Kafka names cannot be rewritten per tenant, see Auto-prefixing.

Auth off ​

With --auth off the listener takes any client, the SASL APIs are not advertised, and the node binds the listener to loopback unless --insecure-allow-remote opts out for a network that is guarded some other way.