Connectors

Connectors move records between PicoMQ and the systems around it. A source reads from somewhere else and produces into topics. A sink consumes topics and writes them out.

Both run inside pico-connectors, a process separate from the node. It speaks only the Kafka protocol to PicoMQ and needs nothing but a bootstrap address. It deploys, scales and restarts on its own schedule, and the cluster never knows it is there.

Plugins, not a monolith

The runtime contains no connector code. Each connector is a shared library, a .so on Linux, loaded at startup from a path in its definition. The runtime drives it through a small C ABI: open with a config blob, exchange batches, close.

The two sides own different things.

Runtime owns	Plugin owns
Consumer groups and offsets	Reading the external system
The producer and topic creation	Writing the external system
Checkpoints for sources	What its checkpoint means
Retries and backoff	Its own connection handling
Decoding, transforms, routing	Its configuration schema
The HTTP API and metrics	Nothing about PicoMQ

That split has consequences worth knowing up front.

• An installation carries exactly the connectors it uses. The image ships the light plugins, and a heavy one is a single file dropped into /usr/local/lib.
• A connector written outside this repository installs the same way as one inside it. The ABI is the contract, not the workspace.
• A plugin that panics takes the runtime down with it. The runtime is meant to be supervised and restarted, and the checkpointing in Delivery guarantees is designed around that.

Topics are the unit

PicoMQ encourages many small streams instead of a few wide ones, and connectors are built for that.

• A source can produce straight into a topic per user, per tenant or per hash bucket by naming the topic from a field of each record.
• A sink can subscribe to a pattern and follow topics into existence as they appear.
• A sink can resolve a table or index per topic from a template, so the fan-out on the way in becomes a fan-out on the way out.

Routing and templating covers this in full. It is the main thing that makes these connectors different from the Kafka Connect model they otherwise resemble.

What you get, and what you do not

Delivery is at-least-once in both directions. Sinks commit their consumer offset only after the plugin confirms the write. Sources advance their cursor only after every record of a batch is acknowledged by the broker. After a crash a record may be seen twice, never zero times, and most sinks upsert on a deterministic id so the duplicate is invisible.

Two things are not offered, and both come from PicoMQ itself rather than the connectors.

• No exactly-once. PicoMQ does not support Kafka transactions, and the sources have no equivalent on their side, so the runtime does not pretend.
• One partition per topic. This is how PicoMQ works, and it simplifies the connectors considerably, since a topic is a single ordered stream and a sink never reasons about partition assignment.

Both hold for any Kafka client against PicoMQ, not only for connectors. The Kafka protocol page has the details.

Where to go next

Page	Read it when
First connector	You want records flowing in ten minutes
Sources and Sinks	You want to know what the runtime does on each side
Routing and templating	You are designing the topic layout
Delivery guarantees	You are about to trust production data to it
Catalog	You need every option of a specific connector
Writing a plugin	The system you need is not in the catalog
Operations	You are deploying and running the runtime