Replay fan-out with Nestor

Nestor is a read-through block cache for S3-compatible object storage. It speaks S3, so PicoMQ uses it as the endpoint for the data bucket. Reads of committed objects are served from Nestor's RAM and disk after the first fetch. Writes pass through to the origin.

Source: examples/storage/nestor-fanout.

Architecture

Path	Endpoint	Why
--storage	Nestor	Committed objects are read many times. Immutable keys, one origin fetch per block.
--wal	Object store	Written once, read on recovery. Nothing to cache.

PicoMQ's own block cache is per node and per process. Nestor is shared by every node and survives PicoMQ restarts and stream transfers.

Configuration

PicoMQ:

--storage='-2@s3://picomq?region=us-east-1&endpoint=http://nestor:9000&pathStyle=true'
--wal='0@s3://picomq?region=us-east-1&endpoint=http://s3:9000&pathStyle=true'

nestor.toml:

[origin]
endpoint = "http://s3:9000"
credentials = { source = "static", access_key = "...", secret_key = "..." }

[auth]
mode = "static"
access_key = "..."
secret_key = "..."

[buckets]
consistency = { mode = "immutable" }   # PicoMQ never reuses an object key
readahead = 0                          # PicoMQ prefetches for itself

PicoMQ signs with the [auth] keys through the usual AWS_* variables.

The example

32 consumers replay one 32 MiB stream. PicoMQ's caches are set to 8 MiB WAL and 1 MiB block so the reads leave the node.

cd examples/storage/nestor-fanout
docker compose up -d
./seed.sh
./verify.sh

verify.sh output from one run:

cold replay: 32 readers from seq=0
  delivered to readers   1024.0 MiB
  fetched from origin    25.3 MiB in 38 requests
  blocks                 miss=38 joined=0 hit=2111

restart PicoMQ, replay once
  delivered to reader    32.0 MiB
  fetched from origin    6.8 MiB in 10 requests
  blocks                 miss=10 hit=125

First pass: 32 readers, one origin fetch per block. Second pass: PicoMQ restarted with empty caches, Nestor did not. The 10 misses are the newest records PicoMQ was serving from its WAL cache before the restart, which Nestor had never been asked for.

Service	Port	Purpose
pico	4437, 9090, 9092	Pico HTTP, admin, Kafka
nestor	9000, 9100	S3 endpoint, Prometheus metrics
rustfs	19000	Origin

Metrics to watch at :9100/metrics: nestor_origin_requests_total, nestor_blocks_hit_total, nestor_blocks_miss_total.