Apache Pulsar
Master Apache Pulsar: multi-tenant messaging, geo-replication, and unified streaming platform.
What is Apache Pulsar?
Apache Pulsar is a distributed publish-and-subscribe platform for durable messaging and retained event streams. Producers publish messages to topics. Consumers attach through named subscriptions, process messages, and acknowledge the subscription cursor after owned work succeeds.
Pulsar matters when one platform must support many tenants, large topic counts, independent consumer applications, queue-style worker pools, long retention, or cross-region delivery. Its defining architectural choice is to separate serving from storage: brokers own client connections and topic dispatch, while Apache BookKeeper stores persistent ledger entries and subscription cursors.
The core invariant is partitioning and subscription type solve different problems. Partitions create producer-side topic lanes that can span brokers. A subscription decides which consumers receive each message and what ordering relationship they may assume.
Follow one persistent message
A persistent topic name has the form persistent://tenant/namespace/topic. The tenant and namespace define administrative and policy boundaries; the topic carries messages; each subscription owns an independent cursor over those messages.
Pulsar persistent message path
The broker serves the topic, BookKeeper stores the entry, and each subscription advances its own cursor.
Publish
Producer
Look up the broker that owns the topic or partition, then send a keyed or unkeyed message using the selected routing policy.
Serve
Owner broker
Accept the client connection, validate policy, append through the managed ledger, and dispatch available entries to subscriptions.
Persist
BookKeeper
Replicate append-only ledger entries across bookies. Persist subscription cursor state separately from the stateless serving broker.
Deliver
Subscription
Apply Exclusive, Failover, Shared, or Key_Shared delivery semantics. Advance the cursor only after the consumer acknowledges successful processing.
Keep the policy hierarchy explicit
- Tenant: the top-level isolation and authorization boundary.
- Namespace: the policy unit for retention, backlog, persistence, quotas, replication, and many other controls.
- Topic or partition: one ordered message lane with one broker owner at a time.
- Subscription: an independently named consumer position and delivery contract.
The metadata store holds ownership, policy, and ledger metadata. Current Pulsar releases can use supported metadata backends such as ZooKeeper or Oxia depending on the deployment. Do not describe ZooKeeper as part of every possible modern Pulsar data path.
Size partitions separately from consumers
A non-partitioned topic has one broker owner, so adding consumers cannot remove a producer-side topic bottleneck. A partitioned topic is implemented as several internal topics that Pulsar can distribute across brokers. Producer routing chooses a partition; subscription type still controls consumer delivery.
Use the lab to preserve two independent inequalities:
partition count x measured publish capacity per partition >= peak publish rateactive consumers x measured service rate per consumer >= peak arrival rate
Loading the partition model...
The lab's rates are transparent planning assumptions, not universal Pulsar limits. Benchmark with the real payload size, batching, compression, persistence policy, network path, backlog reads, and handler behavior.
Choose the subscription from the ordering invariant
All four subscription types can be used with partitioned topics. They differ in consumer ownership and ordering, not in whether the topic is partitioned.
One attached consumer
Exclusive
Only one consumer may attach to the subscription. On a partitioned topic, that one consumer receives from all partitions.
- Choose it when a single process must own the complete subscription.
- Additional connection attempts are rejected; they are not standby consumers.
Active and standby
Failover
Multiple consumers may attach, but one active consumer is selected for each partition. When an active consumer disconnects, another consumer takes over unacknowledged and subsequent messages.
- Choose it when ordered active/standby processing is required.
- Consumer parallelism is bounded by the partition count.
Independent parallel work
Shared
All attached consumers can receive. The broker distributes messages among them, and an unacknowledged message can be redelivered after a consumer disconnects.
- Choose it for independent work with horizontal consumer scaling.
- Do not assume message ordering.
Parallelism across keys
Key_Shared
All attached consumers can receive, but messages with one key are assigned to one consumer at a time.
- Choose a stable key for the smallest entity that needs order.
- Use key-based producer batching or disable batching; default batching can mix keys in a way that breaks the intended distribution semantics.
Transport ordering is not business exactly-once execution. A consumer can commit an external side effect and crash before its acknowledgment reaches Pulsar. Use a stable operation ID and make the destination mutation idempotent.
Trace broker ownership and BookKeeper durability
For a persistent topic, the broker is not the durable system of record. It appends through a managed ledger backed by BookKeeper. A persistence policy uses three related numbers:
- Ensemble size (E): how many bookies participate in a ledger segment.
- Write quorum (W): how many bookies receive each entry.
- Ack quorum (A): how many durable writes must succeed before the broker acknowledges the producer.
- Required relationship:
E >= W >= A.
Change the policy and inject failures below. Watch the serving layer, storage layer, and metadata control plane fail independently.
Loading the broker and storage trace...
Lowering A may restore availability in the model, but it also lowers the number of confirmed durable copies at acknowledgment time. Treat quorum changes as a data-loss and recovery decision, not a generic latency tuning knob.
Separate backlog, retention, and tiered storage
These controls answer different lifecycle questions:
- Backlog is unacknowledged data still required by a subscription cursor.
- Retention deliberately keeps acknowledged data for a configured time or size.
- Tiered storage offloads eligible closed ledger segments from BookKeeper to lower-cost object storage while keeping them readable through Pulsar.
- Expiry can remove unacknowledged messages after a configured TTL; using it is a product data-loss decision.
Design long retention from the read path backward
- Measure replay throughput from offloaded segments, not only hot-cache dispatch.
- Bound subscription backlog separately from acknowledged-message retention.
- Alert on oldest backlog age and BookKeeper disk pressure before emergency offload.
- Test object-store permissions, throttling, and restoration during a real replay.
Use geo-replication as an asynchronous data path
Pulsar brokers can run replicators that read locally published entries and republish them to remote clusters. Applications normally use the local cluster service URL, while namespace or topic policy chooses the replication clusters.
State the regional contract before deployment
- Define whether producers may write in one region or several.
- Measure replication backlog and oldest replication delay per remote cluster.
- Decide how consumers resume when a region fails and remote cursors differ.
- Plan for duplicate or reordered business effects across asynchronous regional paths.
- Keep tested recovery and backup controls; replication copies operational mistakes as well as valid messages.
Operate each namespace as a workload boundary
Pulsar supplies tenant and namespace controls, but teams still need explicit ownership and service objectives.
Rate
Publish and dispatch
Messages and bytes by topic, namespace, and broker
Age
Oldest backlog
User-visible freshness risk for every subscription
Store
BookKeeper health
Add-entry latency, failed bookies, disk, and under-replicated fragments
Replicate
Geo backlog
Remote lag and recovery-point exposure
Production checklist
- Set namespace publish, dispatch, backlog, and storage quotas from measured workload envelopes.
- Authenticate clients, authorize tenant and namespace actions, encrypt network paths, and isolate administrative roles.
- Version schemas and reject incompatible producer changes before they reach consumers.
- Bound negative acknowledgments, acknowledgment timeouts, retry topics, and dead-letter handling.
- Exercise broker loss, bookie loss, metadata quorum loss, hot-key skew, replay from offload, and regional replication lag.
- Track ownership changes and ledger recovery time during broker rebalancing.
Know when Pulsar fits the system
Serving and storage scale apart
Choose Pulsar
Use Pulsar when native multi-tenancy, high topic cardinality, independent subscriptions, mixed queue and stream semantics, long retention, or cross-cluster replication justify operating brokers, BookKeeper, and metadata as separate layers.
Log ecosystem first
Choose Kafka
Prefer Kafka when its partitioned-log model, operational experience, connector ecosystem, and surrounding stream-processing platform already match the workload. Compare current capabilities and measured operations instead of relying on historical feature lists.
One worker owns each task
Choose a managed queue
Use a managed queue when the system primarily needs asynchronous task handoff and the team does not need retained multi-subscriber streams or to operate a separate broker, storage, and metadata stack.
The deciding question is not whether Pulsar can carry the messages. It is whether its independent serving and storage layers, subscription contracts, and policy hierarchy solve enough real requirements to justify their operational surface.