Key takeaway
- How Apache Pulsar works: stateless brokers, BookKeeper storage, subscription types, geo-replication and the 5.0 LTS changes, with a decision table.
Apache Pulsar is an open-source distributed messaging and streaming platform from the Apache Software Foundation, licensed under Apache 2.0. Its architecture separates stateless brokers, which serve producers and consumers, from Apache BookKeeper, which stores the messages, and a metadata store for coordination. That split is the idea to understand before you compare Pulsar with any other platform (source: Pulsar architecture overview).
This guide takes the Pulsar angle for enterprise architects and platform leads: how the components fit, which subscription type to choose, what geo-replication and tiered storage do and do not give you, and what Pulsar 5.0 LTS changes. For the wider landscape, read our data streaming overview and the history in stream processing evolution. For processing engines, see stream processing with Apache Flink and NATS and streaming analytics architecture.
When does Apache Pulsar fit, and when does it not?
| Need | Does Pulsar fit? | Why, from the Pulsar documentation |
|---|---|---|
| Many teams sharing one messaging cluster | Strong fit | Tenants and namespaces give each team its own authentication, authorisation and policies. |
| Replay and long retention without growing broker disks | Good fit | Tiered storage moves older backlog from BookKeeper to cheaper storage while clients can still read it. |
| Data in several regions | Good fit, with limits | Geo-replication is built in, but subscriptions are local to each cluster. |
| Queue-style work sharing and stream-style ordered reading on one platform | Good fit | Exclusive, failover, shared and key_shared subscriptions cover both patterns. |
| A small team that wants the fewest moving parts | Weigh carefully | A full cluster runs brokers, bookies and a metadata store (editorial judgement from the component list). |
How is a Pulsar cluster built?
A Pulsar instance has one or more clusters, and clusters can geo-replicate with each other. Each cluster has brokers, a BookKeeper ensemble of bookies for message storage and a metadata store. An instance-wide configuration store coordinates tasks that span clusters (source: architecture overview).
| Component | What it does | Current status |
|---|---|---|
| Broker | Stateless. Runs an HTTP REST server for admin tasks and topic lookup, plus a dispatcher that uses a binary protocol for data transfer. | Each topic is owned by one broker; other brokers redirect to it. |
| BookKeeper (bookies) | Distributed write-ahead log that stores N replicated copies of messages in ledgers. | Cursors, which record subscription positions, are persisted here too. |
| Metadata store | Holds metadata, cluster configuration and coordination data. | Oxia is recommended for new clusters. ZooKeeper is production-ready and ships with the Pulsar package. RocksDB is for standalone use. |
| Proxy (optional) | A single gateway for clients; you can run several instances. | Optional. |
Because brokers hold no durable message data, a broker can fail without losing messages: the data sits in BookKeeper. Brokers serve most reads from a managed ledger cache and go to BookKeeper only when the backlog exceeds it.
How do topics, tenants and namespaces organise messages?
Topic names follow the pattern persistent://tenant/namespace/topic. If you do not give a type, the topic is persistent. Persistent topics store messages durably on disk, so data survives broker restarts. Non-persistent topics keep messages only in memory, which gives lower publish latency but loses messages in transit if a broker is killed or a subscriber disconnects (source: messaging concepts).
A tenant is the top-level unit for allocating capacity and enforcing authentication and authorisation. A namespace is a logical grouping of topics inside a tenant, and policies set on a namespace apply to every topic in it. Tenants can create namespaces themselves through the REST API or pulsar-admin, for example one namespace per application (source: multi-tenancy).
Which subscription type should you choose?
A subscription is a named configuration that decides how messages are delivered to consumers. Pulsar documents four types (source: messaging concepts).
| Type | Behaviour | Choose it when | Limits |
|---|---|---|---|
| Exclusive (default) | One consumer only. A second consumer on the same subscription gets an error. | You need strict single-reader order. | No parallelism inside the subscription. |
| Failover | Several consumers attach; one master receives messages and the next takes over if it disconnects. | You want a hot standby with order preserved. | Only one consumer is active at a time. |
| Shared | Messages are distributed round-robin, one consumer each. Unacknowledged messages from a disconnected consumer are redelivered. | You want queue-style scale-out. | No ordering and no cumulative acknowledgement. |
| Key_Shared | Messages with the same key go to the same consumer. | You want parallelism with per-key order. | No cumulative acknowledgement. |
How do routing modes work on partitioned topics?
Routing modes decide which partition a producer writes to. RoundRobinPartition is the default: keyless messages are spread across partitions, and keyed messages are hashed to a partition. SinglePartition sends keyless messages to one randomly chosen partition, and keyed messages are still hashed. CustomPartition uses your own MessageRouter implementation, available in the Java client. Messages with the same key stay in order within one partition, and per-producer order needs SinglePartition with no keys (source: messaging concepts).
How does a Pulsar client connect?
Before creating a producer or consumer, the client library runs two steps. First it sends an HTTP lookup request to a broker to find the topic owner; if nobody serves the topic, one broker is assigned. Then it opens a TCP connection to that broker, or reuses a pooled one, authenticates, and asks the broker to create the producer or consumer, which the broker grants after checking authorisation. If the connection drops, the client repeats the setup and retries with exponential backoff. Official clients exist for Java, C++, Go, Python, Node.js and C# (source: client concepts).
What do geo-replication and tiered storage give you?
Geo-replication
Asynchronous replication persists a message in the local cluster first and then replicates it, so latency follows the inter-datacentre round trip and producers and consumers keep working during a partition. The trade-off is weaker consistency, because some data may not have replicated yet. Synchronous replication is built on BookKeeper and acknowledges a write only after a majority of data centres persist it, which needs bookies and brokers spread across sites, a global metadata store and a region-aware placement policy, and costs cross-datacentre latency (source: geo-replication).
Plan for the limits. Subscriptions are not replicated by default, so a consumer that reconnects in another cluster starts a new subscription, and individual acknowledgements are not replicated. Consumers in different clusters do not form one group, so each cluster processes every message unless you shard work by topic or partition per region.
Tiered storage
Tiered storage moves older messages from BookKeeper into cheaper storage while clients read the backlog as before. It offloads sealed segments to Amazon S3, Google Cloud Storage, Azure Blob Storage, Alibaba Cloud OSS, local or network filesystems, and S3-compatible systems such as MinIO and Ceph. Offload can be triggered by backlog size, by age, manually through the REST API or CLI, or by namespace policy (source: tiered storage).
How do schemas and security support governance?
Pulsar applies and enforces schemas at the topic level. When a producer or consumer connects with a new schema, the broker runs the compatibility check defined for the topic, and clients that fail it are rejected. Accepted messages are tagged with the schema version (source: schema overview). Each tenant can apply its own authentication and authorisation scheme, and storage quotas, message TTL and isolation policies are managed at tenant level. Retention and expiry are set per namespace: by default acknowledged messages are deleted immediately, and retention keeps them for a set period.
Streams only help decisions if the data arriving on them carries context and policy. ElixirData Context OS keeps governed context, versioned policy and decision traces in one decision layer above AI agents, and AgentAnalyst lets teams ask questions of enterprise data in plain language and get real-time insights. For agent workflows built on event streams, see ElixirClaw, an agentic OS platform to build, orchestrate and govern AI agents and automation workflows while keeping humans in control.
What changed in Apache Pulsar 5.0 LTS?
The Pulsar project announced 5.0 LTS on 5 October 2026. It introduces Scalable Topics, a topic type that scales capacity up and down with demand while keeping per-key ordering, with a new Java client API. It names Oxia as the recommended metadata store for new clusters and brings performance and reliability improvements to existing topics through a cluster upgrade, with no application changes. You can upgrade a 4.x cluster and keep your current topics, clients and ZooKeeper metadata store, then adopt Scalable Topics later (source: Apache Pulsar 5.0 announcement).
How does Pulsar compare with Kafka, and when should you not choose it?
Compare on your own workload, not on a headline ratio. Pulsar separates serving from storage, builds in multi-tenancy and geo-replication, and offers queue and stream subscriptions in one platform. The cost is more components to run, and a smaller body of in-house skills in many enterprises. Choose another platform if your needs are a single team, a single region and a stream-only pattern that your existing platform already meets. Our Kafka vs Pulsar guide walks through the decision, and our real-time analytics services page shows how we build on streaming platforms. Pulsar Functions, a serverless framework, reads from input topics, applies your logic and writes to output topics in Java, Python or Go.
What is a sensible adoption sequence?
First, define tenants and namespaces per team and set retention and expiry before any data flows. Second, pick the metadata store: Oxia for new clusters, or ZooKeeper if you are upgrading an existing cluster. Third, choose subscription types per consumer group using the table above. Fourth, register schemas and set compatibility checks. Fifth, add tiered storage once backlogs grow, and geo-replication once a second region needs the data. Use our Big Data Maturity Model to judge where your data platform stands first.
To map one streaming decision your agents should make, and the context and policy it depends on, book a Context Workshop. See what happens in the workshop.
Frequently asked questions
What is Apache Pulsar?
Apache Pulsar is an open-source, distributed messaging and streaming platform built for the cloud, and an Apache Software Foundation project under the Apache 2.0 licence.
Are Pulsar brokers stateful?
No. Brokers are stateless. Messages are stored as replicated copies in BookKeeper, and each topic is owned by one broker at a time.
Does Pulsar still need ZooKeeper?
Not for new clusters. Oxia is recommended for new clusters, ZooKeeper remains production-ready and ships with the Pulsar package, and RocksDB is for standalone deployments.
Which Pulsar subscription type preserves order while scaling out?
Key_Shared sends messages with the same key to the same consumer. Shared mode distributes round-robin and does not preserve ordering.
What are the limits of Pulsar geo-replication?
Subscriptions are local to each cluster, individual acknowledgements are not replicated, and asynchronous replication can leave some data unreplicated. Synchronous replication adds cross-datacentre write latency.
Where does tiered storage send old messages?
To Amazon S3, Google Cloud Storage, Azure Blob Storage, Alibaba Cloud OSS, filesystems or S3-compatible systems such as MinIO and Ceph, while clients can still read the backlog.
Follow XenonStack on LinkedIn: XenonStack on LinkedIn for new guides on agentic AI, data and platform engineering.
Explore the XenonStack platform family: ElixirData · ElixirClaw · ElixirHub · NexaStack · MetaSecure · Neural AI · AgentGRC · AgentSRE · AgentAnalyst · AgentInstruct · AgentRAI · AgentScrum · AgentSearch · AgentForce · AI Aviator · SAIF Aviator · CohortIQ · XenonStack AI · XenonStack UAE
