Apache Kafka Architecture Diagram: Producers, Brokers, KRaft Controllers and Consumer Groups

A Kafka architecture diagram has four kinds of boxes: producers that publish records, brokers that store partitioned and replicated topic logs, KRaft controllers that hold cluster metadata, and a consumer group whose members read the partitions. The Apache Kafka 4.0.0 release announcement calls 4.0 the first major release to operate entirely without Apache ZooKeeper, running in KRaft mode by default, so the current picture has no ZooKeeper box. The detail diagram below labels each box with the terms the Apache Kafka documentation uses.

What this Kafka architecture diagram shows

The Kafka documentation describes Kafka as a distributed system of servers and clients that communicate over a high-performance TCP network protocol. Some of the servers form the storage layer and are called brokers. Producers are the client applications that publish events, and consumers are the ones that subscribe to them. The documentation says producers and consumers are fully decoupled and agnostic of each other, and that producers never need to wait for consumers. Events are stored in topics, and topics are partitioned, meaning a topic is spread over a number of buckets located on different brokers so that clients can read and write from many brokers at the same time.

The detail diagram follows that structure. Controllers sit in their own band at the top. The three brokers sit in the middle, each labeled with the partition it leads. Producers are on the left and the consumer group is on the right. The inset at the bottom expands the same three brokers into partition replicas, so the topology and the replica layout can be read separately. On screens that allow motion, a record dot travels from Producer 1 toward the consumer group. With reduced motion requested, the dot stays still and the static drawing carries the same information.

The problem this architecture is solving

The Kafka design documentation says Kafka was built to act as a unified platform for handling all the real-time data feeds a large company might have, which requires high throughput for high-volume event streams. A traditional queue tracks delivery per message. Kafka keeps a log instead. The introduction notes that events in a topic can be read as often as needed and are not deleted after consumption; a per-topic configuration setting defines how long they are retained.

Because each partition is consumed by exactly one consumer within each subscribing consumer group at any given time, the position of a consumer in each partition is a single integer: the offset of the next message to consume. The design documentation says this keeps the consumed state very small, and it lets a consumer deliberately rewind to an old offset and re-consume data, for example after a bug in the consumer code is found.

Main components: producers, brokers, KRaft controllers, consumers

Producers. The design documentation says the producer sends data directly to the broker that is the leader for the partition, without any intervening routing tier. To make that possible, all Kafka nodes can answer a request for metadata about which servers are alive and where the leaders for a topic's partitions are. The client controls which partition it publishes to, either at random for load balancing or by a semantic partitioning function, such as hashing a key so that all data for one user goes to the same partition. In the diagram, Producer 1 sends to the P0 leader on Broker 1 and Producer 2 sends to the P2 leader on Broker 3.

Brokers. Brokers store the partition logs. The design documentation says Kafka attempts to balance partitions within a cluster in a round-robin fashion and to balance leadership so that each node is the leader for a proportional share of its partitions. That is why each broker in the diagram leads a different partition.

KRaft controllers. In KRaft mode each server is configured as a controller, a broker, or both through the process.roles property. The servers selected as controllers participate in the metadata quorum, and each controller is either the active controller or a hot standby for it. The KRaft operations documentation says an admin will typically select 3 or 5 servers for this role, and that a majority of the controllers must be alive to maintain availability: with 3 controllers the cluster can tolerate 1 controller failure, and with 5 it can tolerate 2. It also says combined broker-and-controller mode is not recommended in critical deployment environments, because the controllers cannot be rolled or scaled separately from the brokers. The cluster metadata partition is what the KRaft tools inspect as __cluster_metadata-0 log segments and snapshots.

Consumers. The design documentation says the consumer works by issuing fetch requests to the brokers leading the partitions it wants to consume, specifying its offset with each request and receiving a chunk of log beginning from that position. Kafka follows a pull-based design, so a consumer that falls behind catches up when it can instead of being overwhelmed.

Topics, partitions and replicas (leader, followers, ISR)

When a new event is published to a topic, it is appended to one of the topic's partitions. The introduction says events with the same event key, such as a customer or vehicle ID, are written to the same partition, and Kafka guarantees that any consumer of a given topic-partition reads that partition's events in exactly the order they were written. There is no ordering promise across partitions.

The design documentation says the unit of replication is the topic partition. Under non-failure conditions, each partition has a single leader and zero or more followers, and the total number of replicas including the leader is the replication factor. All writes go to the leader, and reads can go to the leader or the followers. Followers consume messages from the leader just as a normal Kafka consumer would and apply them to their own log. The introduction calls a replication factor of 3 a common production setting. The inset draws exactly that case: P0, P1 and P2 each have one leader and two followers spread across the three brokers, and each follower has a curved arrow to its leader.

The leader keeps track of the set of in-sync replicas, the ISR. For KRaft clusters, a node keeps an active session by sending periodic heartbeats to the controller, and if the controller does not receive a heartbeat before broker.session.timeout.ms expires, the node is considered offline. A follower that has an active session but cannot catch up to the end of the leader's log within replica.lag.time.max.ms is removed from the ISR. Only members of the ISR are eligible to be elected leader, which the inset legend states directly. The documentation says that with this ISR model and f+1 replicas, a topic can tolerate f failures without losing committed messages.

Consumer groups and partition assignment

A consumer group shares the partitions of the topics it subscribes to, and each partition is consumed by exactly one consumer within each subscribing group at a time. The diagram draws one consumer group with three consumers: Consumer 1 is assigned P0, Consumer 2 is assigned P1, and Consumer 3 is assigned P2, and the note under the group reads one consumer per partition in the group. Each consumer has its own fetch arrow from the broker that leads its partition.

Assignment is handled by the group rebalance protocol, which the design documentation says relies on the group coordinator to allocate entity ids to group members. Static membership gives each consumer instance a unique, persistent group instance id, so that restarts during maintenance do not cause the large task reassignments that dynamic membership can trigger. The Kafka 4.0.0 announcement adds that KIP-848, the next generation of the consumer rebalance protocol, is generally available in 4.0. It says the new protocol is enabled by default on the server side and that consumers must opt in by setting group.protocol=consumer.

A record's path from producer to consumer, step by step

1. Find the leader. The producer asks any Kafka node for metadata about which servers are alive and where each partition leader is.
2. Choose the partition. The client picks a partition at random or by hashing the record key, so records with the same key land in the same partition and keep their order.
3. Send to the leader. The producer sends the record directly to the broker leading that partition. In the diagram that is the arrow from Producer 1 to the P0 leader on Broker 1.
4. Replicate. Followers fetch from the leader and append the record to their own logs with the same offsets and in the same order, as the inset's follower arrows show.
5. Commit. The design documentation says a message is considered committed only when all replicas in the ISR for that partition have applied it to their log.
6. Acknowledge. The producer chooses how long to wait with the acks setting. With acks=all, acknowledgement happens as soon as all current in-sync replicas have received the message. With acks=1 the message is committed synchronously only on the leader, and with acks=0 it is committed asynchronously across the ISR. Regardless of the setting, messages are not visible to consumers until they are replicated to all in-sync replicas and the ISR has at least min.insync.replicas members.
7. Consume. Consumer 1 fetches P0 from Broker 1, specifying its offset, and receives the log from that position.
8. Track the position. The consumer's position is its offset in the partition. The design documentation says the position is stored as a message in an internal topic, which lets an application that writes its output to another Kafka topic store the offset in the same transaction.

Two durability settings change what committed protects against. The topic setting min.insync.replicas makes a partition accept writes only if the ISR is above a minimum size, and it only takes effect when the producer uses acks=all. If every replica of a partition dies, the documentation describes a choice between waiting for an ISR member to come back and electing the first replica that returns even if it is behind. By default since 0.11.0.0, Kafka waits for a consistent replica, and unclean.leader.election.enable changes that for use cases that prefer uptime. The guarantee the documentation gives is that a committed message will not be lost as long as at least one in-sync replica is alive. When the controller detects that a broker has failed, it elects one of the remaining ISR members as the new leader for each affected partition, and it batches those leadership changes so the election is cheaper and faster when many partitions are involved.

What changed with KRaft in Kafka 4.0

The Kafka 4.0.0 release announcement describes 4.0 as the first major release to operate entirely without Apache ZooKeeper. It says that by running in KRaft mode by default, Kafka simplifies deployment and management and eliminates the complexity of maintaining a separate ZooKeeper ensemble. That is why the diagram's metadata box is a KRaft controller quorum, with an active controller, standbys, and the cluster metadata log, rather than an external coordination service.

The same announcement lists other changes that touch this diagram. KIP-996 introduces a Pre-Vote mechanism to reduce unnecessary KRaft leader elections. KIP-966 introduces Eligible Leader Replicas in preview, described as a subset of the ISR replicas guaranteed to have complete data up to the high-watermark. KIP-932 offers early access to share groups, a cooperative consumption model for queue-like use cases on regular topics. The announcement also says Kafka brokers now require Java 17, while Kafka clients and Kafka Streams require Java 11.

FAQ

Does Kafka still need ZooKeeper?

Not in Kafka 4.0. The Apache Kafka 4.0.0 release announcement describes 4.0 as the first major release to operate entirely without Apache ZooKeeper, running in KRaft mode by default. In KRaft mode, the servers selected as controllers participate in the metadata quorum.

How many KRaft controllers should a cluster have?

The Kafka documentation says an admin will typically select 3 or 5 servers for the controller role. A majority of the controllers must be alive to maintain availability, so 3 controllers tolerate 1 failure and 5 controllers tolerate 2.

What does acks=all actually wait for?

It waits for all current in-sync replicas, not the full set of assigned replicas. The documentation notes that if the ISR has shrunk, fewer copies acknowledge the write, which is why acks=all is paired with a min.insync.replicas setting when durability matters.

Conclusion

A current Kafka architecture diagram has producers sending records straight to partition leaders, brokers that each lead a share of the partitions while following the rest, a KRaft controller quorum above the brokers that holds the cluster metadata log and receives broker heartbeats, and a consumer group that assigns one consumer to each partition. Followers fetch from leaders to stay in the ISR, a record is committed once every ISR replica has it, and only ISR members can be elected leader. Kafka 4.0 runs in KRaft mode by default without ZooKeeper, so the metadata box is a controller quorum.