In this blogpost you will get a basic understanding about message brokers. We will look at two very popular message brokers, Kafka and RabbitMQ, and learn, how they handle messages.
Let’s start with a short intro about what message brokers are and how they are different from an API: Message brokers allow communication between applications, systems and services. On a high level, can understand message brokers as translators: They receive messages, which they “translate” and pass on to other systems.
While for example REST APIs are designed to expect an immediate response, message brokers can communicate asynchronously. That means, a sending service does not have to wait for the reply of the receiving service.
Let’s go more into detail. There are four components in message brokers:
Producers: The senders of a message. For example, if a user creates an account on your website, a producer, let’s call hom “New User Service” creates a message, that an account has been created.
Consumers: The receiver of a message. Consumers looks at incoming messages and may execute some action, e.g. sending an Email to this new customer to verify his account.
Queues: Storage of the messages until they are consumed.
Exchange: A “config” for routing messages. The producer transmits a message to the exchange using a specified protocol (e.g. AMQP). The echange makes sure to route the message to the correct queue.
The producers can issue messages without knowing where the consumers are, whether or not they are active, or how many of them exist.
Let’s look at two different architectural approaches: RabbitMQ or Kafka Architecture;
Basic Architecture RabbitMQ (AMQP)
RabbitMQ consists of exchanges and queues. Exchanges are responsible for routing incoming messages. The queues store messages.
A producer generates messages and pushes them into an exchange. The exchange receives the messages and routes them into a queue. The routing key is like the address of the message, so the exchange knows to which queue the message should be routed. The queues store the messages until a consumer takes a message off the queue. RabbitMQ is a general purpose message broker that supports different protocols including, MQTT, AMQP, and STOMP.
Basic Architecture Kafka
Instead of pushing messages to a queue, Kafka stores messages in topics. A topic is a partitioned log of messages, where each partition is an ordered, immutable sequence of records. Kafka appends messages to the topics as they arrive. The messages stay for a specified number of time. Kafka does not care if messages are read or not. During the timeframe, consumers can persist and re-process the data as they need (Dumb broker / smart consumer approach). Because partitions are ordered, messages can be processed in batches.
RabbitMQ and Kafka use different approaches for processing messages:
RabbitMQ uses a push model. It distributes messages individually through configurable routing and sends them to the consumers. Once consumed, the messages are gone.
Kafka uses a pull model. Consumers have to request messages from a specific offset. Messages are retained in a partition and can be reread.