Kafka: The Distributed Log That Orders Events in Production

⏱️ Lectura: 13 min

In 2010, LinkedIn had dozens of internal systems that needed to share data with each other in real time, and no available message queue could handle that volume without falling over: that’s how Apache Kafka was born. Today that same distributed log runs behind Netflix, Uber, and Spotify, and it has become the de facto standard for moving events between microservices without losing them.

📑 En este artículo

1. TL;DR
2. What Apache Kafka Is and Why It Matters
3. How It Works Under the Hood
4. Practical Examples: Your First Producer and Consumer
5. Getting Started: Spinning Up Kafka in Minutes
6. Real-World Use Cases
7. Common Mistakes and Best Practices
8. Comparison With Alternatives
9. Going Deeper: Log Compaction and Exactly-Once
10. Frequently Asked Questions
    1. Is Kafka the same as a message queue?
    2. Do I still need ZooKeeper to run Kafka?
    3. How many partitions should a topic have?
    4. Does Kafka guarantee the order of all messages in a topic?
    5. Can I use Kafka as a database?
    6. What happens if a leader broker crashes mid-write?
11. References

TL;DR

• You’ll understand what a topic, a partition, and an offset are, and why they replace a traditional queue.
• You’ll be able to write a Kafka producer and consumer in Node.js in under 20 lines each.
• You’ll know how to spin up a local Kafka broker with Docker in KRaft mode, without ZooKeeper.
• You’ll understand how a consumer group distributes partitions across processes without duplicating work.
• You’ll know how to check a consumer’s lag with a single Kafka CLI command.
• You’ll be able to decide when Kafka makes more sense than RabbitMQ, Amazon SQS, or Apache Pulsar.
• You’ll learn the most common mistakes when designing partitions and how to avoid them from the start.

What Apache Kafka Is and Why It Matters

Apache Kafka is not a traditional message queue. A classic queue, like the ones RabbitMQ handles, deletes a message as soon as a consumer acknowledges it. Kafka does something different: it stores each message in an ordered, immutable log, and keeps it for a configurable period even after it’s been read.

That difference seems small, but it changes everything you can build on top of it. If ten different systems need to read the same event (a new order, a click, a metric), each one can read the log at its own pace, without the producing system knowing or caring who’s reading on the other end.

Kafka was born at LinkedIn in 2010, created by Jay Kreps, Neha Narkhede, and Jun Rao, and was open sourced as an open source project under the Apache Software Foundation in 2011. The original motivation was simple: LinkedIn had an activity feed and operational metrics that no messaging system of the time could move with the volume and latency they needed.

How It Works Under the Hood

Everything in Kafka revolves around four concepts: topics, partitions, offsets, and brokers. A topic is a category of events, for example ordenes-nuevas or eventos-usuario. Each topic is divided into one or more partitions, and each partition is, literally, a log file that content can only be appended to.

Each message within a partition gets an offset: a sequential integer that marks its exact position. A consumer doesn’t ask Kafka for “the next unread message” like in a queue; it explicitly asks “give me everything from offset 4,812 onward,” which lets it rewind, repeat, or skip ahead in its reading at will.

Brokers are the servers that store the partitions. A production Kafka cluster has several brokers, and each partition has a leader broker that handles writes and N replicas on other brokers that copy that log in case the leader goes down. That set of up-to-date replicas is called the ISR, in-sync replicas.

On the producer side, each message is sent with an optional key. Kafka hashes that key to decide which partition it goes to, so all events with the same key (for example, the same usuario_id) always end up in the same partition and are read in the order they were written.

On the consumer side, processes are grouped into consumer groups. Kafka distributes a topic’s partitions among the live members of a group, so that each partition is processed by only one consumer in the group at a time. This is what lets you scale processing by adding more instances, up to the limit of the number of partitions.

flowchart TD
    P1["Producer A"] --> B["Kafka Broker"]
    P2["Producer B"] --> B
    B --> Part0["Partition 0"]
    B --> Part1["Partition 1"]
    B --> Part2["Partition 2"]
    Part0 --> C1["Consumer 1"]
    Part1 --> C2["Consumer 2"]
    Part2 --> C2
    subgraph Group["Consumer group: procesador-ordenes"]
    C1
    C2
    end

Up through version 3.2, Kafka relied on ZooKeeper to coordinate cluster metadata: which broker is the leader of which partition, the list of topics, and so on. Since KRaft reached production in Kafka 3.3, Kafka handles that coordination itself with a Raft-style consensus protocol, without depending on an external system.

Practical Examples: Your First Producer and Consumer

The simplest possible example is a producer that sends a single message. We’ll use kafkajs, the most widely used Node.js library for talking to Kafka:

const { Kafka } = require('kafkajs')

const kafka = new Kafka({
  clientId: 'app-eventos',
  brokers: ['localhost:9092']
})

const productor = kafka.producer()

async function enviarEvento() {
  await productor.connect()
  await productor.send({
    topic: 'eventos-usuario',
    messages: [
      { key: 'usuario-42', value: JSON.stringify({ accion: 'login' }) }
    ]
  })
  await productor.disconnect()
}

enviarEvento()

This script connects to the local broker, sends a message to the eventos-usuario topic with the key usuario-42 so it always lands in the same partition, and closes the connection. The expected result: the message is persisted in the log and available to any consumer that subscribes, even if it didn’t exist yet when the message was sent.

A more realistic case needs fine control over when a message is confirmed as processed. This consumer is part of a consumer group and only advances its offset after saving the order to the database:

const { Kafka } = require('kafkajs')

const kafka = new Kafka({
  clientId: 'procesador-ordenes',
  brokers: ['localhost:9092']
})

const consumidor = kafka.consumer({ groupId: 'procesador-ordenes' })

async function procesarOrdenes() {
  await consumidor.connect()
  await consumidor.subscribe({ topic: 'ordenes-nuevas', fromBeginning: false })

  await consumidor.run({
    autoCommit: false,
    eachMessage: async ({ topic, partition, message }) => {
      const orden = JSON.parse(message.value.toString())
      await guardarEnBaseDeDatos(orden)
      await consumidor.commitOffsets([
        { topic, partition, offset: (Number(message.offset) + 1).toString() }
      ])
    }
  })
}

procesarOrdenes()

Here autoCommit is turned off on purpose. If the offset commit happened automatically before saving to the database, a crash midway through would lose the order forever: with this order, save first and confirm the offset after, if the process crashes, Kafka redelivers that message on restart.

sequenceDiagram
    participant P as Producer
    participant L as Leader broker
    participant R as Replica broker
    participant C as Consumer
    P->>L: sends message with acks=all
    L->>R: replicates the message
    R-->>L: confirms write
    L-->>P: confirms commit
    C->>L: poll requests next offset
    L-->>C: returns the message

Getting Started: Spinning Up Kafka in Minutes

To try out everything above, you don’t need a full cluster. The official Apache Kafka image includes KRaft, so a single container is enough for local development:

docker run -d --name kafka-local -p 9092:9092 \
  -e KAFKA_NODE_ID=1 \
  -e KAFKA_PROCESS_ROLES=broker,controller \
  -e KAFKA_LISTENERS=PLAINTEXT://:9092,CONTROLLER://:9093 \
  -e KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://localhost:9092 \
  -e KAFKA_CONTROLLER_QUORUM_VOTERS=1@localhost:9093 \
  -e KAFKA_CONTROLLER_LISTENER_NAMES=CONTROLLER \
  apache/kafka:3.7.0

With the broker up, create a topic with three partitions so you can scale up to three consumers in parallel:

docker exec kafka-local /opt/kafka/bin/kafka-topics.sh \
  --create --topic eventos-usuario \
  --partitions 3 --replication-factor 1 \
  --bootstrap-server localhost:9092

Install the client library and run the example producer:

npm install kafkajs
node productor.js

To confirm everything works without writing code, Kafka ships console utilities that read directly from the topic:

docker exec kafka-local /opt/kafka/bin/kafka-console-consumer.sh \
  --bootstrap-server localhost:9092 \
  --topic eventos-usuario --from-beginning

💡 Tip: to check whether a consumer group is caught up or falling behind, run kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group procesador-ordenes. The LAG column shows how many messages that group still has to process in each partition.

Real-World Use Cases

LinkedIn’s original case, moving activity events and metrics between internal systems, remains the most common pattern: decoupling whoever generates an event from whoever consumes it, so both sides can scale and fail independently.

Another widespread use is Change Data Capture (CDC): tools like Debezium read a database’s transaction log and publish each change as an event on a Kafka topic, so other systems (caches, search engines, data warehouses) stay in sync without querying the original database.

It’s also the backbone of real-time analytics pipelines: instead of waiting for a nightly batch, every click or transaction is processed as it happens, with tools like Kafka Streams or ksqlDB running aggregations directly on the topics.

And in microservices architectures, Kafka replaces synchronous HTTP calls between services with asynchronous events: an orders service publishes orden-creada and the billing, inventory, and notifications services each react at their own pace, without the orders service needing to know they exist.

Common Mistakes and Best Practices

The costliest mistake is underestimating how many partitions a topic needs. The number of partitions sets the ceiling on parallelism: if a topic has three partitions, you’ll never be able to have more than three active, useful consumers in the same consumer group, no matter how many processes you spin up.

Adding partitions later isn’t free: Kafka can add new partitions to an existing topic, but it doesn’t reorder messages already written, so events with the same key that used to always land in the same partition can start being spread across several, breaking the expected order.

Another common mistake is ignoring the partition key. Without an explicit key, kafkajs and most clients distribute messages round robin, which is fine for independent events, but breaks any logic that depends on ordering between related events, like the state changes of a single order.

It’s also common not to monitor consumer group lag until it’s already a production problem. Lag that keeps growing steadily means consumers are processing slower than producers are writing, and the fix, adding partitions and consumers or optimizing processing, takes time: it needs to be caught early.

Finally, treating Kafka as a database for ad hoc queries is a common design mistake. Kafka is built for sequential reads starting from an offset, not for searching by arbitrary fields: for that you still need databases or search engines fed from those same topics.

⚠️ Watch out: setting acks=all on the producer and min.insync.replicas=2 on the topic is what prevents message loss when a partition’s leader broker crashes mid-write. Without that combination, a leader failure can lose the most recently acknowledged messages.

Comparison With Alternatives

Option	When to use it	Advantage	Limitation
Apache Kafka	High event volume, several independent consumers, you need to replay history	Retains the log and lets you re-read from any offset	Operating partitions, brokers, and replication has a steep learning curve
RabbitMQ	Classic task queues with complex routing	Flexible exchanges (topic, fanout, direct) and very low latency	Messages are deleted once acknowledged: no native history replay
Amazon SQS	Teams without their own infrastructure who need a simple queue	Zero maintenance, automatic scaling, fully managed	No strict ordering in standard queues and no replay of already-consumed messages
Apache Pulsar	Multi-tenancy and separation between compute and storage	Separates brokers from storage bookies, each scaling independently	Smaller ecosystem and community than Kafka’s

Going Deeper: Log Compaction and Exactly-Once

A normal topic deletes messages according to a time- or size-based retention policy, seven days by default. But Kafka supports a different mode, log compaction, where instead of deleting by age, it keeps only the last value written for each key.

This turns a topic into something like a state table: if you publish usuario-42 → {plan: free} and then usuario-42 → {plan: pro}, a compacted topic eventually ends up with only the second value. This is the mechanism behind Kafka Streams’ changelog topics, which rebuild an application’s state by reading the topic end to end.

The other unavoidable advanced topic is exactly-once semantics. By default, if a producer retries a send that actually did arrive, for example due to a network timeout, the message can end up duplicated. By enabling enable.idempotence=true on the producer, Kafka assigns a sequence number to each message and discards duplicates on the broker side.

For operations that write to several partitions or topics as a single unit (reading from one topic, transforming, writing to another), Kafka exposes transactions with a transactional.id, which guarantee that all those writes are committed together or none are, even if the producer process crashes halfway through.

💭 Key point: log compaction and time-based retention aren’t mutually exclusive: a single topic can combine both policies, deleting by age while compacting by key, depending on the cleanup.policy configuration.

flowchart LR
    Leader["Partition 0: leader"] --> Follower1["Replica on broker 2"]
    Leader --> Follower2["Replica on broker 3"]
    subgraph ISR["In-sync replicas"]
    Leader
    Follower1
    Follower2
    end

📖 Summary on Telegram: View summary

Your next step: spin up this article’s Kafka container, create a topic with three partitions, and run both Node.js scripts in parallel to see how consumption is distributed across partitions.

Frequently Asked Questions

Is Kafka the same as a message queue?

Not exactly. A queue deletes a message once it’s acknowledged; Kafka keeps it in a log for a configurable period, which allows re-reading it, having several independent consumer groups on the same topic, and reprocessing from any offset.

Do I still need ZooKeeper to run Kafka?

No. Since KRaft reached production in version 3.3, Kafka can manage its own cluster metadata without depending on ZooKeeper, which simplifies both installation and operation.

How many partitions should a topic have?

It depends on the parallelism you need: the number of partitions is the ceiling on simultaneous useful consumers in a single consumer group. It’s a good idea to start with more partitions than you think you need today, because reducing them later isn’t possible without recreating the topic.

Does Kafka guarantee the order of all messages in a topic?

Only within the same partition. If you need ordering across a group of events, for example all the changes to a single order, you have to use the same partition key for all of them.

Can I use Kafka as a database?

Not for arbitrary queries. Kafka reads sequentially from an offset; for searching by fields or doing ad hoc joins you still need a database or a search engine fed from those same topics.

What happens if a leader broker crashes mid-write?

If the producer set acks=all and the topic has min.insync.replicas greater than one, another synced replica takes over as leader without losing the message. Without that configuration, the message acknowledged before the crash can be lost.

References

• Official Apache Kafka documentation: complete reference for configuration, protocol, and APIs.
• Apache Kafka repository on GitHub: the project’s source code and official Docker images.
• Apache Kafka on Wikipedia: the project’s history from its origins at LinkedIn.
• kafkajs documentation: the Node.js client used in this article’s code examples.

📱 Enjoying this content? Follow @programacion on Telegram for daily tech content in Spanish: quick summaries, fresh content every day.

Imagen destacada: Foto de Emile Perron en Unsplash