Job queues: how retries, idempotency, and DLQs work

TL;DR

• You’ll understand the difference between at-least-once, at-most-once, and exactly-once in job queues.
• You’ll know how to calculate a correct visibility timeout so a slow job doesn’t get processed twice.
• You’ll be able to write an idempotent job with a deduplication key in Redis or Postgres.
• You’ll configure retries with exponential backoff and a dead-letter queue in BullMQ and SQS.
• You’ll be able to choose between SQS, RabbitMQ, Redis/BullMQ, and Kafka based on your volume and use case.
• You’ll know how to diagnose a stuck queue by checking depth, in-flight messages, and the DLQ.
• You’ll learn the outbox pattern to avoid the dual-write problem between your database and the queue.

⏱️ Lectura: 15 min

A job runs twice, and your system charges the same customer twice. There’s no bug in the worker code: the problem is in how you designed the queue.

📑 En este artículo

1. TL;DR
2. What a job queue is and why it matters
3. How a job queue works under the hood
   1. Producer, broker, and consumer
   2. Retries and delivery semantics
   3. Visibility timeout: the race against the clock
   4. Idempotency: the real defense against duplicates
   5. Dead-letter queues: the last resort
4. Practical examples: from zero to a worker with retries and idempotency
5. Getting started: spin up a local queue in minutes
   1. Verifying that it works
6. Real-world use cases
7. Common mistakes and best practices
8. Comparison with alternatives
9. Going deeper: the outbox pattern and effective exactly-once
10. Frequently Asked Questions
    1. What’s the difference between a job queue and an event stream like Kafka?
    2. Is it possible to achieve true exactly-once?
    3. How do I choose the visibility timeout value?
    4. When is a dead-letter queue worth using?
    5. How do I make a job idempotent that I already wrote without thinking about it?
    6. Is Redis enough, or do I need a dedicated broker like RabbitMQ or SQS?
11. References

Job queues decouple who requests work from who runs it, but they hide hard problems: what happens if the worker crashes mid-process, how to avoid processing the same message twice, and what to do with a job that always fails. Here is each piece explained with real code, from the producer to the dead-letter queue.

What a job queue is and why it matters

A job queue is a piece of infrastructure that receives a task, stores it, and hands it to a worker when there’s capacity to process it. The difference from calling a function directly is that whoever requests the work (the producer) doesn’t wait for it to finish: it enqueues the message and moves on. Whoever processes it (the consumer or worker) can be in another process, another server, or scale independently.

The typical case: a user uploads a video in your app. Processing it (transcoding, generating thumbnails, extracting subtitles) can take minutes. If you do it inside the same HTTP request, the user stares at a blank screen and any browser or load balancer timeout cuts the connection midway. With a job queue, the HTTP request responds in milliseconds and the heavy work happens in the background, in a worker you can scale horizontally based on load.

So far it sounds simple. The hard part shows up when something fails midway: the worker crashes, the network drops, the message gets delivered twice or never resolves. That’s where most homegrown job queue implementations break.

How a job queue works under the hood

Producer, broker, and consumer

Every job queue has three roles. The producer creates the message (the job) and sends it to the broker: for example, your API when someone uploads a file. The broker is the system that stores the message durably until someone processes it: it can be Redis, RabbitMQ, Amazon SQS, or even a Postgres table. The consumer (worker) requests messages from the broker, processes them, and confirms that it’s done.

That confirmation is called an ack (acknowledgment). It’s the central piece of the whole system: if the worker doesn’t send the ack, the broker assumes the work wasn’t completed and delivers it again. That single mechanism, redelivering when there’s no ack, is the root of almost every hard problem in job queues.

flowchart TD
    A["Productor (API)"] --> B["Cola / broker"]
    B --> C["Worker"]
    C --> D[("Base de datos")]
    C -. "falla repetidamente" .-> E["Dead-letter queue"]

Retries and delivery semantics

When a message is retried, there are three possible guarantees. At-most-once: the message is delivered once or not at all, never duplicated, but it can be lost if the worker crashes before processing it. At-least-once: the message is delivered one or more times, never lost, but it can be duplicated. Exactly-once: the message is processed exactly one time, with no losses or duplicates.

Most production queue systems (SQS, RabbitMQ, BullMQ on Redis) offer at-least-once by default because it’s the easiest guarantee to uphold: to never lose a message, the broker prefers to resend it too often rather than risk losing it. The cost is that your worker will receive the same job twice at some point, and it has to be ready for that.

💭 Key: true exactly-once in a distributed system doesn’t exist without extra coordination. What does exist, and is good enough in practice, is at-least-once delivery plus idempotent processing: the net result is “effectively once.”

Visibility timeout: the race against the clock

When a worker takes a message from the queue, most brokers don’t delete it right away: they hide it for a set period, the visibility timeout, and wait for the ack. If the ack arrives before that time expires, the message is deleted. If it doesn’t arrive (the worker crashed, took too long, lost the connection), the message becomes visible again so another worker can take it.

The most common mistake is setting a visibility timeout shorter than the actual processing time. If your video transcoding job takes 90 seconds but the timeout is 30, a second worker will take the same message while the first is still processing it: both will transcode the same video in parallel, and you’ll probably end up with two charges, two emails, or two duplicate rows in your database.

sequenceDiagram
    participant P as Productor
    participant Q as Cola
    participant W as Worker
    P->>Q: encola job (id=42)
    Q->>W: entrega job, inicia visibility timeout
    Note over W: procesando (puede tardar)
    W-->>Q: ack, borra el mensaje
    Q-->>P: job completado

⚠️ Watch out: the visibility timeout is calculated based on the worst-case processing time, not the average. If your p99 processing time is 90 seconds, a timeout of 30 will duplicate work constantly.

Idempotency: the real defense against duplicates

Since at-least-once guarantees that a job can arrive more than once, the only reliable defense is to make processing it twice produce the same result as processing it once. That’s idempotency: the property that applying an operation several times has the same effect as applying it once.

In practice it’s implemented with a deduplication key: before running the side effect (charging, sending an email, writing a row), the worker checks whether that key has already been processed. If Redis or your database confirms that it has, the worker discards the job without running the effect again. That key is usually built from the job ID plus some business data, never from a timestamp: two retries of the same job shouldn’t generate two different keys.

Dead-letter queues: the last resort

Not every job can be retried forever. A corrupt message, a payload with a validation bug, or a permanently down external integration will fail on every attempt. Without a limit, that message is retried infinitely and consumes workers without producing anything useful: this is known as a poison message.

The solution is to set a maximum number of attempts (max receive count) and, once it’s exceeded, move the message to a dead-letter queue (DLQ): a separate queue where the message stays visible for manual inspection instead of continuing to burn retry cycles. The DLQ isn’t a graveyard: it’s an inbox someone has to review.

stateDiagram-v2
    [*] --> Visible
    Visible --> EnVuelo: worker lo toma
    EnVuelo --> Visible: timeout expira sin ack
    EnVuelo --> Completado: ack recibido
    EnVuelo --> DLQ: excede maxReceiveCount
    Completado --> [*]
    DLQ --> [*]

Practical examples: from zero to a worker with retries and idempotency

We’re going to build this up progressively with BullMQ, a queue library on top of Redis for Node.js. First the bare minimum: enqueue and process a job.

// producer.js: encola un job minimo
import { Queue } from 'bullmq';

const emailQueue = new Queue('emails', {
  connection: { host: '127.0.0.1', port: 6379 },
});

await emailQueue.add('send-welcome', {
  userId: 1042,
  email: '[email protected]',
});

This creates a queue called emails in Redis and enqueues a job. BullMQ serializes the payload to JSON, assigns it a unique ID, and makes it available for a worker to take. There are no retries configured yet and no idempotency: if this job gets processed twice, two emails are sent.

Now the realistic version: retries with exponential backoff, idempotency with Redis, and handling the final-failure event.

// worker.js: procesa jobs con reintentos e idempotencia
import { Worker } from 'bullmq';
import Redis from 'ioredis';

const redis = new Redis();

const worker = new Worker('emails', async (job) => {
  const dedupeKey = `email:sent:${job.data.userId}:${job.data.email}`;
  const isFirstTime = await redis.set(dedupeKey, '1', 'EX', 86400, 'NX');

  if (!isFirstTime) {
    return { skipped: true }; // ya se proceso este job antes
  }

  await sendWelcomeEmail(job.data.email);
}, {
  connection: { host: '127.0.0.1', port: 6379 },
  concurrency: 5,
});

worker.on('failed', (job, err) => {
  console.error(`job ${job.id} fallo en intento ${job.attemptsMade}: ${err.message}`);
});

redis.set(..., 'NX') only writes the key if it doesn’t exist, atomically. That turns “check and mark” into a single operation, with no race condition between two workers that receive the same job at nearly the same time. The 24-hour expiration keeps the key from growing forever in Redis.

For backoff retries and the attempt limit to kick in, you have to configure them when enqueuing the job, not when processing it:

await emailQueue.add('send-welcome', payload, {
  attempts: 5,
  backoff: { type: 'exponential', delay: 1000 },
  removeOnComplete: true,
  removeOnFail: false, // los jobs fallidos quedan visibles para inspeccionar
});

With attempts: 5 and 1-second exponential backoff, BullMQ retries at 1s, 2s, 4s, 8s, and 16s before marking the job as permanently failed. removeOnFail: false leaves those failed jobs in the failed queue, which acts as a dead-letter queue: you can inspect them with emailQueue.getFailed() and retry them by hand once you’ve fixed the cause.

Getting started: spin up a local queue in minutes

To try everything above on your machine, you need Redis running and the two Node libraries.

docker run -d --name redis-queue -p 6379:6379 redis:7

npm install bullmq ioredis

With Redis up, run the producer.js file above to enqueue a job, then worker.js in another terminal to process it. BullMQ automatically creates the data structures it needs in Redis (lists and sorted sets) the first time you use the queue: no migration or extra configuration required.

Verifying that it works

To confirm that jobs are being processed and that the DLQ (the failed queue) is empty, you can query the counters directly:

const counts = await emailQueue.getJobCounts('active', 'completed', 'failed', 'delayed');
console.log(counts); // { active: 0, completed: 12, failed: 0, delayed: 0 }

If you use Amazon SQS instead of Redis, the equivalent is querying the queue’s attributes via the CLI:

aws sqs get-queue-attributes \
  --queue-url $DLQ_URL \
  --attribute-names ApproximateNumberOfMessages

And if you use RabbitMQ, rabbitmqctl list_queues name messages messages_unacknowledged shows in one line how many messages are pending and how many are in flight without acknowledgment.

Real-world use cases

Job queues show up in almost any system that separates responding fast from doing the heavy work. The most common cases:

• Transactional emails and notifications: sending the welcome email or the “your order shipped” push without blocking the request that triggers them.
• Payment webhooks: processing a payment confirmation from Stripe or Mercado Pago asynchronously, with retries if your server was down when the webhook arrived.
• Media processing: transcoding video, generating thumbnails, or extracting metadata from user-uploaded images.
• Report and export generation: building a large CSV or PDF without keeping an HTTP connection open for several minutes.
• Syncing with external systems: sending data to a CRM or a data warehouse with automatic retries if the destination API responds with a 500 error.

Common mistakes and best practices

The most frequent mistake was already mentioned: a visibility timeout shorter than the actual processing time, which constantly generates duplicates. Measure your jobs’ p99 and set the timeout with margin, not the average.

The second mistake is not having a dead-letter queue configured. Without a maxReceiveCount or a failed queue, a poison message consumes workers in an infinite loop and can hurt the performance of the entire queue for the rest of the legitimate jobs.

The third is storing large payloads inside the message instead of a reference. A job that loads a 50 MB file directly into the message overwhelms the broker; the right approach is to upload the file to object storage (S3, GCS) and enqueue only the URL or the ID.

The fourth, less obvious: treating message order as guaranteed. Unless you explicitly use a FIFO queue (SQS FIFO, or ordered partitions in Kafka), most standard queues don’t guarantee that messages are processed in the order they were enqueued, especially with several workers running in parallel.

Finally: not monitoring the queue. Queue depth (how many messages are waiting) and DLQ size are the two minimum metrics you need to alert on. A DLQ that grows silently is work your system stopped doing without anyone noticing.

Comparison with alternatives

Option	When to use it	Advantage	Limitation
Amazon SQS	you’re already on AWS and want zero infrastructure of your own	managed, scales on its own, native DLQ	vendor lock-in, order not guaranteed in the standard queue
RabbitMQ	you need complex routing (exchanges, fanout, topics)	mature AMQP protocol, very flexible	you have to operate the cluster yourself
Redis + BullMQ	Node.js team, low latency, moderate volume	simple to set up, good dashboards	Redis isn’t a durable log like Kafka; persistence depends on your AOF/RDB configuration
Apache Kafka	high-volume event streams, you need replay	long retention, multiple consumers read the same stream	complex to operate; no per-message ack like a classic queue
Postgres (SKIP LOCKED)	you already have Postgres and low or moderate volume	no new infrastructure, transactional alongside your data	doesn’t scale like a dedicated broker at very high volumes

Going deeper: the outbox pattern and effective exactly-once

There’s a problem that comes before all of the above: what happens if your code writes to the database and then fails right before enqueuing the job. The record is saved but no one will ever process it. The reverse (enqueue first, write after) has the symmetric problem: the worker can start before the database transaction has committed.

This is known as the dual-write problem: two different systems, the database and the queue, that need to stay consistent but don’t share a transaction. The standard solution is the outbox pattern: in the same transaction where you write your business data, you also write a row into an outbox table. A separate process reads that table and enqueues the real message, marking the row as sent. Since both writes (the data and the event) happen in the same Postgres or MySQL transaction, either both are saved or neither is, with no inconsistent intermediate state.

With outbox plus idempotency in the consumer, you achieve what in practice is called exactly-once: not because the message is literally delivered a single time (that’s still at-least-once underneath), but because the net effect on the system is the same as if it had been processed once, no matter how many times it was retried.

📖 Summary on Telegram: View summary

Your next step: spin up Redis with Docker from the “Getting started” section, run the worker with attempts: 3, and kill it mid-job to see the retry and the visibility timeout in action.

Frequently Asked Questions

What’s the difference between a job queue and an event stream like Kafka?

A classic job queue delivers each message to a worker and deletes it once it’s acknowledged (ack): the message disappears after being processed. A stream like Kafka retains messages for a configurable period and lets multiple consumers read the same stream independently, and even re-read messages that were already processed (replay).

Is it possible to achieve true exactly-once?

In the strict sense (the message travels over the network exactly once), no, not without costly extra coordination. What you achieve in practice is at-least-once delivery plus idempotent processing, which produces the same net result.

How do I choose the visibility timeout value?

Measure the actual worst-case time your jobs take (p99), not the average, and set the timeout with margin above that number. If you don’t have that metric yet, start conservative (several minutes) and adjust with real production data.

When is a dead-letter queue worth using?

Whenever you retry automatically. Without an attempt limit and a DLQ, a message that always fails (corrupt payload, validation bug) consumes workers in an infinite loop without producing anything useful.

How do I make a job idempotent that I already wrote without thinking about it?

Identify the real side effect (charge, send, write) and add a prior check with a unique deduplication key, built from business data (not a timestamp), stored in Redis with NX or in a column with a unique constraint in your database.

Is Redis enough, or do I need a dedicated broker like RabbitMQ or SQS?

For moderate volumes and teams already using Redis, BullMQ is enough and simple to operate. If you need complex routing, enterprise-grade guaranteed delivery, or very high-volume event streams, a dedicated broker like RabbitMQ, SQS, or Kafka becomes the more solid option.

References

• Amazon SQS: official documentation on visibility timeout, retries, and dead-letter queues in a managed service.
• RabbitMQ Documentation: guides on acknowledgments, durable queues, and AMQP routing patterns.
• BullMQ Documentation: complete reference for the Redis-based queue library used in this article’s examples.
• Idempotence, Wikipedia: formal definition of idempotency and its mathematical origin.
• Sidekiq: a reference implementation of a job queue system for Ruby, with retries and failed queues.

📱 Enjoying this content? Follow @programacion on Telegram for daily tech content in Spanish: quick summaries, fresh content every day. @programacion

Imagen destacada: Foto de Reza Asadi en Unsplash