Job queues: how retries, idempotency, and DLQs work

TL;DR

• You’ll understand the difference between at-least-once, at-most-once, and exactly-once in job queues.
• You’ll know how to calculate a correct visibility timeout so a slow job isn’t processed twice.
• You’ll be able to write an idempotent job with a deduplication key in Redis or Postgres.
• You’ll configure retries with exponential backoff and a dead-letter queue in BullMQ and SQS.
• You’ll be able to choose between SQS, RabbitMQ, Redis/BullMQ, and Kafka based on your volume and use case.
• You’ll know how to diagnose a stuck queue by checking depth, in-flight messages, and the DLQ.
• You’ll learn the outbox pattern to avoid the dual-write problem between your database and the queue.

⏱️ Lectura: 15 min

A job runs twice and your system charges the same customer twice. The worker code has no bug at all: the problem is how you designed the queue.

📑 En este artículo

1. TL;DR
2. What a job queue is and why it matters
3. How a job queue works under the hood
   1. Producer, broker, and consumer
   2. Retries and delivery semantics
   3. Visibility timeout: the race against the clock
   4. Idempotency: the real defense against duplicates
   5. Dead-letter queues: the last resort
4. Practical examples: from zero to a worker with retries and idempotency
5. Getting started: spin up a local queue in minutes
   1. Verify it works
6. Real-world use cases
7. Common mistakes and best practices
8. Comparison with alternatives
9. Going deeper: the outbox pattern and effective exactly-once
10. Frequently Asked Questions
    1. What’s the difference between a job queue and an event stream like Kafka?
    2. Is real exactly-once actually achievable?
    3. How do I choose the visibility timeout value?
    4. When should I use a dead-letter queue?
    5. How do I make a job idempotent that I already wrote without thinking about it?
    6. Is Redis enough, or do I need a dedicated broker like RabbitMQ or SQS?
11. References

Job queues decouple who requests work from who runs it, but they hide hard problems: what happens if the worker crashes mid-process, how to avoid processing the same message twice, and what to do with a job that always fails. Here is each piece explained with real code, from the producer all the way to the dead-letter queue.

What a job queue is and why it matters

A job queue is a piece of infrastructure that receives a task, stores it, and hands it to a worker when there’s capacity to process it. The difference from calling a function directly is that whoever requests the work (the producer) doesn’t wait for it to finish: it enqueues the message and moves on. Whoever processes it (the consumer or worker) can be in another process, another server, or scaled independently.

The classic case: a user uploads a video to your app. Processing it (transcoding, generating thumbnails, extracting subtitles) can take minutes. If you do it inside the same HTTP request, the user stares at a blank screen and any browser or load balancer timeout cuts the connection halfway through. With a job queue, the HTTP request responds in milliseconds and the heavy work happens in the background, on a worker you can scale horizontally with the load.

So far it sounds simple. The hard part shows up when something fails midway: the worker crashes, the network drops, the message gets delivered twice, or it never resolves. That’s where most homegrown job queue implementations break.

How a job queue works under the hood

Producer, broker, and consumer

Every job queue has three roles. The producer creates the message (the job) and sends it to the broker: for example, your API when someone uploads a file. The broker is the system that stores the message durably until someone processes it: it can be Redis, RabbitMQ, Amazon SQS, or even a Postgres table. The consumer (worker) pulls messages from the broker, processes them, and confirms it’s done.

That confirmation is called an ack (acknowledgment). It’s the central piece of the whole system: if the worker doesn’t send the ack, the broker assumes the work didn’t complete and delivers it again. That single mechanism, redelivering when there’s no ack, is the root of almost every hard problem in job queues.

flowchart TD
    A["Producer (API)"] --> B["Queue / broker"]
    B --> C["Worker"]
    C --> D[("Database")]
    C -. "fails repeatedly" .-> E["Dead-letter queue"]

Retries and delivery semantics

When a message is retried, there are three possible guarantees. At-most-once: the message is delivered once or not at all, never duplicated, but it can be lost if the worker crashes before processing it. At-least-once: the message is delivered one or more times, never lost, but it can be duplicated. Exactly-once: the message is processed exactly once, with no losses and no duplicates.

Most production queue systems (SQS, RabbitMQ, BullMQ on Redis) offer at-least-once by default because it’s the easiest guarantee to uphold: to never lose a message, the broker prefers to resend it too often rather than risk losing it. The cost is that your worker will receive the same job twice at some point, and it has to be ready for that.

💭 Key point: true exactly-once in a distributed system does not exist without additional coordination. What does exist, and is enough in practice, is at-least-once delivery plus idempotent processing: the net result is “effectively once.”

Visibility timeout: the race against the clock

When a worker takes a message from the queue, most brokers don’t delete it right away: they hide it for a set period, the visibility timeout, and wait for the ack. If the ack arrives before that time expires, the message is deleted. If it doesn’t (the worker crashed, took too long, lost the connection), the message becomes visible again so another worker can take it.

The most common mistake is setting a visibility timeout shorter than the actual processing time. If your video transcoding job takes 90 seconds but the timeout is 30, a second worker will take the same message while the first one is still processing it: both will transcode the same video in parallel, and you’ll probably end up with two charges, two emails, or two duplicate rows in your database.

sequenceDiagram
    participant P as Producer
    participant Q as Queue
    participant W as Worker
    P->>Q: enqueue job (id=42)
    Q->>W: deliver job, start visibility timeout
    Note over W: processing (may take a while)
    W-->>Q: ack, delete the message
    Q-->>P: job completed

⚠️ Watch out: the visibility timeout is calculated from the worst-case processing time, not the average. If your p99 processing time is 90 seconds, a 30-second timeout will duplicate work constantly.

Idempotency: the real defense against duplicates

Since at-least-once guarantees a job can arrive more than once, the only reliable defense is to make processing it twice produce the same result as processing it once. That’s idempotency: the property that applying an operation multiple times has the same effect as applying it once.

In practice it’s implemented with a deduplication key: before running the side effect (charging, sending an email, writing a row), the worker checks whether that key has already been processed. If Redis or your database confirms it has, the worker discards the job without running the effect again. That key is usually built from the job ID plus some business data, never from a timestamp: two retries of the same job shouldn’t generate two different keys.

Dead-letter queues: the last resort

Not every job can be retried forever. A corrupt message, a payload with a validation bug, or a permanently down external integration will fail on every attempt. Without a limit, that message is retried infinitely and consumes workers without producing anything useful: it’s known as a poison message.

The fix is to set a maximum number of attempts (max receive count) and, once it’s exceeded, move the message to a dead-letter queue (DLQ): a separate queue where the message stays visible for manual inspection instead of continuing to burn retry cycles. The DLQ isn’t a graveyard: it’s an inbox someone has to review.

stateDiagram-v2
    [*] --> Visible
    Visible --> InFlight: worker takes it
    InFlight --> Visible: timeout expires without ack
    InFlight --> Completed: ack received
    InFlight --> DLQ: exceeds maxReceiveCount
    Completed --> [*]
    DLQ --> [*]

Practical examples: from zero to a worker with retries and idempotency

We’ll build this up progressively with BullMQ, a Redis-based queue library for Node.js. First the bare minimum: enqueue and process a job.

// producer.js: enqueue a minimal job
import { Queue } from 'bullmq';

const emailQueue = new Queue('emails', {
  connection: { host: '127.0.0.1', port: 6379 },
});

await emailQueue.add('send-welcome', {
  userId: 1042,
  email: '[email protected]',
});

This creates a queue called emails in Redis and enqueues a job. BullMQ serializes the payload to JSON, assigns it a unique ID, and makes it available for a worker to pick up. There are no retries configured yet and no idempotency: if this job is processed twice, two emails are sent.

Now the realistic version: retries with exponential backoff, idempotency with Redis, and handling the final-failure event.

// worker.js: process jobs with retries and idempotency
import { Worker } from 'bullmq';
import Redis from 'ioredis';

const redis = new Redis();

const worker = new Worker('emails', async (job) => {
  const dedupeKey = `email:sent:${job.data.userId}:${job.data.email}`;
  const isFirstTime = await redis.set(dedupeKey, '1', 'EX', 86400, 'NX');

  if (!isFirstTime) {
    return { skipped: true }; // this job was already processed before
  }

  await sendWelcomeEmail(job.data.email);
}, {
  connection: { host: '127.0.0.1', port: 6379 },
  concurrency: 5,
});

worker.on('failed', (job, err) => {
  console.error(`job ${job.id} failed on attempt ${job.attemptsMade}: ${err.message}`);
});

redis.set(..., 'NX') only writes the key if it doesn’t exist, atomically. That turns “check and mark” into a single operation, with no race condition between two workers that receive the same job almost at the same time. The 24-hour expiration keeps the key from growing forever in Redis.

For the backoff retries and the attempt limit to kick in, you have to configure them when enqueuing the job, not when processing it:

await emailQueue.add('send-welcome', payload, {
  attempts: 5,
  backoff: { type: 'exponential', delay: 1000 },
  removeOnComplete: true,
  removeOnFail: false, // failed jobs stay visible for inspection
});

With attempts: 5 and a 1-second exponential backoff, BullMQ retries at 1s, 2s, 4s, 8s, and 16s before marking the job as permanently failed. removeOnFail: false leaves those failed jobs in the failed queue, which acts as a dead-letter queue: you can inspect them with emailQueue.getFailed() and retry them by hand once you fix the root cause.

Getting started: spin up a local queue in minutes

To try everything above on your machine, you need Redis running and the two Node libraries.

docker run -d --name redis-queue -p 6379:6379 redis:7

npm install bullmq ioredis

With Redis up, run the producer.js file from above to enqueue a job, then worker.js in another terminal to process it. BullMQ automatically creates the data structures it needs in Redis (lists and sorted sets) the first time you use the queue: no migration or extra configuration required.

Verify it works

To confirm jobs are being processed and the DLQ (the failed queue) is empty, you can query the counters directly:

const counts = await emailQueue.getJobCounts('active', 'completed', 'failed', 'delayed');
console.log(counts); // { active: 0, completed: 12, failed: 0, delayed: 0 }

If you use Amazon SQS instead of Redis, the equivalent is querying the queue attributes via CLI:

aws sqs get-queue-attributes \
  --queue-url $DLQ_URL \
  --attribute-names ApproximateNumberOfMessages

And if you use RabbitMQ, rabbitmqctl list_queues name messages messages_unacknowledged shows in one line how many messages are pending and how many are in flight without confirmation.

Real-world use cases

Job queues show up in almost any system that separates responding fast from doing the heavy work. The most common cases:

• Transactional emails and notifications: sending the welcome email or the “your order shipped” push without blocking the request that triggers them.
• Payment webhooks: processing a Stripe or Mercado Pago payment confirmation asynchronously, with retries if your server was down when the webhook arrived.
• Media processing: transcoding video, generating thumbnails, or extracting metadata from user-uploaded images.
• Report and export generation: building a large CSV or PDF without keeping an HTTP connection open for several minutes.
• Syncing with external systems: pushing data to a CRM or a data warehouse with automatic retries if the destination API responds with a 500 error.

Common mistakes and best practices

The most frequent mistake was already mentioned: a visibility timeout shorter than the actual processing time, which generates constant duplicates. Measure your jobs’ p99 and set the timeout with a margin, not the average.

The second mistake is not having a dead-letter queue configured. Without a maxReceiveCount or a failed queue, a poison message consumes workers in an infinite loop and can hurt the performance of the entire queue for the rest of the legitimate jobs.

The third is storing large payloads inside the message instead of a reference. A job that loads a 50 MB file directly into the message saturates the broker; the right approach is to upload the file to object storage (S3, GCS) and enqueue only the URL or the ID.

The fourth, less obvious one: treating message order as guaranteed. Unless you explicitly use a FIFO queue (SQS FIFO, or ordered partitions in Kafka), most standard queues don’t guarantee that messages are processed in the order they were enqueued, especially with several workers running in parallel.

Finally: not monitoring the queue. Queue depth (how many messages are waiting) and DLQ size are the two minimum metrics you need to alert on. A DLQ that grows silently is work your system stopped doing without anyone noticing.

Comparison with alternatives

Option	When to use it	Advantage	Limitation
Amazon SQS	you’re already on AWS and want zero infrastructure of your own	managed, scales on its own, native DLQ	vendor lock-in, order not guaranteed in the standard queue
RabbitMQ	you need complex routing (exchanges, fanout, topics)	mature AMQP protocol, very flexible	you have to operate the cluster yourself
Redis + BullMQ	Node.js team, low latency, moderate volume	simple to stand up, good dashboards	Redis isn’t a durable log like Kafka; persistence depends on your AOF/RDB configuration
Apache Kafka	high-volume event streams, you need replay	long retention, multiple consumers read the same stream	complex to operate; no per-message ack like a classic queue
Postgres (SKIP LOCKED)	you already have Postgres and low or moderate volume	no new infrastructure, transactional alongside your data	doesn’t scale like a dedicated broker at very high volumes

Going deeper: the outbox pattern and effective exactly-once

There’s a problem that comes before all of the above: what happens if your code writes to the database and then fails right before enqueuing the job. The record is saved but nobody will ever process it. The reverse (enqueue first, write later) has the symmetric problem: the worker can start before the database transaction has committed.

This is known as the dual-write problem: two different systems, the database and the queue, that need to stay consistent but don’t share a transaction. The standard solution is the outbox pattern: in the same transaction where you write your business data, you also write a row into an outbox table. A separate process reads that table and enqueues the actual message, marking the row as sent. Since both writes (the data and the event) happen in the same Postgres or MySQL transaction, either both are saved or neither is, with no inconsistent intermediate state.

With outbox plus idempotency on the consumer, you get what’s called exactly-once in practice: not because the message is literally delivered only once (that’s still at-least-once underneath), but because the net effect on the system is the same as if it had been processed once, no matter how many times it was retried.

📖 Summary on Telegram: View summary

Your next step: spin up Redis with Docker from the “Getting started” section, run the worker with attempts: 3, and kill it in the middle of a job to see the retry and the visibility timeout in action.

Frequently Asked Questions

What’s the difference between a job queue and an event stream like Kafka?

A classic job queue delivers each message to a worker and deletes it when it’s confirmed (ack): the message disappears after being processed. A stream like Kafka retains messages for a configurable period and lets multiple consumers read the same stream independently, even re-read already-processed messages (replay).

Is real exactly-once actually achievable?

In the strict sense (the message travels over the network exactly once), no, not without costly additional coordination. What you achieve in practice is at-least-once delivery plus idempotent processing, which produces the same net result.

How do I choose the visibility timeout value?

Measure the actual worst-case time your jobs take (p99), not the average, and set the timeout with a margin over that number. If you don’t have that metric yet, start conservative (several minutes) and tune with real production data.

When should I use a dead-letter queue?

Whenever you retry automatically. Without an attempt limit and a DLQ, a message that always fails (corrupt payload, validation bug) consumes workers in an infinite loop without producing anything useful.

How do I make a job idempotent that I already wrote without thinking about it?

Identify the actual side effect (charge, send, write) and add a prior check with a unique deduplication key, built from business data (not a timestamp), stored in Redis with NX or in a column with a unique constraint in your database.

Is Redis enough, or do I need a dedicated broker like RabbitMQ or SQS?

For moderate volumes and teams already using Redis, BullMQ is enough and simple to operate. If you need complex routing, enterprise-grade guaranteed delivery, or very high-volume event streams, a dedicated broker like RabbitMQ, SQS, or Kafka becomes the more solid choice.

References

• Amazon SQS: official documentation on visibility timeout, retries, and dead-letter queues in a managed service.
• RabbitMQ Documentation: guides on acknowledgments, durable queues, and routing patterns with AMQP.
• BullMQ Documentation: complete reference for the Redis-based queue library used in this article’s examples.
• Idempotence, Wikipedia: formal definition of idempotency and its mathematical origin.
• Sidekiq: reference implementation of a job queue system for Ruby, with retries and failed queues.

📱 Enjoying this content? Follow @programacion on Telegram for daily tech content in Spanish: quick summaries, fresh content every day. @programacion

Featured image: Photo by Reza Asadi on Unsplash