Skip to content
FLAVIO COPES
flaviocopes.com
2026

Background jobs in Node.js with BullMQ

By

Build a reliable Node.js background job queue with BullMQ and Redis, including workers, retries, backoff, progress, concurrency, and idempotency.

~~~

Some work should not happen while a person waits for an HTTP response.

Sending an email, resizing an image, generating a PDF, and importing a large CSV file can all take time. They can also fail for reasons outside our application.

If we perform that work inside a request handler, the request becomes slow and fragile.

A background job queue lets the web process describe the work and return quickly. A separate worker performs it.

In this tutorial we’ll build an email queue with BullMQ and Redis.

We’ll add retries, exponential backoff, progress, concurrency, and the most important production feature: making jobs safe to run more than once.

How the queue fits together

Our system has three parts:

HTTP application -> Redis queue -> worker

The application is the producer. It adds jobs.

Redis stores job data and queue state.

The worker reserves waiting jobs and processes them.

BullMQ is not a hosted queue. It is a Node.js library that coordinates queue behavior using Redis.

Start Redis

The easiest local setup uses Docker:

docker run --name jobs-redis \
  -p 6379:6379 \
  redis:7-alpine

Create the Node.js project:

mkdir bullmq-email-queue
cd bullmq-email-queue
npm init -y
npm install bullmq express

Add "type": "module" to package.json.

Create the queue

Create queue.js:

import { Queue } from 'bullmq'

export const connection = {
  host: process.env.REDIS_HOST || '127.0.0.1',
  port: Number(process.env.REDIS_PORT || 6379),
}

export const emailQueue = new Queue('emails', {
  connection,
  defaultJobOptions: {
    attempts: 5,
    backoff: {
      type: 'exponential',
      delay: 1000,
    },
    removeOnComplete: {
      age: 24 * 60 * 60,
      count: 1000,
    },
    removeOnFail: {
      age: 7 * 24 * 60 * 60,
    },
  },
})

The queue name, emails, identifies its Redis keys. Producers and workers must use the same name and Redis connection.

The defaults say:

Without cleanup rules, old job records can grow forever.

Add jobs from an API

Create server.js:

import express from 'express'
import { emailQueue } from './queue.js'

const app = express()
app.use(express.json())

app.post('/welcome-email', async (request, response) => {
  const { userId, email } = request.body

  if (!userId || !email) {
    return response.status(400).json({
      error: 'userId and email are required',
    })
  }

  const job = await emailQueue.add(
    'send-welcome-email',
    {
      userId,
      email,
    },
    {
      jobId: `welcome-${userId}`,
    }
  )

  response.status(202).json({
    jobId: job.id,
    status: 'queued',
  })
})

app.listen(3000, () => {
  console.log('API listening on http://localhost:3000')
})

Run it:

node server.js

Add a job:

curl -X POST http://localhost:3000/welcome-email \
  -H 'content-type: application/json' \
  -d '{"userId":"42","email":"me@example.com"}'

The API returns 202 Accepted.

That status is a useful promise: the request was accepted for processing, but the work is not finished yet.

Create a worker

Create worker.js:

import { Worker } from 'bullmq'
import { connection } from './queue.js'

async function sendWelcomeEmail({ email }) {
  console.log(`Sending welcome email to ${email}`)

  await new Promise((resolve) => {
    setTimeout(resolve, 1000)
  })
}

const worker = new Worker(
  'emails',
  async (job) => {
    if (job.name !== 'send-welcome-email') {
      throw new Error(`Unknown job: ${job.name}`)
    }

    await job.updateProgress(10)
    await sendWelcomeEmail(job.data)
    await job.updateProgress(100)

    return {
      sentAt: new Date().toISOString(),
    }
  },
  {
    connection,
    concurrency: 5,
  }
)

worker.on('completed', (job, result) => {
  console.log(`Job ${job.id} completed`, result)
})

worker.on('failed', (job, error) => {
  console.error(`Job ${job?.id} failed`, error.message)
})

const shutdown = async () => {
  await worker.close()
  process.exit(0)
}

process.on('SIGTERM', shutdown)
process.on('SIGINT', shutdown)

Run it in another terminal:

node worker.js

The waiting job is processed.

Throwing an error marks the attempt as failed. If attempts remain, BullMQ moves the job back to wait after its backoff delay.

Why retries need backoff

Imagine an email provider is unavailable for one minute.

Immediate retries hit the same failure repeatedly and add more pressure. Exponential backoff spaces attempts out:

1 second
2 seconds
4 seconds
8 seconds

Add a little randomness, called jitter, when a large number of jobs might fail together. Otherwise they can all retry at the same instant.

Retry errors that may disappear:

Do not retry permanent errors forever:

Validate job data before adding it, and classify worker errors when the external service gives enough information.

Make jobs idempotent

Queues provide at-least-once processing.

A worker can finish an external action and crash before Redis records completion. The job may run again.

This means send welcome email must be safe to repeat.

The jobId in the producer prevents two currently retained jobs with the same ID from being added. It helps, but it is not a complete idempotency strategy. Completed jobs may eventually be removed.

For a durable guarantee, record the business action in your database:

async function sendWelcomeEmail({ userId, email }) {
  const existing = await database.emailDelivery.findUnique({
    where: {
      operationKey: `welcome-${userId}`,
    },
  })

  if (existing) {
    return
  }

  const providerMessageId = await emailProvider.send({
    to: email,
    template: 'welcome',
  })

  await database.emailDelivery.create({
    data: {
      operationKey: `welcome-${userId}`,
      providerMessageId,
    },
  })
}

The exact transaction depends on the external service. If the provider supports idempotency keys, send the same operation key there too.

Never design a background job under the assumption that it runs exactly once.

Report progress and status

Add a status endpoint:

app.get('/jobs/:id', async (request, response) => {
  const job = await emailQueue.getJob(request.params.id)

  if (!job) {
    return response.status(404).json({
      error: 'Job not found',
    })
  }

  response.json({
    id: job.id,
    name: job.name,
    state: await job.getState(),
    progress: job.progress,
    result: job.returnvalue,
    failedReason: job.failedReason,
  })
})

This is useful for work that has a visible result, such as a report export.

Do not expose raw job data publicly. It may contain email addresses, file paths, or other private information. Authorize the request and return only the fields the owner needs.

Listen to queue-wide events

Worker events are local to that worker process.

Use QueueEvents when another process needs queue-wide completion or failure events:

import { QueueEvents } from 'bullmq'
import { connection } from './queue.js'

const queueEvents = new QueueEvents('emails', {
  connection,
})

queueEvents.on('completed', ({ jobId }) => {
  console.log(`Job ${jobId} completed`)
})

queueEvents.on('failed', ({ jobId, failedReason }) => {
  console.error(`Job ${jobId} failed: ${failedReason}`)
})

BullMQ uses Redis streams for these events, which gives them different behavior from ordinary Redis pub/sub.

Modern BullMQ does not require a QueueScheduler for delayed jobs and stalled-job recovery. Old tutorials that add one by default are outdated.

Concurrency and multiple workers

Our worker uses:

concurrency: 5

One process can work on five jobs at once.

This helps I/O-heavy jobs that spend time waiting for an API. It does not make CPU-heavy JavaScript run in parallel on one event loop.

For CPU-heavy image or document processing, use multiple worker processes or worker threads.

You can also run several copies of worker.js. BullMQ ensures that one worker reserves a particular job at a time.

Start with low concurrency and measure:

More concurrency can make the whole system slower when the database becomes the bottleneck.

Priorities, delays, and scheduled work

Delay one job:

await emailQueue.add(
  'send-reminder',
  {
    userId: '42',
  },
  {
    delay: 60 * 60 * 1000,
  }
)

Give an urgent job higher priority:

await emailQueue.add('send-password-reset', data, {
  priority: 1,
})

Lower numeric values have higher priority.

BullMQ also supports repeatable job schedulers. Use those for recurring application jobs, but remember that Redis is then part of the scheduling infrastructure. Monitor it and back it up according to your recovery needs.

Deploy producers and workers separately

The web application and worker should be separate processes:

web:    node server.js
worker: node worker.js

This lets you scale them independently.

The API may need ten instances while two workers are enough. A large import could require twenty temporary workers without changing the web tier.

In production:

If Redis evicts queue keys because it ran out of memory, the queue loses state. A job queue is not cache data.

The important design rule

Moving work to a queue does not make failure disappear.

It gives failure a place to be retried, inspected, and recovered without holding an HTTP request open.

A reliable job should have:

With those pieces, BullMQ turns Redis into a practical background processing system for Node.js, while keeping slow and failure-prone work away from the request that started it.

Tagged: Node.js · All topics
~~~

Related posts about node: