Background tasks with Node.js

Sometimes we are faced with a CPU-intensive task when handling HTTP requests. Handling these straight away would cause the event loop to halt therefore preventing us from handling additional requests.

To prevent this we can leverage queues and workers to delegate the execution of said task.

Queues are a powerful design pattern that help us deal with scalability issues. These are some problems they can help solve:

1. Smooth out processing peaks.
2. Break up tasks that may block the event loop.
3. Provide a reliable communications channel across multiple services.

There are two main components with this pattern — queues and processors. A queue will hold tasks and related information while processors can be assigned to those queues in order to process the tasks.

Bull is a Node library that implements a performant queueing system based on Redis. Let's get started with it!

Installation

Before we start, we need to have a Redis instance running. You can follow RedisLabs' Get Started guide.

Then let's go ahead and install Bull:

yarn add bull
yarn add -D @types/bull

Creating a queue

Create a simple queue by instantiating a new instance.

import Queue, { Job } from 'bull';

const q = new Queue('sample', 'redis://localhost:6379');

Scheduling a task

Sending emails is an operation with an unpredictable duration. Sometimes it's useful to delay or to delegate its execution. We'll create such a task.

interface EmailTask {
  email: string;
  content: string;
}

const data: EmailTask = {
  email: 'hello@example.com',
  content: '...'
};

await q.add(data);

Processing a task

We can have the same service processing the tasks but we can also delegate that responsibility to another service. We simply need to make sure they're connecting to the same queue:

q.process(async (job: Job<EmailTask>) => {
  const { email, content } = job.data;
  // TODO use your preferred email provider to send that email.
}).catch((e) => console.log(`Something went wrong: ${e}`));

Recurring tasks

Bull also allows us to schedule recurring tasks. Quite often we're faced with certain maintenance tasks that should be performed, for instance, every day at a given time.

We can create such a task as follows:

await q.add('my_recurring_task', {}, { repeat: { cron: '0 0 * * *' } });

We should also specify a named processor which will only pick up tasks of the my_recurring_task kind:

q.process('my_recurring_task', async (job: Job) => {
  // TODO handle your maintenance operation.
}).catch((e) => console.log(`Something went wrong: ${e}`));

Conclusion

By now you should have a good overview of Bull's capabilities and how to leverage them to improve your product.

You can check out more patterns and examples in Bull's GitHub page.

If you liked this article or have something to add, we are available, as always, via our Support Channel.