Blog /

PM2 Modules & Resolving Lifecycle Hooks

Wouter van der Meulen

Jan 13 2023

Posted in Engineering & Technology

Building a PM2 module to resolve lifecycle hooks.

Recently I talked about resolving lifecycle hooks. But that required setting up a bash script and a cron job to properly function. What if you already have a process manager like PM2 running your processes? You might as well use that to resolve lifecycle hooks, right?

In this post, I'll show you how I'm building a PM2 module to resolve lifecycle hooks. Hopefully this will help you create your own modules for PM2.

What is PM2

If you're not familiar with PM2, it's a process manager for Node.js applications. Besides simply managing a process, it offers bidirectional communication, which allows you to let your application give PM2 status updates. This is convenient for handling errors, as you can tell PM2 your application is in a broken state. Most of these features are designed to work with PM2's enterprise dashboard, but you can also use them without it.

PM2 has a built-in module system, which allows you to run your own monitoring logic. This is what we're going to use to monitor and manage lifecycle hooks. The modules are primarily used to monitor 3rd party services, such as databases, but we're using it to monitor and resolve Lifecycle Hooks.

The documentation for PM2 is a bit confusing, there's pm2.io and pmw.keymetrics.io, both of which are the same tool. But the documentation is split over these websites, so you'll have to look around a bit to find what you're looking for.

Caveat

There's 2 versions of the module package, the legacy one and the new one. The first npm package is pmx, and the new one is @pm2/io. Keymetrics claims that the new one is all-inclusive and will replace pmx, but there's an open issue that's blocking important functionality. All the current 3rd party modules are still using the old package to circumvent this.

For this use case, we don't need that functionality, but it's something to keep in mind.

The PM2 Lifecycle Module

Now, unto the module itself. I've created a GitHub repository for the module, so you can check it out there in full. Just keep in mind that it's an ongoing project.

It's main purpose is to show you how to build a PM2 module.

The module is pretty simple, it just monitors a termination lifecycle hook and resolves it when it has shut down all the pm2 workers. Since this module is ongoing, I don't want to just copy-paste the code here, so I'll just want chop it up a bit and explain what's going on.

Note: I'm using Typescript in this project, but that obviously isn't a requirement. It's just my preference.

AWS Metadata

As of writing, the AWS SDK for Javascript is at version 3. Which doesn't really matter much for the module, but it does mean that retrieving metadata is a bit different. From what I can tell, the new SDK doesn't have a way to get the metadata, so we're going to have to get it ourselves through the metadata service.

EC2 instances have a metadata service that you can use to get information about the instance. You can find the documentation here. It's essentially a basic HTTP server that you can query for information about the instance.

Since the SDK doesn't have a retrieval method, we're going to create a class that will do it for us.

import axios, { AxiosResponse } from 'axios';
import Context from '../types';

export default class Metadata {
  private readonly metadataURL: string;

  constructor(config?: { metadataURL: string; } | undefined) {
    this.metadataURL = config?.metadataURL || 'http://169.254.169.254';
  }

  async getAWSMetadata(path : string) : Promise<string | undefined> {
    try {
      const resp: AxiosResponse = await axios.get(
        `${this.metadataURL}/latest/meta-data${path}`,
        {
          timeout: 10000,
          responseType: 'json',
        },
      );
      return resp.data.toString();
    } catch (err) {
      if (axios.isAxiosError(err)) {
        if (err?.response?.status === 404) {
          console.error('Error retrieving metadata', err.message);
        } else {
          console.error('Unknown error retrieving metadata', err.message);
        }
      }
      return undefined;
    }
  }

  async isAWS() : Promise<boolean> {
    try {
      const resp: AxiosResponse = await axios.get(`${this.metadataURL}/latest/meta-data/instance-id`, { timeout: 10000 });
      return resp.status === 200;
    } catch (err) {
      if (axios.isAxiosError(err)) {
        if (err?.response?.status === 404) {
          console.error('Error retrieving metadata', err.message);
        } else {
          console.error('Unable to fetch -', err.message);
        }
      }
      return false;
    }
  }

  async getLifecycleState(): Promise<string | undefined> {
    try {
      return await this.getAWSMetadata('/autoscaling/target-lifecycle-state');
    } catch (e: unknown) {
      const err = e as Error;
      console.error('Error retrieving lifecycle state', err.message);
      return undefined;
    }
  }

  async getInstanceContext(): Promise<Context> {
    const ctx = new Context();
    const resp = await axios.get(`${this.metadataURL}/latest/dynamic/instance-identity/document`, { timeout: 10000 });
    ctx.instanceId = resp.data.instanceId;
    ctx.region = resp.data.region;
    return ctx;
  }
}

Now you can simply query the metadata service with getLifecycleState and it will return the current state of the lifecycle hook. We're also using getInstanceContext for setting a context with instanceId and region.

// Assume the following type to be available.
export default class Context {
  instanceId!: string
  region!: string
}

For the autoscaling group name we need to actually use the AWS SDK.

const command = new DescribeAutoScalingInstancesCommand({
  InstanceIds: [this.instanceId],
});
const result = await this.client.send(command);
const instance = result.AutoScalingInstances?.[0];
return instance?.AutoScalingGroupName || undefined;

Lifecycle Hooks

Now that we have all the relevant data, we can look for and resolve our hooks. This is basically the same as the previous example, just using the data we collected.

  console.log('Retrieving autoscaling group info...');
  const groupName = await this.getAutoScalingGroup();
  if (groupName) {
    console.log('Checking lifecycle hooks for ', groupName);
    const input = { AutoScalingGroupName: groupName };
    const command = new DescribeLifecycleHooksCommand(input);
    const result = await this.client.send(command);
    const hooks = result.LifecycleHooks ?? [];
    const hook = hooks.find((h: LifecycleHook) => h.LifecycleTransition === 'autoscaling:EC2_INSTANCE_TERMINATING');
    if (hook) {
      console.log('Completing lifecycle hook: ', hook.LifecycleHookName);
      await this.completeLifecycleHook(hook);
    } else {
      console.log('No lifecycle hooks to complete');
    }
  } else {
    console.log('No autoscaling group for this instance');
  }

And complete the Lifecycle hook:

const input = {
  AutoScalingGroupName: hook.AutoScalingGroupName,
  LifecycleActionResult: 'CONTINUE',
  LifecycleHookName: hook.LifecycleHookName,
  InstanceId: this.instanceId,
};
const command = new CompleteLifecycleActionCommand(input);
await this.client.send(command);

That takes care of the lifecycle hooks, now we just need to stop the processes.

PM2 API

import pm2 from 'pm2';

export default async function stopPM2Processes(): Promise<void> {
  return new Promise((resolve, reject) => {
    pm2.connect((err) => {
      if (err) {
        reject(err);
      } else {
        pm2.stop('all', (stopErr) => {
          pm2.disconnect();
          if (stopErr) {
            reject(stopErr);
          } else {
            resolve();
          }
        });
      }
    });
  });
}

Putting it all together

Now that we have all the pieces, we can put them together. You want to create a loop that checks if the instance is in the Terminated state. If it is, you want to stop all the processes and complete the lifecycle hook. In my case, it's a timer function that executes every 60 seconds unless the instance is in the Terminated state.

import {
  AutoScalingClient,
  CompleteLifecycleActionCommand, DescribeAutoScalingInstancesCommand,
  DescribeLifecycleHooksCommand,
  LifecycleHook,
} from '@aws-sdk/client-auto-scaling';
import Context from '../types';
import Metadata from './Metadata';
import stopPM2Processes from '../pm2';

/**
 * Class to handle Terminating lifecycle hooks for AWS Autoscaling
 */
export default class LifecycleHandler {
  private client: AutoScalingClient;

  private readonly instanceId: string;

  private terminating: boolean;

  private timer: NodeJS.Timer | undefined;

  private metadata: Metadata;

  private readonly checkInterval: number;

  /**
   * Constructor
   * @param ctx The EC2 instance context
   * @param metadata The metadata instance
   * @param options
   */
  constructor(ctx: Context, metadata: Metadata, options?: { checkInterval: number; } | undefined) {
    this.instanceId = ctx.instanceId;
    this.client = new AutoScalingClient({ region: ctx.region });
    this.metadata = metadata || new Metadata();
    this.terminating = false;
    this.checkInterval = options?.checkInterval || 60000;
  }

  start() {
    this.timer = setInterval(async () => {
      try {
        await this.checkLifecycles();
      } catch (e) {
        const err = e as Error;
        console.error('Error handling lifecycle hooks: ', err.message);
      }
    }, this.checkInterval);
  }

  private stop() {
    if (this.timer) {
      clearInterval(this.timer);
    }
  }

  /**
   * Get the name of the autoscaling group this instance  is in
   */
  async getAutoScalingGroup(): Promise<string | undefined> {
    const command = new DescribeAutoScalingInstancesCommand({
      InstanceIds: [this.instanceId],
    });
    const result = await this.client.send(command);
    const instance = result.AutoScalingInstances?.[0];
    return instance?.AutoScalingGroupName || undefined;
  }

  /**
   * Check if there is a pending Terminating:Wait lifecycle
   * If so, stop all PM2 processes and complete the lifecycle hook
   */
  async checkLifecycles(): Promise<void> {
    if (await this.metadata.getLifecycleState() === 'Terminated') {
      console.log('Instance lifecycle is: Terminated');
      this.stop();
      try {
        console.log('Stopping all PM2 processes');
        await stopPM2Processes();
        console.log('Successfully stopped all PM2 processes');
      } catch (e) {
        // PM2 failed to stop processes, let's still continue and complete lifecycle hooks
        const err = e as Error;
        console.error('Error stopping PM2 processes: ', err.message);
      }
      console.log('Retrieving autoscaling group info...');
      const groupName = await this.getAutoScalingGroup();
      if (groupName) {
        console.log('Checking lifecycle hooks for ', groupName);
        const input = { AutoScalingGroupName: groupName };
        const command = new DescribeLifecycleHooksCommand(input);
        const result = await this.client.send(command);
        const hooks = result.LifecycleHooks ?? [];
        const hook = hooks.find((h: LifecycleHook) => h.LifecycleTransition === 'autoscaling:EC2_INSTANCE_TERMINATING');
        if (hook) {
          console.log('Completing lifecycle hook: ', hook.LifecycleHookName);
          await this.completeLifecycleHook(hook);
        } else {
          console.log('No lifecycle hooks to complete');
        }
      } else {
        console.log('No autoscaling group for this instance');
      }
    }
  }

  /**
   * Complete a lifecycle hook to continue with termination process
   * @param hook
   */
  async completeLifecycleHook(hook: LifecycleHook): Promise<void> {
    const input = {
      AutoScalingGroupName: hook.AutoScalingGroupName,
      LifecycleActionResult: 'CONTINUE',
      LifecycleHookName: hook.LifecycleHookName,
      InstanceId: this.instanceId,
    };
    const command = new CompleteLifecycleActionCommand(input);
    await this.client.send(command);
  }
}

For development, you can run pm2 install . in your project to install the module. This will load the script as a PM2 module and refresh it every time you make a change. You can run pm2 install <module name> to install a published module from NPM, or pm2 install <username/repository> to install a module from a GitHub repository.

Done

And there you have it, a PM2 module that resolves lifecycle hooks for you. You can find the example module on GitHub.

As always, we hope you liked this article and if you have anything to add, we are available via our Support Channel.