Custom Metrics with AWS CloudWatch

AWS CloudWatch gathers metrics from different parts of the AWS services spectrum. It allows you to monitor your applications and trigger actions based on thresholds and rules.

Typically, you would use some of these metrics, for example, CPU usage, to take action and scale up your capacity to handle the increase in load.

But what if you want to take action based on some custom metric that AWS does not give you by default? Or what if you just want to see a nice graph that shows your application's performance over time?

Generate Custom Metrics

Luckily, the AWS CLI and the AWS SDKs allow you to log your own metrics to CloudWatch.

Let's take a look on how you can add logging of custom metrics to your application from code.

As an example, let's assume we want to log every invocation of a task that is performed by a specific service. Let's call this metric Tasks. CloudWatch allows you to add multiple dimensions to your data points, but in this example we only use one: ServiceId.

const AWS = require('aws')
const cloudwatch = new AWS.CloudWatch({
  region: 'eu-west-1'
})

function logMetricData(value) {
  cloudwatch.putMetricData({
    Namespace: 'MyCustomMetrics', // our namespace, can be anything you want
    MetricData: [{
      MetricName: 'Tasks', // we want to count task invocations
      Dimensions: [{
        Name: 'ServiceId', // we want to group by service id
        Value: 'TaskService01' // the service id that executed this task
      }],
      Value: value,
      Timestamp: new Date(),
      Unit: 'Count' // we use a count, this can also be other units, e.g., bytes
    }]
  })
}

If we then finish a task, we can log it to CloudWatch by calling:

logMetricData(1)

CloudWatch will then take care of storing and aggregating these data points for every minute it received data from your metrics logger.

Monitor Custom Metrics

If we keep calling the CloudWatch API for each single data point, we are going to run into throttling limits of AWS. Since we are counting task invocations, and since AWS will aggregate this data per minute anyway, we can decide to send the data to CloudWatch at a set interval, let's say 10 seconds. We can then also send an explicit value of 0 when there are no task invocations. That way we can later set up alarms based on missing data points, so we can use CloudWatch to both monitor for performance (i.e., task invocations per minute) as well as availability of our service.

Let's turn this into a utility class:

class MetricsLogger {

  /**
   * Log metrics for a specific service
   * @param serviceId
   */
  constructor(serviceId) {
    this.cloudwatch = new AWS.CloudWatch({
      region: 'eu-west-1'
    })
    this.serviceId = serviceId
    this._metrics = {}
    // Send metics every 10 seconds
    setInterval(this.sendMetrics.bind(this), 10000)
  }

  /**
   * Add a metric to be monitored
   * @param name
   */
  addMetric(name) {
    if (!this._metrics[name]) {
      this._metrics[name] = 0
    }
  }

  /**
   * Log an invocation to metrics
   * @param name
   * @param value
   */
  logMetric(name) {
    if (this._metrics[name] == null) {
      this._metrics[name] = 1;
    } else {
      this._metrics[name] += 1;
    }
  }

  /**
   * Send the mtrics to AWS
   */
  sendMetrics() {
    const data = []
    const now = new Date()
    Object.keys(this._metrics).forEach((name) => {
      data.push({
        MetricName: name,
        Dimensions: [{
          Name: 'ServiceId',
          Value: this.serviceId
        }],
        Value: this._metrics[name],
        Timestamp: now,
        Unit: 'Count'
      });
      this._metrics[name] = 0;
    })
    this.cloudwatch.putMetricData({
      Namespace: 'MyCustomMetrics',
      MetricData: data
    }, (err, result) => {
      if (err) {
        console.log('error logging custom metrics: ' + err.message)
      }
    });
  }

}

Then, inside your task worker code, you initialize this logger:

const myMetricsLogger = new MetricsLogger('TaskService01')
myMetricsLogger.addMetric('Tasks')

And then after each task invocation, you log it as a metrics data point:

myMetricsLogger.logMetric('Tasks')

Now if we go into the AWS CloudWatch console, we will see our namespace and the custom metrics:

If we pick the Tasks metric, we can plot a graph with our metric over time:

Like with any metric in CloudWatch, you can then set alarms that trigger based on values or on the absence of metrics data:

Conclusion

If your application already uses AWS CloudWatch to monitor standard performance metrics, you can easily add custom metrics to your list from within your application. You can then use CloudWatch as a central place to monitor your application and set alarms that send out warnings and/or scale your application to handle the required capacity.

If you would like to know more about this subject, you can take a look at the official AWS CloudWatch Documentation and JavaScript SDK documentation.

We hope you found this article useful, and as always, we are available via our Support Channel for any questions you might have.