Autoscaling Group Notifications with Terraform and AWS Lambda
I use Autoscaling Groups in AWS for all of my systems. The main benefit for me here was to make sure that when a node died in AWS, the Autoscaling Group policy made sure that the node was replaced. I wanted to get some visibility of when the Autoscaling Group was launching and terminating nodes and decided that posting notifications to Slack would be a good way of getting this. With Terraform and AWS Lambda, I was able to make this happen.
This post assumes that you are already setup and running with Terraform
Create an IAM Role that allows access to AWS Lambda:
resource "aws_iam_role" "slack_iam_lambda" {
name = "slack-iam-lambda"
assume_role_policy = <<EOF
{
"Version": "2012-10-17",
"Statement": [
{
"Action": "sts:AssumeRole",
"Principal": {
"Service": "lambda.amazonaws.com"
},
"Effect": "Allow",
"Sid": ""
}
]
}
EOF
}
Create a lambda function as follows:
resource "aws_lambda_function" "slack_notify" {
filename = "slackNotify.zip"
function_name = "slackNotify"
role = "${aws_iam_role.slack_iam_lambda.arn}"
handler = "slackNotify.handler"
}
We assume here, that you have already created a Slack Integration. The hook URL from that integration is required for the lambda contents.
The filename slackNotify.zip
is a zip of a file called slackNotify.js
. The contents of that js file are available
Terraform currently does not support hooking AWS Lambda up to SNS Event Sources. Therefore, unfortunately, there is a manual step required to configure the Lambda to talk to the SNS Topic. There is a PR in Terraform to allow this to be automated as well.
In the AWS Console, go to Lambda and then chose the Lambda function.
Go to the Event Sources
tab:
Click on Add Event Source
and then choose SNS
from the dropdown and then make sure you chose the correct SNS Topic name:
We then use another Terraform resource to attach the Autoscale Groups to the Lambda as follows:
resource "aws_autoscaling_notification" "slack_notifications" {
group_names = [
"admin-api-autoscale-group",
"rundeck-autoscale-group",
]
notifications = [
"autoscaling:EC2_INSTANCE_LAUNCH",
"autoscaling:EC2_INSTANCE_TERMINATE",
"autoscaling:EC2_INSTANCE_LAUNCH_ERROR",
"autoscaling:EC2_INSTANCE_TERMINATE_ERROR",
"autoscaling:TEST_NOTIFICATION"
]
topic_arn = "${aws_sns_topic.asg_slack_notifications.arn}"
}
As we have configured notifications for autoscaling:TEST_NOTIFICATION, when you apply this configuration with Terraform, you will see something similar to the following in Slack:
In the current infrastructure I manage, there are 27 Autoscale groups. I don’t really want to add 27 hardcoded group_names in the aws_autoscaling_notifcation in Terraform.
I wanted to take advantage of a Terraform module. In a nutshell, the module does a lookup of all the Autoscaling Groups in a region and then passes that list into the Terraform resource.
The output of the module looks as follows:
{
"variable": {
"autoscalegroup_names": {
"description": "List of autoscalegroup names for a region",
"default": {
"eu-west-1": "admin-api-autoscale-group,dash-autoscale-group,demo-autoscale-group,docker-v2-autoscale-group,elasticsearch-autoscale-group,faces-autoscale-group,internal-api-autoscale-group,jenkins-master-autoscale-group,kafka-autoscale-group,landscapes-autoscale-group",
"ap-southeast-1": "",
"ap-southeast-2": "",
"eu-central-1": "",
"ap-northeast-1": "",
"us-east-1": "",
"sa-east-1": "",
"us-west-1": "",
"us-west-2": ""
}
}
}
}
I then pass this into the Terraform resource as follows:
module "autoscalegroups" {
source = "github.com/stack72/tf_aws_autoscalegroup_names"
region = "${var.aws_region}"
}
resource "aws_autoscaling_notification" "slack_notifications" {
group_names = [
"${split(",", module.autoscalegroups.asg_names)}",
]
notifications = [
"autoscaling:EC2_INSTANCE_LAUNCH",
"autoscaling:EC2_INSTANCE_TERMINATE",
"autoscaling:EC2_INSTANCE_LAUNCH_ERROR",
"autoscaling:EC2_INSTANCE_TERMINATE_ERROR",
"autoscaling:TEST_NOTIFICATION"
]
topic_arn = "${aws_sns_topic.asg_slack_notifications.arn}"
}
When anything happens within an Autoscaling Group, I now get notifications as follows: