Welcome to yet another blog post from the series "AWS security at scale". In this post, we are going to learn how to receive alerts whenever an important AWS resource event occurs.
For example, we want an alert whenever there's a new IAM user created, or we want an alert whenever a Security Group is created, or if the Security Group rules are modified. The list can go on and on...
In this blog, we will mainly focus on how to achieve this task. Once you understand how we can get alerts for some resources, you can go ahead and tweak the implementation as per your Organisation's use case.
Overview
To achieve our goal, and make sure that our solution is scalable, we will use the following AWS resources:
AWS Stack and StackSet
AWS Stack and StackSet will help us deploy our solution to all the accounts that are a part of the Organization. (Assuming AWS Organization is enabled).
If AWS organization is not enabled then the manual overhead of deploying the Stack to every account increases. (Feel free to correct me in the comments)
AWS Eventbridge
- We will use AWS Eventbridge to create an event pattern rule. Using this event pattern, whenever a defined event occurs, the event pattern rule will trigger our Lambda.
AWS Lambda
- We will parse the data sent by AWS Eventbridge and send an appropriate slack alert.
Implementation
Lambda
import json
import boto3
import requests
import os
url = os.environ['WEBHOOK_URL']
slack_channel = os.environ['SLACK_CHANNEL_NAME']
slack_bot_username = "AWS Audit Bot"
my_session = boto3.session.Session()
headers = {
'Content-Type': "application/json",
'User-Agent': "PostmanRuntime/7.19.0",
'Accept': "*/*",
'Cache-Control': "no-cache",
'Postman-Token': "56df98df-XXXX-XXXX-XXXX-9a2k5q56b8gf,458sadwa-XXXX-XXXX-XXXX-p456z4564a45",
'Host': "hooks.slack.com",
'Accept-Encoding': "gzip, deflate",
'Content-Length': "497",
'Connection': "keep-alive",
'cache-control': "no-cache"
}
def get_iam_client():
"""
Get identity and access management client
"""
return boto3.client(
'iam'
)
def get_account_alias():
paginator = get_iam_client().get_paginator('list_account_aliases')
for resp in paginator.paginate():
return resp['AccountAliases']
def lambda_handler(event, context):
print("Event: ", event)
eventSource = event['source']
if eventSource == "aws.iam":
handle_iam_events(event)
elif eventSource == "aws.ec2":
handle_ec2_events(event)
def handle_iam_events(event):
eventName = event["detail"]["eventName"]
if eventName == "CreateUser":
slack_data = dict()
slack_data['title'] = "New IAM User Created"
print(slack_data['title'])
slack_data['created_by'] = event["detail"]["userIdentity"]["arn"]
slack_data['username'] = event["detail"]["requestParameters"]["userName"]
slack_response = requests.request("POST", url, data=get_slack_payload_iam(1, slack_data), headers=headers)
elif eventName == "DeleteUser":
slack_data = dict()
slack_data['title'] = "IAM User Deleted"
print(slack_data['title'])
slack_data['deleted_by'] = event["detail"]["userIdentity"]["arn"]
slack_data['username'] = event["detail"]["requestParameters"]["userName"]
slack_response = requests.request("POST", url, data=get_slack_payload_iam(2, slack_data), headers=headers)
elif eventName == "CreateRole":
slack_data = dict()
slack_data['title'] = "New IAM Role Created"
print(slack_data['title'])
slack_data['created_by'] = event["detail"]["userIdentity"]["arn"]
slack_data['role_name'] = event["detail"]["requestParameters"]["roleName"]
slack_response = requests.request("POST", url, data=get_slack_payload_iam(3, slack_data), headers=headers)
elif eventName == "DeleteRole":
slack_data = dict()
slack_data['title'] = "IAM Role Deleted"
print(slack_data['title'])
slack_data['deleted_by'] = event["detail"]["userIdentity"]["arn"]
slack_data['role_name'] = event["detail"]["requestParameters"]["roleName"]
slack_response = requests.request("POST", url, data=get_slack_payload_iam(4, slack_data), headers=headers)
def handle_ec2_events(event):
slack_data = dict()
eventName = event["detail"]["eventName"]
slack_data['region'] = event["detail"]["awsRegion"]
if eventName == "CreateSecurityGroup":
slack_data['title'] = "New Security Group Created"
print(slack_data['title'])
slack_data['created_by'] = event["detail"]["userIdentity"]["arn"]
slack_data['sg_id'] = event['detail']['responseElements']['groupId']
slack_data['sg_name'] = event['detail']['requestParameters']['groupName']
slack_response = requests.request("POST", url, data=get_slack_payload_ec2(1, slack_data), headers=headers)
elif eventName == "DeleteSecurityGroup":
slack_data['title'] = "Security Group Deleted"
print(slack_data['title'])
slack_data['deleted_by'] = event["detail"]["userIdentity"]["arn"]
slack_data['sg_id'] = event['detail']['requestParameters']['groupId']
slack_response = requests.request("POST", url, data=get_slack_payload_ec2(2, slack_data), headers=headers)
elif eventName == "ModifySecurityGroupRules":
slack_data['title'] = "Security Group Modified"
print(slack_data['title'])
slack_data['modified_by'] = event["detail"]["userIdentity"]["arn"]
slack_data['sg_id'] = event['detail']['requestParameters']['ModifySecurityGroupRulesRequest']['GroupId']
slack_response = requests.request("POST", url, data=get_slack_payload_ec2(3, slack_data), headers=headers)
def get_slack_payload_iam(operation_code, data):
slack_emoji = ":aws-iam:"
slack_message_title = data['title']
payload = ""
account_alias = get_account_alias()[0]
if operation_code == 1:
payload = """{
\n\t\"channel\": \"#""" + slack_channel + """\",
\n\t\"username\": \"""" + slack_bot_username + """\",
\n\t\"icon_emoji\": \"""" + slack_emoji + """\",
\n\t\"attachments\":[\n
{\n
\"fallback\":\"""" + slack_message_title + """\",\n
\"pretext\":\"""" + slack_message_title + """\",\n
\"color\":\"#34bb13\",\n
\"fields\":[\n
{\n
\"value\":\"*Account:* """ + account_alias + """\n*User:* """ + data['username'] + """\n*Created By:* """ + data['created_by'] + """\",\n
}\n
]\n
}\n
]\n
}"""
elif operation_code == 2:
payload = """{
\n\t\"channel\": \"#""" + slack_channel + """\",
\n\t\"username\": \"""" + slack_bot_username + """\",
\n\t\"icon_emoji\": \"""" + slack_emoji + """\",
\n\t\"attachments\":[\n
{\n
\"fallback\":\"""" + slack_message_title + """\",\n
\"pretext\":\"""" + slack_message_title + """\",\n
\"color\":\"#34bb13\",\n
\"fields\":[\n
{\n
\"value\":\"*Account:* """ + account_alias + """\n*User:* """ + data['username'] + """\n*Deleted By:* """ + data['deleted_by'] + """\",\n
}\n
]\n
}\n
]\n
}"""
elif operation_code == 3:
payload = """{
\n\t\"channel\": \"#""" + slack_channel + """\",
\n\t\"username\": \"""" + slack_bot_username + """\",
\n\t\"icon_emoji\": \"""" + slack_emoji + """\",
\n\t\"attachments\":[\n
{\n
\"fallback\":\"""" + slack_message_title + """\",\n
\"pretext\":\"""" + slack_message_title + """\",\n
\"color\":\"#34bb13\",\n
\"fields\":[\n
{\n
\"value\":\"*Account:* """ + account_alias + """\n*Role Name:* """ + data['role_name'] + """\n*Created By:* """ + data['created_by'] + """\",\n
}\n
]\n
}\n
]\n
}"""
elif operation_code == 4:
payload = """{
\n\t\"channel\": \"#""" + slack_channel + """\",
\n\t\"username\": \"""" + slack_bot_username + """\",
\n\t\"icon_emoji\": \"""" + slack_emoji + """\",
\n\t\"attachments\":[\n
{\n
\"fallback\":\"""" + slack_message_title + """\",\n
\"pretext\":\"""" + slack_message_title + """\",\n
\"color\":\"#34bb13\",\n
\"fields\":[\n
{\n
\"value\":\"*Account:* """ + account_alias + """\n*Role Name:* """ + data['role_name'] + """\n*Deleted By:* """ + data['deleted_by'] + """\",\n
}\n
]\n
}\n
]\n
}"""
def get_slack_payload_ec2(operation_code, data):
slack_emoji = ":aws-ec2:"
slack_message_title = data['title']
current_region = data['region']
payload = ""
account_alias = get_account_alias()[0]
if operation_code == 1:
payload = """{
\n\t\"channel\": \"#""" + slack_channel + """\",
\n\t\"username\": \"""" + slack_bot_username + """\",
\n\t\"icon_emoji\": \"""" + slack_emoji + """\",
\n\t\"attachments\":[\n
{\n
\"fallback\":\"""" + slack_message_title + """\",\n
\"pretext\":\"""" + slack_message_title + """\",\n
\"color\":\"#34bb13\",\n
\"fields\":[\n
{\n
\"value\":\"*Account:* """ + account_alias + """\n*Security Group:* """ + data['sg_name'] + """\n*Security Group ID:* """ + data['sg_id'] + """\n*Created By:* """ + data['created_by'] + """\n*Region:* """ + current_region + """\",\n
}\n
]\n
}\n
]\n
}"""
elif operation_code == 2:
payload = """{
\n\t\"channel\": \"#""" + slack_channel + """\",
\n\t\"username\": \"""" + slack_bot_username + """\",
\n\t\"icon_emoji\": \"""" + slack_emoji + """\",
\n\t\"attachments\":[\n
{\n
\"fallback\":\"""" + slack_message_title + """\",\n
\"pretext\":\"""" + slack_message_title + """\",\n
\"color\":\"#34bb13\",\n
\"fields\":[\n
{\n
\"value\":\"*Account:* """ + account_alias + """\n*Security Group ID:* """ + data['sg_id'] + """\n*Deleted By:* """ + data['deleted_by'] + """\n*Region:* """ + current_region + """\",\n
}\n
]\n
}\n
]\n
}"""
elif operation_code == 3:
payload = """{
\n\t\"channel\": \"#""" + slack_channel + """\",
\n\t\"username\": \"""" + slack_bot_username + """\",
\n\t\"icon_emoji\": \"""" + slack_emoji + """\",
\n\t\"attachments\":[\n
{\n
\"fallback\":\"""" + slack_message_title + """\",\n
\"pretext\":\"""" + slack_message_title + """\",\n
\"color\":\"#34bb13\",\n
\"fields\":[\n
{\n
\"value\":\"*Account:* """ + account_alias + """\n*Security Group ID:* """ + data['sg_id'] + """\n*Modified By:* """ + data['modified_by'] + """\n*Region:* """ + current_region + """\",\n
}\n
]\n
}\n
]\n
}"""
- This template will be similar to the one we defined in the Previous Blog. The only change is the Eventbridge rule.
- This time, we will be creating an Event Pattern
- To create an Event Pattern, we need to mention the resources (source).
- For the sake of this blog, let's consider some of the
IAM and EC2 resources.
- Now, we don't want all the IAM and EC2 events, we want to filter them based on the Actions. We can add this in the
eventName parameter under the detail JSON object.
- This is what the entire CloudFormation template looks like:
{
"AWSTemplateFormatVersion": "2010-09-09",
"Description" : "Deploy Lambda Function to trigger Slack alert for various AWS IAM and Network events.",
"Parameters" : {
"SlackWebhookParameter" : {
"Type" : "String",
"Default" : "",
"Description" : "Webhook for Slack Channel"
},
"SlackChannelName" : {
"Type" : "String",
"Default" : "",
"Description" : "Name of the slack channel where you want alerts"
},
"S3Bucket" : {
"Type" : "String",
"Default" : "",
"Description" : "Name of the S3 bucket where the lambda is stored"
},
"S3Key" : {
"Type" : "String",
"Default" : "",
"Description" : "Key name of the S3 object"
},
"LambdaHandler" : {
"Type" : "String",
"Default" : "",
"Description" : "Lambda Handler name E.g: <file_name>.lambda_handler"
},
"LambdaRoleName" : {
"Type" : "String",
"Default" : "",
"Description" : "Role that will be attached to the lambda"
}
},
"Resources": {
"awsAuditAlertLambda": {
"Type": "AWS::Lambda::Function",
"Properties": {
"FunctionName": "awsAuditAlert",
"Tags": [
{
"Key": "CreatedBy",
"Value": "Security Team"
}
],
"Handler": {"Ref": "LambdaHandler"},
"Environment" : {
"Variables": { "WEBHOOK_URL": {"Ref": "SlackWebhookParameter"}, "SLACK_CHANNEL_NAME": {"Ref": "SlackChannelName"} }
},
"Role": {"Fn::Join": ["",[{"Fn::Sub": "arn:aws:iam::${AWS::AccountId}:role/"}, {"Ref": "LambdaRoleName"}]]},
"Code": {
"S3Bucket": {"Ref": "S3Bucket"},
"S3Key": {"Ref": "S3Key"}
},
"Runtime": "python3.7",
"Timeout": 900
}
},
"ScheduledRule": {
"Type": "AWS::Events::Rule",
"Properties": {
"Description": "Rule to trigger awsAuditAlert Lambda",
"Name" : "awsAuditAlertLambdaEventRule",
"EventPattern": {
"source": ["aws.iam", "aws.ec2"],
"detail-type": ["AWS API Call via CloudTrail"],
"detail": {
"eventSource": ["iam.amazonaws.com", "ec2.amazonaws.com"],
"eventName": ["CreateUser", "DeleteUser", "CreateRole", "DeleteRole", "AttachRolePolicy", "PutRolePolicy", "DetachRolePolicy", "DeleteRolePolicy", "CreateSecurityGroup", "DeleteSecurityGroup", "ModifySecurityGroupRules", "CreateNetworkAcl", "CreateNetworkAclEntry", "ReplaceNetworkAclEntry", "DeleteNetworkAcl", "DeleteNetworkAclEntry"]
}
},
"State": "ENABLED",
"Targets": [{
"Arn": { "Fn::GetAtt": ["awsAuditAlertLambda", "Arn"] },
"Id": "TargetFunctionV1"
}]
}
},
"PermissionForEventsToInvokeLambda": {
"Type": "AWS::Lambda::Permission",
"Properties": {
"FunctionName": { "Ref": "awsAuditAlertLambda" },
"Action": "lambda:InvokeFunction",
"Principal": "events.amazonaws.com",
"SourceArn": { "Fn::GetAtt": ["ScheduledRule", "Arn"] }
}
}
}
}
- Save this file as
aws-audit-alerts-cft.template
- This template is still not complete. As you can see, we have not defined the role that should be given to the Lambda function.
- We will create one more CloudFormation template just to create a role. This role will be attached to our Lambda using the above template.
- The reason we are doing this is that the above template will be deployed in multiple regions. We are creating EventRule and Lambda in multiple regions because we want alerts for EC2 resources as well, and EC2 resources are region-specific.
Note: IAM-specific event patterns work only in the N.Virginia region. I couldn't find out the reason (But feel free to give one in the comments).
- Following is the Cloudformation template to create a role for the Lambda function
{
"AWSTemplateFormatVersion": "2010-09-09",
"Description" : "Create the required IAM Role.",
"Parameters" : {
"RoleName" : {
"Type" : "String",
"Default" : "",
"Description" : "Name of the Role to be created"
}
},
"Resources": {
"awsAuditAlertLambdaRole": {
"Type": "AWS::IAM::Role",
"Properties": {
"RoleName": {"Ref": "RoleName"},
"AssumeRolePolicyDocument": {
"Version": "2012-10-17",
"Statement": [{
"Effect": "Allow",
"Principal": {
"Service": [ "lambda.amazonaws.com" ]
},
"Action": [ "sts:AssumeRole" ]
}]
},
"Path": "/",
"Policies": [{
"PolicyName": "awsAuditAlertLambdaPolicy",
"PolicyDocument": {
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"logs:CreateLogStream",
"logs:CreateLogGroup",
"iam:ListAccountAliases",
"logs:PutLogEvents"
],
"Resource": "*"
}
]
}
}]
}
}
},
"Outputs": {
"RoleArn" : {
"Description": "ARN of the Role",
"Value" : { "Fn::GetAtt": ["awsAuditAlertLambdaRole","Arn"] }
}
}
}
- Save this file as
aws-audit-role-cft.template
- Create a StackSet and deploy the
aws-audit-role-cft.template StackSet in any region.
- Once the roles are created, go ahead and deploy the StackSet
aws-audit-alerts-cft.template which will use the previously created role.
- Next, don't forget to create a Stack in the Root Account, since StackSet does not create resources in the Root Account.
Slack Alerts
- Whenever a particular event is triggered, you will receive an alert on the slack.
Conclusion
We have successfully deployed our solution to all the cloud accounts using Cloudformation Stacks and StackSets. To verify that we have deployed it correctly, you can visit any account and check for the Lambda that is created, the Cloudwatch event-pattern rule, and the IAM Role created for the Lambda.
We used Cloudformation StackSets to implement AWS security measures that are scalable. In the future, if there is any new account added to the Organization, the same stack will be created for that account automatically.
