How to centrally monitor failed backup jobs in Security Hub

Customers love to build and innovate their business in their AWS accounts with fast ,easy and scalable services which AWS provides. Also you can manage the security postures by using different controls， such as: Preventive controls, Detective controls and Corrective controls. For data loss prevention, using backup is a very effective measure as both Preventive and Corrective controls.

AWS Backup offers a cost-effective, fully managed, policy-based managed service that simplifies data protection at scale. AWS Backup leverages AWS Organizations to centrally automate backup policies to implement, configure, manage, and govern backup activity across supported AWS resources.

But if a you do not properly back things up, they are more at risk of suffering consequences from things like a ransomware attack, so you should consider failed backups from a security risk and compliance perspective.

AWS Security Hub is designed to give you a comprehensive view of your security posture across your AWS accounts. With Security Hub, you have a single pane of glass that aggregates, organizes, correlates and prioritizes your security finding from multiple AWS services, multiple account and regions.

In this post, I will walk through with you on how to centrally monitor your backup failure alerts across your single or multi-account in multi-region AWS environment in Security Hub with different design architectures. Also I will provide sample code in this Github repository(https://github.com/jessicawyc/securityhub-custom-finding/tree/main/backupfailure) for you to automatically deploy the solution.

Solution overview

Generally speaking, the solution includes a serverless architecture for getting backup event logged by AWS CloudTrail, the Amazon EventBridge Rule will trigger an AWS Lambda function which will analysis the event and then generate a Critical finding into Security hub if it is a failed backup job.

Figure 1. Basic Architecture
As you may have different AWS account structures ,I also provide below different deployment architecture:

• Single Account with multiple regions
• Multiple Accounts with multiple regions
  • Architecture 1 - multiple lambda functions
  • Architecture 2 -one central lambda function

Prerequisites

Before getting started, make sure that you have a basic understanding of the following:

• Amazon EventBridge rule
• AWS Lambda Function
• Python and Boto3
• AWS Command Line Interface(CLI) and make sure your version is above aws-cli/2.7.35
• AWS Organizations

You will also need to enable Security Hub with Aggregation Region

Solution Walkthrough

You can download all the related templates from Github repository in the local folder where you will run CLI command. Then choose one deployment architecture for your AWS accounts to follow.
I will guide you to use CLI command to run Cloudformation stack or stacksets to create related resources in the following chapters. If you prefer to use Cloudformation template in AWS console for each region or account, please refer to user guide

Single Account with multiple regions

The CloudFormation template will create Eventbridge rule and Lambda function in each region, and Security Hub will automatically aggregates all findings into Aggregation Region.

Figure 2. Single AWS account architecture

Deploying

• Set Paramenter

stackname=backup-sechub-cfn
regions=($(aws ec2 describe-regions --query 'Regions[*].RegionName' --output text --region=us-east-1))
echo $regions

If you do not want to deploy it in all regions, you may adjust the "regions" list manually.

• Run Command

stacktemplate=Arch1-memberaccounts.yaml
for region in $regions; do
aws cloudformation create-stack --stack-name $stackname --template-body file://$stacktemplate --capabilities CAPABILITY_NAMED_IAM --region=$region
echo $region
done

Multiple Accounts with multiple regions

For multiple accounts in AWS Organizations, we will use below CLI commands to create CloudFormation stacksets to create resources.So you should config your CLI profile with the Organizations management account user or role.

Architecture 1 - multiple lambda functions

Figure 3. Multiple AWS accounts Architecture 1

Deploying

• Set Paramenter
  You may decide which region to deploy the CloudFormation by changing the parameter 'region'.
  'regions' is a list for all the regions you want to deploy in each member accounts ,you can adjust it if you would like to limit some regions.

stacksetname=backup-sechub-org
region=us-east-1
regions=($(aws ec2 describe-regions --query 'Regions[*].RegionName' --output text))
echo $regions

• Run Command in Management Account

stacksettemplate=Arch1-memberaccounts.yaml
aws cloudformation create-stack-set \
    --stack-set-name $stacksetname\
    --template-body file://$stacksettemplate  \
    --permission-model SERVICE_MANAGED \
    --auto-deployment Enabled=true,RetainStacksOnAccountRemoval=true \
    --capabilities CAPABILITY_NAMED_IAM \
    --region=$region
aws cloudformation create-stack-instances \
    --stack-set-name $stacksetname\
    --deployment-targets OrganizationalUnitIds=$(aws organizations list-roots --query "Roots[].Id" --output text)  \
    --regions $regions  --region=$region

If the output looks like below format, means you successfully created a StackSet and Operation of it for your whole organizations member accounts.

{
    "StackSetId": "backup-sechub-org:3092ef5d-5d89-4876-9e20-bc15446c9594"
}
{
    "OperationId": "ab088733-f7a7-47a9-b7f7-1810409bcd58"
}

Architecture 2 -one central lambda function

For this architecture,I will use a central Event Bus to receive events from all the member accounts. This central event bus will be in the Delegated admin account of security hub aggregation region.

Figure 4. Multiple AWS accounts Architecture 2

Deploying

1. Set Parameter in Security Hub delegated admin account
   The 'region' should be your aggregation region of Security Hub

region=eu-west-2
ebarn=$(aws events list-event-buses --region=$region --output text --query "EventBuses[*].Arn") 
echo $ebarn

2. Deploy CloudFormation template in management account

• Set Parameter

region=us-east-1
stacksetname=backup-sechub-member
regions=($(aws ec2 describe-regions --query 'Regions[*].RegionName' --output text --region=us-east-1))
echo regions

If you do not want to deploy it in all regions, you may adjust the 'regions' list manually.

• Run Command in Management Account
  Below commands will deploy an EventBridge rule in all member accounts in each region.
  (https://docs.aws.amazon.com/cli/latest/reference/cloudformation/create-stack-instances.html) accordingly.
  There will be a special Stack instance which will be failed, it is the one in aggression region of the delegated admin account, because the target EventBridge bus is there, so you will see a Status Reason :Source EventBus and Target EventBus must not be the same. In below commands, I set FailureToleranceCount=1, so there will be no impact for the final deployment result. We will deploy in aggregation region sperately in step 3 later.

stacksettemplate=Arch2-memberaccounts.yaml
root=$(aws organizations list-roots --query "Roots[].Id" --output text) 
admin=$(aws securityhub list-organization-admin-accounts --region=$region --output text --query 'AdminAccounts[*].AccountId')
aws cloudformation create-stack-set \
    --stack-set-name $stacksetname\
    --template-body file://$stacksettemplate  \
    --parameters  \
    ParameterKey=EBARN,ParameterValue=$ebarn  \
    --permission-model SERVICE_MANAGED \
    --auto-deployment Enabled=true,RetainStacksOnAccountRemoval=true \
    --capabilities CAPABILITY_NAMED_IAM \
    --region=$region
aws cloudformation create-stack-instances --stack-set-name $stacksetname \
        --deployment-targets OrganizationalUnitIds=$root \
        --operation-preferences FailureToleranceCount=1 --regions $regions --region=$region

The output should look like below:

{
    "StackSetId": "backup-sechub-member:21191bfb-b235-4247-bbac-a2bb8acb865f"
}
{
    "OperationId": "7e21e52c-9b80-4449-89a5-9a26fddb9558"
}

3. Deploy lambda function in Delegated Admin account of Security Hub

• Set Parameter
  The 'region' should be your aggregation region of Security Hub

stackname=backup-sechub-admin
region=eu-west-2

• Run Command in delegated admin account

stacktemplate=Arch1-memberaccounts.yaml
aws cloudformation create-stack --stack-name $stackname --template-body file://$stacktemplate --capabilities CAPABILITY_NAMED_IAM --region=$region
--region=$region

China Region

If you use the two China regions, for Single Account and Multiple Account Architecture 1, above steps are also applied to. Only for Architecture 2, you need to adjust the architecture a little because so far China regions Event Bridge has not supported cross region event bus aggregation yet, so you will need to deploy two lambda functions in each region as below picture shows the architecture:

Figure 5. China region architecture 2

The deployment steps only need to repeat once again in the other region in your delegated admin account

Review the result

Once there is failed backup job, in Security Hub, you will see a new critical finding. Let's create a custom insight by using below CLI command in Security Hub delegated admin account:
region should be your aggregation region

• Set Parameter

region='eu-west-2'
insight='BackupAlert'

• Run Command

aws securityhub create-insight \
--filters \
 '{"RecordState": [{ "Comparison": "EQUALS", "Value": "ACTIVE"}], "WorkflowStatus": [{"Comparison": "EQUALS", "Value": "NEW"}], "ProductName": [{"Comparison": "EQUALS", "Value": "Default"}], "Type": [{"Comparison": "EQUALS", "Value": "Effects/Data Destruction/Backup Data"}]}' \
 --group-by-attribute "ResourceId" \
--name $insight \
--query 'InsightArn' --output text --region=$region

It is easier for security admin to watch this insight for backup failure alert and take follow up actions.

Figure 6. Custom Insight

Cleanup

It is very easy to delete the CloudFormation stack or stacksets you have deployed in previous steps in your AWS console.

Conclusion

In this post, I demonstrated how easily it is to centrally monitor AWS backup job failure alert in Security Hub for security team in different architectures.You can also modify the sample code to add more customized information of the new finding.