From 41c5150f666a709efe2b686f89eb3f174b865943 Mon Sep 17 00:00:00 2001 From: charlie keegan Date: Tue, 7 Feb 2023 11:29:43 +0000 Subject: [PATCH 01/48] initialize terraform repo --- .gitignore | 34 ++++++++++++++++++++++++++++++++++ 1 file changed, 34 insertions(+) create mode 100644 .gitignore diff --git a/.gitignore b/.gitignore new file mode 100644 index 0000000..dc82526 --- /dev/null +++ b/.gitignore @@ -0,0 +1,34 @@ +# Local .terraform directories +**/.terraform/* + +# .tfstate files +*.tfstate +*.tfstate.* + +# Crash log files +crash.log +crash.*.log + +# Exclude all .tfvars files, which are likely to contain sensitive data, such as +# password, private keys, and other secrets. These should not be part of version +# control as they are data points which are potentially sensitive and subject +# to change depending on the environment. +*.tfvars +*.tfvars.json + +# Ignore override files as they are usually used to override resources locally and so +# are not checked in +override.tf +override.tf.json +*_override.tf +*_override.tf.json + +# Include override files you do wish to add to version control using negated pattern +# !example_override.tf + +# Include tfplan files to ignore the plan output of command: terraform plan -out=tfplan +# example: *tfplan* + +# Ignore CLI configuration files +.terraformrc +terraform.rc \ No newline at end of file From ee77950ff7719299370362260bffec9073a76766 Mon Sep 17 00:00:00 2001 From: charlie keegan Date: Tue, 7 Feb 2023 11:30:50 +0000 Subject: [PATCH 02/48] add yaml translation for cfn --- cftemplates/snapshots_tool_rds_dest.yaml | 496 +++++++++++++++++ cftemplates/snapshots_tool_rds_source.yaml | 601 +++++++++++++++++++++ terraform/main.tf | 0 3 files changed, 1097 insertions(+) create mode 100644 cftemplates/snapshots_tool_rds_dest.yaml create mode 100644 cftemplates/snapshots_tool_rds_source.yaml create mode 100644 terraform/main.tf diff --git a/cftemplates/snapshots_tool_rds_dest.yaml b/cftemplates/snapshots_tool_rds_dest.yaml new file mode 100644 index 0000000..d047fdc --- /dev/null +++ b/cftemplates/snapshots_tool_rds_dest.yaml @@ -0,0 +1,496 @@ +--- +AWSTemplateFormatVersion: '2010-09-09' +Parameters: + CodeBucket: + Type: String + Description: Name of the bucket that contains the lambda functions to deploy. + SnapshotPattern: + Type: String + Default: ALL_SNAPSHOTS + Description: Python regex for matching instance names to backup. Use "ALL_SNAPSHOTS" + to back up every RDS instance in the region. + RetentionDays: + Type: Number + Default: '7' + Description: Number of days to keep snapshots in retention before deleting them + DestinationRegion: + Type: String + Description: Destination region for snapshots. + LogLevel: + Type: String + Default: ERROR + Description: Log level for Lambda functions (DEBUG, INFO, WARN, ERROR, CRITICAL + are valid values). + LambdaCWLogRetention: + Type: Number + Default: '7' + Description: Number of days to retain logs from the lambda functions in CloudWatch + Logs + SourceRegionOverride: + Type: String + Default: 'NO' + Description: Set to the region where your RDS instances run, only if such region + does not support Step Functions. Leave as NO otherwise + KmsKeyDestination: + Type: String + Default: None + Description: Set to the ARN for the KMS key in the destination region to re-encrypt + encrypted snapshots. Leave None if you are not using encryption + KmsKeySource: + Type: String + Default: None + Description: Set to the ARN for the KMS key in the SOURCE region to re-encrypt + encrypted snapshots. Leave None if you are not using encryption + DeleteOldSnapshots: + Type: String + Default: 'TRUE' + Description: Set to TRUE to enable deletion of snapshot based on RetentionDays. + Set to FALSE to disable + AllowedValues: + - 'TRUE' + - 'FALSE' + CrossAccountCopy: + Type: String + AllowedValues: + - 'TRUE' + - 'FALSE' + Default: 'TRUE' + Description: Enable copying snapshots across accounts. Set to FALSE if your source + snapshosts are not on a different account + LogGroupName: + Type: String + Default: lambdaDeleteOldSnapshotsRDS-dest + Description: Name for RDS snapshot log group. +Conditions: + DeleteOld: + Fn::Equals: + - Ref: DeleteOldSnapshots + - 'TRUE' + CrossAccount: + Fn::Equals: + - Ref: CrossAccountCopy + - 'TRUE' +Resources: + topicCopyFailedDest: + Type: AWS::SNS::Topic + Properties: + DisplayName: copies_failed_dest_rds + topicDeleteOldFailedDest: + Type: AWS::SNS::Topic + Properties: + DisplayName: delete_old_failed_dest_rds + snspolicyCopyFailedDest: + Type: AWS::SNS::TopicPolicy + Properties: + Topics: + - Ref: topicCopyFailedDest + - Ref: topicDeleteOldFailedDest + PolicyDocument: + Version: '2008-10-17' + Id: __default_policy_ID + Statement: + - Sid: __default_statement_ID + Effect: Allow + Principal: + AWS: "*" + Action: + - SNS:GetTopicAttributes + - SNS:SetTopicAttributes + - SNS:AddPermission + - SNS:RemovePermission + - SNS:DeleteTopic + - SNS:Subscribe + - SNS:ListSubscriptionsByTopic + - SNS:Publish + - SNS:Receive + Resource: "*" + Condition: + StringEquals: + AWS:SourceOwner: + Ref: AWS::AccountId + alarmcwCopyFailedDest: + Type: AWS::CloudWatch::Alarm + Properties: + ActionsEnabled: 'true' + ComparisonOperator: GreaterThanOrEqualToThreshold + EvaluationPeriods: '1' + MetricName: ExecutionsFailed + Namespace: AWS/States + Period: '300' + Statistic: Sum + Threshold: '1.0' + AlarmActions: + - Ref: topicCopyFailedDest + Dimensions: + - Name: StateMachineArn + Value: + Ref: statemachineCopySnapshotsDestRDS + alarmcwDeleteOldFailedDest: + Type: AWS::CloudWatch::Alarm + Condition: DeleteOld + Properties: + ActionsEnabled: 'true' + ComparisonOperator: GreaterThanOrEqualToThreshold + EvaluationPeriods: '2' + MetricName: ExecutionsFailed + Namespace: AWS/States + Period: '3600' + Statistic: Sum + Threshold: '2.0' + AlarmActions: + - Ref: topicDeleteOldFailedDest + Dimensions: + - Name: StateMachineArn + Value: + Ref: statemachineDeleteOldSnapshotsDestRDS + iamroleSnapshotsRDS: + Type: AWS::IAM::Role + Properties: + AssumeRolePolicyDocument: + Version: '2012-10-17' + Statement: + - Effect: Allow + Principal: + Service: lambda.amazonaws.com + Action: sts:AssumeRole + Policies: + - PolicyName: inline_policy_snapshots_rds_cw_logs + PolicyDocument: + Version: '2012-10-17' + Statement: + - Effect: Allow + Action: + - logs:CreateLogGroup + - logs:CreateLogStream + - logs:PutLogEvents + Resource: arn:aws:logs:*:*:* + - PolicyName: inline_policy_snapshots_rds + PolicyDocument: + Version: '2012-10-17' + Statement: + - Effect: Allow + Action: + - rds:CreateDBSnapshot + - rds:DeleteDBSnapshot + - rds:DescribeDBInstances + - rds:DescribeDBSnapshots + - rds:ModifyDBSnapshotAttribute + - rds:DescribeDBSnapshotAttributes + - rds:CopyDBSnapshot + - rds:ListTagsForResource + - rds:AddTagsToResource + Resource: "*" + - PolicyName: inline_policy_snapshot_rds_kms_access + PolicyDocument: + Version: '2012-10-17' + Statement: + - Sid: AllowUseOfTheKey + Effect: Allow + Action: + - kms:Encrypt + - kms:Decrypt + - kms:ReEncrypt* + - kms:GenerateDataKey* + - kms:DescribeKey + Resource: + - "*" + - Sid: AllowAttachmentOfPersistentResources + Effect: Allow + Action: + - kms:CreateGrant + - kms:ListGrants + - kms:RevokeGrant + Resource: + - "*" + Condition: + Bool: + kms:GrantIsForAWSResource: true + lambdaCopySnapshotsRDS: + Type: AWS::Lambda::Function + Properties: + Code: + S3Bucket: + Ref: CodeBucket + S3Key: + Fn::If: + - CrossAccount + - copy_snapshots_dest_rds.zip + - copy_snapshots_no_x_account_rds.zip + MemorySize: 512 + Description: 'This functions copies snapshots for RDS Instances shared with + this account. It checks for existing snapshots following the pattern specified + in the environment variables with the following format: -YYYY-MM-DD-HH-MM' + Environment: + Variables: + SNAPSHOT_PATTERN: + Ref: SnapshotPattern + DEST_REGION: + Ref: DestinationRegion + LOG_LEVEL: + Ref: LogLevel + REGION_OVERRIDE: + Ref: SourceRegionOverride + KMS_KEY_DEST_REGION: + Ref: KmsKeyDestination + KMS_KEY_SOURCE_REGION: + Ref: KmsKeySource + RETENTION_DAYS: + Ref: RetentionDays + Role: + Fn::GetAtt: + - iamroleSnapshotsRDS + - Arn + Runtime: python3.7 + Handler: lambda_function.lambda_handler + Timeout: 300 + lambdaDeleteOldDestRDS: + Type: AWS::Lambda::Function + Condition: DeleteOld + Properties: + Code: + S3Bucket: + Ref: CodeBucket + S3Key: + Fn::If: + - CrossAccount + - delete_old_snapshots_dest_rds.zip + - delete_old_snapshots_no_x_account_rds.zip + MemorySize: 512 + Description: 'This function enforces retention on the snapshots shared with + the destination account. ' + Environment: + Variables: + SNAPSHOT_PATTERN: + Ref: SnapshotPattern + DEST_REGION: + Ref: DestinationRegion + RETENTION_DAYS: + Ref: RetentionDays + LOG_LEVEL: + Ref: LogLevel + Role: + Fn::GetAtt: + - iamroleSnapshotsRDS + - Arn + Runtime: python3.7 + Handler: lambda_function.lambda_handler + Timeout: 300 + iamroleStateExecution: + Type: AWS::IAM::Role + Properties: + AssumeRolePolicyDocument: + Version: '2012-10-17' + Statement: + - Effect: Allow + Principal: + Service: + Fn::Join: + - '' + - - states. + - Ref: AWS::Region + - ".amazonaws.com" + Action: sts:AssumeRole + Policies: + - PolicyName: inline_policy_rds_snapshot + PolicyDocument: + Version: '2012-10-17' + Statement: + - Effect: Allow + Action: + - lambda:InvokeFunction + Resource: "*" + statemachineCopySnapshotsDestRDS: + Type: AWS::StepFunctions::StateMachine + Properties: + DefinitionString: + Fn::Join: + - '' + - - Fn::Join: + - "\n" + - - ' {"Comment":"Copies snapshots locally and then to DEST_REGION",' + - ' "StartAt":"CopySnapshots",' + - ' "States":{' + - ' "CopySnapshots":{' + - ' "Type":"Task",' + - ' "Resource": ' + - "\"" + - Fn::GetAtt: + - lambdaCopySnapshotsRDS + - Arn + - |- + " + , + - Fn::Join: + - "\n" + - - ' "Retry":[' + - " {" + - ' "ErrorEquals":[ ' + - ' "SnapshotToolException"' + - " ]," + - ' "IntervalSeconds":300,' + - ' "MaxAttempts":5,' + - ' "BackoffRate":1' + - " }," + - " {" + - ' "ErrorEquals":[ ' + - ' "States.ALL"], ' + - ' "IntervalSeconds": 30,' + - ' "MaxAttempts": 20,' + - ' "BackoffRate": 1' + - " }" + - " ]," + - ' "End": true ' + - " }" + - " }}" + RoleArn: + Fn::GetAtt: + - iamroleStateExecution + - Arn + statemachineDeleteOldSnapshotsDestRDS: + Type: AWS::StepFunctions::StateMachine + Condition: DeleteOld + Properties: + DefinitionString: + Fn::Join: + - '' + - - Fn::Join: + - "\n" + - - ' {"Comment":"DeleteOld for RDS snapshots in destination region",' + - ' "StartAt":"DeleteOldDestRegion",' + - ' "States":{' + - ' "DeleteOldDestRegion":{' + - ' "Type":"Task",' + - ' "Resource": ' + - "\"" + - Fn::GetAtt: + - lambdaDeleteOldDestRDS + - Arn + - |- + " + , + - Fn::Join: + - "\n" + - - ' "Retry":[' + - " {" + - ' "ErrorEquals":[ ' + - ' "SnapshotToolException"' + - " ]," + - ' "IntervalSeconds":600,' + - ' "MaxAttempts":5,' + - ' "BackoffRate":1' + - " }," + - " {" + - ' "ErrorEquals":[ ' + - ' "States.ALL"], ' + - ' "IntervalSeconds": 30,' + - ' "MaxAttempts": 20,' + - ' "BackoffRate": 1' + - " }" + - " ]," + - ' "End": true ' + - " }" + - " }}" + RoleArn: + Fn::GetAtt: + - iamroleStateExecution + - Arn + iamroleStepInvocation: + Type: AWS::IAM::Role + Properties: + AssumeRolePolicyDocument: + Version: '2012-10-17' + Statement: + - Effect: Allow + Principal: + Service: events.amazonaws.com + Action: sts:AssumeRole + Policies: + - PolicyName: inline_policy_state_invocation + PolicyDocument: + Version: '2012-10-17' + Statement: + - Effect: Allow + Action: + - states:StartExecution + Resource: "*" + cwEventCopySnapshotsRDS: + Type: AWS::Events::Rule + Properties: + Description: Triggers the RDS Copy state machine in the destination account + ScheduleExpression: + Fn::Join: + - '' + - - cron( + - "/30 * * * ? *" + - ")" + State: ENABLED + Targets: + - Arn: + Ref: statemachineCopySnapshotsDestRDS + Id: Target1 + RoleArn: + Fn::GetAtt: + - iamroleStepInvocation + - Arn + cwEventDeleteOldSnapshotsRDS: + Type: AWS::Events::Rule + Condition: DeleteOld + Properties: + Description: Triggers the RDS DeleteOld state machine in the destination account + ScheduleExpression: + Fn::Join: + - '' + - - cron( + - 0 /1 * * ? * + - ")" + State: ENABLED + Targets: + - Arn: + Ref: statemachineDeleteOldSnapshotsDestRDS + Id: Target1 + RoleArn: + Fn::GetAtt: + - iamroleStepInvocation + - Arn + cwloggroupDeleteOldSnapshotsDestRDS: + Type: AWS::Logs::LogGroup + Description: Log group for the lambdaCopySnapshotsRDS function's logs + Condition: DeleteOld + DependsOn: lambdaDeleteOldDestRDS + Properties: + RetentionInDays: + Ref: LambdaCWLogRetention + LogGroupName: + Fn::Sub: + - "/aws/lambda/${func}" + - func: + Ref: LogGroupName + cwloggrouplambdaCopySnapshotsRDS: + Type: AWS::Logs::LogGroup + Description: Log group for the lambdaCopySnapshotsRDS function's logs + DependsOn: lambdaCopySnapshotsRDS + Properties: + RetentionInDays: + Ref: LambdaCWLogRetention + LogGroupName: + Fn::Sub: + - "/aws/lambda/${func}" + - func: + Ref: lambdaCopySnapshotsRDS +Outputs: + CopyFailedTopic: + Description: Subscribe to this topic to receive alerts of failed copies + Value: + Ref: topicCopyFailedDest + DeleteOldFailedTopic: + Condition: DeleteOld + Description: Subscribe to this topic to receive alerts of failures at deleting + old snapshots + Value: + Ref: topicDeleteOldFailedDest + SourceURL: + Description: For more information and documentation, see the source repository + at GitHub. + Value: https://github.com/awslabs/rds-snapshot-tool +Description: Snapshots Tool for RDS cross-region and cross-account (destination account + stack) diff --git a/cftemplates/snapshots_tool_rds_source.yaml b/cftemplates/snapshots_tool_rds_source.yaml new file mode 100644 index 0000000..bb0a3a9 --- /dev/null +++ b/cftemplates/snapshots_tool_rds_source.yaml @@ -0,0 +1,601 @@ +--- +AWSTemplateFormatVersion: '2010-09-09' +Parameters: + CodeBucket: + Type: String + Description: Name of the bucket that contains the lambda functions to deploy. + InstanceNamePattern: + Type: String + Default: ALL_INSTANCES + Description: Python regex for matching cluster identifiers to backup. Use "ALL_INSTANCES" + to back up every RDS instance in the region. + BackupInterval: + Type: Number + Default: '24' + Description: Interval for backups in hours. Default is 24 + DestinationAccount: + Type: Number + Default: '000000000000' + Description: Destination account with no dashes. + ShareSnapshots: + Type: String + Default: 'TRUE' + AllowedValues: + - 'TRUE' + - 'FALSE' + BackupSchedule: + Type: String + Default: 0 1 * * ? * + Description: 'Backup schedule in Cloudwatch Event cron format. Needs to run at + least once for every Interval. The default value runs once every at 1AM UTC. + More information: http://docs.aws.amazon.com/AmazonCloudWatch/latest/events/ScheduledEvents.html' + RetentionDays: + Type: Number + Default: '7' + Description: Number of days to keep snapshots in retention before deleting them + LogLevel: + Type: String + Default: ERROR + Description: Log level for Lambda functions (DEBUG, INFO, WARN, ERROR, CRITICAL + are valid values). + LambdaCWLogRetention: + Type: Number + Default: '7' + Description: Number of days to retain logs from the lambda functions in CloudWatch + Logs + SourceRegionOverride: + Type: String + Default: 'NO' + Description: Set to the region where your RDS instances run, only if such region + does not support Step Functions. Leave as NO otherwise + DeleteOldSnapshots: + Type: String + Default: 'TRUE' + Description: Set to TRUE to enable deletion of snapshot based on RetentionDays. + Set to FALSE to disable + AllowedValues: + - 'TRUE' + - 'FALSE' + TaggedInstance: + Type: String + Default: 'FALSE' + Description: Set to TRUE to filter instances that have tag CopyDBSnapshot set + to True. Set to FALSE to disable + AllowedValues: + - 'TRUE' + - 'FALSE' + LogGroupName: + Type: String + Default: lambdaDeleteOldSnapshotsRDS-source + Description: Name for RDS snapshot log group. +Conditions: + Share: + Fn::Equals: + - Ref: ShareSnapshots + - 'TRUE' + DeleteOld: + Fn::Equals: + - Ref: DeleteOldSnapshots + - 'TRUE' +Resources: + topicBackupsFailed: + Type: AWS::SNS::Topic + Properties: + DisplayName: backups_failed_rds + topicShareFailed: + Type: AWS::SNS::Topic + Properties: + DisplayName: share_failed_rds + topicDeleteOldFailed: + Type: AWS::SNS::Topic + Properties: + DisplayName: delete_old_failed_rds + snspolicySnapshotsRDS: + Type: AWS::SNS::TopicPolicy + Properties: + Topics: + - Ref: topicBackupsFailed + - Ref: topicShareFailed + - Ref: topicDeleteOldFailed + PolicyDocument: + Version: '2008-10-17' + Id: __default_policy_ID + Statement: + - Sid: __default_statement_ID + Effect: Allow + Principal: + AWS: "*" + Action: + - SNS:GetTopicAttributes + - SNS:SetTopicAttributes + - SNS:AddPermission + - SNS:RemovePermission + - SNS:DeleteTopic + - SNS:Subscribe + - SNS:ListSubscriptionsByTopic + - SNS:Publish + - SNS:Receive + Resource: "*" + Condition: + StringEquals: + AWS:SourceOwner: + Ref: AWS::AccountId + alarmcwBackupsFailed: + Type: AWS::CloudWatch::Alarm + Properties: + ActionsEnabled: 'true' + ComparisonOperator: GreaterThanOrEqualToThreshold + EvaluationPeriods: '1' + MetricName: ExecutionsFailed + Namespace: AWS/States + Period: '300' + Statistic: Sum + Threshold: '1.0' + AlarmActions: + - Ref: topicBackupsFailed + Dimensions: + - Name: StateMachineArn + Value: + Ref: stateMachineTakeSnapshotsRDS + alarmcwShareFailed: + Condition: Share + Type: AWS::CloudWatch::Alarm + Properties: + ActionsEnabled: 'true' + ComparisonOperator: GreaterThanOrEqualToThreshold + EvaluationPeriods: '2' + MetricName: ExecutionsFailed + Namespace: AWS/States + Period: '3600' + Statistic: Sum + Threshold: '2.0' + AlarmActions: + - Ref: topicShareFailed + Dimensions: + - Name: StateMachineArn + Value: + Ref: statemachineShareSnapshotsRDS + alarmcwDeleteOldFailed: + Condition: DeleteOld + Type: AWS::CloudWatch::Alarm + Properties: + ActionsEnabled: 'true' + ComparisonOperator: GreaterThanOrEqualToThreshold + EvaluationPeriods: '2' + MetricName: ExecutionsFailed + Namespace: AWS/States + Period: '3600' + Statistic: Sum + Threshold: '2.0' + AlarmActions: + - Ref: topicDeleteOldFailed + Dimensions: + - Name: StateMachineArn + Value: + Ref: statemachineDeleteOldSnapshotsRDS + iamroleSnapshotsRDS: + Type: AWS::IAM::Role + Properties: + AssumeRolePolicyDocument: + Version: '2012-10-17' + Statement: + - Effect: Allow + Principal: + Service: lambda.amazonaws.com + Action: sts:AssumeRole + Policies: + - PolicyName: inline_policy_snapshots_rds_cw_logs + PolicyDocument: + Version: '2012-10-17' + Statement: + - Effect: Allow + Action: + - logs:CreateLogGroup + - logs:CreateLogStream + - logs:PutLogEvents + Resource: arn:aws:logs:*:*:* + - PolicyName: inline_policy_snapshots_rds + PolicyDocument: + Version: '2012-10-17' + Statement: + - Effect: Allow + Action: + - rds:CreateDBSnapshot + - rds:DeleteDBSnapshot + - rds:DescribeDBInstances + - rds:DescribeDBSnapshots + - rds:ModifyDBSnapshotAttribute + - rds:DescribeDBSnapshotAttributes + - rds:ListTagsForResource + - rds:AddTagsToResource + Resource: "*" + lambdaTakeSnapshotsRDS: + Type: AWS::Lambda::Function + Properties: + Code: + S3Bucket: + Ref: CodeBucket + S3Key: take_snapshots_rds.zip + MemorySize: 512 + Description: 'This functions triggers snapshots creation for RDS instances. + It checks for existing snapshots following the pattern and interval specified + in the environment variables with the following format: -YYYY-MM-DD-HH-MM' + Environment: + Variables: + INTERVAL: + Ref: BackupInterval + PATTERN: + Ref: InstanceNamePattern + LOG_LEVEL: + Ref: LogLevel + REGION_OVERRIDE: + Ref: SourceRegionOverride + TAGGEDINSTANCE: + Ref: TaggedInstance + Role: + Fn::GetAtt: + - iamroleSnapshotsRDS + - Arn + Runtime: python3.7 + Handler: lambda_function.lambda_handler + Timeout: 300 + lambdaShareSnapshotsRDS: + Type: AWS::Lambda::Function + Condition: Share + Properties: + Code: + S3Bucket: + Ref: CodeBucket + S3Key: share_snapshots_rds.zip + MemorySize: 512 + Description: 'This function shares snapshots created by the take_snapshots_rds + function with DEST_ACCOUNT specified in the environment variables. ' + Environment: + Variables: + DEST_ACCOUNT: + Ref: DestinationAccount + LOG_LEVEL: + Ref: LogLevel + PATTERN: + Ref: InstanceNamePattern + REGION_OVERRIDE: + Ref: SourceRegionOverride + Role: + Fn::GetAtt: + - iamroleSnapshotsRDS + - Arn + Runtime: python3.7 + Handler: lambda_function.lambda_handler + Timeout: 300 + lambdaDeleteOldSnapshotsRDS: + Type: AWS::Lambda::Function + Condition: DeleteOld + Properties: + Code: + S3Bucket: + Ref: CodeBucket + S3Key: delete_old_snapshots_rds.zip + MemorySize: 512 + Description: 'This function deletes snapshots created by the take_snapshots_rds + function. ' + Environment: + Variables: + RETENTION_DAYS: + Ref: RetentionDays + PATTERN: + Ref: InstanceNamePattern + LOG_LEVEL: + Ref: LogLevel + REGION_OVERRIDE: + Ref: SourceRegionOverride + Role: + Fn::GetAtt: + - iamroleSnapshotsRDS + - Arn + Runtime: python3.7 + Handler: lambda_function.lambda_handler + Timeout: 300 + iamroleStateExecution: + Type: AWS::IAM::Role + Properties: + AssumeRolePolicyDocument: + Version: '2012-10-17' + Statement: + - Effect: Allow + Principal: + Service: + Fn::Join: + - '' + - - states. + - Ref: AWS::Region + - ".amazonaws.com" + Action: sts:AssumeRole + Policies: + - PolicyName: inline_policy_snapshots_rds + PolicyDocument: + Version: '2012-10-17' + Statement: + - Effect: Allow + Action: + - lambda:InvokeFunction + Resource: "*" + stateMachineTakeSnapshotsRDS: + Type: AWS::StepFunctions::StateMachine + Properties: + DefinitionString: + Fn::Join: + - '' + - - Fn::Join: + - "\n" + - - ' {"Comment":"Triggers snapshot backup for RDS instances",' + - ' "StartAt":"TakeSnapshots",' + - ' "States":{' + - ' "TakeSnapshots":{' + - ' "Type":"Task",' + - ' "Resource": ' + - "\"" + - Fn::GetAtt: + - lambdaTakeSnapshotsRDS + - Arn + - |- + " + , + - Fn::Join: + - "\n" + - - ' "Retry":[' + - " {" + - ' "ErrorEquals":[ ' + - ' "SnapshotToolException"' + - " ]," + - ' "IntervalSeconds":300,' + - ' "MaxAttempts":20,' + - ' "BackoffRate":1' + - " }," + - " {" + - ' "ErrorEquals":[ ' + - ' "States.ALL"], ' + - ' "IntervalSeconds": 30,' + - ' "MaxAttempts": 20,' + - ' "BackoffRate": 1' + - " }" + - " ]," + - ' "End": true ' + - " }" + - " }}" + RoleArn: + Fn::GetAtt: + - iamroleStateExecution + - Arn + statemachineShareSnapshotsRDS: + Type: AWS::StepFunctions::StateMachine + Condition: Share + Properties: + DefinitionString: + Fn::Join: + - '' + - - Fn::Join: + - "\n" + - - ' {"Comment":"Shares snapshots with DEST_ACCOUNT",' + - ' "StartAt":"ShareSnapshots",' + - ' "States":{' + - ' "ShareSnapshots":{' + - ' "Type":"Task",' + - ' "Resource": ' + - "\"" + - Fn::GetAtt: + - lambdaShareSnapshotsRDS + - Arn + - |- + " + , + - Fn::Join: + - "\n" + - - ' "Retry":[' + - " {" + - ' "ErrorEquals":[ ' + - ' "SnapshotToolException"' + - " ]," + - ' "IntervalSeconds":300,' + - ' "MaxAttempts":3,' + - ' "BackoffRate":1' + - " }," + - " {" + - ' "ErrorEquals":[ ' + - ' "States.ALL"], ' + - ' "IntervalSeconds": 30,' + - ' "MaxAttempts": 20,' + - ' "BackoffRate": 1' + - " }" + - " ]," + - ' "End": true ' + - " }" + - " }}" + RoleArn: + Fn::GetAtt: + - iamroleStateExecution + - Arn + statemachineDeleteOldSnapshotsRDS: + Type: AWS::StepFunctions::StateMachine + Condition: DeleteOld + Properties: + DefinitionString: + Fn::Join: + - '' + - - Fn::Join: + - "\n" + - - ' {"Comment":"DeleteOld management for RDS snapshots",' + - ' "StartAt":"DeleteOld",' + - ' "States":{' + - ' "DeleteOld":{' + - ' "Type":"Task",' + - ' "Resource": ' + - "\"" + - Fn::GetAtt: + - lambdaDeleteOldSnapshotsRDS + - Arn + - |- + " + , + - Fn::Join: + - "\n" + - - ' "Retry":[' + - " {" + - ' "ErrorEquals":[ ' + - ' "SnapshotToolException"' + - " ]," + - ' "IntervalSeconds":300,' + - ' "MaxAttempts":7,' + - ' "BackoffRate":1' + - " }," + - " {" + - ' "ErrorEquals":[ ' + - ' "States.ALL"], ' + - ' "IntervalSeconds": 30,' + - ' "MaxAttempts": 20,' + - ' "BackoffRate": 1' + - " }" + - " ]," + - ' "End": true ' + - " }" + - " }}" + RoleArn: + Fn::GetAtt: + - iamroleStateExecution + - Arn + iamroleStepInvocation: + Type: AWS::IAM::Role + Properties: + AssumeRolePolicyDocument: + Version: '2012-10-17' + Statement: + - Effect: Allow + Principal: + Service: events.amazonaws.com + Action: sts:AssumeRole + Policies: + - PolicyName: inline_policy_state_invocation + PolicyDocument: + Version: '2012-10-17' + Statement: + - Effect: Allow + Action: + - states:StartExecution + Resource: "*" + cwEventBackupRDS: + Type: AWS::Events::Rule + Properties: + Description: Triggers the TakeSnapshotsRDS state machine + ScheduleExpression: + Fn::Join: + - '' + - - cron( + - Ref: BackupSchedule + - ")" + State: ENABLED + Targets: + - Arn: + Ref: stateMachineTakeSnapshotsRDS + Id: Target1 + RoleArn: + Fn::GetAtt: + - iamroleStepInvocation + - Arn + cwEventShareSnapshotsRDS: + Type: AWS::Events::Rule + Condition: Share + Properties: + Description: Triggers the ShareSnapshotsRDS state machine + ScheduleExpression: + Fn::Join: + - '' + - - cron( + - "/10 * * * ? *" + - ")" + State: ENABLED + Targets: + - Arn: + Ref: statemachineShareSnapshotsRDS + Id: Target1 + RoleArn: + Fn::GetAtt: + - iamroleStepInvocation + - Arn + cwEventDeleteOldSnapshotsRDS: + Type: AWS::Events::Rule + Condition: DeleteOld + Properties: + Description: Triggers the DeleteOldSnapshotsRDS state machine + ScheduleExpression: + Fn::Join: + - '' + - - cron( + - 0 /1 * * ? * + - ")" + State: ENABLED + Targets: + - Arn: + Ref: statemachineDeleteOldSnapshotsRDS + Id: Target1 + RoleArn: + Fn::GetAtt: + - iamroleStepInvocation + - Arn + cwloggrouplambdaTakeSnapshotsRDS: + Type: AWS::Logs::LogGroup + Description: Log group for the lambdaTakeSnapshotsRDS function's logs + DependsOn: lambdaTakeSnapshotsRDS + Properties: + RetentionInDays: + Ref: LambdaCWLogRetention + LogGroupName: + Fn::Sub: + - "/aws/lambda/${func}" + - func: + Ref: lambdaTakeSnapshotsRDS + cwloggrouplambdaShareSnapshotsRDS: + Condition: Share + Type: AWS::Logs::LogGroup + Description: Log group for the lambdaShareSnapshotsRDS function's logs + DependsOn: lambdaShareSnapshotsRDS + Properties: + RetentionInDays: + Ref: LambdaCWLogRetention + LogGroupName: + Fn::Sub: + - "/aws/lambda/${func}" + - func: + Ref: lambdaShareSnapshotsRDS + cwloggrouplambdaDeleteOldSnapshotsRDS: + Type: AWS::Logs::LogGroup + Description: Log group for the lambdaDeleteOldSnapshotsRDS function's logs + Properties: + RetentionInDays: + Ref: LambdaCWLogRetention + LogGroupName: + Fn::Sub: + - "/aws/lambda/${func}" + - func: + Ref: LogGroupName +Outputs: + BackupFailedTopic: + Description: Subscribe to this topic to receive alerts of failed backups + Value: + Ref: topicBackupsFailed + ShareFailedTopic: + Condition: Share + Description: Subscribe to this topic to receive alerts of failures at sharing + snapshots with destination account + Value: + Ref: topicShareFailed + DeleteOldFailedTopic: + Condition: DeleteOld + Description: Subscribe to this topic to receive alerts of failures at deleting + old snapshots + Value: + Ref: topicDeleteOldFailed + SourceURL: + Description: For more information and documentation, see the source repository + at GitHub. + Value: https://github.com/awslabs/rds-snapshot-tool +Description: Snapshots Tool for RDS cross-region and cross-account (source account + stack) diff --git a/terraform/main.tf b/terraform/main.tf new file mode 100644 index 0000000..e69de29 From dd4707109148e0dc1cb255e97a54016da1ee2adb Mon Sep 17 00:00:00 2001 From: charlie keegan Date: Tue, 7 Feb 2023 11:36:31 +0000 Subject: [PATCH 03/48] initial cf2tf conversion --- terraform/outputs.tf | 14 + terraform/snapshots_tool_rds_dest.tf | 440 +++++++++++++++++++++++++++ terraform/variables.tf | 69 +++++ 3 files changed, 523 insertions(+) create mode 100644 terraform/outputs.tf create mode 100644 terraform/snapshots_tool_rds_dest.tf create mode 100644 terraform/variables.tf diff --git a/terraform/outputs.tf b/terraform/outputs.tf new file mode 100644 index 0000000..2d18814 --- /dev/null +++ b/terraform/outputs.tf @@ -0,0 +1,14 @@ +output "copy_failed_topic" { + description = "Subscribe to this topic to receive alerts of failed copies" + value = aws_sns_topic.topic_copy_failed_dest.id +} + +output "delete_old_failed_topic" { + description = "Subscribe to this topic to receive alerts of failures at deleting old snapshots" + value = aws_sns_topic.topic_delete_old_failed_dest.id +} + +output "source_url" { + description = "For more information and documentation, see the source repository at GitHub." + value = "https://github.com/awslabs/rds-snapshot-tool" +} diff --git a/terraform/snapshots_tool_rds_dest.tf b/terraform/snapshots_tool_rds_dest.tf new file mode 100644 index 0000000..7bfcd78 --- /dev/null +++ b/terraform/snapshots_tool_rds_dest.tf @@ -0,0 +1,440 @@ +data "aws_region" "current" {} + +data "aws_caller_identity" "current" {} + +locals { + DeleteOld = var.delete_old_snapshots == "TRUE" + CrossAccount = var.cross_account_copy == "TRUE" +} + +variable "code_bucket" { + description = "Name of the bucket that contains the lambda functions to deploy." + type = string +} + +variable "snapshot_pattern" { + description = "Python regex for matching instance names to backup. Use "ALL_SNAPSHOTS" to back up every RDS instance in the region." + type = string + default = "ALL_SNAPSHOTS" +} + +variable "retention_days" { + description = "Number of days to keep snapshots in retention before deleting them" + type = string + default = "7" +} + +variable "destination_region" { + description = "Destination region for snapshots." + type = string +} + +variable "log_level" { + description = "Log level for Lambda functions (DEBUG, INFO, WARN, ERROR, CRITICAL are valid values)." + type = string + default = "ERROR" +} + +variable "lambda_cw_log_retention" { + description = "Number of days to retain logs from the lambda functions in CloudWatch Logs" + type = string + default = "7" +} + +variable "source_region_override" { + description = "Set to the region where your RDS instances run, only if such region does not support Step Functions. Leave as NO otherwise" + type = string + default = "NO" +} + +variable "kms_key_destination" { + description = "Set to the ARN for the KMS key in the destination region to re-encrypt encrypted snapshots. Leave None if you are not using encryption" + type = string + default = "None" +} + +variable "kms_key_source" { + description = "Set to the ARN for the KMS key in the SOURCE region to re-encrypt encrypted snapshots. Leave None if you are not using encryption" + type = string + default = "None" +} + +variable "delete_old_snapshots" { + description = "Set to TRUE to enable deletion of snapshot based on RetentionDays. Set to FALSE to disable" + type = string + default = "TRUE" +} + +variable "cross_account_copy" { + description = "Enable copying snapshots across accounts. Set to FALSE if your source snapshosts are not on a different account" + type = string + default = "TRUE" +} + +variable "log_group_name" { + description = "Name for RDS snapshot log group." + type = string + default = "lambdaDeleteOldSnapshotsRDS-dest" +} + +resource "aws_sns_topic" "topic_copy_failed_dest" { + display_name = "copies_failed_dest_rds" +} + +resource "aws_sns_topic" "topic_delete_old_failed_dest" { + display_name = "delete_old_failed_dest_rds" +} + +resource "aws_sns_topic_policy" "snspolicy_copy_failed_dest" { + // CF Property(Topics) = [ + // aws_sns_topic.topic_copy_failed_dest.id, + // aws_sns_topic.topic_delete_old_failed_dest.id + // ] + policy = { + Version = "2008-10-17" + Id = "__default_policy_ID" + Statement = [ + { + Sid = "__default_statement_ID" + Effect = "Allow" + Principal = { + AWS = "*" + } + Action = [ + "SNS:GetTopicAttributes", + "SNS:SetTopicAttributes", + "SNS:AddPermission", + "SNS:RemovePermission", + "SNS:DeleteTopic", + "SNS:Subscribe", + "SNS:ListSubscriptionsByTopic", + "SNS:Publish", + "SNS:Receive" + ] + Resource = "*" + Condition = { + StringEquals = { + AWS:SourceOwner = data.aws_caller_identity.current.account_id + } + } + } + ] + } +} + +resource "aws_cloudwatch_composite_alarm" "alarmcw_copy_failed_dest" { + actions_enabled = "true" + // CF Property(ComparisonOperator) = "GreaterThanOrEqualToThreshold" + // CF Property(EvaluationPeriods) = "1" + // CF Property(MetricName) = "ExecutionsFailed" + // CF Property(Namespace) = "AWS/States" + // CF Property(Period) = "300" + // CF Property(Statistic) = "Sum" + // CF Property(Threshold) = "1.0" + alarm_actions = [ + aws_sns_topic.topic_copy_failed_dest.id + ] + // CF Property(Dimensions) = [ + // { + // Name = "StateMachineArn" + // Value = aws_ec2_instance_state.statemachine_copy_snapshots_dest_rds.id + // } + // ] +} + +resource "aws_cloudwatch_composite_alarm" "alarmcw_delete_old_failed_dest" { + count = locals.DeleteOld ? 1 : 0 + actions_enabled = "true" + // CF Property(ComparisonOperator) = "GreaterThanOrEqualToThreshold" + // CF Property(EvaluationPeriods) = "2" + // CF Property(MetricName) = "ExecutionsFailed" + // CF Property(Namespace) = "AWS/States" + // CF Property(Period) = "3600" + // CF Property(Statistic) = "Sum" + // CF Property(Threshold) = "2.0" + alarm_actions = [ + aws_sns_topic.topic_delete_old_failed_dest.id + ] + // CF Property(Dimensions) = [ + // { + // Name = "StateMachineArn" + // Value = aws_ec2_instance_state.statemachine_delete_old_snapshots_dest_rds[0].id + // } + // ] +} + +resource "aws_iam_role" "iamrole_snapshots_rds" { + assume_role_policy = { + Version = "2012-10-17" + Statement = [ + { + Effect = "Allow" + Principal = { + Service = "lambda.amazonaws.com" + } + Action = "sts:AssumeRole" + } + ] + } + force_detach_policies = [ + { + PolicyName = "inline_policy_snapshots_rds_cw_logs" + PolicyDocument = { + Version = "2012-10-17" + Statement = [ + { + Effect = "Allow" + Action = [ + "logs:CreateLogGroup", + "logs:CreateLogStream", + "logs:PutLogEvents" + ] + Resource = "arn:aws:logs:*:*:*" + } + ] + } + }, + { + PolicyName = "inline_policy_snapshots_rds" + PolicyDocument = { + Version = "2012-10-17" + Statement = [ + { + Effect = "Allow" + Action = [ + "rds:CreateDBSnapshot", + "rds:DeleteDBSnapshot", + "rds:DescribeDBInstances", + "rds:DescribeDBSnapshots", + "rds:ModifyDBSnapshotAttribute", + "rds:DescribeDBSnapshotAttributes", + "rds:CopyDBSnapshot", + "rds:ListTagsForResource", + "rds:AddTagsToResource" + ] + Resource = "*" + } + ] + } + }, + { + PolicyName = "inline_policy_snapshot_rds_kms_access" + PolicyDocument = { + Version = "2012-10-17" + Statement = [ + { + Sid = "AllowUseOfTheKey" + Effect = "Allow" + Action = [ + "kms:Encrypt", + "kms:Decrypt", + "kms:ReEncrypt*", + "kms:GenerateDataKey*", + "kms:DescribeKey" + ] + Resource = [ + "*" + ] + }, + { + Sid = "AllowAttachmentOfPersistentResources" + Effect = "Allow" + Action = [ + "kms:CreateGrant", + "kms:ListGrants", + "kms:RevokeGrant" + ] + Resource = [ + "*" + ] + Condition = { + Bool = { + kms:GrantIsForAWSResource = True + } + } + } + ] + } + } + ] +} + +resource "aws_lambda_function" "lambda_copy_snapshots_rds" { + code_signing_config_arn = { + S3Bucket = var.code_bucket + S3Key = local.CrossAccount ? "copy_snapshots_dest_rds.zip" : "copy_snapshots_no_x_account_rds.zip" + } + memory_size = 512 + description = "This functions copies snapshots for RDS Instances shared with this account. It checks for existing snapshots following the pattern specified in the environment variables with the following format: -YYYY-MM-DD-HH-MM" + environment { + variables = { + SNAPSHOT_PATTERN = var.snapshot_pattern + DEST_REGION = var.destination_region + LOG_LEVEL = var.log_level + REGION_OVERRIDE = var.source_region_override + KMS_KEY_DEST_REGION = var.kms_key_destination + KMS_KEY_SOURCE_REGION = var.kms_key_source + RETENTION_DAYS = var.retention_days + } + } + role = aws_iam_role.iamrole_snapshots_rds.arn + runtime = "python3.7" + handler = "lambda_function.lambda_handler" + timeout = 300 +} + +resource "aws_lambda_function" "lambda_delete_old_dest_rds" { + count = locals.DeleteOld ? 1 : 0 + code_signing_config_arn = { + S3Bucket = var.code_bucket + S3Key = local.CrossAccount ? "delete_old_snapshots_dest_rds.zip" : "delete_old_snapshots_no_x_account_rds.zip" + } + memory_size = 512 + description = "This function enforces retention on the snapshots shared with the destination account. " + environment { + variables = { + SNAPSHOT_PATTERN = var.snapshot_pattern + DEST_REGION = var.destination_region + RETENTION_DAYS = var.retention_days + LOG_LEVEL = var.log_level + } + } + role = aws_iam_role.iamrole_snapshots_rds.arn + runtime = "python3.7" + handler = "lambda_function.lambda_handler" + timeout = 300 +} + +resource "aws_iam_role" "iamrole_state_execution" { + assume_role_policy = { + Version = "2012-10-17" + Statement = [ + { + Effect = "Allow" + Principal = { + Service = join("", ["states.", data.aws_region.current.name, ".amazonaws.com"]) + } + Action = "sts:AssumeRole" + } + ] + } + force_detach_policies = [ + { + PolicyName = "inline_policy_rds_snapshot" + PolicyDocument = { + Version = "2012-10-17" + Statement = [ + { + Effect = "Allow" + Action = [ + "lambda:InvokeFunction" + ] + Resource = "*" + } + ] + } + } + ] +} + +resource "aws_ec2_instance_state" "statemachine_copy_snapshots_dest_rds" { + // CF Property(DefinitionString) = join("", [join(" + // ", [" {"Comment":"Copies snapshots locally and then to DEST_REGION",", " "StartAt":"CopySnapshots",", " "States":{", " "CopySnapshots":{", " "Type":"Task",", " "Resource": "]), """, aws_lambda_function.lambda_copy_snapshots_rds.arn, "" + // ,", join(" + // ", [" "Retry":[", " {", " "ErrorEquals":[ ", " "SnapshotToolException"", " ],", " "IntervalSeconds":300,", " "MaxAttempts":5,", " "BackoffRate":1", " },", " {", " "ErrorEquals":[ ", " "States.ALL"], ", " "IntervalSeconds": 30,", " "MaxAttempts": 20,", " "BackoffRate": 1", " }", " ],", " "End": true ", " }", " }}"])]) + // CF Property(RoleArn) = aws_iam_role.iamrole_state_execution.arn +} + +resource "aws_ec2_instance_state" "statemachine_delete_old_snapshots_dest_rds" { + count = locals.DeleteOld ? 1 : 0 + // CF Property(DefinitionString) = join("", [join(" + // ", [" {"Comment":"DeleteOld for RDS snapshots in destination region",", " "StartAt":"DeleteOldDestRegion",", " "States":{", " "DeleteOldDestRegion":{", " "Type":"Task",", " "Resource": "]), """, aws_lambda_function.lambda_delete_old_dest_rds.arn, "" + // ,", join(" + // ", [" "Retry":[", " {", " "ErrorEquals":[ ", " "SnapshotToolException"", " ],", " "IntervalSeconds":600,", " "MaxAttempts":5,", " "BackoffRate":1", " },", " {", " "ErrorEquals":[ ", " "States.ALL"], ", " "IntervalSeconds": 30,", " "MaxAttempts": 20,", " "BackoffRate": 1", " }", " ],", " "End": true ", " }", " }}"])]) + // CF Property(RoleArn) = aws_iam_role.iamrole_state_execution.arn +} + +resource "aws_iam_role" "iamrole_step_invocation" { + assume_role_policy = { + Version = "2012-10-17" + Statement = [ + { + Effect = "Allow" + Principal = { + Service = "events.amazonaws.com" + } + Action = "sts:AssumeRole" + } + ] + } + force_detach_policies = [ + { + PolicyName = "inline_policy_state_invocation" + PolicyDocument = { + Version = "2012-10-17" + Statement = [ + { + Effect = "Allow" + Action = [ + "states:StartExecution" + ] + Resource = "*" + } + ] + } + } + ] +} + +resource "aws_iot_topic_rule_destination" "cw_event_copy_snapshots_rds" { + // CF Property(Description) = "Triggers the RDS Copy state machine in the destination account" + // CF Property(ScheduleExpression) = join("", ["cron(", "/30 * * * ? *", ")"]) + // CF Property(State) = "ENABLED" + // CF Property(Targets) = [ + // { + // Arn = aws_ec2_instance_state.statemachine_copy_snapshots_dest_rds.id + // Id = "Target1" + // RoleArn = aws_iam_role.iamrole_step_invocation.arn + // } + // ] +} + +resource "aws_iot_topic_rule_destination" "cw_event_delete_old_snapshots_rds" { + count = locals.DeleteOld ? 1 : 0 + // CF Property(Description) = "Triggers the RDS DeleteOld state machine in the destination account" + // CF Property(ScheduleExpression) = join("", ["cron(", "0 /1 * * ? *", ")"]) + // CF Property(State) = "ENABLED" + // CF Property(Targets) = [ + // { + // Arn = aws_ec2_instance_state.statemachine_delete_old_snapshots_dest_rds[0].id + // Id = "Target1" + // RoleArn = aws_iam_role.iamrole_step_invocation.arn + // } + // ] +} + +resource "aws_inspector_resource_group" "cwloggroup_delete_old_snapshots_dest_rds" { + count = locals.DeleteOld ? 1 : 0 + // CF Property(RetentionInDays) = var.lambda_cw_log_retention + // CF Property(LogGroupName) = "/aws/lambda/${var.log_group_name}" +} + +resource "aws_inspector_resource_group" "cwloggrouplambda_copy_snapshots_rds" { + // CF Property(RetentionInDays) = var.lambda_cw_log_retention + // CF Property(LogGroupName) = "/aws/lambda/${aws_lambda_function.lambda_copy_snapshots_rds.arn}" +} + +output "copy_failed_topic" { + description = "Subscribe to this topic to receive alerts of failed copies" + value = aws_sns_topic.topic_copy_failed_dest.id +} + +output "delete_old_failed_topic" { + description = "Subscribe to this topic to receive alerts of failures at deleting old snapshots" + value = aws_sns_topic.topic_delete_old_failed_dest.id +} + +output "source_url" { + description = "For more information and documentation, see the source repository at GitHub." + value = "https://github.com/awslabs/rds-snapshot-tool" +} diff --git a/terraform/variables.tf b/terraform/variables.tf new file mode 100644 index 0000000..58bad23 --- /dev/null +++ b/terraform/variables.tf @@ -0,0 +1,69 @@ +variable "code_bucket" { + description = "Name of the bucket that contains the lambda functions to deploy." + type = string +} + +variable "snapshot_pattern" { + description = "Python regex for matching instance names to backup. Use "ALL_SNAPSHOTS" to back up every RDS instance in the region." + type = string + default = "ALL_SNAPSHOTS" +} + +variable "retention_days" { + description = "Number of days to keep snapshots in retention before deleting them" + type = string + default = "7" +} + +variable "destination_region" { + description = "Destination region for snapshots." + type = string +} + +variable "log_level" { + description = "Log level for Lambda functions (DEBUG, INFO, WARN, ERROR, CRITICAL are valid values)." + type = string + default = "ERROR" +} + +variable "lambda_cw_log_retention" { + description = "Number of days to retain logs from the lambda functions in CloudWatch Logs" + type = string + default = "7" +} + +variable "source_region_override" { + description = "Set to the region where your RDS instances run, only if such region does not support Step Functions. Leave as NO otherwise" + type = string + default = "NO" +} + +variable "kms_key_destination" { + description = "Set to the ARN for the KMS key in the destination region to re-encrypt encrypted snapshots. Leave None if you are not using encryption" + type = string + default = "None" +} + +variable "kms_key_source" { + description = "Set to the ARN for the KMS key in the SOURCE region to re-encrypt encrypted snapshots. Leave None if you are not using encryption" + type = string + default = "None" +} + +variable "delete_old_snapshots" { + description = "Set to TRUE to enable deletion of snapshot based on RetentionDays. Set to FALSE to disable" + type = string + default = "TRUE" +} + +variable "cross_account_copy" { + description = "Enable copying snapshots across accounts. Set to FALSE if your source snapshosts are not on a different account" + type = string + default = "TRUE" +} + +variable "log_group_name" { + description = "Name for RDS snapshot log group." + type = string + default = "lambdaDeleteOldSnapshotsRDS-dest" +} From 96afafe453cf3336fc61c634dfb33a1da1d57a2a Mon Sep 17 00:00:00 2001 From: charlie keegan Date: Tue, 7 Feb 2023 11:47:24 +0000 Subject: [PATCH 04/48] begin debug and add test dbs --- .../{ => snapshots_tool_rds_dest}/main.tf | 0 .../{ => snapshots_tool_rds_dest}/outputs.tf | 5 - .../snapshots_tool_rds_dest.tf | 143 ++++-------------- .../variables.tf | 0 terraform/test_dbs/main.tf | 24 +++ 5 files changed, 55 insertions(+), 117 deletions(-) rename terraform/{ => snapshots_tool_rds_dest}/main.tf (100%) rename terraform/{ => snapshots_tool_rds_dest}/outputs.tf (66%) rename terraform/{ => snapshots_tool_rds_dest}/snapshots_tool_rds_dest.tf (73%) rename terraform/{ => snapshots_tool_rds_dest}/variables.tf (100%) create mode 100644 terraform/test_dbs/main.tf diff --git a/terraform/main.tf b/terraform/snapshots_tool_rds_dest/main.tf similarity index 100% rename from terraform/main.tf rename to terraform/snapshots_tool_rds_dest/main.tf diff --git a/terraform/outputs.tf b/terraform/snapshots_tool_rds_dest/outputs.tf similarity index 66% rename from terraform/outputs.tf rename to terraform/snapshots_tool_rds_dest/outputs.tf index 2d18814..6e9e194 100644 --- a/terraform/outputs.tf +++ b/terraform/snapshots_tool_rds_dest/outputs.tf @@ -7,8 +7,3 @@ output "delete_old_failed_topic" { description = "Subscribe to this topic to receive alerts of failures at deleting old snapshots" value = aws_sns_topic.topic_delete_old_failed_dest.id } - -output "source_url" { - description = "For more information and documentation, see the source repository at GitHub." - value = "https://github.com/awslabs/rds-snapshot-tool" -} diff --git a/terraform/snapshots_tool_rds_dest.tf b/terraform/snapshots_tool_rds_dest/snapshots_tool_rds_dest.tf similarity index 73% rename from terraform/snapshots_tool_rds_dest.tf rename to terraform/snapshots_tool_rds_dest/snapshots_tool_rds_dest.tf index 7bfcd78..31e8cfe 100644 --- a/terraform/snapshots_tool_rds_dest.tf +++ b/terraform/snapshots_tool_rds_dest/snapshots_tool_rds_dest.tf @@ -3,84 +3,12 @@ data "aws_region" "current" {} data "aws_caller_identity" "current" {} locals { - DeleteOld = var.delete_old_snapshots == "TRUE" + DeleteOld = var.delete_old_snapshots == "TRUE" CrossAccount = var.cross_account_copy == "TRUE" } - -variable "code_bucket" { - description = "Name of the bucket that contains the lambda functions to deploy." - type = string -} - -variable "snapshot_pattern" { - description = "Python regex for matching instance names to backup. Use "ALL_SNAPSHOTS" to back up every RDS instance in the region." - type = string - default = "ALL_SNAPSHOTS" -} - -variable "retention_days" { - description = "Number of days to keep snapshots in retention before deleting them" - type = string - default = "7" -} - -variable "destination_region" { - description = "Destination region for snapshots." - type = string -} - -variable "log_level" { - description = "Log level for Lambda functions (DEBUG, INFO, WARN, ERROR, CRITICAL are valid values)." - type = string - default = "ERROR" -} - -variable "lambda_cw_log_retention" { - description = "Number of days to retain logs from the lambda functions in CloudWatch Logs" - type = string - default = "7" -} - -variable "source_region_override" { - description = "Set to the region where your RDS instances run, only if such region does not support Step Functions. Leave as NO otherwise" - type = string - default = "NO" -} - -variable "kms_key_destination" { - description = "Set to the ARN for the KMS key in the destination region to re-encrypt encrypted snapshots. Leave None if you are not using encryption" - type = string - default = "None" -} - -variable "kms_key_source" { - description = "Set to the ARN for the KMS key in the SOURCE region to re-encrypt encrypted snapshots. Leave None if you are not using encryption" - type = string - default = "None" -} - -variable "delete_old_snapshots" { - description = "Set to TRUE to enable deletion of snapshot based on RetentionDays. Set to FALSE to disable" - type = string - default = "TRUE" -} - -variable "cross_account_copy" { - description = "Enable copying snapshots across accounts. Set to FALSE if your source snapshosts are not on a different account" - type = string - default = "TRUE" -} - -variable "log_group_name" { - description = "Name for RDS snapshot log group." - type = string - default = "lambdaDeleteOldSnapshotsRDS-dest" -} - resource "aws_sns_topic" "topic_copy_failed_dest" { display_name = "copies_failed_dest_rds" } - resource "aws_sns_topic" "topic_delete_old_failed_dest" { display_name = "delete_old_failed_dest_rds" } @@ -92,10 +20,10 @@ resource "aws_sns_topic_policy" "snspolicy_copy_failed_dest" { // ] policy = { Version = "2008-10-17" - Id = "__default_policy_ID" + Id = "__default_policy_ID" Statement = [ { - Sid = "__default_statement_ID" + Sid = "__default_statement_ID" Effect = "Allow" Principal = { AWS = "*" @@ -113,9 +41,9 @@ resource "aws_sns_topic_policy" "snspolicy_copy_failed_dest" { ] Resource = "*" Condition = { - StringEquals = { - AWS:SourceOwner = data.aws_caller_identity.current.account_id - } + test = "StringEquals" + variable = "AWS:SourceOwner" + values = [data.aws_caller_identity.current.account_id] } } ] @@ -143,7 +71,7 @@ resource "aws_cloudwatch_composite_alarm" "alarmcw_copy_failed_dest" { } resource "aws_cloudwatch_composite_alarm" "alarmcw_delete_old_failed_dest" { - count = locals.DeleteOld ? 1 : 0 + count = locals.DeleteOld ? 1 : 0 actions_enabled = "true" // CF Property(ComparisonOperator) = "GreaterThanOrEqualToThreshold" // CF Property(EvaluationPeriods) = "2" @@ -223,7 +151,7 @@ resource "aws_iam_role" "iamrole_snapshots_rds" { Version = "2012-10-17" Statement = [ { - Sid = "AllowUseOfTheKey" + Sid = "AllowUseOfTheKey" Effect = "Allow" Action = [ "kms:Encrypt", @@ -237,7 +165,7 @@ resource "aws_iam_role" "iamrole_snapshots_rds" { ] }, { - Sid = "AllowAttachmentOfPersistentResources" + Sid = "AllowAttachmentOfPersistentResources" Effect = "Allow" Action = [ "kms:CreateGrant", @@ -248,9 +176,9 @@ resource "aws_iam_role" "iamrole_snapshots_rds" { "*" ] Condition = { - Bool = { - kms:GrantIsForAWSResource = True - } + test = "Bool" + variable = "kms:GrantIsForAWSResource" + values = ["True"] } } ] @@ -262,22 +190,22 @@ resource "aws_iam_role" "iamrole_snapshots_rds" { resource "aws_lambda_function" "lambda_copy_snapshots_rds" { code_signing_config_arn = { S3Bucket = var.code_bucket - S3Key = local.CrossAccount ? "copy_snapshots_dest_rds.zip" : "copy_snapshots_no_x_account_rds.zip" + S3Key = local.CrossAccount ? "copy_snapshots_dest_rds.zip" : "copy_snapshots_no_x_account_rds.zip" } memory_size = 512 description = "This functions copies snapshots for RDS Instances shared with this account. It checks for existing snapshots following the pattern specified in the environment variables with the following format: -YYYY-MM-DD-HH-MM" environment { variables = { - SNAPSHOT_PATTERN = var.snapshot_pattern - DEST_REGION = var.destination_region - LOG_LEVEL = var.log_level - REGION_OVERRIDE = var.source_region_override - KMS_KEY_DEST_REGION = var.kms_key_destination + SNAPSHOT_PATTERN = var.snapshot_pattern + DEST_REGION = var.destination_region + LOG_LEVEL = var.log_level + REGION_OVERRIDE = var.source_region_override + KMS_KEY_DEST_REGION = var.kms_key_destination KMS_KEY_SOURCE_REGION = var.kms_key_source - RETENTION_DAYS = var.retention_days + RETENTION_DAYS = var.retention_days } } - role = aws_iam_role.iamrole_snapshots_rds.arn + role = aws_iam_role.iamrole_snapshots_rds.arn runtime = "python3.7" handler = "lambda_function.lambda_handler" timeout = 300 @@ -287,19 +215,19 @@ resource "aws_lambda_function" "lambda_delete_old_dest_rds" { count = locals.DeleteOld ? 1 : 0 code_signing_config_arn = { S3Bucket = var.code_bucket - S3Key = local.CrossAccount ? "delete_old_snapshots_dest_rds.zip" : "delete_old_snapshots_no_x_account_rds.zip" + S3Key = local.CrossAccount ? "delete_old_snapshots_dest_rds.zip" : "delete_old_snapshots_no_x_account_rds.zip" } memory_size = 512 description = "This function enforces retention on the snapshots shared with the destination account. " environment { variables = { SNAPSHOT_PATTERN = var.snapshot_pattern - DEST_REGION = var.destination_region - RETENTION_DAYS = var.retention_days - LOG_LEVEL = var.log_level + DEST_REGION = var.destination_region + RETENTION_DAYS = var.retention_days + LOG_LEVEL = var.log_level } } - role = aws_iam_role.iamrole_snapshots_rds.arn + role = aws_iam_role.iamrole_snapshots_rds.arn runtime = "python3.7" handler = "lambda_function.lambda_handler" timeout = 300 @@ -401,6 +329,12 @@ resource "aws_iot_topic_rule_destination" "cw_event_copy_snapshots_rds" { resource "aws_iot_topic_rule_destination" "cw_event_delete_old_snapshots_rds" { count = locals.DeleteOld ? 1 : 0 + vpc_configuration { + role_arn = "" + security_groups = "" + subnet_ids = "" + vpc_id = "" + } // CF Property(Description) = "Triggers the RDS DeleteOld state machine in the destination account" // CF Property(ScheduleExpression) = join("", ["cron(", "0 /1 * * ? *", ")"]) // CF Property(State) = "ENABLED" @@ -423,18 +357,3 @@ resource "aws_inspector_resource_group" "cwloggrouplambda_copy_snapshots_rds" { // CF Property(RetentionInDays) = var.lambda_cw_log_retention // CF Property(LogGroupName) = "/aws/lambda/${aws_lambda_function.lambda_copy_snapshots_rds.arn}" } - -output "copy_failed_topic" { - description = "Subscribe to this topic to receive alerts of failed copies" - value = aws_sns_topic.topic_copy_failed_dest.id -} - -output "delete_old_failed_topic" { - description = "Subscribe to this topic to receive alerts of failures at deleting old snapshots" - value = aws_sns_topic.topic_delete_old_failed_dest.id -} - -output "source_url" { - description = "For more information and documentation, see the source repository at GitHub." - value = "https://github.com/awslabs/rds-snapshot-tool" -} diff --git a/terraform/variables.tf b/terraform/snapshots_tool_rds_dest/variables.tf similarity index 100% rename from terraform/variables.tf rename to terraform/snapshots_tool_rds_dest/variables.tf diff --git a/terraform/test_dbs/main.tf b/terraform/test_dbs/main.tf new file mode 100644 index 0000000..fee010a --- /dev/null +++ b/terraform/test_dbs/main.tf @@ -0,0 +1,24 @@ +resource "aws_db_instance" "source" { + allocated_storage = 10 + db_name = "db_migration_source_db" + engine = "mysql" + engine_version = "5.7" + instance_class = "db.t3.micro" + username = "foo" + password = "foobarbaz" + parameter_group_name = "default.mysql5.7" + skip_final_snapshot = true +} + +resource "aws_db_instance" "destination" { + allocated_storage = 10 + db_name = "db_migration_destination_db" + engine = "mysql" + engine_version = "5.7" + instance_class = "db.t3.micro" + username = "foo" + password = "foobarbaz" + parameter_group_name = "default.mysql5.7" + skip_final_snapshot = true +} + From 87bbbf8f522693313f5f26046e8fb89a17012172 Mon Sep 17 00:00:00 2001 From: charlie keegan Date: Tue, 7 Feb 2023 11:51:05 +0000 Subject: [PATCH 05/48] change file structure and add provider --- terraform/main.tf | 8 ++++ .../snapshots_tool_rds_dest/main.tf | 0 .../snapshots_tool_rds_dest/outputs.tf | 0 .../snapshots_tool_rds_dest.tf | 0 .../snapshots_tool_rds_dest/variables.tf | 46 +++++++++---------- terraform/{ => modules}/test_dbs/main.tf | 0 terraform/provider.tf | 13 ++++++ 7 files changed, 44 insertions(+), 23 deletions(-) create mode 100644 terraform/main.tf rename terraform/{ => modules}/snapshots_tool_rds_dest/main.tf (100%) rename terraform/{ => modules}/snapshots_tool_rds_dest/outputs.tf (100%) rename terraform/{ => modules}/snapshots_tool_rds_dest/snapshots_tool_rds_dest.tf (100%) rename terraform/{ => modules}/snapshots_tool_rds_dest/variables.tf (72%) rename terraform/{ => modules}/test_dbs/main.tf (100%) create mode 100644 terraform/provider.tf diff --git a/terraform/main.tf b/terraform/main.tf new file mode 100644 index 0000000..393e916 --- /dev/null +++ b/terraform/main.tf @@ -0,0 +1,8 @@ +module "test_dbs" { + source = "./modules/test_dbs" +} + + +module "test_dbs" { + source = "./modules/snapshots_tool_rds_dest" +} diff --git a/terraform/snapshots_tool_rds_dest/main.tf b/terraform/modules/snapshots_tool_rds_dest/main.tf similarity index 100% rename from terraform/snapshots_tool_rds_dest/main.tf rename to terraform/modules/snapshots_tool_rds_dest/main.tf diff --git a/terraform/snapshots_tool_rds_dest/outputs.tf b/terraform/modules/snapshots_tool_rds_dest/outputs.tf similarity index 100% rename from terraform/snapshots_tool_rds_dest/outputs.tf rename to terraform/modules/snapshots_tool_rds_dest/outputs.tf diff --git a/terraform/snapshots_tool_rds_dest/snapshots_tool_rds_dest.tf b/terraform/modules/snapshots_tool_rds_dest/snapshots_tool_rds_dest.tf similarity index 100% rename from terraform/snapshots_tool_rds_dest/snapshots_tool_rds_dest.tf rename to terraform/modules/snapshots_tool_rds_dest/snapshots_tool_rds_dest.tf diff --git a/terraform/snapshots_tool_rds_dest/variables.tf b/terraform/modules/snapshots_tool_rds_dest/variables.tf similarity index 72% rename from terraform/snapshots_tool_rds_dest/variables.tf rename to terraform/modules/snapshots_tool_rds_dest/variables.tf index 58bad23..66573bd 100644 --- a/terraform/snapshots_tool_rds_dest/variables.tf +++ b/terraform/modules/snapshots_tool_rds_dest/variables.tf @@ -1,69 +1,69 @@ variable "code_bucket" { description = "Name of the bucket that contains the lambda functions to deploy." - type = string + type = string } variable "snapshot_pattern" { - description = "Python regex for matching instance names to backup. Use "ALL_SNAPSHOTS" to back up every RDS instance in the region." - type = string - default = "ALL_SNAPSHOTS" + description = "Python regex for matching instance names to backup. Use 'ALL_SNAPSHOTS' to back up every RDS instance in the region." + type = string + default = "ALL_SNAPSHOTS" } variable "retention_days" { description = "Number of days to keep snapshots in retention before deleting them" - type = string - default = "7" + type = string + default = "7" } variable "destination_region" { description = "Destination region for snapshots." - type = string + type = string } variable "log_level" { description = "Log level for Lambda functions (DEBUG, INFO, WARN, ERROR, CRITICAL are valid values)." - type = string - default = "ERROR" + type = string + default = "ERROR" } variable "lambda_cw_log_retention" { description = "Number of days to retain logs from the lambda functions in CloudWatch Logs" - type = string - default = "7" + type = string + default = "7" } variable "source_region_override" { description = "Set to the region where your RDS instances run, only if such region does not support Step Functions. Leave as NO otherwise" - type = string - default = "NO" + type = string + default = "NO" } variable "kms_key_destination" { description = "Set to the ARN for the KMS key in the destination region to re-encrypt encrypted snapshots. Leave None if you are not using encryption" - type = string - default = "None" + type = string + default = "None" } variable "kms_key_source" { description = "Set to the ARN for the KMS key in the SOURCE region to re-encrypt encrypted snapshots. Leave None if you are not using encryption" - type = string - default = "None" + type = string + default = "None" } variable "delete_old_snapshots" { description = "Set to TRUE to enable deletion of snapshot based on RetentionDays. Set to FALSE to disable" - type = string - default = "TRUE" + type = string + default = "TRUE" } variable "cross_account_copy" { description = "Enable copying snapshots across accounts. Set to FALSE if your source snapshosts are not on a different account" - type = string - default = "TRUE" + type = string + default = "TRUE" } variable "log_group_name" { description = "Name for RDS snapshot log group." - type = string - default = "lambdaDeleteOldSnapshotsRDS-dest" + type = string + default = "lambdaDeleteOldSnapshotsRDS-dest" } diff --git a/terraform/test_dbs/main.tf b/terraform/modules/test_dbs/main.tf similarity index 100% rename from terraform/test_dbs/main.tf rename to terraform/modules/test_dbs/main.tf diff --git a/terraform/provider.tf b/terraform/provider.tf new file mode 100644 index 0000000..8e4ce6e --- /dev/null +++ b/terraform/provider.tf @@ -0,0 +1,13 @@ +terraform { + required_providers { + aws = { + source = "hashicorp/aws" + version = "~> 4.0" + } + } +} + +# Configure the AWS Provider +provider "aws" { + region = "us-east-1" +} From 93e1cc2511b92b5762786151cf88fd5ba4b76e09 Mon Sep 17 00:00:00 2001 From: charlie keegan Date: Tue, 7 Feb 2023 11:52:39 +0000 Subject: [PATCH 06/48] cf2tf conversion of source --- .../modules/snapshots_tool_rds_source/data.tf | 4 + .../snapshots_tool_rds_source/locals.tf | 5 + .../snapshots_tool_rds_source/output.tf | 20 + .../snapshots_tool_rds_source/resource.tf | 381 ++++++++++++++++++ .../snapshots_tool_rds_source/variable.tf | 76 ++++ 5 files changed, 486 insertions(+) create mode 100644 terraform/modules/snapshots_tool_rds_source/data.tf create mode 100644 terraform/modules/snapshots_tool_rds_source/locals.tf create mode 100644 terraform/modules/snapshots_tool_rds_source/output.tf create mode 100644 terraform/modules/snapshots_tool_rds_source/resource.tf create mode 100644 terraform/modules/snapshots_tool_rds_source/variable.tf diff --git a/terraform/modules/snapshots_tool_rds_source/data.tf b/terraform/modules/snapshots_tool_rds_source/data.tf new file mode 100644 index 0000000..cfd929b --- /dev/null +++ b/terraform/modules/snapshots_tool_rds_source/data.tf @@ -0,0 +1,4 @@ +data "aws_region" "current" {} + +data "aws_caller_identity" "current" {} + diff --git a/terraform/modules/snapshots_tool_rds_source/locals.tf b/terraform/modules/snapshots_tool_rds_source/locals.tf new file mode 100644 index 0000000..283731b --- /dev/null +++ b/terraform/modules/snapshots_tool_rds_source/locals.tf @@ -0,0 +1,5 @@ +locals { + Share = var.share_snapshots == "TRUE" + DeleteOld = var.delete_old_snapshots == "TRUE" +} + diff --git a/terraform/modules/snapshots_tool_rds_source/output.tf b/terraform/modules/snapshots_tool_rds_source/output.tf new file mode 100644 index 0000000..5a4e15a --- /dev/null +++ b/terraform/modules/snapshots_tool_rds_source/output.tf @@ -0,0 +1,20 @@ +output "backup_failed_topic" { + description = "Subscribe to this topic to receive alerts of failed backups" + value = aws_sns_topic.topic_backups_failed.id +} + +output "share_failed_topic" { + description = "Subscribe to this topic to receive alerts of failures at sharing snapshots with destination account" + value = aws_sns_topic.topic_share_failed.id +} + +output "delete_old_failed_topic" { + description = "Subscribe to this topic to receive alerts of failures at deleting old snapshots" + value = aws_sns_topic.topic_delete_old_failed.id +} + +output "source_url" { + description = "For more information and documentation, see the source repository at GitHub." + value = "https://github.com/awslabs/rds-snapshot-tool" +} + diff --git a/terraform/modules/snapshots_tool_rds_source/resource.tf b/terraform/modules/snapshots_tool_rds_source/resource.tf new file mode 100644 index 0000000..b7ac9a4 --- /dev/null +++ b/terraform/modules/snapshots_tool_rds_source/resource.tf @@ -0,0 +1,381 @@ +resource "aws_sns_topic" "topic_backups_failed" { + display_name = "backups_failed_rds" +} + +resource "aws_sns_topic" "topic_share_failed" { + display_name = "share_failed_rds" +} + +resource "aws_sns_topic" "topic_delete_old_failed" { + display_name = "delete_old_failed_rds" +} + +resource "aws_sns_topic_policy" "snspolicy_snapshots_rds" { + // CF Property(Topics) = [ + // aws_sns_topic.topic_backups_failed.id, + // aws_sns_topic.topic_share_failed.id, + // aws_sns_topic.topic_delete_old_failed.id + // ] + policy = { + Version = "2008-10-17" + Id = "__default_policy_ID" + Statement = [ + { + Sid = "__default_statement_ID" + Effect = "Allow" + Principal = { + AWS = "*" + } + Action = [ + "SNS:GetTopicAttributes", + "SNS:SetTopicAttributes", + "SNS:AddPermission", + "SNS:RemovePermission", + "SNS:DeleteTopic", + "SNS:Subscribe", + "SNS:ListSubscriptionsByTopic", + "SNS:Publish", + "SNS:Receive" + ] + Resource = "*" + Condition = { + StringEquals = { + AWS:SourceOwner = data.aws_caller_identity.current.account_id + } + } + } + ] + } +} + +resource "aws_cloudwatch_composite_alarm" "alarmcw_backups_failed" { + actions_enabled = "true" + // CF Property(ComparisonOperator) = "GreaterThanOrEqualToThreshold" + // CF Property(EvaluationPeriods) = "1" + // CF Property(MetricName) = "ExecutionsFailed" + // CF Property(Namespace) = "AWS/States" + // CF Property(Period) = "300" + // CF Property(Statistic) = "Sum" + // CF Property(Threshold) = "1.0" + alarm_actions = [ + aws_sns_topic.topic_backups_failed.id + ] + // CF Property(Dimensions) = [ + // { + // Name = "StateMachineArn" + // Value = aws_ec2_instance_state.state_machine_take_snapshots_rds.id + // } + // ] +} + +resource "aws_cloudwatch_composite_alarm" "alarmcw_share_failed" { + count = locals.Share ? 1 : 0 + actions_enabled = "true" + // CF Property(ComparisonOperator) = "GreaterThanOrEqualToThreshold" + // CF Property(EvaluationPeriods) = "2" + // CF Property(MetricName) = "ExecutionsFailed" + // CF Property(Namespace) = "AWS/States" + // CF Property(Period) = "3600" + // CF Property(Statistic) = "Sum" + // CF Property(Threshold) = "2.0" + alarm_actions = [ + aws_sns_topic.topic_share_failed.id + ] + // CF Property(Dimensions) = [ + // { + // Name = "StateMachineArn" + // Value = aws_ec2_instance_state.statemachine_share_snapshots_rds[0].id + // } + // ] +} + +resource "aws_cloudwatch_composite_alarm" "alarmcw_delete_old_failed" { + count = locals.DeleteOld ? 1 : 0 + actions_enabled = "true" + // CF Property(ComparisonOperator) = "GreaterThanOrEqualToThreshold" + // CF Property(EvaluationPeriods) = "2" + // CF Property(MetricName) = "ExecutionsFailed" + // CF Property(Namespace) = "AWS/States" + // CF Property(Period) = "3600" + // CF Property(Statistic) = "Sum" + // CF Property(Threshold) = "2.0" + alarm_actions = [ + aws_sns_topic.topic_delete_old_failed.id + ] + // CF Property(Dimensions) = [ + // { + // Name = "StateMachineArn" + // Value = aws_ec2_instance_state.statemachine_delete_old_snapshots_rds[0].id + // } + // ] +} + +resource "aws_iam_role" "iamrole_snapshots_rds" { + assume_role_policy = { + Version = "2012-10-17" + Statement = [ + { + Effect = "Allow" + Principal = { + Service = "lambda.amazonaws.com" + } + Action = "sts:AssumeRole" + } + ] + } + force_detach_policies = [ + { + PolicyName = "inline_policy_snapshots_rds_cw_logs" + PolicyDocument = { + Version = "2012-10-17" + Statement = [ + { + Effect = "Allow" + Action = [ + "logs:CreateLogGroup", + "logs:CreateLogStream", + "logs:PutLogEvents" + ] + Resource = "arn:aws:logs:*:*:*" + } + ] + } + }, + { + PolicyName = "inline_policy_snapshots_rds" + PolicyDocument = { + Version = "2012-10-17" + Statement = [ + { + Effect = "Allow" + Action = [ + "rds:CreateDBSnapshot", + "rds:DeleteDBSnapshot", + "rds:DescribeDBInstances", + "rds:DescribeDBSnapshots", + "rds:ModifyDBSnapshotAttribute", + "rds:DescribeDBSnapshotAttributes", + "rds:ListTagsForResource", + "rds:AddTagsToResource" + ] + Resource = "*" + } + ] + } + } + ] +} + +resource "aws_lambda_function" "lambda_take_snapshots_rds" { + code_signing_config_arn = { + S3Bucket = var.code_bucket + S3Key = "take_snapshots_rds.zip" + } + memory_size = 512 + description = "This functions triggers snapshots creation for RDS instances. It checks for existing snapshots following the pattern and interval specified in the environment variables with the following format: -YYYY-MM-DD-HH-MM" + environment { + variables = { + INTERVAL = var.backup_interval + PATTERN = var.instance_name_pattern + LOG_LEVEL = var.log_level + REGION_OVERRIDE = var.source_region_override + TAGGEDINSTANCE = var.tagged_instance + } + } + role = aws_iam_role.iamrole_snapshots_rds.arn + runtime = "python3.7" + handler = "lambda_function.lambda_handler" + timeout = 300 +} + +resource "aws_lambda_function" "lambda_share_snapshots_rds" { + count = locals.Share ? 1 : 0 + code_signing_config_arn = { + S3Bucket = var.code_bucket + S3Key = "share_snapshots_rds.zip" + } + memory_size = 512 + description = "This function shares snapshots created by the take_snapshots_rds function with DEST_ACCOUNT specified in the environment variables. " + environment { + variables = { + DEST_ACCOUNT = var.destination_account + LOG_LEVEL = var.log_level + PATTERN = var.instance_name_pattern + REGION_OVERRIDE = var.source_region_override + } + } + role = aws_iam_role.iamrole_snapshots_rds.arn + runtime = "python3.7" + handler = "lambda_function.lambda_handler" + timeout = 300 +} + +resource "aws_lambda_function" "lambda_delete_old_snapshots_rds" { + count = locals.DeleteOld ? 1 : 0 + code_signing_config_arn = { + S3Bucket = var.code_bucket + S3Key = "delete_old_snapshots_rds.zip" + } + memory_size = 512 + description = "This function deletes snapshots created by the take_snapshots_rds function. " + environment { + variables = { + RETENTION_DAYS = var.retention_days + PATTERN = var.instance_name_pattern + LOG_LEVEL = var.log_level + REGION_OVERRIDE = var.source_region_override + } + } + role = aws_iam_role.iamrole_snapshots_rds.arn + runtime = "python3.7" + handler = "lambda_function.lambda_handler" + timeout = 300 +} + +resource "aws_iam_role" "iamrole_state_execution" { + assume_role_policy = { + Version = "2012-10-17" + Statement = [ + { + Effect = "Allow" + Principal = { + Service = join("", ["states.", data.aws_region.current.name, ".amazonaws.com"]) + } + Action = "sts:AssumeRole" + } + ] + } + force_detach_policies = [ + { + PolicyName = "inline_policy_snapshots_rds" + PolicyDocument = { + Version = "2012-10-17" + Statement = [ + { + Effect = "Allow" + Action = [ + "lambda:InvokeFunction" + ] + Resource = "*" + } + ] + } + } + ] +} + +resource "aws_ec2_instance_state" "state_machine_take_snapshots_rds" { + // CF Property(DefinitionString) = join("", [join(" + // ", [" {"Comment":"Triggers snapshot backup for RDS instances",", " "StartAt":"TakeSnapshots",", " "States":{", " "TakeSnapshots":{", " "Type":"Task",", " "Resource": "]), """, aws_lambda_function.lambda_take_snapshots_rds.arn, "" + // ,", join(" + // ", [" "Retry":[", " {", " "ErrorEquals":[ ", " "SnapshotToolException"", " ],", " "IntervalSeconds":300,", " "MaxAttempts":20,", " "BackoffRate":1", " },", " {", " "ErrorEquals":[ ", " "States.ALL"], ", " "IntervalSeconds": 30,", " "MaxAttempts": 20,", " "BackoffRate": 1", " }", " ],", " "End": true ", " }", " }}"])]) + // CF Property(RoleArn) = aws_iam_role.iamrole_state_execution.arn +} + +resource "aws_ec2_instance_state" "statemachine_share_snapshots_rds" { + count = locals.Share ? 1 : 0 + // CF Property(DefinitionString) = join("", [join(" + // ", [" {"Comment":"Shares snapshots with DEST_ACCOUNT",", " "StartAt":"ShareSnapshots",", " "States":{", " "ShareSnapshots":{", " "Type":"Task",", " "Resource": "]), """, aws_lambda_function.lambda_share_snapshots_rds.arn, "" + // ,", join(" + // ", [" "Retry":[", " {", " "ErrorEquals":[ ", " "SnapshotToolException"", " ],", " "IntervalSeconds":300,", " "MaxAttempts":3,", " "BackoffRate":1", " },", " {", " "ErrorEquals":[ ", " "States.ALL"], ", " "IntervalSeconds": 30,", " "MaxAttempts": 20,", " "BackoffRate": 1", " }", " ],", " "End": true ", " }", " }}"])]) + // CF Property(RoleArn) = aws_iam_role.iamrole_state_execution.arn +} + +resource "aws_ec2_instance_state" "statemachine_delete_old_snapshots_rds" { + count = locals.DeleteOld ? 1 : 0 + // CF Property(DefinitionString) = join("", [join(" + // ", [" {"Comment":"DeleteOld management for RDS snapshots",", " "StartAt":"DeleteOld",", " "States":{", " "DeleteOld":{", " "Type":"Task",", " "Resource": "]), """, aws_lambda_function.lambda_delete_old_snapshots_rds.arn, "" + // ,", join(" + // ", [" "Retry":[", " {", " "ErrorEquals":[ ", " "SnapshotToolException"", " ],", " "IntervalSeconds":300,", " "MaxAttempts":7,", " "BackoffRate":1", " },", " {", " "ErrorEquals":[ ", " "States.ALL"], ", " "IntervalSeconds": 30,", " "MaxAttempts": 20,", " "BackoffRate": 1", " }", " ],", " "End": true ", " }", " }}"])]) + // CF Property(RoleArn) = aws_iam_role.iamrole_state_execution.arn +} + +resource "aws_iam_role" "iamrole_step_invocation" { + assume_role_policy = { + Version = "2012-10-17" + Statement = [ + { + Effect = "Allow" + Principal = { + Service = "events.amazonaws.com" + } + Action = "sts:AssumeRole" + } + ] + } + force_detach_policies = [ + { + PolicyName = "inline_policy_state_invocation" + PolicyDocument = { + Version = "2012-10-17" + Statement = [ + { + Effect = "Allow" + Action = [ + "states:StartExecution" + ] + Resource = "*" + } + ] + } + } + ] +} + +resource "aws_iot_topic_rule_destination" "cw_event_backup_rds" { + // CF Property(Description) = "Triggers the TakeSnapshotsRDS state machine" + // CF Property(ScheduleExpression) = join("", ["cron(", var.backup_schedule, ")"]) + // CF Property(State) = "ENABLED" + // CF Property(Targets) = [ + // { + // Arn = aws_ec2_instance_state.state_machine_take_snapshots_rds.id + // Id = "Target1" + // RoleArn = aws_iam_role.iamrole_step_invocation.arn + // } + // ] +} + +resource "aws_iot_topic_rule_destination" "cw_event_share_snapshots_rds" { + count = locals.Share ? 1 : 0 + // CF Property(Description) = "Triggers the ShareSnapshotsRDS state machine" + // CF Property(ScheduleExpression) = join("", ["cron(", "/10 * * * ? *", ")"]) + // CF Property(State) = "ENABLED" + // CF Property(Targets) = [ + // { + // Arn = aws_ec2_instance_state.statemachine_share_snapshots_rds[0].id + // Id = "Target1" + // RoleArn = aws_iam_role.iamrole_step_invocation.arn + // } + // ] +} + +resource "aws_iot_topic_rule_destination" "cw_event_delete_old_snapshots_rds" { + count = locals.DeleteOld ? 1 : 0 + // CF Property(Description) = "Triggers the DeleteOldSnapshotsRDS state machine" + // CF Property(ScheduleExpression) = join("", ["cron(", "0 /1 * * ? *", ")"]) + // CF Property(State) = "ENABLED" + // CF Property(Targets) = [ + // { + // Arn = aws_ec2_instance_state.statemachine_delete_old_snapshots_rds[0].id + // Id = "Target1" + // RoleArn = aws_iam_role.iamrole_step_invocation.arn + // } + // ] +} + +resource "aws_inspector_resource_group" "cwloggrouplambda_take_snapshots_rds" { + // CF Property(RetentionInDays) = var.lambda_cw_log_retention + // CF Property(LogGroupName) = "/aws/lambda/${aws_lambda_function.lambda_take_snapshots_rds.arn}" +} + +resource "aws_inspector_resource_group" "cwloggrouplambda_share_snapshots_rds" { + count = locals.Share ? 1 : 0 + // CF Property(RetentionInDays) = var.lambda_cw_log_retention + // CF Property(LogGroupName) = "/aws/lambda/${aws_lambda_function.lambda_share_snapshots_rds[0].arn}" +} + +resource "aws_inspector_resource_group" "cwloggrouplambda_delete_old_snapshots_rds" { + // CF Property(RetentionInDays) = var.lambda_cw_log_retention + // CF Property(LogGroupName) = "/aws/lambda/${var.log_group_name}" +} + diff --git a/terraform/modules/snapshots_tool_rds_source/variable.tf b/terraform/modules/snapshots_tool_rds_source/variable.tf new file mode 100644 index 0000000..71eab88 --- /dev/null +++ b/terraform/modules/snapshots_tool_rds_source/variable.tf @@ -0,0 +1,76 @@ +variable "code_bucket" { + description = "Name of the bucket that contains the lambda functions to deploy." + type = string +} + +variable "instance_name_pattern" { + description = "Python regex for matching cluster identifiers to backup. Use "ALL_INSTANCES" to back up every RDS instance in the region." + type = string + default = "ALL_INSTANCES" +} + +variable "backup_interval" { + description = "Interval for backups in hours. Default is 24" + type = string + default = "24" +} + +variable "destination_account" { + description = "Destination account with no dashes." + type = string + default = "000000000000" +} + +variable "share_snapshots" { + type = string + default = "TRUE" +} + +variable "backup_schedule" { + description = "Backup schedule in Cloudwatch Event cron format. Needs to run at least once for every Interval. The default value runs once every at 1AM UTC. More information: http://docs.aws.amazon.com/AmazonCloudWatch/latest/events/ScheduledEvents.html" + type = string + default = "0 1 * * ? *" +} + +variable "retention_days" { + description = "Number of days to keep snapshots in retention before deleting them" + type = string + default = "7" +} + +variable "log_level" { + description = "Log level for Lambda functions (DEBUG, INFO, WARN, ERROR, CRITICAL are valid values)." + type = string + default = "ERROR" +} + +variable "lambda_cw_log_retention" { + description = "Number of days to retain logs from the lambda functions in CloudWatch Logs" + type = string + default = "7" +} + +variable "source_region_override" { + description = "Set to the region where your RDS instances run, only if such region does not support Step Functions. Leave as NO otherwise" + type = string + default = "NO" +} + +variable "delete_old_snapshots" { + description = "Set to TRUE to enable deletion of snapshot based on RetentionDays. Set to FALSE to disable" + type = string + default = "TRUE" +} + +variable "tagged_instance" { + description = "Set to TRUE to filter instances that have tag CopyDBSnapshot set to True. Set to FALSE to disable" + type = string + default = "FALSE" +} + +variable "log_group_name" { + description = "Name for RDS snapshot log group." + type = string + default = "lambdaDeleteOldSnapshotsRDS-source" +} + From fb6ca289f2d7c52dee386360119bc9c0aacec5a8 Mon Sep 17 00:00:00 2001 From: charlie keegan Date: Tue, 7 Feb 2023 11:54:03 +0000 Subject: [PATCH 07/48] fix typos --- .../modules/snapshots_tool_rds_dest/main.tf | 355 +++++++++++++++++ .../snapshots_tool_rds_dest.tf | 359 ------------------ .../snapshots_tool_rds_source/resource.tf | 46 +-- .../snapshots_tool_rds_source/variable.tf | 50 +-- 4 files changed, 403 insertions(+), 407 deletions(-) delete mode 100644 terraform/modules/snapshots_tool_rds_dest/snapshots_tool_rds_dest.tf diff --git a/terraform/modules/snapshots_tool_rds_dest/main.tf b/terraform/modules/snapshots_tool_rds_dest/main.tf index e69de29..f4455d0 100644 --- a/terraform/modules/snapshots_tool_rds_dest/main.tf +++ b/terraform/modules/snapshots_tool_rds_dest/main.tf @@ -0,0 +1,355 @@ +locals { + DeleteOld = var.delete_old_snapshots == "TRUE" + CrossAccount = var.cross_account_copy == "TRUE" +} +resource "aws_sns_topic" "topic_copy_failed_dest" { + display_name = "copies_failed_dest_rds" +} +resource "aws_sns_topic" "topic_delete_old_failed_dest" { + display_name = "delete_old_failed_dest_rds" +} + +resource "aws_sns_topic_policy" "snspolicy_copy_failed_dest" { + // CF Property(Topics) = [ + // aws_sns_topic.topic_copy_failed_dest.id, + // aws_sns_topic.topic_delete_old_failed_dest.id + // ] + policy = { + Version = "2008-10-17" + Id = "__default_policy_ID" + Statement = [ + { + Sid = "__default_statement_ID" + Effect = "Allow" + Principal = { + AWS = "*" + } + Action = [ + "SNS:GetTopicAttributes", + "SNS:SetTopicAttributes", + "SNS:AddPermission", + "SNS:RemovePermission", + "SNS:DeleteTopic", + "SNS:Subscribe", + "SNS:ListSubscriptionsByTopic", + "SNS:Publish", + "SNS:Receive" + ] + Resource = "*" + Condition = { + test = "StringEquals" + variable = "AWS:SourceOwner" + values = [data.aws_caller_identity.current.account_id] + } + } + ] + } +} + +resource "aws_cloudwatch_composite_alarm" "alarmcw_copy_failed_dest" { + actions_enabled = "true" + // CF Property(ComparisonOperator) = "GreaterThanOrEqualToThreshold" + // CF Property(EvaluationPeriods) = "1" + // CF Property(MetricName) = "ExecutionsFailed" + // CF Property(Namespace) = "AWS/States" + // CF Property(Period) = "300" + // CF Property(Statistic) = "Sum" + // CF Property(Threshold) = "1.0" + alarm_actions = [ + aws_sns_topic.topic_copy_failed_dest.id + ] + // CF Property(Dimensions) = [ + // { + // Name = "StateMachineArn" + // Value = aws_ec2_instance_state.statemachine_copy_snapshots_dest_rds.id + // } + // ] +} + +resource "aws_cloudwatch_composite_alarm" "alarmcw_delete_old_failed_dest" { + count = locals.DeleteOld ? 1 : 0 + actions_enabled = "true" + // CF Property(ComparisonOperator) = "GreaterThanOrEqualToThreshold" + // CF Property(EvaluationPeriods) = "2" + // CF Property(MetricName) = "ExecutionsFailed" + // CF Property(Namespace) = "AWS/States" + // CF Property(Period) = "3600" + // CF Property(Statistic) = "Sum" + // CF Property(Threshold) = "2.0" + alarm_actions = [ + aws_sns_topic.topic_delete_old_failed_dest.id + ] + // CF Property(Dimensions) = [ + // { + // Name = "StateMachineArn" + // Value = aws_ec2_instance_state.statemachine_delete_old_snapshots_dest_rds[0].id + // } + // ] +} + +resource "aws_iam_role" "iamrole_snapshots_rds" { + assume_role_policy = { + Version = "2012-10-17" + Statement = [ + { + Effect = "Allow" + Principal = { + Service = "lambda.amazonaws.com" + } + Action = "sts:AssumeRole" + } + ] + } + force_detach_policies = [ + { + PolicyName = "inline_policy_snapshots_rds_cw_logs" + PolicyDocument = { + Version = "2012-10-17" + Statement = [ + { + Effect = "Allow" + Action = [ + "logs:CreateLogGroup", + "logs:CreateLogStream", + "logs:PutLogEvents" + ] + Resource = "arn:aws:logs:*:*:*" + } + ] + } + }, + { + PolicyName = "inline_policy_snapshots_rds" + PolicyDocument = { + Version = "2012-10-17" + Statement = [ + { + Effect = "Allow" + Action = [ + "rds:CreateDBSnapshot", + "rds:DeleteDBSnapshot", + "rds:DescribeDBInstances", + "rds:DescribeDBSnapshots", + "rds:ModifyDBSnapshotAttribute", + "rds:DescribeDBSnapshotAttributes", + "rds:CopyDBSnapshot", + "rds:ListTagsForResource", + "rds:AddTagsToResource" + ] + Resource = "*" + } + ] + } + }, + { + PolicyName = "inline_policy_snapshot_rds_kms_access" + PolicyDocument = { + Version = "2012-10-17" + Statement = [ + { + Sid = "AllowUseOfTheKey" + Effect = "Allow" + Action = [ + "kms:Encrypt", + "kms:Decrypt", + "kms:ReEncrypt*", + "kms:GenerateDataKey*", + "kms:DescribeKey" + ] + Resource = [ + "*" + ] + }, + { + Sid = "AllowAttachmentOfPersistentResources" + Effect = "Allow" + Action = [ + "kms:CreateGrant", + "kms:ListGrants", + "kms:RevokeGrant" + ] + Resource = [ + "*" + ] + Condition = { + test = "Bool" + variable = "kms:GrantIsForAWSResource" + values = ["True"] + } + } + ] + } + } + ] +} + +resource "aws_lambda_function" "lambda_copy_snapshots_rds" { + code_signing_config_arn = { + S3Bucket = var.code_bucket + S3Key = local.CrossAccount ? "copy_snapshots_dest_rds.zip" : "copy_snapshots_no_x_account_rds.zip" + } + memory_size = 512 + description = "This functions copies snapshots for RDS Instances shared with this account. It checks for existing snapshots following the pattern specified in the environment variables with the following format: -YYYY-MM-DD-HH-MM" + environment { + variables = { + SNAPSHOT_PATTERN = var.snapshot_pattern + DEST_REGION = var.destination_region + LOG_LEVEL = var.log_level + REGION_OVERRIDE = var.source_region_override + KMS_KEY_DEST_REGION = var.kms_key_destination + KMS_KEY_SOURCE_REGION = var.kms_key_source + RETENTION_DAYS = var.retention_days + } + } + role = aws_iam_role.iamrole_snapshots_rds.arn + runtime = "python3.7" + handler = "lambda_function.lambda_handler" + timeout = 300 +} + +resource "aws_lambda_function" "lambda_delete_old_dest_rds" { + count = locals.DeleteOld ? 1 : 0 + code_signing_config_arn = { + S3Bucket = var.code_bucket + S3Key = local.CrossAccount ? "delete_old_snapshots_dest_rds.zip" : "delete_old_snapshots_no_x_account_rds.zip" + } + memory_size = 512 + description = "This function enforces retention on the snapshots shared with the destination account. " + environment { + variables = { + SNAPSHOT_PATTERN = var.snapshot_pattern + DEST_REGION = var.destination_region + RETENTION_DAYS = var.retention_days + LOG_LEVEL = var.log_level + } + } + role = aws_iam_role.iamrole_snapshots_rds.arn + runtime = "python3.7" + handler = "lambda_function.lambda_handler" + timeout = 300 +} + +resource "aws_iam_role" "iamrole_state_execution" { + assume_role_policy = { + Version = "2012-10-17" + Statement = [ + { + Effect = "Allow" + Principal = { + Service = join("", ["states.", data.aws_region.current.name, ".amazonaws.com"]) + } + Action = "sts:AssumeRole" + } + ] + } + force_detach_policies = [ + { + PolicyName = "inline_policy_rds_snapshot" + PolicyDocument = { + Version = "2012-10-17" + Statement = [ + { + Effect = "Allow" + Action = [ + "lambda:InvokeFunction" + ] + Resource = "*" + } + ] + } + } + ] +} + +resource "aws_ec2_instance_state" "statemachine_copy_snapshots_dest_rds" { + // CF Property(DefinitionString) = join("", [join(" + // ", [" {"Comment":"Copies snapshots locally and then to DEST_REGION",", " "StartAt":"CopySnapshots",", " "States":{", " "CopySnapshots":{", " "Type":"Task",", " "Resource": "]), """, aws_lambda_function.lambda_copy_snapshots_rds.arn, "" + // ,", join(" + // ", [" "Retry":[", " {", " "ErrorEquals":[ ", " "SnapshotToolException"", " ],", " "IntervalSeconds":300,", " "MaxAttempts":5,", " "BackoffRate":1", " },", " {", " "ErrorEquals":[ ", " "States.ALL"], ", " "IntervalSeconds": 30,", " "MaxAttempts": 20,", " "BackoffRate": 1", " }", " ],", " "End": true ", " }", " }}"])]) + // CF Property(RoleArn) = aws_iam_role.iamrole_state_execution.arn +} + +resource "aws_ec2_instance_state" "statemachine_delete_old_snapshots_dest_rds" { + count = locals.DeleteOld ? 1 : 0 + // CF Property(DefinitionString) = join("", [join(" + // ", [" {"Comment":"DeleteOld for RDS snapshots in destination region",", " "StartAt":"DeleteOldDestRegion",", " "States":{", " "DeleteOldDestRegion":{", " "Type":"Task",", " "Resource": "]), """, aws_lambda_function.lambda_delete_old_dest_rds.arn, "" + // ,", join(" + // ", [" "Retry":[", " {", " "ErrorEquals":[ ", " "SnapshotToolException"", " ],", " "IntervalSeconds":600,", " "MaxAttempts":5,", " "BackoffRate":1", " },", " {", " "ErrorEquals":[ ", " "States.ALL"], ", " "IntervalSeconds": 30,", " "MaxAttempts": 20,", " "BackoffRate": 1", " }", " ],", " "End": true ", " }", " }}"])]) + // CF Property(RoleArn) = aws_iam_role.iamrole_state_execution.arn +} + +resource "aws_iam_role" "iamrole_step_invocation" { + assume_role_policy = { + Version = "2012-10-17" + Statement = [ + { + Effect = "Allow" + Principal = { + Service = "events.amazonaws.com" + } + Action = "sts:AssumeRole" + } + ] + } + force_detach_policies = [ + { + PolicyName = "inline_policy_state_invocation" + PolicyDocument = { + Version = "2012-10-17" + Statement = [ + { + Effect = "Allow" + Action = [ + "states:StartExecution" + ] + Resource = "*" + } + ] + } + } + ] +} + +resource "aws_iot_topic_rule_destination" "cw_event_copy_snapshots_rds" { + // CF Property(Description) = "Triggers the RDS Copy state machine in the destination account" + // CF Property(ScheduleExpression) = join("", ["cron(", "/30 * * * ? *", ")"]) + // CF Property(State) = "ENABLED" + // CF Property(Targets) = [ + // { + // Arn = aws_ec2_instance_state.statemachine_copy_snapshots_dest_rds.id + // Id = "Target1" + // RoleArn = aws_iam_role.iamrole_step_invocation.arn + // } + // ] +} + +resource "aws_iot_topic_rule_destination" "cw_event_delete_old_snapshots_rds" { + count = locals.DeleteOld ? 1 : 0 + vpc_configuration { + role_arn = "" + security_groups = "" + subnet_ids = "" + vpc_id = "" + } + // CF Property(Description) = "Triggers the RDS DeleteOld state machine in the destination account" + // CF Property(ScheduleExpression) = join("", ["cron(", "0 /1 * * ? *", ")"]) + // CF Property(State) = "ENABLED" + // CF Property(Targets) = [ + // { + // Arn = aws_ec2_instance_state.statemachine_delete_old_snapshots_dest_rds[0].id + // Id = "Target1" + // RoleArn = aws_iam_role.iamrole_step_invocation.arn + // } + // ] +} + +resource "aws_inspector_resource_group" "cwloggroup_delete_old_snapshots_dest_rds" { + count = locals.DeleteOld ? 1 : 0 + // CF Property(RetentionInDays) = var.lambda_cw_log_retention + // CF Property(LogGroupName) = "/aws/lambda/${var.log_group_name}" +} + +resource "aws_inspector_resource_group" "cwloggrouplambda_copy_snapshots_rds" { + // CF Property(RetentionInDays) = var.lambda_cw_log_retention + // CF Property(LogGroupName) = "/aws/lambda/${aws_lambda_function.lambda_copy_snapshots_rds.arn}" +} diff --git a/terraform/modules/snapshots_tool_rds_dest/snapshots_tool_rds_dest.tf b/terraform/modules/snapshots_tool_rds_dest/snapshots_tool_rds_dest.tf deleted file mode 100644 index 31e8cfe..0000000 --- a/terraform/modules/snapshots_tool_rds_dest/snapshots_tool_rds_dest.tf +++ /dev/null @@ -1,359 +0,0 @@ -data "aws_region" "current" {} - -data "aws_caller_identity" "current" {} - -locals { - DeleteOld = var.delete_old_snapshots == "TRUE" - CrossAccount = var.cross_account_copy == "TRUE" -} -resource "aws_sns_topic" "topic_copy_failed_dest" { - display_name = "copies_failed_dest_rds" -} -resource "aws_sns_topic" "topic_delete_old_failed_dest" { - display_name = "delete_old_failed_dest_rds" -} - -resource "aws_sns_topic_policy" "snspolicy_copy_failed_dest" { - // CF Property(Topics) = [ - // aws_sns_topic.topic_copy_failed_dest.id, - // aws_sns_topic.topic_delete_old_failed_dest.id - // ] - policy = { - Version = "2008-10-17" - Id = "__default_policy_ID" - Statement = [ - { - Sid = "__default_statement_ID" - Effect = "Allow" - Principal = { - AWS = "*" - } - Action = [ - "SNS:GetTopicAttributes", - "SNS:SetTopicAttributes", - "SNS:AddPermission", - "SNS:RemovePermission", - "SNS:DeleteTopic", - "SNS:Subscribe", - "SNS:ListSubscriptionsByTopic", - "SNS:Publish", - "SNS:Receive" - ] - Resource = "*" - Condition = { - test = "StringEquals" - variable = "AWS:SourceOwner" - values = [data.aws_caller_identity.current.account_id] - } - } - ] - } -} - -resource "aws_cloudwatch_composite_alarm" "alarmcw_copy_failed_dest" { - actions_enabled = "true" - // CF Property(ComparisonOperator) = "GreaterThanOrEqualToThreshold" - // CF Property(EvaluationPeriods) = "1" - // CF Property(MetricName) = "ExecutionsFailed" - // CF Property(Namespace) = "AWS/States" - // CF Property(Period) = "300" - // CF Property(Statistic) = "Sum" - // CF Property(Threshold) = "1.0" - alarm_actions = [ - aws_sns_topic.topic_copy_failed_dest.id - ] - // CF Property(Dimensions) = [ - // { - // Name = "StateMachineArn" - // Value = aws_ec2_instance_state.statemachine_copy_snapshots_dest_rds.id - // } - // ] -} - -resource "aws_cloudwatch_composite_alarm" "alarmcw_delete_old_failed_dest" { - count = locals.DeleteOld ? 1 : 0 - actions_enabled = "true" - // CF Property(ComparisonOperator) = "GreaterThanOrEqualToThreshold" - // CF Property(EvaluationPeriods) = "2" - // CF Property(MetricName) = "ExecutionsFailed" - // CF Property(Namespace) = "AWS/States" - // CF Property(Period) = "3600" - // CF Property(Statistic) = "Sum" - // CF Property(Threshold) = "2.0" - alarm_actions = [ - aws_sns_topic.topic_delete_old_failed_dest.id - ] - // CF Property(Dimensions) = [ - // { - // Name = "StateMachineArn" - // Value = aws_ec2_instance_state.statemachine_delete_old_snapshots_dest_rds[0].id - // } - // ] -} - -resource "aws_iam_role" "iamrole_snapshots_rds" { - assume_role_policy = { - Version = "2012-10-17" - Statement = [ - { - Effect = "Allow" - Principal = { - Service = "lambda.amazonaws.com" - } - Action = "sts:AssumeRole" - } - ] - } - force_detach_policies = [ - { - PolicyName = "inline_policy_snapshots_rds_cw_logs" - PolicyDocument = { - Version = "2012-10-17" - Statement = [ - { - Effect = "Allow" - Action = [ - "logs:CreateLogGroup", - "logs:CreateLogStream", - "logs:PutLogEvents" - ] - Resource = "arn:aws:logs:*:*:*" - } - ] - } - }, - { - PolicyName = "inline_policy_snapshots_rds" - PolicyDocument = { - Version = "2012-10-17" - Statement = [ - { - Effect = "Allow" - Action = [ - "rds:CreateDBSnapshot", - "rds:DeleteDBSnapshot", - "rds:DescribeDBInstances", - "rds:DescribeDBSnapshots", - "rds:ModifyDBSnapshotAttribute", - "rds:DescribeDBSnapshotAttributes", - "rds:CopyDBSnapshot", - "rds:ListTagsForResource", - "rds:AddTagsToResource" - ] - Resource = "*" - } - ] - } - }, - { - PolicyName = "inline_policy_snapshot_rds_kms_access" - PolicyDocument = { - Version = "2012-10-17" - Statement = [ - { - Sid = "AllowUseOfTheKey" - Effect = "Allow" - Action = [ - "kms:Encrypt", - "kms:Decrypt", - "kms:ReEncrypt*", - "kms:GenerateDataKey*", - "kms:DescribeKey" - ] - Resource = [ - "*" - ] - }, - { - Sid = "AllowAttachmentOfPersistentResources" - Effect = "Allow" - Action = [ - "kms:CreateGrant", - "kms:ListGrants", - "kms:RevokeGrant" - ] - Resource = [ - "*" - ] - Condition = { - test = "Bool" - variable = "kms:GrantIsForAWSResource" - values = ["True"] - } - } - ] - } - } - ] -} - -resource "aws_lambda_function" "lambda_copy_snapshots_rds" { - code_signing_config_arn = { - S3Bucket = var.code_bucket - S3Key = local.CrossAccount ? "copy_snapshots_dest_rds.zip" : "copy_snapshots_no_x_account_rds.zip" - } - memory_size = 512 - description = "This functions copies snapshots for RDS Instances shared with this account. It checks for existing snapshots following the pattern specified in the environment variables with the following format: -YYYY-MM-DD-HH-MM" - environment { - variables = { - SNAPSHOT_PATTERN = var.snapshot_pattern - DEST_REGION = var.destination_region - LOG_LEVEL = var.log_level - REGION_OVERRIDE = var.source_region_override - KMS_KEY_DEST_REGION = var.kms_key_destination - KMS_KEY_SOURCE_REGION = var.kms_key_source - RETENTION_DAYS = var.retention_days - } - } - role = aws_iam_role.iamrole_snapshots_rds.arn - runtime = "python3.7" - handler = "lambda_function.lambda_handler" - timeout = 300 -} - -resource "aws_lambda_function" "lambda_delete_old_dest_rds" { - count = locals.DeleteOld ? 1 : 0 - code_signing_config_arn = { - S3Bucket = var.code_bucket - S3Key = local.CrossAccount ? "delete_old_snapshots_dest_rds.zip" : "delete_old_snapshots_no_x_account_rds.zip" - } - memory_size = 512 - description = "This function enforces retention on the snapshots shared with the destination account. " - environment { - variables = { - SNAPSHOT_PATTERN = var.snapshot_pattern - DEST_REGION = var.destination_region - RETENTION_DAYS = var.retention_days - LOG_LEVEL = var.log_level - } - } - role = aws_iam_role.iamrole_snapshots_rds.arn - runtime = "python3.7" - handler = "lambda_function.lambda_handler" - timeout = 300 -} - -resource "aws_iam_role" "iamrole_state_execution" { - assume_role_policy = { - Version = "2012-10-17" - Statement = [ - { - Effect = "Allow" - Principal = { - Service = join("", ["states.", data.aws_region.current.name, ".amazonaws.com"]) - } - Action = "sts:AssumeRole" - } - ] - } - force_detach_policies = [ - { - PolicyName = "inline_policy_rds_snapshot" - PolicyDocument = { - Version = "2012-10-17" - Statement = [ - { - Effect = "Allow" - Action = [ - "lambda:InvokeFunction" - ] - Resource = "*" - } - ] - } - } - ] -} - -resource "aws_ec2_instance_state" "statemachine_copy_snapshots_dest_rds" { - // CF Property(DefinitionString) = join("", [join(" - // ", [" {"Comment":"Copies snapshots locally and then to DEST_REGION",", " "StartAt":"CopySnapshots",", " "States":{", " "CopySnapshots":{", " "Type":"Task",", " "Resource": "]), """, aws_lambda_function.lambda_copy_snapshots_rds.arn, "" - // ,", join(" - // ", [" "Retry":[", " {", " "ErrorEquals":[ ", " "SnapshotToolException"", " ],", " "IntervalSeconds":300,", " "MaxAttempts":5,", " "BackoffRate":1", " },", " {", " "ErrorEquals":[ ", " "States.ALL"], ", " "IntervalSeconds": 30,", " "MaxAttempts": 20,", " "BackoffRate": 1", " }", " ],", " "End": true ", " }", " }}"])]) - // CF Property(RoleArn) = aws_iam_role.iamrole_state_execution.arn -} - -resource "aws_ec2_instance_state" "statemachine_delete_old_snapshots_dest_rds" { - count = locals.DeleteOld ? 1 : 0 - // CF Property(DefinitionString) = join("", [join(" - // ", [" {"Comment":"DeleteOld for RDS snapshots in destination region",", " "StartAt":"DeleteOldDestRegion",", " "States":{", " "DeleteOldDestRegion":{", " "Type":"Task",", " "Resource": "]), """, aws_lambda_function.lambda_delete_old_dest_rds.arn, "" - // ,", join(" - // ", [" "Retry":[", " {", " "ErrorEquals":[ ", " "SnapshotToolException"", " ],", " "IntervalSeconds":600,", " "MaxAttempts":5,", " "BackoffRate":1", " },", " {", " "ErrorEquals":[ ", " "States.ALL"], ", " "IntervalSeconds": 30,", " "MaxAttempts": 20,", " "BackoffRate": 1", " }", " ],", " "End": true ", " }", " }}"])]) - // CF Property(RoleArn) = aws_iam_role.iamrole_state_execution.arn -} - -resource "aws_iam_role" "iamrole_step_invocation" { - assume_role_policy = { - Version = "2012-10-17" - Statement = [ - { - Effect = "Allow" - Principal = { - Service = "events.amazonaws.com" - } - Action = "sts:AssumeRole" - } - ] - } - force_detach_policies = [ - { - PolicyName = "inline_policy_state_invocation" - PolicyDocument = { - Version = "2012-10-17" - Statement = [ - { - Effect = "Allow" - Action = [ - "states:StartExecution" - ] - Resource = "*" - } - ] - } - } - ] -} - -resource "aws_iot_topic_rule_destination" "cw_event_copy_snapshots_rds" { - // CF Property(Description) = "Triggers the RDS Copy state machine in the destination account" - // CF Property(ScheduleExpression) = join("", ["cron(", "/30 * * * ? *", ")"]) - // CF Property(State) = "ENABLED" - // CF Property(Targets) = [ - // { - // Arn = aws_ec2_instance_state.statemachine_copy_snapshots_dest_rds.id - // Id = "Target1" - // RoleArn = aws_iam_role.iamrole_step_invocation.arn - // } - // ] -} - -resource "aws_iot_topic_rule_destination" "cw_event_delete_old_snapshots_rds" { - count = locals.DeleteOld ? 1 : 0 - vpc_configuration { - role_arn = "" - security_groups = "" - subnet_ids = "" - vpc_id = "" - } - // CF Property(Description) = "Triggers the RDS DeleteOld state machine in the destination account" - // CF Property(ScheduleExpression) = join("", ["cron(", "0 /1 * * ? *", ")"]) - // CF Property(State) = "ENABLED" - // CF Property(Targets) = [ - // { - // Arn = aws_ec2_instance_state.statemachine_delete_old_snapshots_dest_rds[0].id - // Id = "Target1" - // RoleArn = aws_iam_role.iamrole_step_invocation.arn - // } - // ] -} - -resource "aws_inspector_resource_group" "cwloggroup_delete_old_snapshots_dest_rds" { - count = locals.DeleteOld ? 1 : 0 - // CF Property(RetentionInDays) = var.lambda_cw_log_retention - // CF Property(LogGroupName) = "/aws/lambda/${var.log_group_name}" -} - -resource "aws_inspector_resource_group" "cwloggrouplambda_copy_snapshots_rds" { - // CF Property(RetentionInDays) = var.lambda_cw_log_retention - // CF Property(LogGroupName) = "/aws/lambda/${aws_lambda_function.lambda_copy_snapshots_rds.arn}" -} diff --git a/terraform/modules/snapshots_tool_rds_source/resource.tf b/terraform/modules/snapshots_tool_rds_source/resource.tf index b7ac9a4..41cf650 100644 --- a/terraform/modules/snapshots_tool_rds_source/resource.tf +++ b/terraform/modules/snapshots_tool_rds_source/resource.tf @@ -18,10 +18,10 @@ resource "aws_sns_topic_policy" "snspolicy_snapshots_rds" { // ] policy = { Version = "2008-10-17" - Id = "__default_policy_ID" + Id = "__default_policy_ID" Statement = [ { - Sid = "__default_statement_ID" + Sid = "__default_statement_ID" Effect = "Allow" Principal = { AWS = "*" @@ -39,9 +39,9 @@ resource "aws_sns_topic_policy" "snspolicy_snapshots_rds" { ] Resource = "*" Condition = { - StringEquals = { - AWS:SourceOwner = data.aws_caller_identity.current.account_id - } + test = "StringEquals" + variable = "AWS:SourceOwner" + values = [data.aws_caller_identity.current.account_id] } } ] @@ -69,7 +69,7 @@ resource "aws_cloudwatch_composite_alarm" "alarmcw_backups_failed" { } resource "aws_cloudwatch_composite_alarm" "alarmcw_share_failed" { - count = locals.Share ? 1 : 0 + count = locals.Share ? 1 : 0 actions_enabled = "true" // CF Property(ComparisonOperator) = "GreaterThanOrEqualToThreshold" // CF Property(EvaluationPeriods) = "2" @@ -90,7 +90,7 @@ resource "aws_cloudwatch_composite_alarm" "alarmcw_share_failed" { } resource "aws_cloudwatch_composite_alarm" "alarmcw_delete_old_failed" { - count = locals.DeleteOld ? 1 : 0 + count = locals.DeleteOld ? 1 : 0 actions_enabled = "true" // CF Property(ComparisonOperator) = "GreaterThanOrEqualToThreshold" // CF Property(EvaluationPeriods) = "2" @@ -169,20 +169,20 @@ resource "aws_iam_role" "iamrole_snapshots_rds" { resource "aws_lambda_function" "lambda_take_snapshots_rds" { code_signing_config_arn = { S3Bucket = var.code_bucket - S3Key = "take_snapshots_rds.zip" + S3Key = "take_snapshots_rds.zip" } memory_size = 512 description = "This functions triggers snapshots creation for RDS instances. It checks for existing snapshots following the pattern and interval specified in the environment variables with the following format: -YYYY-MM-DD-HH-MM" environment { variables = { - INTERVAL = var.backup_interval - PATTERN = var.instance_name_pattern - LOG_LEVEL = var.log_level + INTERVAL = var.backup_interval + PATTERN = var.instance_name_pattern + LOG_LEVEL = var.log_level REGION_OVERRIDE = var.source_region_override - TAGGEDINSTANCE = var.tagged_instance + TAGGEDINSTANCE = var.tagged_instance } } - role = aws_iam_role.iamrole_snapshots_rds.arn + role = aws_iam_role.iamrole_snapshots_rds.arn runtime = "python3.7" handler = "lambda_function.lambda_handler" timeout = 300 @@ -192,19 +192,19 @@ resource "aws_lambda_function" "lambda_share_snapshots_rds" { count = locals.Share ? 1 : 0 code_signing_config_arn = { S3Bucket = var.code_bucket - S3Key = "share_snapshots_rds.zip" + S3Key = "share_snapshots_rds.zip" } memory_size = 512 description = "This function shares snapshots created by the take_snapshots_rds function with DEST_ACCOUNT specified in the environment variables. " environment { variables = { - DEST_ACCOUNT = var.destination_account - LOG_LEVEL = var.log_level - PATTERN = var.instance_name_pattern + DEST_ACCOUNT = var.destination_account + LOG_LEVEL = var.log_level + PATTERN = var.instance_name_pattern REGION_OVERRIDE = var.source_region_override } } - role = aws_iam_role.iamrole_snapshots_rds.arn + role = aws_iam_role.iamrole_snapshots_rds.arn runtime = "python3.7" handler = "lambda_function.lambda_handler" timeout = 300 @@ -214,19 +214,19 @@ resource "aws_lambda_function" "lambda_delete_old_snapshots_rds" { count = locals.DeleteOld ? 1 : 0 code_signing_config_arn = { S3Bucket = var.code_bucket - S3Key = "delete_old_snapshots_rds.zip" + S3Key = "delete_old_snapshots_rds.zip" } memory_size = 512 description = "This function deletes snapshots created by the take_snapshots_rds function. " environment { variables = { - RETENTION_DAYS = var.retention_days - PATTERN = var.instance_name_pattern - LOG_LEVEL = var.log_level + RETENTION_DAYS = var.retention_days + PATTERN = var.instance_name_pattern + LOG_LEVEL = var.log_level REGION_OVERRIDE = var.source_region_override } } - role = aws_iam_role.iamrole_snapshots_rds.arn + role = aws_iam_role.iamrole_snapshots_rds.arn runtime = "python3.7" handler = "lambda_function.lambda_handler" timeout = 300 diff --git a/terraform/modules/snapshots_tool_rds_source/variable.tf b/terraform/modules/snapshots_tool_rds_source/variable.tf index 71eab88..10857d8 100644 --- a/terraform/modules/snapshots_tool_rds_source/variable.tf +++ b/terraform/modules/snapshots_tool_rds_source/variable.tf @@ -1,76 +1,76 @@ variable "code_bucket" { description = "Name of the bucket that contains the lambda functions to deploy." - type = string + type = string } variable "instance_name_pattern" { - description = "Python regex for matching cluster identifiers to backup. Use "ALL_INSTANCES" to back up every RDS instance in the region." - type = string - default = "ALL_INSTANCES" + description = "Python regex for matching cluster identifiers to backup. Use 'ALL_INSTANCES' to back up every RDS instance in the region." + type = string + default = "ALL_INSTANCES" } variable "backup_interval" { description = "Interval for backups in hours. Default is 24" - type = string - default = "24" + type = string + default = "24" } variable "destination_account" { description = "Destination account with no dashes." - type = string - default = "000000000000" + type = string + default = "000000000000" } variable "share_snapshots" { - type = string + type = string default = "TRUE" } variable "backup_schedule" { description = "Backup schedule in Cloudwatch Event cron format. Needs to run at least once for every Interval. The default value runs once every at 1AM UTC. More information: http://docs.aws.amazon.com/AmazonCloudWatch/latest/events/ScheduledEvents.html" - type = string - default = "0 1 * * ? *" + type = string + default = "0 1 * * ? *" } variable "retention_days" { description = "Number of days to keep snapshots in retention before deleting them" - type = string - default = "7" + type = string + default = "7" } variable "log_level" { description = "Log level for Lambda functions (DEBUG, INFO, WARN, ERROR, CRITICAL are valid values)." - type = string - default = "ERROR" + type = string + default = "ERROR" } variable "lambda_cw_log_retention" { description = "Number of days to retain logs from the lambda functions in CloudWatch Logs" - type = string - default = "7" + type = string + default = "7" } variable "source_region_override" { description = "Set to the region where your RDS instances run, only if such region does not support Step Functions. Leave as NO otherwise" - type = string - default = "NO" + type = string + default = "NO" } variable "delete_old_snapshots" { description = "Set to TRUE to enable deletion of snapshot based on RetentionDays. Set to FALSE to disable" - type = string - default = "TRUE" + type = string + default = "TRUE" } variable "tagged_instance" { description = "Set to TRUE to filter instances that have tag CopyDBSnapshot set to True. Set to FALSE to disable" - type = string - default = "FALSE" + type = string + default = "FALSE" } variable "log_group_name" { description = "Name for RDS snapshot log group." - type = string - default = "lambdaDeleteOldSnapshotsRDS-source" + type = string + default = "lambdaDeleteOldSnapshotsRDS-source" } From fda79e4499144e34884f6a4556c2315c5fe49224 Mon Sep 17 00:00:00 2001 From: charlie keegan Date: Tue, 7 Feb 2023 11:55:49 +0000 Subject: [PATCH 08/48] initialse tf --- terraform/main.tf | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/terraform/main.tf b/terraform/main.tf index 393e916..8bbd06a 100644 --- a/terraform/main.tf +++ b/terraform/main.tf @@ -2,7 +2,11 @@ module "test_dbs" { source = "./modules/test_dbs" } - -module "test_dbs" { +module "snapshots_tool_rds_dest" { source = "./modules/snapshots_tool_rds_dest" } + + +module "snapshots_tool_rds_source" { + source = "./modules/snapshots_tool_rds_source" +} From db0e26d9b2bbfe99884c015dac02fad59b242fbc Mon Sep 17 00:00:00 2001 From: charlie keegan Date: Tue, 7 Feb 2023 12:04:38 +0000 Subject: [PATCH 09/48] solve for cfn topics list attribute --- terraform/modules/snapshots_tool_rds_dest/main.tf | 11 +++-------- 1 file changed, 3 insertions(+), 8 deletions(-) diff --git a/terraform/modules/snapshots_tool_rds_dest/main.tf b/terraform/modules/snapshots_tool_rds_dest/main.tf index f4455d0..201dc2a 100644 --- a/terraform/modules/snapshots_tool_rds_dest/main.tf +++ b/terraform/modules/snapshots_tool_rds_dest/main.tf @@ -1,7 +1,3 @@ -locals { - DeleteOld = var.delete_old_snapshots == "TRUE" - CrossAccount = var.cross_account_copy == "TRUE" -} resource "aws_sns_topic" "topic_copy_failed_dest" { display_name = "copies_failed_dest_rds" } @@ -10,10 +6,9 @@ resource "aws_sns_topic" "topic_delete_old_failed_dest" { } resource "aws_sns_topic_policy" "snspolicy_copy_failed_dest" { - // CF Property(Topics) = [ - // aws_sns_topic.topic_copy_failed_dest.id, - // aws_sns_topic.topic_delete_old_failed_dest.id - // ] + count = 2 + arn = element(toset(aws_sns_topc.topic_copy_failed_dest.arn, aws_sns_topic.topic_delete_old_failed_dest.arn), count.index) // in tf there is not a cfn "topcs" attribute which allows a list. + policy = { Version = "2008-10-17" Id = "__default_policy_ID" From 7949762069d3cd7b5e1ae5ebbe755bffd00fb41d Mon Sep 17 00:00:00 2001 From: charlie keegan Date: Tue, 7 Feb 2023 13:34:16 +0000 Subject: [PATCH 10/48] update variables and first pass at destination work --- .../modules/snapshots_tool_rds_dest/main.tf | 350 ------------------ .../snapshots_tool_rds_dest/variables.tf | 2 + .../snapshots_tool_rds_source/variable.tf | 1 + 3 files changed, 3 insertions(+), 350 deletions(-) delete mode 100644 terraform/modules/snapshots_tool_rds_dest/main.tf diff --git a/terraform/modules/snapshots_tool_rds_dest/main.tf b/terraform/modules/snapshots_tool_rds_dest/main.tf deleted file mode 100644 index 201dc2a..0000000 --- a/terraform/modules/snapshots_tool_rds_dest/main.tf +++ /dev/null @@ -1,350 +0,0 @@ -resource "aws_sns_topic" "topic_copy_failed_dest" { - display_name = "copies_failed_dest_rds" -} -resource "aws_sns_topic" "topic_delete_old_failed_dest" { - display_name = "delete_old_failed_dest_rds" -} - -resource "aws_sns_topic_policy" "snspolicy_copy_failed_dest" { - count = 2 - arn = element(toset(aws_sns_topc.topic_copy_failed_dest.arn, aws_sns_topic.topic_delete_old_failed_dest.arn), count.index) // in tf there is not a cfn "topcs" attribute which allows a list. - - policy = { - Version = "2008-10-17" - Id = "__default_policy_ID" - Statement = [ - { - Sid = "__default_statement_ID" - Effect = "Allow" - Principal = { - AWS = "*" - } - Action = [ - "SNS:GetTopicAttributes", - "SNS:SetTopicAttributes", - "SNS:AddPermission", - "SNS:RemovePermission", - "SNS:DeleteTopic", - "SNS:Subscribe", - "SNS:ListSubscriptionsByTopic", - "SNS:Publish", - "SNS:Receive" - ] - Resource = "*" - Condition = { - test = "StringEquals" - variable = "AWS:SourceOwner" - values = [data.aws_caller_identity.current.account_id] - } - } - ] - } -} - -resource "aws_cloudwatch_composite_alarm" "alarmcw_copy_failed_dest" { - actions_enabled = "true" - // CF Property(ComparisonOperator) = "GreaterThanOrEqualToThreshold" - // CF Property(EvaluationPeriods) = "1" - // CF Property(MetricName) = "ExecutionsFailed" - // CF Property(Namespace) = "AWS/States" - // CF Property(Period) = "300" - // CF Property(Statistic) = "Sum" - // CF Property(Threshold) = "1.0" - alarm_actions = [ - aws_sns_topic.topic_copy_failed_dest.id - ] - // CF Property(Dimensions) = [ - // { - // Name = "StateMachineArn" - // Value = aws_ec2_instance_state.statemachine_copy_snapshots_dest_rds.id - // } - // ] -} - -resource "aws_cloudwatch_composite_alarm" "alarmcw_delete_old_failed_dest" { - count = locals.DeleteOld ? 1 : 0 - actions_enabled = "true" - // CF Property(ComparisonOperator) = "GreaterThanOrEqualToThreshold" - // CF Property(EvaluationPeriods) = "2" - // CF Property(MetricName) = "ExecutionsFailed" - // CF Property(Namespace) = "AWS/States" - // CF Property(Period) = "3600" - // CF Property(Statistic) = "Sum" - // CF Property(Threshold) = "2.0" - alarm_actions = [ - aws_sns_topic.topic_delete_old_failed_dest.id - ] - // CF Property(Dimensions) = [ - // { - // Name = "StateMachineArn" - // Value = aws_ec2_instance_state.statemachine_delete_old_snapshots_dest_rds[0].id - // } - // ] -} - -resource "aws_iam_role" "iamrole_snapshots_rds" { - assume_role_policy = { - Version = "2012-10-17" - Statement = [ - { - Effect = "Allow" - Principal = { - Service = "lambda.amazonaws.com" - } - Action = "sts:AssumeRole" - } - ] - } - force_detach_policies = [ - { - PolicyName = "inline_policy_snapshots_rds_cw_logs" - PolicyDocument = { - Version = "2012-10-17" - Statement = [ - { - Effect = "Allow" - Action = [ - "logs:CreateLogGroup", - "logs:CreateLogStream", - "logs:PutLogEvents" - ] - Resource = "arn:aws:logs:*:*:*" - } - ] - } - }, - { - PolicyName = "inline_policy_snapshots_rds" - PolicyDocument = { - Version = "2012-10-17" - Statement = [ - { - Effect = "Allow" - Action = [ - "rds:CreateDBSnapshot", - "rds:DeleteDBSnapshot", - "rds:DescribeDBInstances", - "rds:DescribeDBSnapshots", - "rds:ModifyDBSnapshotAttribute", - "rds:DescribeDBSnapshotAttributes", - "rds:CopyDBSnapshot", - "rds:ListTagsForResource", - "rds:AddTagsToResource" - ] - Resource = "*" - } - ] - } - }, - { - PolicyName = "inline_policy_snapshot_rds_kms_access" - PolicyDocument = { - Version = "2012-10-17" - Statement = [ - { - Sid = "AllowUseOfTheKey" - Effect = "Allow" - Action = [ - "kms:Encrypt", - "kms:Decrypt", - "kms:ReEncrypt*", - "kms:GenerateDataKey*", - "kms:DescribeKey" - ] - Resource = [ - "*" - ] - }, - { - Sid = "AllowAttachmentOfPersistentResources" - Effect = "Allow" - Action = [ - "kms:CreateGrant", - "kms:ListGrants", - "kms:RevokeGrant" - ] - Resource = [ - "*" - ] - Condition = { - test = "Bool" - variable = "kms:GrantIsForAWSResource" - values = ["True"] - } - } - ] - } - } - ] -} - -resource "aws_lambda_function" "lambda_copy_snapshots_rds" { - code_signing_config_arn = { - S3Bucket = var.code_bucket - S3Key = local.CrossAccount ? "copy_snapshots_dest_rds.zip" : "copy_snapshots_no_x_account_rds.zip" - } - memory_size = 512 - description = "This functions copies snapshots for RDS Instances shared with this account. It checks for existing snapshots following the pattern specified in the environment variables with the following format: -YYYY-MM-DD-HH-MM" - environment { - variables = { - SNAPSHOT_PATTERN = var.snapshot_pattern - DEST_REGION = var.destination_region - LOG_LEVEL = var.log_level - REGION_OVERRIDE = var.source_region_override - KMS_KEY_DEST_REGION = var.kms_key_destination - KMS_KEY_SOURCE_REGION = var.kms_key_source - RETENTION_DAYS = var.retention_days - } - } - role = aws_iam_role.iamrole_snapshots_rds.arn - runtime = "python3.7" - handler = "lambda_function.lambda_handler" - timeout = 300 -} - -resource "aws_lambda_function" "lambda_delete_old_dest_rds" { - count = locals.DeleteOld ? 1 : 0 - code_signing_config_arn = { - S3Bucket = var.code_bucket - S3Key = local.CrossAccount ? "delete_old_snapshots_dest_rds.zip" : "delete_old_snapshots_no_x_account_rds.zip" - } - memory_size = 512 - description = "This function enforces retention on the snapshots shared with the destination account. " - environment { - variables = { - SNAPSHOT_PATTERN = var.snapshot_pattern - DEST_REGION = var.destination_region - RETENTION_DAYS = var.retention_days - LOG_LEVEL = var.log_level - } - } - role = aws_iam_role.iamrole_snapshots_rds.arn - runtime = "python3.7" - handler = "lambda_function.lambda_handler" - timeout = 300 -} - -resource "aws_iam_role" "iamrole_state_execution" { - assume_role_policy = { - Version = "2012-10-17" - Statement = [ - { - Effect = "Allow" - Principal = { - Service = join("", ["states.", data.aws_region.current.name, ".amazonaws.com"]) - } - Action = "sts:AssumeRole" - } - ] - } - force_detach_policies = [ - { - PolicyName = "inline_policy_rds_snapshot" - PolicyDocument = { - Version = "2012-10-17" - Statement = [ - { - Effect = "Allow" - Action = [ - "lambda:InvokeFunction" - ] - Resource = "*" - } - ] - } - } - ] -} - -resource "aws_ec2_instance_state" "statemachine_copy_snapshots_dest_rds" { - // CF Property(DefinitionString) = join("", [join(" - // ", [" {"Comment":"Copies snapshots locally and then to DEST_REGION",", " "StartAt":"CopySnapshots",", " "States":{", " "CopySnapshots":{", " "Type":"Task",", " "Resource": "]), """, aws_lambda_function.lambda_copy_snapshots_rds.arn, "" - // ,", join(" - // ", [" "Retry":[", " {", " "ErrorEquals":[ ", " "SnapshotToolException"", " ],", " "IntervalSeconds":300,", " "MaxAttempts":5,", " "BackoffRate":1", " },", " {", " "ErrorEquals":[ ", " "States.ALL"], ", " "IntervalSeconds": 30,", " "MaxAttempts": 20,", " "BackoffRate": 1", " }", " ],", " "End": true ", " }", " }}"])]) - // CF Property(RoleArn) = aws_iam_role.iamrole_state_execution.arn -} - -resource "aws_ec2_instance_state" "statemachine_delete_old_snapshots_dest_rds" { - count = locals.DeleteOld ? 1 : 0 - // CF Property(DefinitionString) = join("", [join(" - // ", [" {"Comment":"DeleteOld for RDS snapshots in destination region",", " "StartAt":"DeleteOldDestRegion",", " "States":{", " "DeleteOldDestRegion":{", " "Type":"Task",", " "Resource": "]), """, aws_lambda_function.lambda_delete_old_dest_rds.arn, "" - // ,", join(" - // ", [" "Retry":[", " {", " "ErrorEquals":[ ", " "SnapshotToolException"", " ],", " "IntervalSeconds":600,", " "MaxAttempts":5,", " "BackoffRate":1", " },", " {", " "ErrorEquals":[ ", " "States.ALL"], ", " "IntervalSeconds": 30,", " "MaxAttempts": 20,", " "BackoffRate": 1", " }", " ],", " "End": true ", " }", " }}"])]) - // CF Property(RoleArn) = aws_iam_role.iamrole_state_execution.arn -} - -resource "aws_iam_role" "iamrole_step_invocation" { - assume_role_policy = { - Version = "2012-10-17" - Statement = [ - { - Effect = "Allow" - Principal = { - Service = "events.amazonaws.com" - } - Action = "sts:AssumeRole" - } - ] - } - force_detach_policies = [ - { - PolicyName = "inline_policy_state_invocation" - PolicyDocument = { - Version = "2012-10-17" - Statement = [ - { - Effect = "Allow" - Action = [ - "states:StartExecution" - ] - Resource = "*" - } - ] - } - } - ] -} - -resource "aws_iot_topic_rule_destination" "cw_event_copy_snapshots_rds" { - // CF Property(Description) = "Triggers the RDS Copy state machine in the destination account" - // CF Property(ScheduleExpression) = join("", ["cron(", "/30 * * * ? *", ")"]) - // CF Property(State) = "ENABLED" - // CF Property(Targets) = [ - // { - // Arn = aws_ec2_instance_state.statemachine_copy_snapshots_dest_rds.id - // Id = "Target1" - // RoleArn = aws_iam_role.iamrole_step_invocation.arn - // } - // ] -} - -resource "aws_iot_topic_rule_destination" "cw_event_delete_old_snapshots_rds" { - count = locals.DeleteOld ? 1 : 0 - vpc_configuration { - role_arn = "" - security_groups = "" - subnet_ids = "" - vpc_id = "" - } - // CF Property(Description) = "Triggers the RDS DeleteOld state machine in the destination account" - // CF Property(ScheduleExpression) = join("", ["cron(", "0 /1 * * ? *", ")"]) - // CF Property(State) = "ENABLED" - // CF Property(Targets) = [ - // { - // Arn = aws_ec2_instance_state.statemachine_delete_old_snapshots_dest_rds[0].id - // Id = "Target1" - // RoleArn = aws_iam_role.iamrole_step_invocation.arn - // } - // ] -} - -resource "aws_inspector_resource_group" "cwloggroup_delete_old_snapshots_dest_rds" { - count = locals.DeleteOld ? 1 : 0 - // CF Property(RetentionInDays) = var.lambda_cw_log_retention - // CF Property(LogGroupName) = "/aws/lambda/${var.log_group_name}" -} - -resource "aws_inspector_resource_group" "cwloggrouplambda_copy_snapshots_rds" { - // CF Property(RetentionInDays) = var.lambda_cw_log_retention - // CF Property(LogGroupName) = "/aws/lambda/${aws_lambda_function.lambda_copy_snapshots_rds.arn}" -} diff --git a/terraform/modules/snapshots_tool_rds_dest/variables.tf b/terraform/modules/snapshots_tool_rds_dest/variables.tf index 66573bd..1981e96 100644 --- a/terraform/modules/snapshots_tool_rds_dest/variables.tf +++ b/terraform/modules/snapshots_tool_rds_dest/variables.tf @@ -1,6 +1,7 @@ variable "code_bucket" { description = "Name of the bucket that contains the lambda functions to deploy." type = string + default = "code-bucket-816708952522" } variable "snapshot_pattern" { @@ -18,6 +19,7 @@ variable "retention_days" { variable "destination_region" { description = "Destination region for snapshots." type = string + default = "us-east-2" } variable "log_level" { diff --git a/terraform/modules/snapshots_tool_rds_source/variable.tf b/terraform/modules/snapshots_tool_rds_source/variable.tf index 10857d8..e1092a1 100644 --- a/terraform/modules/snapshots_tool_rds_source/variable.tf +++ b/terraform/modules/snapshots_tool_rds_source/variable.tf @@ -1,6 +1,7 @@ variable "code_bucket" { description = "Name of the bucket that contains the lambda functions to deploy." type = string + default = "code-bucket-816708952522" } variable "instance_name_pattern" { From a8a560c6d1478ce780c80ce42d87216cd0b87af3 Mon Sep 17 00:00:00 2001 From: charlie keegan Date: Tue, 7 Feb 2023 13:42:30 +0000 Subject: [PATCH 11/48] correct sns topic arn policies --- terraform/main.tf | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/terraform/main.tf b/terraform/main.tf index 8bbd06a..3cfa1fe 100644 --- a/terraform/main.tf +++ b/terraform/main.tf @@ -7,6 +7,6 @@ module "snapshots_tool_rds_dest" { } -module "snapshots_tool_rds_source" { - source = "./modules/snapshots_tool_rds_source" -} +# module "snapshots_tool_rds_source" { +# source = "./modules/snapshots_tool_rds_source" +# } From b805eae017747cb1bc6f58d5f5684c8f541793bf Mon Sep 17 00:00:00 2001 From: charlie keegan Date: Tue, 7 Feb 2023 13:44:48 +0000 Subject: [PATCH 12/48] fix invoke state machine policy --- terraform/.terraform.lock.hcl | 25 + .../modules/snapshots_tool_rds_dest/data.tf | 3 + .../modules/snapshots_tool_rds_dest/locals.tf | 4 + .../snapshots_tool_rds_dest/resource.tf | 453 ++++++++++++++++++ 4 files changed, 485 insertions(+) create mode 100644 terraform/.terraform.lock.hcl create mode 100644 terraform/modules/snapshots_tool_rds_dest/data.tf create mode 100644 terraform/modules/snapshots_tool_rds_dest/locals.tf create mode 100644 terraform/modules/snapshots_tool_rds_dest/resource.tf diff --git a/terraform/.terraform.lock.hcl b/terraform/.terraform.lock.hcl new file mode 100644 index 0000000..4a85ae8 --- /dev/null +++ b/terraform/.terraform.lock.hcl @@ -0,0 +1,25 @@ +# This file is maintained automatically by "terraform init". +# Manual edits may be lost in future updates. + +provider "registry.terraform.io/hashicorp/aws" { + version = "4.53.0" + constraints = "~> 4.0" + hashes = [ + "h1:P6ZZ716SRIimw0t/SAgYbOMZtO0HDvwVQKxyHEW6aaE=", + "zh:0d44171544a916adf0fa96b7d0851a49d8dec98f71f0229dfd2d178958b3996b", + "zh:16945808ce26b86af7f5a77c4ab1154da786208c793abb95b8f918b4f48daded", + "zh:1a57a5a30cef9a5867579d894b74f60bb99afc7ca0d030d49a80ad776958b428", + "zh:2c718734ae17430d7f598ca0b4e4f86d43d66569c72076a10f4ace3ff8dfc605", + "zh:46fdf6301cb2fa0a4d122d1a8f75f047b6660c24851d6a4537ee38926a86485d", + "zh:53a53920b38a9e1648e85c6ee33bccf95bfcd067bffc4934a2af55621e6a6bd9", + "zh:548d927b234b1914c43169224b03f641d0961a4e312e5c6508657fce27b66db4", + "zh:57c847b2a5ae41ddea20b18ef006369d36bfdc4dec7f542f60e22a47f7b6f347", + "zh:79f7402b581621ba69f5a07ce70299735c678beb265d114d58955d04f0d39f87", + "zh:8970109a692dc4ecbda98a0969da472da4759db90ce22f2a196356ea85bb2cf7", + "zh:9b12af85486a96aedd8d7984b0ff811a4b42e3d88dad1a3fb4c0b580d04fa425", + "zh:a500cc4ffcad854dec0cf6f97751930a53c9f278f143a4355fa8892aa77c77bf", + "zh:b687c20b42a8b9e9e9f56c42e3b3c6859c043ec72b8907a6e4d4b64068e11df5", + "zh:e2c592e96822b78287554be43c66398f658c74c4ae3796f6b9e6d4b0f1f7f626", + "zh:ff1c4a46fdc988716c6fc28925549600093fc098828237cb1a30264e15cf730f", + ] +} diff --git a/terraform/modules/snapshots_tool_rds_dest/data.tf b/terraform/modules/snapshots_tool_rds_dest/data.tf new file mode 100644 index 0000000..eb58f21 --- /dev/null +++ b/terraform/modules/snapshots_tool_rds_dest/data.tf @@ -0,0 +1,3 @@ +data "aws_region" "current" {} + +data "aws_caller_identity" "current" {} diff --git a/terraform/modules/snapshots_tool_rds_dest/locals.tf b/terraform/modules/snapshots_tool_rds_dest/locals.tf new file mode 100644 index 0000000..440067b --- /dev/null +++ b/terraform/modules/snapshots_tool_rds_dest/locals.tf @@ -0,0 +1,4 @@ +locals { + DeleteOld = var.delete_old_snapshots == 0 + CrossAccount = var.cross_account_copy == 0 +} diff --git a/terraform/modules/snapshots_tool_rds_dest/resource.tf b/terraform/modules/snapshots_tool_rds_dest/resource.tf new file mode 100644 index 0000000..f3c0f9a --- /dev/null +++ b/terraform/modules/snapshots_tool_rds_dest/resource.tf @@ -0,0 +1,453 @@ +resource "aws_sns_topic" "topic_copy_failed_dest" { + display_name = "copies_failed_dest_rds" +} +resource "aws_sns_topic" "topic_delete_old_failed_dest" { + display_name = "delete_old_failed_dest_rds" +} + +resource "aws_sns_topic_policy" "snspolicy_delete_failed_dest" { + arn = aws_sns_topic.topic_delete_old_failed_dest.arn // in tf there is not a cfn "topcs" attribute which allows a list. + + policy = { + Version = "2008-10-17" + Id = "destination_rds_failed_snapshot_delete" + Statement = [ + { + Sid = "sns-permissions" + Effect = "Allow" + Principal = { + AWS = "*" + } + Action = [ + "SNS:GetTopicAttributes", + "SNS:SetTopicAttributes", + "SNS:AddPermission", + "SNS:RemovePermission", + "SNS:DeleteTopic", + "SNS:Subscribe", + "SNS:ListSubscriptionsByTopic", + "SNS:Publish", + "SNS:Receive" + ] + Resource = "*" + Condition = { + test = "StringEquals" + variable = "AWS:SourceOwner" + values = [data.aws_caller_identity.current.account_id] + } + } + ] + } +} + + +resource "aws_sns_topic_policy" "snspolicy_copy_failed_dest" { + arn = aws_sns_topic.topic_copy_failed_dest.arn + + policy = { + Version = "2008-10-17" + Id = "destination_rds_failed_snapshot_copy" + Statement = [ + { + Sid = "sns-permissions" + Effect = "Allow" + Principal = { + AWS = "*" + } + Action = [ + "SNS:GetTopicAttributes", + "SNS:SetTopicAttributes", + "SNS:AddPermission", + "SNS:RemovePermission", + "SNS:DeleteTopic", + "SNS:Subscribe", + "SNS:ListSubscriptionsByTopic", + "SNS:Publish", + "SNS:Receive" + ] + Resource = "*" + Condition = { + test = "StringEquals" + variable = "AWS:SourceOwner" + values = [data.aws_caller_identity.current.account_id] + } + } + ] + } +} + +resource "aws_cloudwatch_metric_alarm" "alarmcw_copy_failed_dest" { + alarm_name = "failed-rds-copy" + comparison_operator = "GreaterThanOrEqualToThreshold" + evaluation_periods = "1" + metric_name = "ExecutionsFailed" + namespace = "AWS/States" + period = "300" + statistic = "Sum" + threshold = "1.0" + + dimensions = { + StateMachineArn = aws_sfn_state_machine.statemachine_copy_snapshots_dest_rds.id + } + + alarm_description = "This metric monitors state machine failure for copying snapshots" + alarm_actions = [ + aws_sns_topic.topic_copy_failed_dest.id + ] +} + +resource "aws_cloudwatch_metric_alarm" "alarmcw_delete_old_failed_dest" { + alarm_name = "failed-rds-delete-old-snapshot" + comparison_operator = "GreaterThanOrEqualToThreshold" + evaluation_periods = "2" + metric_name = "ExecutionsFailed" + namespace = "AWS/States" + period = "3600" + statistic = "Sum" + threshold = "2.0" + + dimensions = { + StateMachineArn = aws_sfn_state_machine.statemachine_copy_snapshots_dest_rds.id + } + + alarm_description = "This metric monitors state machine failure for deleting old snapshots" + alarm_actions = [ + aws_sns_topic.topic_delete_old_failed_dest.id + ] +} + +resource "aws_iam_role" "iamrole_snapshots_rds" { + assume_role_policy = { + Version = "2012-10-17" + Statement = [ + { + Effect = "Allow" + Principal = { + Service = "lambda.amazonaws.com" + } + Action = "sts:AssumeRole" + } + ] + } + force_detach_policies = [ + { + PolicyName = "inline_policy_snapshots_rds_cw_logs" + PolicyDocument = { + Version = "2012-10-17" + Statement = [ + { + Effect = "Allow" + Action = [ + "logs:CreateLogGroup", + "logs:CreateLogStream", + "logs:PutLogEvents" + ] + Resource = "arn:aws:logs:*:*:*" + } + ] + } + }, + { + PolicyName = "inline_policy_snapshots_rds" + PolicyDocument = { + Version = "2012-10-17" + Statement = [ + { + Effect = "Allow" + Action = [ + "rds:CreateDBSnapshot", + "rds:DeleteDBSnapshot", + "rds:DescribeDBInstances", + "rds:DescribeDBSnapshots", + "rds:ModifyDBSnapshotAttribute", + "rds:DescribeDBSnapshotAttributes", + "rds:CopyDBSnapshot", + "rds:ListTagsForResource", + "rds:AddTagsToResource" + ] + Resource = "*" + } + ] + } + }, + { + PolicyName = "inline_policy_snapshot_rds_kms_access" + PolicyDocument = { + Version = "2012-10-17" + Statement = [ + { + Sid = "AllowUseOfTheKey" + Effect = "Allow" + Action = [ + "kms:Encrypt", + "kms:Decrypt", + "kms:ReEncrypt*", + "kms:GenerateDataKey*", + "kms:DescribeKey" + ] + Resource = [ + "*" + ] + }, + { + Sid = "AllowAttachmentOfPersistentResources" + Effect = "Allow" + Action = [ + "kms:CreateGrant", + "kms:ListGrants", + "kms:RevokeGrant" + ] + Resource = [ + "*" + ] + Condition = { + test = "Bool" + variable = "kms:GrantIsForAWSResource" + values = ["True"] + } + } + ] + } + } + ] +} + +resource "aws_lambda_function" "lambda_copy_snapshots_rds" { + code_signing_config_arn = { + S3Bucket = var.code_bucket + S3Key = local.CrossAccount ? "copy_snapshots_dest_rds.zip" : "copy_snapshots_no_x_account_rds.zip" + } + memory_size = 512 + description = "This functions copies snapshots for RDS Instances shared with this account. It checks for existing snapshots following the pattern specified in the environment variables with the following format: -YYYY-MM-DD-HH-MM" + environment { + variables = { + SNAPSHOT_PATTERN = var.snapshot_pattern + DEST_REGION = var.destination_region + LOG_LEVEL = var.log_level + REGION_OVERRIDE = var.source_region_override + KMS_KEY_DEST_REGION = var.kms_key_destination + KMS_KEY_SOURCE_REGION = var.kms_key_source + RETENTION_DAYS = var.retention_days + } + } + role = aws_iam_role.iamrole_snapshots_rds.arn + runtime = "python3.7" + handler = "lambda_function.lambda_handler" + timeout = 300 +} + +resource "aws_lambda_function" "lambda_delete_old_dest_rds" { + count = locals.DeleteOld ? 1 : 0 + code_signing_config_arn = { + S3Bucket = var.code_bucket + S3Key = local.CrossAccount ? "delete_old_snapshots_dest_rds.zip" : "delete_old_snapshots_no_x_account_rds.zip" + } + memory_size = 512 + description = "This function enforces retention on the snapshots shared with the destination account. " + environment { + variables = { + SNAPSHOT_PATTERN = var.snapshot_pattern + DEST_REGION = var.destination_region + RETENTION_DAYS = var.retention_days + LOG_LEVEL = var.log_level + } + } + role = aws_iam_role.iamrole_snapshots_rds.arn + runtime = "python3.7" + handler = "lambda_function.lambda_handler" + timeout = 300 +} + +resource "aws_iam_role" "iamrole_state_execution" { + assume_role_policy = { + Version = "2012-10-17" + Statement = [ + { + Effect = "Allow" + Principal = { + Service = join("", ["states.", data.aws_region.current.name, ".amazonaws.com"]) + } + Action = "sts:AssumeRole" + } + ] + } + force_detach_policies = [ + { + PolicyName = "inline_policy_rds_snapshot" + PolicyDocument = { + Version = "2012-10-17" + Statement = [ + { + Effect = "Allow" + Action = [ + "lambda:InvokeFunction" + ] + Resource = "*" + } + ] + } + } + ] +} + +resource "aws_sfn_state_machine" "statemachine_delete_old_snapshots_dest_rds" { + name = "delete-old-snapshots-destination-rds" + role_arn = aws_iam_role.iamrole_state_execution.arn + + definition = < Date: Tue, 7 Feb 2023 13:45:11 +0000 Subject: [PATCH 13/48] fix invoke state machine policy --- terraform/modules/snapshots_tool_rds_dest/resource.tf | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/terraform/modules/snapshots_tool_rds_dest/resource.tf b/terraform/modules/snapshots_tool_rds_dest/resource.tf index f3c0f9a..a68287d 100644 --- a/terraform/modules/snapshots_tool_rds_dest/resource.tf +++ b/terraform/modules/snapshots_tool_rds_dest/resource.tf @@ -380,7 +380,7 @@ resource "aws_iam_role" "iamrole_step_invocation" { ] } force_detach_policies = true - inlinline_policy { + inline_policy { PolicyName = "inline_policy_state_invocation" PolicyDocument = { Version = "2012-10-17" From d3641565b757e08de74102f9dccf0efed7d76be3 Mon Sep 17 00:00:00 2001 From: charlie keegan Date: Tue, 7 Feb 2023 13:46:33 +0000 Subject: [PATCH 14/48] fix invoke state machine policy --- terraform/modules/snapshots_tool_rds_dest/resource.tf | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/terraform/modules/snapshots_tool_rds_dest/resource.tf b/terraform/modules/snapshots_tool_rds_dest/resource.tf index a68287d..dd855db 100644 --- a/terraform/modules/snapshots_tool_rds_dest/resource.tf +++ b/terraform/modules/snapshots_tool_rds_dest/resource.tf @@ -367,7 +367,7 @@ EOF resource "aws_iam_role" "iamrole_step_invocation" { name = "invoke-state-machines" - assume_role_policy = { + assume_role_policy = jsonencode({ Version = "2012-10-17" Statement = [ { @@ -378,11 +378,11 @@ resource "aws_iam_role" "iamrole_step_invocation" { Action = "sts:AssumeRole" } ] - } + }) force_detach_policies = true inline_policy { - PolicyName = "inline_policy_state_invocation" - PolicyDocument = { + name = "inline_policy_state_invocation" + policy = jsonencode({ Version = "2012-10-17" Statement = [ { @@ -393,7 +393,7 @@ resource "aws_iam_role" "iamrole_step_invocation" { Resource = "*" } ] - } + }) } } From 47412530ed0000854ac51aeffad51403f377ae5a Mon Sep 17 00:00:00 2001 From: charlie keegan Date: Tue, 7 Feb 2023 13:47:50 +0000 Subject: [PATCH 15/48] fix state execution policy --- .../snapshots_tool_rds_dest/resource.tf | 35 +++++++++---------- 1 file changed, 17 insertions(+), 18 deletions(-) diff --git a/terraform/modules/snapshots_tool_rds_dest/resource.tf b/terraform/modules/snapshots_tool_rds_dest/resource.tf index dd855db..66ef6e4 100644 --- a/terraform/modules/snapshots_tool_rds_dest/resource.tf +++ b/terraform/modules/snapshots_tool_rds_dest/resource.tf @@ -259,7 +259,7 @@ resource "aws_lambda_function" "lambda_delete_old_dest_rds" { } resource "aws_iam_role" "iamrole_state_execution" { - assume_role_policy = { + assume_role_policy = jsonencode({ Version = "2012-10-17" Statement = [ { @@ -270,24 +270,23 @@ resource "aws_iam_role" "iamrole_state_execution" { Action = "sts:AssumeRole" } ] + }) + force_detach_policies = true + inline_policy { + name = "inline_policy_rds_snapshot" + policy = jsonencode({ + Version = "2012-10-17" + Statement = [ + { + Effect = "Allow" + Action = [ + "lambda:InvokeFunction" + ] + Resource = "*" + } + ] + }) } - force_detach_policies = [ - { - PolicyName = "inline_policy_rds_snapshot" - PolicyDocument = { - Version = "2012-10-17" - Statement = [ - { - Effect = "Allow" - Action = [ - "lambda:InvokeFunction" - ] - Resource = "*" - } - ] - } - } - ] } resource "aws_sfn_state_machine" "statemachine_delete_old_snapshots_dest_rds" { From b5c56036a0f0b484d8396e8ed9fa27d504eeb698 Mon Sep 17 00:00:00 2001 From: charlie keegan Date: Tue, 7 Feb 2023 13:55:20 +0000 Subject: [PATCH 16/48] fix lambda iam role --- .../snapshots_tool_rds_dest/resource.tf | 175 ++++++++++-------- 1 file changed, 95 insertions(+), 80 deletions(-) diff --git a/terraform/modules/snapshots_tool_rds_dest/resource.tf b/terraform/modules/snapshots_tool_rds_dest/resource.tf index 66ef6e4..7d8f43c 100644 --- a/terraform/modules/snapshots_tool_rds_dest/resource.tf +++ b/terraform/modules/snapshots_tool_rds_dest/resource.tf @@ -116,8 +116,8 @@ resource "aws_cloudwatch_metric_alarm" "alarmcw_delete_old_failed_dest" { ] } -resource "aws_iam_role" "iamrole_snapshots_rds" { - assume_role_policy = { +resource "aws_iam_role" "snapshots_rds" { + assume_role_policy = jsonencode({ Version = "2012-10-17" Statement = [ { @@ -128,88 +128,103 @@ resource "aws_iam_role" "iamrole_snapshots_rds" { Action = "sts:AssumeRole" } ] - } - force_detach_policies = [ - { - PolicyName = "inline_policy_snapshots_rds_cw_logs" - PolicyDocument = { - Version = "2012-10-17" - Statement = [ - { - Effect = "Allow" - Action = [ - "logs:CreateLogGroup", - "logs:CreateLogStream", - "logs:PutLogEvents" - ] - Resource = "arn:aws:logs:*:*:*" - } + }) + force_detach_policies = true +} + +resource "aws_iam_policy" "snapshots_rds_cw_logs" { + name = "snapshots_rds_cw_logs" + + policy = jsonencode({ + Version = "2012-10-17" + Statement = [ + { + Effect = "Allow" + Action = [ + "logs:CreateLogGroup", + "logs:CreateLogStream", + "logs:PutLogEvents" ] + Resource = "arn:aws:logs:*:*:*" } - }, - { - PolicyName = "inline_policy_snapshots_rds" - PolicyDocument = { - Version = "2012-10-17" - Statement = [ - { - Effect = "Allow" - Action = [ - "rds:CreateDBSnapshot", - "rds:DeleteDBSnapshot", - "rds:DescribeDBInstances", - "rds:DescribeDBSnapshots", - "rds:ModifyDBSnapshotAttribute", - "rds:DescribeDBSnapshotAttributes", - "rds:CopyDBSnapshot", - "rds:ListTagsForResource", - "rds:AddTagsToResource" - ] - Resource = "*" - } + ] + }) +} + +resource "aws_iam_policy" "snapshots_rds" { + name = "snapshots_rds" + + policy = jsonencode({ + Version = "2012-10-17" + Statement = [ + { + Effect = "Allow" + Action = [ + "rds:CreateDBSnapshot", + "rds:DeleteDBSnapshot", + "rds:DescribeDBInstances", + "rds:DescribeDBSnapshots", + "rds:ModifyDBSnapshotAttribute", + "rds:DescribeDBSnapshotAttributes", + "rds:CopyDBSnapshot", + "rds:ListTagsForResource", + "rds:AddTagsToResource" ] + Resource = "*" } - }, - { - PolicyName = "inline_policy_snapshot_rds_kms_access" - PolicyDocument = { - Version = "2012-10-17" - Statement = [ - { - Sid = "AllowUseOfTheKey" - Effect = "Allow" - Action = [ - "kms:Encrypt", - "kms:Decrypt", - "kms:ReEncrypt*", - "kms:GenerateDataKey*", - "kms:DescribeKey" - ] - Resource = [ - "*" - ] - }, - { - Sid = "AllowAttachmentOfPersistentResources" - Effect = "Allow" - Action = [ - "kms:CreateGrant", - "kms:ListGrants", - "kms:RevokeGrant" - ] - Resource = [ - "*" - ] - Condition = { - test = "Bool" - variable = "kms:GrantIsForAWSResource" - values = ["True"] - } - } + ] + }) +} + +resource "aws_iam_policy" "rds_kms_access" { + name = "rds_kms_access" + + policy = jsonencode({ + Version = "2012-10-17" + Statement = [ + { + Sid = "AllowUseOfTheKey" + Effect = "Allow" + Action = [ + "kms:Encrypt", + "kms:Decrypt", + "kms:ReEncrypt*", + "kms:GenerateDataKey*", + "kms:DescribeKey" ] + Resource = [ + "*" + ] + }, + { + Sid = "AllowAttachmentOfPersistentResources" + Effect = "Allow" + Action = [ + "kms:CreateGrant", + "kms:ListGrants", + "kms:RevokeGrant" + ] + Resource = [ + "*" + ] + Condition = { + test = "Bool" + variable = "kms:GrantIsForAWSResource" + values = ["True"] + } } - } - ] + ] + }) +} +resource "aws_iam_role_policy_attachment" "attachment" { + for_each = toset([ + aws_iam_policy.snapshots_rds_cw_logs.arn, + aws_iam_policy.rds_kms_access.arn, + aws_iam_policy.snapshots_rds.arn + ]) + + role = aws_iam_role.snapshots_rds.name + policy_arn = each.value } resource "aws_lambda_function" "lambda_copy_snapshots_rds" { @@ -420,9 +435,9 @@ PATTERN } resource "aws_cloudwatch_event_target" "copy_snapshots_rds" { - rule = aws_cloudwatch_event_rule.dest_copy_events.name + rule = aws_cloudwatch_event_rule.copy_snapshots_rds.name target_id = "DestTarget1" - arn = ws_sfn_state_machine.statemachine_copy_snapshots_dest_rds.id + arn = aws_sfn_state_machine.statemachine_copy_snapshots_dest_rds.id role_arn = aws_iam_role.iamrole_step_invocation.arn } From 56f516ecf962431578f852e1b0c2fae224d17de7 Mon Sep 17 00:00:00 2001 From: charlie keegan Date: Tue, 7 Feb 2023 14:07:40 +0000 Subject: [PATCH 17/48] accurately reference s3 buckets --- .../modules/snapshots_tool_rds_dest/locals.tf | 4 ++-- .../snapshots_tool_rds_dest/resource.tf | 23 ++++++++----------- 2 files changed, 12 insertions(+), 15 deletions(-) diff --git a/terraform/modules/snapshots_tool_rds_dest/locals.tf b/terraform/modules/snapshots_tool_rds_dest/locals.tf index 440067b..0b868ae 100644 --- a/terraform/modules/snapshots_tool_rds_dest/locals.tf +++ b/terraform/modules/snapshots_tool_rds_dest/locals.tf @@ -1,4 +1,4 @@ locals { - DeleteOld = var.delete_old_snapshots == 0 - CrossAccount = var.cross_account_copy == 0 + DeleteOld = var.delete_old_snapshots == "FALSE" + CrossAccount = var.cross_account_copy == "FALSE" } diff --git a/terraform/modules/snapshots_tool_rds_dest/resource.tf b/terraform/modules/snapshots_tool_rds_dest/resource.tf index 7d8f43c..a97140d 100644 --- a/terraform/modules/snapshots_tool_rds_dest/resource.tf +++ b/terraform/modules/snapshots_tool_rds_dest/resource.tf @@ -228,10 +228,9 @@ resource "aws_iam_role_policy_attachment" "attachment" { } resource "aws_lambda_function" "lambda_copy_snapshots_rds" { - code_signing_config_arn = { - S3Bucket = var.code_bucket - S3Key = local.CrossAccount ? "copy_snapshots_dest_rds.zip" : "copy_snapshots_no_x_account_rds.zip" - } + s3_bucket = var.code_bucket + s3_key = local.CrossAccount ? "copy_snapshots_dest_rds.zip" : "copy_snapshots_no_x_account_rds.zip" + memory_size = 512 description = "This functions copies snapshots for RDS Instances shared with this account. It checks for existing snapshots following the pattern specified in the environment variables with the following format: -YYYY-MM-DD-HH-MM" environment { @@ -245,18 +244,16 @@ resource "aws_lambda_function" "lambda_copy_snapshots_rds" { RETENTION_DAYS = var.retention_days } } - role = aws_iam_role.iamrole_snapshots_rds.arn + role = aws_iam_role.snapshots_rds.arn runtime = "python3.7" handler = "lambda_function.lambda_handler" timeout = 300 } -resource "aws_lambda_function" "lambda_delete_old_dest_rds" { - count = locals.DeleteOld ? 1 : 0 - code_signing_config_arn = { - S3Bucket = var.code_bucket - S3Key = local.CrossAccount ? "delete_old_snapshots_dest_rds.zip" : "delete_old_snapshots_no_x_account_rds.zip" - } +resource "aws_lambda_function" "delete_old_dest_rds" { + count = locals.DeleteOld ? 1 : 0 + s3_bucket = var.code_bucket + s3_key = local.CrossAccount ? "delete_old_snapshots_dest_rds.zip" : "delete_old_snapshots_no_x_account_rds.zip" memory_size = 512 description = "This function enforces retention on the snapshots shared with the destination account. " environment { @@ -267,7 +264,7 @@ resource "aws_lambda_function" "lambda_delete_old_dest_rds" { LOG_LEVEL = var.log_level } } - role = aws_iam_role.iamrole_snapshots_rds.arn + role = aws_iam_role.snapshots_rds.arn runtime = "python3.7" handler = "lambda_function.lambda_handler" timeout = 300 @@ -315,7 +312,7 @@ resource "aws_sfn_state_machine" "statemachine_delete_old_snapshots_dest_rds" { "States": { "DeleteOldDestRegion": { "Type": "Task", - "Resource": "${aws_lambda_function.lambda_delete_old_dest_rds.arn}", + "Resource": "${aws_lambda_function.delete_old_dest_rds.arn}", "Retry": [ { "ErrorEquals": [ From a5829aed29807f0fe0993cbcaf2d90f64fd91a9d Mon Sep 17 00:00:00 2001 From: charlie keegan Date: Tue, 7 Feb 2023 14:10:11 +0000 Subject: [PATCH 18/48] rationalise sns topic creation --- .../modules/snapshots_tool_rds_dest/locals.tf | 4 ++++ .../modules/snapshots_tool_rds_dest/resource.tf | 17 +++++++++-------- 2 files changed, 13 insertions(+), 8 deletions(-) diff --git a/terraform/modules/snapshots_tool_rds_dest/locals.tf b/terraform/modules/snapshots_tool_rds_dest/locals.tf index 0b868ae..635cecc 100644 --- a/terraform/modules/snapshots_tool_rds_dest/locals.tf +++ b/terraform/modules/snapshots_tool_rds_dest/locals.tf @@ -1,4 +1,8 @@ locals { DeleteOld = var.delete_old_snapshots == "FALSE" CrossAccount = var.cross_account_copy == "FALSE" + sns_topic_names = [ + "copies_failed_dest_rds", + "delete_old_failed_dest_rds" + ] } diff --git a/terraform/modules/snapshots_tool_rds_dest/resource.tf b/terraform/modules/snapshots_tool_rds_dest/resource.tf index a97140d..dc8c416 100644 --- a/terraform/modules/snapshots_tool_rds_dest/resource.tf +++ b/terraform/modules/snapshots_tool_rds_dest/resource.tf @@ -1,16 +1,16 @@ -resource "aws_sns_topic" "topic_copy_failed_dest" { - display_name = "copies_failed_dest_rds" -} -resource "aws_sns_topic" "topic_delete_old_failed_dest" { - display_name = "delete_old_failed_dest_rds" +resource "aws_sns_topic" "this" { + for_each = local.sns_topic_names + display_name = each.value } -resource "aws_sns_topic_policy" "snspolicy_delete_failed_dest" { - arn = aws_sns_topic.topic_delete_old_failed_dest.arn // in tf there is not a cfn "topcs" attribute which allows a list. +resource "aws_sns_topic_policy" "this" { + for_each = aws_sns_topic.this + + arn = each.value.arn policy = { Version = "2008-10-17" - Id = "destination_rds_failed_snapshot_delete" + Id = each.value.name Statement = [ { Sid = "sns-permissions" @@ -41,6 +41,7 @@ resource "aws_sns_topic_policy" "snspolicy_delete_failed_dest" { } + resource "aws_sns_topic_policy" "snspolicy_copy_failed_dest" { arn = aws_sns_topic.topic_copy_failed_dest.arn From 992d2eeedd9eb559b02ce1d33814006aaa9dc267 Mon Sep 17 00:00:00 2001 From: charlie keegan Date: Tue, 7 Feb 2023 14:10:59 +0000 Subject: [PATCH 19/48] change directory names --- terraform/main.tf | 2 +- .../{snapshots_tool_rds_dest => rds_destination}/data.tf | 0 .../{snapshots_tool_rds_dest => rds_destination}/locals.tf | 0 .../{snapshots_tool_rds_dest => rds_destination}/outputs.tf | 0 .../{snapshots_tool_rds_dest => rds_destination}/resource.tf | 0 .../{snapshots_tool_rds_dest => rds_destination}/variables.tf | 0 .../modules/{snapshots_tool_rds_source => rds_source}/data.tf | 0 .../modules/{snapshots_tool_rds_source => rds_source}/locals.tf | 0 .../modules/{snapshots_tool_rds_source => rds_source}/output.tf | 0 .../{snapshots_tool_rds_source => rds_source}/resource.tf | 0 .../{snapshots_tool_rds_source => rds_source}/variable.tf | 0 11 files changed, 1 insertion(+), 1 deletion(-) rename terraform/modules/{snapshots_tool_rds_dest => rds_destination}/data.tf (100%) rename terraform/modules/{snapshots_tool_rds_dest => rds_destination}/locals.tf (100%) rename terraform/modules/{snapshots_tool_rds_dest => rds_destination}/outputs.tf (100%) rename terraform/modules/{snapshots_tool_rds_dest => rds_destination}/resource.tf (100%) rename terraform/modules/{snapshots_tool_rds_dest => rds_destination}/variables.tf (100%) rename terraform/modules/{snapshots_tool_rds_source => rds_source}/data.tf (100%) rename terraform/modules/{snapshots_tool_rds_source => rds_source}/locals.tf (100%) rename terraform/modules/{snapshots_tool_rds_source => rds_source}/output.tf (100%) rename terraform/modules/{snapshots_tool_rds_source => rds_source}/resource.tf (100%) rename terraform/modules/{snapshots_tool_rds_source => rds_source}/variable.tf (100%) diff --git a/terraform/main.tf b/terraform/main.tf index 3cfa1fe..3ea9603 100644 --- a/terraform/main.tf +++ b/terraform/main.tf @@ -3,7 +3,7 @@ module "test_dbs" { } module "snapshots_tool_rds_dest" { - source = "./modules/snapshots_tool_rds_dest" + source = "./modules/rds_destination" } diff --git a/terraform/modules/snapshots_tool_rds_dest/data.tf b/terraform/modules/rds_destination/data.tf similarity index 100% rename from terraform/modules/snapshots_tool_rds_dest/data.tf rename to terraform/modules/rds_destination/data.tf diff --git a/terraform/modules/snapshots_tool_rds_dest/locals.tf b/terraform/modules/rds_destination/locals.tf similarity index 100% rename from terraform/modules/snapshots_tool_rds_dest/locals.tf rename to terraform/modules/rds_destination/locals.tf diff --git a/terraform/modules/snapshots_tool_rds_dest/outputs.tf b/terraform/modules/rds_destination/outputs.tf similarity index 100% rename from terraform/modules/snapshots_tool_rds_dest/outputs.tf rename to terraform/modules/rds_destination/outputs.tf diff --git a/terraform/modules/snapshots_tool_rds_dest/resource.tf b/terraform/modules/rds_destination/resource.tf similarity index 100% rename from terraform/modules/snapshots_tool_rds_dest/resource.tf rename to terraform/modules/rds_destination/resource.tf diff --git a/terraform/modules/snapshots_tool_rds_dest/variables.tf b/terraform/modules/rds_destination/variables.tf similarity index 100% rename from terraform/modules/snapshots_tool_rds_dest/variables.tf rename to terraform/modules/rds_destination/variables.tf diff --git a/terraform/modules/snapshots_tool_rds_source/data.tf b/terraform/modules/rds_source/data.tf similarity index 100% rename from terraform/modules/snapshots_tool_rds_source/data.tf rename to terraform/modules/rds_source/data.tf diff --git a/terraform/modules/snapshots_tool_rds_source/locals.tf b/terraform/modules/rds_source/locals.tf similarity index 100% rename from terraform/modules/snapshots_tool_rds_source/locals.tf rename to terraform/modules/rds_source/locals.tf diff --git a/terraform/modules/snapshots_tool_rds_source/output.tf b/terraform/modules/rds_source/output.tf similarity index 100% rename from terraform/modules/snapshots_tool_rds_source/output.tf rename to terraform/modules/rds_source/output.tf diff --git a/terraform/modules/snapshots_tool_rds_source/resource.tf b/terraform/modules/rds_source/resource.tf similarity index 100% rename from terraform/modules/snapshots_tool_rds_source/resource.tf rename to terraform/modules/rds_source/resource.tf diff --git a/terraform/modules/snapshots_tool_rds_source/variable.tf b/terraform/modules/rds_source/variable.tf similarity index 100% rename from terraform/modules/snapshots_tool_rds_source/variable.tf rename to terraform/modules/rds_source/variable.tf From 9b5a9a19fed206ac2664db75d5adf395770d82c1 Mon Sep 17 00:00:00 2001 From: charlie keegan Date: Tue, 7 Feb 2023 15:02:36 +0000 Subject: [PATCH 20/48] cleanup of terraform resources --- terraform/modules/rds_destination/locals.tf | 4 - terraform/modules/rds_destination/resource.tf | 91 +++++++++++++------ 2 files changed, 63 insertions(+), 32 deletions(-) diff --git a/terraform/modules/rds_destination/locals.tf b/terraform/modules/rds_destination/locals.tf index 635cecc..0b868ae 100644 --- a/terraform/modules/rds_destination/locals.tf +++ b/terraform/modules/rds_destination/locals.tf @@ -1,8 +1,4 @@ locals { DeleteOld = var.delete_old_snapshots == "FALSE" CrossAccount = var.cross_account_copy == "FALSE" - sns_topic_names = [ - "copies_failed_dest_rds", - "delete_old_failed_dest_rds" - ] } diff --git a/terraform/modules/rds_destination/resource.tf b/terraform/modules/rds_destination/resource.tf index dc8c416..472a712 100644 --- a/terraform/modules/rds_destination/resource.tf +++ b/terraform/modules/rds_destination/resource.tf @@ -1,16 +1,16 @@ -resource "aws_sns_topic" "this" { - for_each = local.sns_topic_names - display_name = each.value +resource "aws_sns_topic" "copy_failed_dest" { + display_name = "copies_failed_dest_rds" +} +resource "aws_sns_topic" "delete_old_failed_dest" { + display_name = "delete_old_failed_dest_rds" } -resource "aws_sns_topic_policy" "this" { - for_each = aws_sns_topic.this - - arn = each.value.arn +resource "aws_sns_topic_policy" "copy_failed_dest" { + arn = aws_sns_topic.copy_failed_dest.arn // in tf there is not a cfn "topcs" attribute which allows a list. - policy = { + policy = jsonencode({ Version = "2008-10-17" - Id = each.value.name + Id = "destination_rds_failed_snapshot_delete" Statement = [ { Sid = "sns-permissions" @@ -37,15 +37,48 @@ resource "aws_sns_topic_policy" "this" { } } ] - } + }) } +resource "aws_sns_topic_policy" "delete_failed_dest" { + arn = aws_sns_topic.delete_old_failed_dest.arn // in tf there is not a cfn "topcs" attribute which allows a list. + policy = jsonencode({ + Version = "2008-10-17" + Id = "destination_rds_failed_snapshot_delete" + Statement = [ + { + Sid = "sns-permissions" + Effect = "Allow" + Principal = { + AWS = "*" + } + Action = [ + "SNS:GetTopicAttributes", + "SNS:SetTopicAttributes", + "SNS:AddPermission", + "SNS:RemovePermission", + "SNS:DeleteTopic", + "SNS:Subscribe", + "SNS:ListSubscriptionsByTopic", + "SNS:Publish", + "SNS:Receive" + ] + Resource = "*" + Condition = { + test = "StringEquals" + variable = "AWS:SourceOwner" + values = [data.aws_caller_identity.current.account_id] + } + } + ] + }) +} resource "aws_sns_topic_policy" "snspolicy_copy_failed_dest" { - arn = aws_sns_topic.topic_copy_failed_dest.arn + arn = aws_sns_topic.copy_failed_dest.arn - policy = { + policy = jsonencode({ Version = "2008-10-17" Id = "destination_rds_failed_snapshot_copy" Statement = [ @@ -74,7 +107,7 @@ resource "aws_sns_topic_policy" "snspolicy_copy_failed_dest" { } } ] - } + }) } resource "aws_cloudwatch_metric_alarm" "alarmcw_copy_failed_dest" { @@ -88,12 +121,12 @@ resource "aws_cloudwatch_metric_alarm" "alarmcw_copy_failed_dest" { threshold = "1.0" dimensions = { - StateMachineArn = aws_sfn_state_machine.statemachine_copy_snapshots_dest_rds.id + StateMachineArn = aws_sfn_state_machine.statemachine_delete_old_snapshots_dest_rds.arn } alarm_description = "This metric monitors state machine failure for copying snapshots" alarm_actions = [ - aws_sns_topic.topic_copy_failed_dest.id + aws_sns_topic.copy_failed_dest.id ] } @@ -108,12 +141,12 @@ resource "aws_cloudwatch_metric_alarm" "alarmcw_delete_old_failed_dest" { threshold = "2.0" dimensions = { - StateMachineArn = aws_sfn_state_machine.statemachine_copy_snapshots_dest_rds.id + StateMachineArn = aws_sfn_state_machine.statemachine_copy_old_snapshots_dest_rds.arn } alarm_description = "This metric monitors state machine failure for deleting old snapshots" alarm_actions = [ - aws_sns_topic.topic_delete_old_failed_dest.id + aws_sns_topic.delete_old_failed_dest.id ] } @@ -229,11 +262,11 @@ resource "aws_iam_role_policy_attachment" "attachment" { } resource "aws_lambda_function" "lambda_copy_snapshots_rds" { - s3_bucket = var.code_bucket - s3_key = local.CrossAccount ? "copy_snapshots_dest_rds.zip" : "copy_snapshots_no_x_account_rds.zip" - - memory_size = 512 - description = "This functions copies snapshots for RDS Instances shared with this account. It checks for existing snapshots following the pattern specified in the environment variables with the following format: -YYYY-MM-DD-HH-MM" + function_name = "snapshot-copier" + description = "This functions copies snapshots for RDS Instances shared with this account. It checks for existing snapshots following the pattern specified in the environment variables with the following format: -YYYY-MM-DD-HH-MM" + s3_bucket = var.code_bucket + s3_key = local.CrossAccount ? "copy_snapshots_dest_rds.zip" : "copy_snapshots_no_x_account_rds.zip" + memory_size = 512 environment { variables = { SNAPSHOT_PATTERN = var.snapshot_pattern @@ -252,11 +285,13 @@ resource "aws_lambda_function" "lambda_copy_snapshots_rds" { } resource "aws_lambda_function" "delete_old_dest_rds" { - count = locals.DeleteOld ? 1 : 0 - s3_bucket = var.code_bucket - s3_key = local.CrossAccount ? "delete_old_snapshots_dest_rds.zip" : "delete_old_snapshots_no_x_account_rds.zip" - memory_size = 512 - description = "This function enforces retention on the snapshots shared with the destination account. " + function_name = "dest-snapshot-retention" + description = "This function enforces retention on the snapshots shared with the destination account. " + count = locals.DeleteOld ? 1 : 0 + s3_bucket = var.code_bucket + s3_key = local.CrossAccount ? "delete_old_snapshots_dest_rds.zip" : "delete_old_snapshots_no_x_account_rds.zip" + memory_size = 512 + environment { variables = { SNAPSHOT_PATTERN = var.snapshot_pattern @@ -435,7 +470,7 @@ PATTERN resource "aws_cloudwatch_event_target" "copy_snapshots_rds" { rule = aws_cloudwatch_event_rule.copy_snapshots_rds.name target_id = "DestTarget1" - arn = aws_sfn_state_machine.statemachine_copy_snapshots_dest_rds.id + arn = aws_sfn_state_machine.statemachine_copy_old_snapshots_dest_rds.id role_arn = aws_iam_role.iamrole_step_invocation.arn } From e81ea279772219941e905dfebaee7a45bc98416d Mon Sep 17 00:00:00 2001 From: charlie keegan Date: Tue, 7 Feb 2023 15:35:41 +0000 Subject: [PATCH 21/48] fix lambda function arn count --- terraform/modules/rds_destination/outputs.tf | 4 ++-- terraform/modules/rds_destination/resource.tf | 4 ++-- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/terraform/modules/rds_destination/outputs.tf b/terraform/modules/rds_destination/outputs.tf index 6e9e194..5c653d0 100644 --- a/terraform/modules/rds_destination/outputs.tf +++ b/terraform/modules/rds_destination/outputs.tf @@ -1,9 +1,9 @@ output "copy_failed_topic" { description = "Subscribe to this topic to receive alerts of failed copies" - value = aws_sns_topic.topic_copy_failed_dest.id + value = aws_sns_topic.copy_failed_dest.id } output "delete_old_failed_topic" { description = "Subscribe to this topic to receive alerts of failures at deleting old snapshots" - value = aws_sns_topic.topic_delete_old_failed_dest.id + value = aws_sns_topic.delete_old_failed_dest.id } diff --git a/terraform/modules/rds_destination/resource.tf b/terraform/modules/rds_destination/resource.tf index 472a712..d289f28 100644 --- a/terraform/modules/rds_destination/resource.tf +++ b/terraform/modules/rds_destination/resource.tf @@ -287,7 +287,7 @@ resource "aws_lambda_function" "lambda_copy_snapshots_rds" { resource "aws_lambda_function" "delete_old_dest_rds" { function_name = "dest-snapshot-retention" description = "This function enforces retention on the snapshots shared with the destination account. " - count = locals.DeleteOld ? 1 : 0 + count = local.DeleteOld ? 1 : 0 s3_bucket = var.code_bucket s3_key = local.CrossAccount ? "delete_old_snapshots_dest_rds.zip" : "delete_old_snapshots_no_x_account_rds.zip" memory_size = 512 @@ -348,7 +348,7 @@ resource "aws_sfn_state_machine" "statemachine_delete_old_snapshots_dest_rds" { "States": { "DeleteOldDestRegion": { "Type": "Task", - "Resource": "${aws_lambda_function.delete_old_dest_rds.arn}", + "Resource": "${aws_lambda_function.delete_old_dest_rds[*].arn}", "Retry": [ { "ErrorEquals": [ From b2f251261f8e50444a088969626fecaab78b5fbd Mon Sep 17 00:00:00 2001 From: charlie keegan Date: Tue, 7 Feb 2023 15:58:18 +0000 Subject: [PATCH 22/48] fix state machine references --- terraform/modules/rds_destination/resource.tf | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/terraform/modules/rds_destination/resource.tf b/terraform/modules/rds_destination/resource.tf index d289f28..8c07952 100644 --- a/terraform/modules/rds_destination/resource.tf +++ b/terraform/modules/rds_destination/resource.tf @@ -111,6 +111,7 @@ resource "aws_sns_topic_policy" "snspolicy_copy_failed_dest" { } resource "aws_cloudwatch_metric_alarm" "alarmcw_copy_failed_dest" { + alarm_name = "failed-rds-copy" comparison_operator = "GreaterThanOrEqualToThreshold" evaluation_periods = "1" @@ -121,7 +122,7 @@ resource "aws_cloudwatch_metric_alarm" "alarmcw_copy_failed_dest" { threshold = "1.0" dimensions = { - StateMachineArn = aws_sfn_state_machine.statemachine_delete_old_snapshots_dest_rds.arn + StateMachineArn = aws_sfn_state_machine.statemachine_copy_old_snapshots_dest_rds[*].arn } alarm_description = "This metric monitors state machine failure for copying snapshots" @@ -131,6 +132,7 @@ resource "aws_cloudwatch_metric_alarm" "alarmcw_copy_failed_dest" { } resource "aws_cloudwatch_metric_alarm" "alarmcw_delete_old_failed_dest" { + count = local.DeleteOld ? 1 : 0 alarm_name = "failed-rds-delete-old-snapshot" comparison_operator = "GreaterThanOrEqualToThreshold" evaluation_periods = "2" @@ -141,7 +143,7 @@ resource "aws_cloudwatch_metric_alarm" "alarmcw_delete_old_failed_dest" { threshold = "2.0" dimensions = { - StateMachineArn = aws_sfn_state_machine.statemachine_copy_old_snapshots_dest_rds.arn + StateMachineArn = aws_sfn_state_machine.statemachine_delete_old_snapshots_dest_rds[*].arn } alarm_description = "This metric monitors state machine failure for deleting old snapshots" @@ -285,9 +287,9 @@ resource "aws_lambda_function" "lambda_copy_snapshots_rds" { } resource "aws_lambda_function" "delete_old_dest_rds" { + count = local.DeleteOld ? 1 : 0 function_name = "dest-snapshot-retention" description = "This function enforces retention on the snapshots shared with the destination account. " - count = local.DeleteOld ? 1 : 0 s3_bucket = var.code_bucket s3_key = local.CrossAccount ? "delete_old_snapshots_dest_rds.zip" : "delete_old_snapshots_no_x_account_rds.zip" memory_size = 512 @@ -338,6 +340,7 @@ resource "aws_iam_role" "iamrole_state_execution" { } resource "aws_sfn_state_machine" "statemachine_delete_old_snapshots_dest_rds" { + count = local.DeleteOld ? 1 : 0 name = "delete-old-snapshots-destination-rds" role_arn = aws_iam_role.iamrole_state_execution.arn From 96385a18452f98301925755e805ee6fe74e7262f Mon Sep 17 00:00:00 2001 From: charlie keegan Date: Wed, 8 Feb 2023 10:01:57 +0000 Subject: [PATCH 23/48] fix deployment and split iam into separate file --- terraform/modules/rds_destination/iam.tf | 84 +++++++++++++ terraform/modules/rds_destination/resource.tf | 113 +----------------- 2 files changed, 85 insertions(+), 112 deletions(-) create mode 100644 terraform/modules/rds_destination/iam.tf diff --git a/terraform/modules/rds_destination/iam.tf b/terraform/modules/rds_destination/iam.tf new file mode 100644 index 0000000..1b0555b --- /dev/null +++ b/terraform/modules/rds_destination/iam.tf @@ -0,0 +1,84 @@ +resource "aws_iam_role" "snapshots_rds" { + assume_role_policy = jsonencode({ + Version = "2012-10-17" + Statement = [ + { + Effect = "Allow" + Principal = { + Service = "lambda.amazonaws.com" + } + Action = "sts:AssumeRole" + } + ] + }) + force_detach_policies = true +} + +resource "aws_iam_policy" "snapshot_rds" { + name = "rds_kms_access" + + policy = jsonencode({ + Version = "2012-10-17" + Statement = [ + { + Sid = "AllowUseOfTheKey" + Effect = "Allow" + Action = [ + "kms:Encrypt", + "kms:Decrypt", + "kms:ReEncrypt*", + "kms:GenerateDataKey*", + "kms:DescribeKey" + ] + Resource = [ + "*" + ] + }, + { + Sid = "AllowAttachmentOfPersistentResources" + Effect = "Allow" + Action = [ + "kms:CreateGrant", + "kms:ListGrants", + "kms:RevokeGrant" + ] + Resource = [ + "*" + ] + Condition = { + test = "Bool" + variable = "kms:GrantIsForAWSResource" + values = ["True"] + } + }, + { + Effect = "Allow" + Action = [ + "rds:CreateDBSnapshot", + "rds:DeleteDBSnapshot", + "rds:DescribeDBInstances", + "rds:DescribeDBSnapshots", + "rds:ModifyDBSnapshotAttribute", + "rds:DescribeDBSnapshotAttributes", + "rds:CopyDBSnapshot", + "rds:ListTagsForResource", + "rds:AddTagsToResource" + ] + Resource = "*" + }, + { + Effect = "Allow" + Action = [ + "logs:CreateLogGroup", + "logs:CreateLogStream", + "logs:PutLogEvents" + ] + Resource = "arn:aws:logs:*:*:*" + } + ] + }) +} +resource "aws_iam_role_policy_attachment" "attachment" { + role = aws_iam_role.snapshots_rds.name + policy_arn = aws_iam_policy.snapshot_rds.arn +} diff --git a/terraform/modules/rds_destination/resource.tf b/terraform/modules/rds_destination/resource.tf index 8c07952..48efdc2 100644 --- a/terraform/modules/rds_destination/resource.tf +++ b/terraform/modules/rds_destination/resource.tf @@ -122,7 +122,7 @@ resource "aws_cloudwatch_metric_alarm" "alarmcw_copy_failed_dest" { threshold = "1.0" dimensions = { - StateMachineArn = aws_sfn_state_machine.statemachine_copy_old_snapshots_dest_rds[*].arn + StateMachineArn = aws_sfn_state_machine.statemachine_copy_old_snapshots_dest_rds.arn } alarm_description = "This metric monitors state machine failure for copying snapshots" @@ -152,117 +152,6 @@ resource "aws_cloudwatch_metric_alarm" "alarmcw_delete_old_failed_dest" { ] } -resource "aws_iam_role" "snapshots_rds" { - assume_role_policy = jsonencode({ - Version = "2012-10-17" - Statement = [ - { - Effect = "Allow" - Principal = { - Service = "lambda.amazonaws.com" - } - Action = "sts:AssumeRole" - } - ] - }) - force_detach_policies = true -} - -resource "aws_iam_policy" "snapshots_rds_cw_logs" { - name = "snapshots_rds_cw_logs" - - policy = jsonencode({ - Version = "2012-10-17" - Statement = [ - { - Effect = "Allow" - Action = [ - "logs:CreateLogGroup", - "logs:CreateLogStream", - "logs:PutLogEvents" - ] - Resource = "arn:aws:logs:*:*:*" - } - ] - }) -} - -resource "aws_iam_policy" "snapshots_rds" { - name = "snapshots_rds" - - policy = jsonencode({ - Version = "2012-10-17" - Statement = [ - { - Effect = "Allow" - Action = [ - "rds:CreateDBSnapshot", - "rds:DeleteDBSnapshot", - "rds:DescribeDBInstances", - "rds:DescribeDBSnapshots", - "rds:ModifyDBSnapshotAttribute", - "rds:DescribeDBSnapshotAttributes", - "rds:CopyDBSnapshot", - "rds:ListTagsForResource", - "rds:AddTagsToResource" - ] - Resource = "*" - } - ] - }) -} - -resource "aws_iam_policy" "rds_kms_access" { - name = "rds_kms_access" - - policy = jsonencode({ - Version = "2012-10-17" - Statement = [ - { - Sid = "AllowUseOfTheKey" - Effect = "Allow" - Action = [ - "kms:Encrypt", - "kms:Decrypt", - "kms:ReEncrypt*", - "kms:GenerateDataKey*", - "kms:DescribeKey" - ] - Resource = [ - "*" - ] - }, - { - Sid = "AllowAttachmentOfPersistentResources" - Effect = "Allow" - Action = [ - "kms:CreateGrant", - "kms:ListGrants", - "kms:RevokeGrant" - ] - Resource = [ - "*" - ] - Condition = { - test = "Bool" - variable = "kms:GrantIsForAWSResource" - values = ["True"] - } - } - ] - }) -} -resource "aws_iam_role_policy_attachment" "attachment" { - for_each = toset([ - aws_iam_policy.snapshots_rds_cw_logs.arn, - aws_iam_policy.rds_kms_access.arn, - aws_iam_policy.snapshots_rds.arn - ]) - - role = aws_iam_role.snapshots_rds.name - policy_arn = each.value -} - resource "aws_lambda_function" "lambda_copy_snapshots_rds" { function_name = "snapshot-copier" description = "This functions copies snapshots for RDS Instances shared with this account. It checks for existing snapshots following the pattern specified in the environment variables with the following format: -YYYY-MM-DD-HH-MM" From 69ce803c3644496ecadbaede34a87957083827d2 Mon Sep 17 00:00:00 2001 From: charlie keegan Date: Wed, 8 Feb 2023 10:06:41 +0000 Subject: [PATCH 24/48] enable old snapshots cw events --- terraform/modules/rds_destination/iam.tf | 66 ++++++++++++ terraform/modules/rds_destination/resource.tf | 101 ++++-------------- 2 files changed, 87 insertions(+), 80 deletions(-) diff --git a/terraform/modules/rds_destination/iam.tf b/terraform/modules/rds_destination/iam.tf index 1b0555b..4a0fdd4 100644 --- a/terraform/modules/rds_destination/iam.tf +++ b/terraform/modules/rds_destination/iam.tf @@ -82,3 +82,69 @@ resource "aws_iam_role_policy_attachment" "attachment" { role = aws_iam_role.snapshots_rds.name policy_arn = aws_iam_policy.snapshot_rds.arn } + + + +resource "aws_iam_role" "iamrole_state_execution" { + assume_role_policy = jsonencode({ + Version = "2012-10-17" + Statement = [ + { + Effect = "Allow" + Principal = { + Service = join("", ["states.", data.aws_region.current.name, ".amazonaws.com"]) + } + Action = "sts:AssumeRole" + } + ] + }) + force_detach_policies = true + inline_policy { + name = "inline_policy_rds_snapshot" + policy = jsonencode({ + Version = "2012-10-17" + Statement = [ + { + Effect = "Allow" + Action = [ + "lambda:InvokeFunction" + ] + Resource = "*" + } + ] + }) + } +} + + +resource "aws_iam_role" "iamrole_step_invocation" { + name = "invoke-state-machines" + assume_role_policy = jsonencode({ + Version = "2012-10-17" + Statement = [ + { + Effect = "Allow" + Principal = { + Service = "events.amazonaws.com" + } + Action = "sts:AssumeRole" + } + ] + }) + force_detach_policies = true + inline_policy { + name = "inline_policy_state_invocation" + policy = jsonencode({ + Version = "2012-10-17" + Statement = [ + { + Effect = "Allow" + Action = [ + "states:StartExecution" + ] + Resource = "*" + } + ] + }) + } +} diff --git a/terraform/modules/rds_destination/resource.tf b/terraform/modules/rds_destination/resource.tf index 48efdc2..03203c6 100644 --- a/terraform/modules/rds_destination/resource.tf +++ b/terraform/modules/rds_destination/resource.tf @@ -197,36 +197,6 @@ resource "aws_lambda_function" "delete_old_dest_rds" { timeout = 300 } -resource "aws_iam_role" "iamrole_state_execution" { - assume_role_policy = jsonencode({ - Version = "2012-10-17" - Statement = [ - { - Effect = "Allow" - Principal = { - Service = join("", ["states.", data.aws_region.current.name, ".amazonaws.com"]) - } - Action = "sts:AssumeRole" - } - ] - }) - force_detach_policies = true - inline_policy { - name = "inline_policy_rds_snapshot" - policy = jsonencode({ - Version = "2012-10-17" - Statement = [ - { - Effect = "Allow" - Action = [ - "lambda:InvokeFunction" - ] - Resource = "*" - } - ] - }) - } -} resource "aws_sfn_state_machine" "statemachine_delete_old_snapshots_dest_rds" { count = local.DeleteOld ? 1 : 0 @@ -304,37 +274,6 @@ EOF } -resource "aws_iam_role" "iamrole_step_invocation" { - name = "invoke-state-machines" - assume_role_policy = jsonencode({ - Version = "2012-10-17" - Statement = [ - { - Effect = "Allow" - Principal = { - Service = "events.amazonaws.com" - } - Action = "sts:AssumeRole" - } - ] - }) - force_detach_policies = true - inline_policy { - name = "inline_policy_state_invocation" - policy = jsonencode({ - Version = "2012-10-17" - Statement = [ - { - Effect = "Allow" - Action = [ - "states:StartExecution" - ] - Resource = "*" - } - ] - }) - } -} resource "aws_cloudwatch_event_rule" "copy_snapshots_rds" { @@ -366,25 +305,27 @@ resource "aws_cloudwatch_event_target" "copy_snapshots_rds" { role_arn = aws_iam_role.iamrole_step_invocation.arn } -# resource "aws_iot_topic_rule_destination" "cw_event_delete_old_snapshots_rds" { -# count = locals.DeleteOld ? 1 : 0 -# // CF Property(Description) = "Triggers the RDS DeleteOld state machine in the destination account" -# // CF Property(ScheduleExpression) = join("", ["cron(", "0 /1 * * ? *", ")"]) -# // CF Property(State) = "ENABLED" -# // CF Property(Targets) = [ -# // { -# // Arn = aws_sfn_state_machine.statemachine_delete_old_snapshots_dest_rds[0].id -# // Id = "Target1" -# // RoleArn = aws_iam_role.iamrole_step_invocation.arn -# // } -# // ] -# } - -# resource "aws_inspector_resource_group" "cwloggroup_delete_old_snapshots_dest_rds" { -# count = locals.DeleteOld ? 1 : 0 -# // CF Property(RetentionInDays) = var.lambda_cw_log_retention -# // CF Property(LogGroupName) = "/aws/lambda/${var.log_group_name}" -# } +resource "aws_cloudwatch_event_rule" "delete_old_snapshots_rds" { + name = "trigger-snapshot-delete-in-dest-account" + description = "Triggers the RDS DeleteOld state machine in the destination account" + schedule_expression = "cron(0 /1 * * ? *)" + is_enabled = true +} + +resource "aws_cloudwatch_event_target" "delete_old_snapshots_rds" { + count = locals.DeleteOld ? 1 : 0 + rule = aws_cloudwatch_event_rule.delete_old_snapshots_rds.name + target_id = "DestTarget1" + arn = aws_sfn_state_machine.statemachine_delete_old_snapshots_dest_rds[0].id + role_arn = aws_iam_role.iamrole_step_invocation.arn +} + +resource "aws_cloudwatch_log_group" "cwloggroup_delete_old_snapshots_dest_rds" { + count = locals.DeleteOld ? 1 : 0 + name = "/aws/lambda/${var.log_group_name}" + retention_in_days = var.lambda_cw_log_retention +} + resource "aws_cloudwatch_log_group" "copy_snapshots_rds" { name = "/aws/lambda/${aws_lambda_function.lambda_copy_snapshots_rds.arn}" From 8f77df9977a8fd4733aba4525c3aab1c80369d77 Mon Sep 17 00:00:00 2001 From: charlie keegan Date: Wed, 8 Feb 2023 10:09:27 +0000 Subject: [PATCH 25/48] fix log group naming --- terraform/modules/rds_destination/resource.tf | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/terraform/modules/rds_destination/resource.tf b/terraform/modules/rds_destination/resource.tf index 03203c6..74990f5 100644 --- a/terraform/modules/rds_destination/resource.tf +++ b/terraform/modules/rds_destination/resource.tf @@ -313,7 +313,7 @@ resource "aws_cloudwatch_event_rule" "delete_old_snapshots_rds" { } resource "aws_cloudwatch_event_target" "delete_old_snapshots_rds" { - count = locals.DeleteOld ? 1 : 0 + count = local.DeleteOld ? 1 : 0 rule = aws_cloudwatch_event_rule.delete_old_snapshots_rds.name target_id = "DestTarget1" arn = aws_sfn_state_machine.statemachine_delete_old_snapshots_dest_rds[0].id @@ -321,13 +321,13 @@ resource "aws_cloudwatch_event_target" "delete_old_snapshots_rds" { } resource "aws_cloudwatch_log_group" "cwloggroup_delete_old_snapshots_dest_rds" { - count = locals.DeleteOld ? 1 : 0 + count = local.DeleteOld ? 1 : 0 name = "/aws/lambda/${var.log_group_name}" retention_in_days = var.lambda_cw_log_retention } resource "aws_cloudwatch_log_group" "copy_snapshots_rds" { - name = "/aws/lambda/${aws_lambda_function.lambda_copy_snapshots_rds.arn}" + name = "/aws/lambda/${aws_lambda_function.lambda_copy_snapshots_rds.function_name}" retention_in_days = var.lambda_cw_log_retention } From 259af753bfc26c88a55977ef2899fc3486da84a7 Mon Sep 17 00:00:00 2001 From: charlie keegan Date: Wed, 8 Feb 2023 10:12:19 +0000 Subject: [PATCH 26/48] move stepfuncs and lambdas into their own files --- terraform/Makefile | 2 + terraform/modules/rds_destination/lambda.tf | 45 +++++++ terraform/modules/rds_destination/resource.tf | 124 ------------------ .../modules/rds_destination/stepfunction.tf | 75 +++++++++++ 4 files changed, 122 insertions(+), 124 deletions(-) create mode 100644 terraform/Makefile create mode 100644 terraform/modules/rds_destination/lambda.tf create mode 100644 terraform/modules/rds_destination/stepfunction.tf diff --git a/terraform/Makefile b/terraform/Makefile new file mode 100644 index 0000000..451f93f --- /dev/null +++ b/terraform/Makefile @@ -0,0 +1,2 @@ +apply: + terraform apply --auto-approve \ No newline at end of file diff --git a/terraform/modules/rds_destination/lambda.tf b/terraform/modules/rds_destination/lambda.tf new file mode 100644 index 0000000..c8aa6c5 --- /dev/null +++ b/terraform/modules/rds_destination/lambda.tf @@ -0,0 +1,45 @@ +resource "aws_lambda_function" "delete_old_dest_rds" { + count = local.DeleteOld ? 1 : 0 + function_name = "dest-snapshot-retention" + description = "This function enforces retention on the snapshots shared with the destination account. " + s3_bucket = var.code_bucket + s3_key = local.CrossAccount ? "delete_old_snapshots_dest_rds.zip" : "delete_old_snapshots_no_x_account_rds.zip" + memory_size = 512 + + environment { + variables = { + SNAPSHOT_PATTERN = var.snapshot_pattern + DEST_REGION = var.destination_region + RETENTION_DAYS = var.retention_days + LOG_LEVEL = var.log_level + } + } + role = aws_iam_role.snapshots_rds.arn + runtime = "python3.7" + handler = "lambda_function.lambda_handler" + timeout = 300 +} + + +resource "aws_lambda_function" "lambda_copy_snapshots_rds" { + function_name = "snapshot-copier" + description = "This functions copies snapshots for RDS Instances shared with this account. It checks for existing snapshots following the pattern specified in the environment variables with the following format: -YYYY-MM-DD-HH-MM" + s3_bucket = var.code_bucket + s3_key = local.CrossAccount ? "copy_snapshots_dest_rds.zip" : "copy_snapshots_no_x_account_rds.zip" + memory_size = 512 + environment { + variables = { + SNAPSHOT_PATTERN = var.snapshot_pattern + DEST_REGION = var.destination_region + LOG_LEVEL = var.log_level + REGION_OVERRIDE = var.source_region_override + KMS_KEY_DEST_REGION = var.kms_key_destination + KMS_KEY_SOURCE_REGION = var.kms_key_source + RETENTION_DAYS = var.retention_days + } + } + role = aws_iam_role.snapshots_rds.arn + runtime = "python3.7" + handler = "lambda_function.lambda_handler" + timeout = 300 +} diff --git a/terraform/modules/rds_destination/resource.tf b/terraform/modules/rds_destination/resource.tf index 74990f5..e5b502f 100644 --- a/terraform/modules/rds_destination/resource.tf +++ b/terraform/modules/rds_destination/resource.tf @@ -152,130 +152,6 @@ resource "aws_cloudwatch_metric_alarm" "alarmcw_delete_old_failed_dest" { ] } -resource "aws_lambda_function" "lambda_copy_snapshots_rds" { - function_name = "snapshot-copier" - description = "This functions copies snapshots for RDS Instances shared with this account. It checks for existing snapshots following the pattern specified in the environment variables with the following format: -YYYY-MM-DD-HH-MM" - s3_bucket = var.code_bucket - s3_key = local.CrossAccount ? "copy_snapshots_dest_rds.zip" : "copy_snapshots_no_x_account_rds.zip" - memory_size = 512 - environment { - variables = { - SNAPSHOT_PATTERN = var.snapshot_pattern - DEST_REGION = var.destination_region - LOG_LEVEL = var.log_level - REGION_OVERRIDE = var.source_region_override - KMS_KEY_DEST_REGION = var.kms_key_destination - KMS_KEY_SOURCE_REGION = var.kms_key_source - RETENTION_DAYS = var.retention_days - } - } - role = aws_iam_role.snapshots_rds.arn - runtime = "python3.7" - handler = "lambda_function.lambda_handler" - timeout = 300 -} - -resource "aws_lambda_function" "delete_old_dest_rds" { - count = local.DeleteOld ? 1 : 0 - function_name = "dest-snapshot-retention" - description = "This function enforces retention on the snapshots shared with the destination account. " - s3_bucket = var.code_bucket - s3_key = local.CrossAccount ? "delete_old_snapshots_dest_rds.zip" : "delete_old_snapshots_no_x_account_rds.zip" - memory_size = 512 - - environment { - variables = { - SNAPSHOT_PATTERN = var.snapshot_pattern - DEST_REGION = var.destination_region - RETENTION_DAYS = var.retention_days - LOG_LEVEL = var.log_level - } - } - role = aws_iam_role.snapshots_rds.arn - runtime = "python3.7" - handler = "lambda_function.lambda_handler" - timeout = 300 -} - - -resource "aws_sfn_state_machine" "statemachine_delete_old_snapshots_dest_rds" { - count = local.DeleteOld ? 1 : 0 - name = "delete-old-snapshots-destination-rds" - role_arn = aws_iam_role.iamrole_state_execution.arn - - definition = < Date: Wed, 8 Feb 2023 10:39:40 +0000 Subject: [PATCH 27/48] fix iam role definitions --- terraform/modules/rds_destination/iam.tf | 121 +++++++++++------------ 1 file changed, 60 insertions(+), 61 deletions(-) diff --git a/terraform/modules/rds_destination/iam.tf b/terraform/modules/rds_destination/iam.tf index 4a0fdd4..58e7b95 100644 --- a/terraform/modules/rds_destination/iam.tf +++ b/terraform/modules/rds_destination/iam.tf @@ -14,69 +14,68 @@ resource "aws_iam_role" "snapshots_rds" { force_detach_policies = true } -resource "aws_iam_policy" "snapshot_rds" { - name = "rds_kms_access" +data "aws_iam_policy_document" "snapshot_rds" { - policy = jsonencode({ - Version = "2012-10-17" - Statement = [ - { - Sid = "AllowUseOfTheKey" - Effect = "Allow" - Action = [ - "kms:Encrypt", - "kms:Decrypt", - "kms:ReEncrypt*", - "kms:GenerateDataKey*", - "kms:DescribeKey" - ] - Resource = [ - "*" - ] - }, - { - Sid = "AllowAttachmentOfPersistentResources" - Effect = "Allow" - Action = [ - "kms:CreateGrant", - "kms:ListGrants", - "kms:RevokeGrant" - ] - Resource = [ - "*" - ] - Condition = { - test = "Bool" - variable = "kms:GrantIsForAWSResource" - values = ["True"] - } - }, - { - Effect = "Allow" - Action = [ - "rds:CreateDBSnapshot", - "rds:DeleteDBSnapshot", - "rds:DescribeDBInstances", - "rds:DescribeDBSnapshots", - "rds:ModifyDBSnapshotAttribute", - "rds:DescribeDBSnapshotAttributes", - "rds:CopyDBSnapshot", - "rds:ListTagsForResource", - "rds:AddTagsToResource" - ] - Resource = "*" - }, - { - Effect = "Allow" - Action = [ - "logs:CreateLogGroup", - "logs:CreateLogStream", - "logs:PutLogEvents" - ] - Resource = "arn:aws:logs:*:*:*" - } + statement { + sid = "AllowUseOfTheKey" + effect = "Allow" + actions = [ + "kms:Encrypt", + "kms:Decrypt", + "kms:ReEncrypt*", + "kms:GenerateDataKey*", + "kms:DescribeKey" ] - }) + resources = [ + "*" + ] + } + statement { + sid = "AllowAttachmentOfPersistentResources" + effect = "Allow" + actions = [ + "kms:CreateGrant", + "kms:ListGrants", + "kms:RevokeGrant" + ] + resources = [ + "*" + ] + condition { + test = "Bool" + variable = "kms:GrantIsForAWSResource" + values = ["True"] + } + } + statement { + effect = "Allow" + actions = [ + "rds:CreateDBSnapshot", + "rds:DeleteDBSnapshot", + "rds:DescribeDBInstances", + "rds:DescribeDBSnapshots", + "rds:ModifyDBSnapshotAttribute", + "rds:DescribeDBSnapshotAttributes", + "rds:CopyDBSnapshot", + "rds:ListTagsForResource", + "rds:AddTagsToResource" + ] + resources = ["*"] + } + statement { + effect = "Allow" + actions = [ + "logs:CreateLogGroup", + "logs:CreateLogStream", + "logs:PutLogEvents" + ] + resources = ["arn:aws:logs:*:*:*"] + } +} + +resource "aws_iam_policy" "snapshot_rds" { + name = "snapshot_rds" + policy = data.aws_iam_policy_document.snapshot_rds.json } resource "aws_iam_role_policy_attachment" "attachment" { role = aws_iam_role.snapshots_rds.name From fb08161ab371f5973c0a122fb92e0bc24530ca67 Mon Sep 17 00:00:00 2001 From: charlie keegan Date: Wed, 8 Feb 2023 13:40:34 +0000 Subject: [PATCH 28/48] finialise first deployment --- terraform/modules/rds_destination/resource.tf | 209 ------------------ 1 file changed, 209 deletions(-) delete mode 100644 terraform/modules/rds_destination/resource.tf diff --git a/terraform/modules/rds_destination/resource.tf b/terraform/modules/rds_destination/resource.tf deleted file mode 100644 index e5b502f..0000000 --- a/terraform/modules/rds_destination/resource.tf +++ /dev/null @@ -1,209 +0,0 @@ -resource "aws_sns_topic" "copy_failed_dest" { - display_name = "copies_failed_dest_rds" -} -resource "aws_sns_topic" "delete_old_failed_dest" { - display_name = "delete_old_failed_dest_rds" -} - -resource "aws_sns_topic_policy" "copy_failed_dest" { - arn = aws_sns_topic.copy_failed_dest.arn // in tf there is not a cfn "topcs" attribute which allows a list. - - policy = jsonencode({ - Version = "2008-10-17" - Id = "destination_rds_failed_snapshot_delete" - Statement = [ - { - Sid = "sns-permissions" - Effect = "Allow" - Principal = { - AWS = "*" - } - Action = [ - "SNS:GetTopicAttributes", - "SNS:SetTopicAttributes", - "SNS:AddPermission", - "SNS:RemovePermission", - "SNS:DeleteTopic", - "SNS:Subscribe", - "SNS:ListSubscriptionsByTopic", - "SNS:Publish", - "SNS:Receive" - ] - Resource = "*" - Condition = { - test = "StringEquals" - variable = "AWS:SourceOwner" - values = [data.aws_caller_identity.current.account_id] - } - } - ] - }) -} - -resource "aws_sns_topic_policy" "delete_failed_dest" { - arn = aws_sns_topic.delete_old_failed_dest.arn // in tf there is not a cfn "topcs" attribute which allows a list. - - policy = jsonencode({ - Version = "2008-10-17" - Id = "destination_rds_failed_snapshot_delete" - Statement = [ - { - Sid = "sns-permissions" - Effect = "Allow" - Principal = { - AWS = "*" - } - Action = [ - "SNS:GetTopicAttributes", - "SNS:SetTopicAttributes", - "SNS:AddPermission", - "SNS:RemovePermission", - "SNS:DeleteTopic", - "SNS:Subscribe", - "SNS:ListSubscriptionsByTopic", - "SNS:Publish", - "SNS:Receive" - ] - Resource = "*" - Condition = { - test = "StringEquals" - variable = "AWS:SourceOwner" - values = [data.aws_caller_identity.current.account_id] - } - } - ] - }) -} - -resource "aws_sns_topic_policy" "snspolicy_copy_failed_dest" { - arn = aws_sns_topic.copy_failed_dest.arn - - policy = jsonencode({ - Version = "2008-10-17" - Id = "destination_rds_failed_snapshot_copy" - Statement = [ - { - Sid = "sns-permissions" - Effect = "Allow" - Principal = { - AWS = "*" - } - Action = [ - "SNS:GetTopicAttributes", - "SNS:SetTopicAttributes", - "SNS:AddPermission", - "SNS:RemovePermission", - "SNS:DeleteTopic", - "SNS:Subscribe", - "SNS:ListSubscriptionsByTopic", - "SNS:Publish", - "SNS:Receive" - ] - Resource = "*" - Condition = { - test = "StringEquals" - variable = "AWS:SourceOwner" - values = [data.aws_caller_identity.current.account_id] - } - } - ] - }) -} - -resource "aws_cloudwatch_metric_alarm" "alarmcw_copy_failed_dest" { - - alarm_name = "failed-rds-copy" - comparison_operator = "GreaterThanOrEqualToThreshold" - evaluation_periods = "1" - metric_name = "ExecutionsFailed" - namespace = "AWS/States" - period = "300" - statistic = "Sum" - threshold = "1.0" - - dimensions = { - StateMachineArn = aws_sfn_state_machine.statemachine_copy_old_snapshots_dest_rds.arn - } - - alarm_description = "This metric monitors state machine failure for copying snapshots" - alarm_actions = [ - aws_sns_topic.copy_failed_dest.id - ] -} - -resource "aws_cloudwatch_metric_alarm" "alarmcw_delete_old_failed_dest" { - count = local.DeleteOld ? 1 : 0 - alarm_name = "failed-rds-delete-old-snapshot" - comparison_operator = "GreaterThanOrEqualToThreshold" - evaluation_periods = "2" - metric_name = "ExecutionsFailed" - namespace = "AWS/States" - period = "3600" - statistic = "Sum" - threshold = "2.0" - - dimensions = { - StateMachineArn = aws_sfn_state_machine.statemachine_delete_old_snapshots_dest_rds[*].arn - } - - alarm_description = "This metric monitors state machine failure for deleting old snapshots" - alarm_actions = [ - aws_sns_topic.delete_old_failed_dest.id - ] -} - -resource "aws_cloudwatch_event_rule" "copy_snapshots_rds" { - name = "capture-snapshot-copy-cw-events" - description = "Capture all CW state change events and triggers RDS copy state machine in destination account" - schedule_expression = "cron(30 * * * ? *)" - is_enabled = true - - - event_pattern = < Date: Wed, 8 Feb 2023 17:08:05 +0000 Subject: [PATCH 29/48] split out source sns docs --- lambda/Makefile | 4 +- .../modules/rds_destination/cloudwatch.tf | 97 +++++++++++++++++++ terraform/modules/rds_destination/sns.tf | 57 +++++++++++ terraform/modules/rds_source/resource.tf | 50 ---------- terraform/modules/rds_source/sns.tf | 59 +++++++++++ 5 files changed, 215 insertions(+), 52 deletions(-) create mode 100644 terraform/modules/rds_destination/cloudwatch.tf create mode 100644 terraform/modules/rds_destination/sns.tf create mode 100644 terraform/modules/rds_source/sns.tf diff --git a/lambda/Makefile b/lambda/Makefile index f284746..48367f5 100644 --- a/lambda/Makefile +++ b/lambda/Makefile @@ -16,11 +16,11 @@ # Override S3 destination by changing this variable or setting it in the # environment -S3DEST?=[YOUR BUCKET HERE] +S3DEST?=code-bucket-816708952522 # Set these if, for example, you use profiles on the AWS command line # or if your 'aws' executable is in a weird place. -AWSARGS=--region [YOUR REGION] --profile [YOUR PROFILE, or 'default', or remove this] +AWSARGS=--region us-east-1 --profile default AWSCMD=aws ZIPCMD=zip diff --git a/terraform/modules/rds_destination/cloudwatch.tf b/terraform/modules/rds_destination/cloudwatch.tf new file mode 100644 index 0000000..7298a79 --- /dev/null +++ b/terraform/modules/rds_destination/cloudwatch.tf @@ -0,0 +1,97 @@ +resource "aws_cloudwatch_metric_alarm" "alarmcw_copy_failed_dest" { + + alarm_name = "failed-rds-copy" + comparison_operator = "GreaterThanOrEqualToThreshold" + evaluation_periods = "1" + metric_name = "ExecutionsFailed" + namespace = "AWS/States" + period = "300" + statistic = "Sum" + threshold = "1.0" + + dimensions = { + StateMachineArn = aws_sfn_state_machine.statemachine_copy_old_snapshots_dest_rds.arn + } + + alarm_description = "This metric monitors state machine failure for copying snapshots" + alarm_actions = [ + aws_sns_topic.copy_failed_dest.id + ] +} + +resource "aws_cloudwatch_metric_alarm" "alarmcw_delete_old_failed_dest" { + count = local.DeleteOld ? 1 : 0 + alarm_name = "failed-rds-delete-old-snapshot" + comparison_operator = "GreaterThanOrEqualToThreshold" + evaluation_periods = "2" + metric_name = "ExecutionsFailed" + namespace = "AWS/States" + period = "3600" + statistic = "Sum" + threshold = "2.0" + + dimensions = { + StateMachineArn = aws_sfn_state_machine.statemachine_delete_old_snapshots_dest_rds[*].arn + } + + alarm_description = "This metric monitors state machine failure for deleting old snapshots" + alarm_actions = [ + aws_sns_topic.delete_old_failed_dest.id + ] +} + +resource "aws_cloudwatch_event_rule" "copy_snapshots_rds" { + name = "capture-snapshot-copy-cw-events" + description = "Capture all CW state change events and triggers RDS copy state machine in destination account" + schedule_expression = "cron(30 * * * ? *)" + is_enabled = true + + + event_pattern = < Date: Wed, 8 Feb 2023 17:19:47 +0000 Subject: [PATCH 30/48] split raw conversion into separate files --- lambda/.DS_Store | Bin 0 -> 10244 bytes lambda/copy_snapshots_dest_rds/.DS_Store | Bin 0 -> 6148 bytes .../snapshots_tool_utils.py | 364 ++++++++++++++++++ .../copy_snapshots_no_x_account_rds/.DS_Store | Bin 0 -> 6148 bytes .../snapshots_tool_utils.py | 364 ++++++++++++++++++ .../delete_old_snapshots_dest_rds/.DS_Store | Bin 0 -> 6148 bytes .../snapshots_tool_utils.py | 364 ++++++++++++++++++ .../.DS_Store | Bin 0 -> 6148 bytes .../snapshots_tool_utils.py | 364 ++++++++++++++++++ lambda/delete_old_snapshots_rds/.DS_Store | Bin 0 -> 6148 bytes .../snapshots_tool_utils.py | 364 ++++++++++++++++++ lambda/packaged_lambdas/.DS_Store | Bin 0 -> 6148 bytes lambda/share_snapshots_rds/.DS_Store | Bin 0 -> 6148 bytes .../snapshots_tool_utils.py | 364 ++++++++++++++++++ lambda/take_snapshots_rds/.DS_Store | Bin 0 -> 6148 bytes .../snapshots_tool_utils.py | 364 ++++++++++++++++++ terraform/modules/rds_source/resource.tf | 331 ---------------- 17 files changed, 2548 insertions(+), 331 deletions(-) create mode 100644 lambda/.DS_Store create mode 100644 lambda/copy_snapshots_dest_rds/.DS_Store create mode 100644 lambda/copy_snapshots_dest_rds/snapshots_tool_utils.py create mode 100644 lambda/copy_snapshots_no_x_account_rds/.DS_Store create mode 100644 lambda/copy_snapshots_no_x_account_rds/snapshots_tool_utils.py create mode 100644 lambda/delete_old_snapshots_dest_rds/.DS_Store create mode 100644 lambda/delete_old_snapshots_dest_rds/snapshots_tool_utils.py create mode 100644 lambda/delete_old_snapshots_no_x_account_rds/.DS_Store create mode 100644 lambda/delete_old_snapshots_no_x_account_rds/snapshots_tool_utils.py create mode 100644 lambda/delete_old_snapshots_rds/.DS_Store create mode 100644 lambda/delete_old_snapshots_rds/snapshots_tool_utils.py create mode 100644 lambda/packaged_lambdas/.DS_Store create mode 100644 lambda/share_snapshots_rds/.DS_Store create mode 100644 lambda/share_snapshots_rds/snapshots_tool_utils.py create mode 100644 lambda/take_snapshots_rds/.DS_Store create mode 100644 lambda/take_snapshots_rds/snapshots_tool_utils.py delete mode 100644 terraform/modules/rds_source/resource.tf diff --git a/lambda/.DS_Store b/lambda/.DS_Store new file mode 100644 index 0000000000000000000000000000000000000000..c2defc924c9df9d209960bb19b8840bb2b4b1519 GIT binary patch literal 10244 zcmeHM&2G~`5T0!VbybieepG~1u!N8}l#)Uf32_M_95`?x2qb=F>?UoZI(8I04Je9~ zJMaqd2;4by;sJOUBqaD|*J)EXRcunBV(m)1v$1!^+3)l2&Wwpjl)JTcB9n*=RF?TO zXeJcC&-GB5i>_RO6woKKC?Jn)YD4Ox5zq)|1T+E~0gb>(L;zUqE-J@Xh720g~Pw1Zu6i<;O4>j!;=IX>ma&!arN4L-+q&~8JE zcTE@l7CGqKaUVnMLjUuak1*sX91EUjC|7wvYaBHkTR zrA8nr0`qF1$@%|>*#3|B{-4yG9;Xq|2>gc#h+L&wDdUfi(p&L4aIW1%eS^w{#|@Q~ yDrjUn9#W>`@q^RxH&A(Ofh8C8OOYHZi7P1o{A0k;c^V!6A9r3<$NwiXcl`%4dM*6` literal 0 HcmV?d00001 diff --git a/lambda/copy_snapshots_dest_rds/.DS_Store b/lambda/copy_snapshots_dest_rds/.DS_Store new file mode 100644 index 0000000000000000000000000000000000000000..16bdc9699ea620373d61b248dfd50765062681a2 GIT binary patch literal 6148 zcmeHKyH3ME5S)V)k!T_skZA4;g7^ayg@T5L4=}bvL?jbO#3`WikMV6VyL$nLq(%bm zN_Vr_yPaE)lzRXOlg;c37y%fv2?lWvn1l}w9i_>~#Go8G)>xvziugB7YQI2*hBd#U zU;h(hEx#kj43ET?-V6OT#(2X5tJ}J%%DO4ZAo3M#cOPo>fK$|rJ)>k!!Q44E)TScO zg3L>@aK#@S*UTIDIL^4*R?qKxTIM%o&&eokwbr!H_D;6T9-CC<<_fq1u7E4x3h>Mp z8=e???+UmAuE3W9vOh#N!7O2Cs9Og+eF6{%46D(X%dq6cgjvGQkR!BUsYFW+>4_06 zo$;9FWeGb&OGikLkC0a0K3;;ZPX4IV5i&#XT>)2MR{?qZ(nfOr-|#PU@R6@mymJLy zf&Z#N7) 0: + return max(timestamps) + + else: + return None + + + +def requires_backup(backup_interval, instance, filtered_snapshots): +# Returns True if latest snapshot is older than INTERVAL + latest = get_latest_snapshot_ts(instance['DBInstanceIdentifier'], filtered_snapshots) + + if latest is not None: + backup_age = datetime.now() - latest + + if backup_age.total_seconds() >= (backup_interval * 60 * 60): + return True + + else: + return False + + elif latest is None: + return True + + +def paginate_api_call(client, api_call, objecttype, *args, **kwargs): +#Takes an RDS boto client and paginates through api_call calls and returns a list of objects of objecttype + response = {} + response[objecttype] = [] + + # Create a paginator + paginator = client.get_paginator(api_call) + + # Create a PageIterator from the Paginator + page_iterator = paginator.paginate(**kwargs) + for page in page_iterator: + for item in page[objecttype]: + response[objecttype].append(item) + + return response + + +def copy_local(snapshot_identifier, snapshot_object): + client = boto3.client('rds', region_name=_REGION) + + tags = [{ + 'Key': 'CopiedBy', + 'Value': 'Snapshot Tool for RDS' + }] + + if snapshot_object['Encrypted']: + logger.info('Copying encrypted snapshot %s locally' % snapshot_identifier) + response = client.copy_db_snapshot( + SourceDBSnapshotIdentifier = snapshot_object['Arn'], + TargetDBSnapshotIdentifier = snapshot_identifier, + KmsKeyId = _KMS_KEY_SOURCE_REGION, + Tags = tags) + + else: + logger.info('Copying snapshot %s locally' %snapshot_identifier) + response = client.copy_db_snapshot( + SourceDBSnapshotIdentifier = snapshot_object['Arn'], + TargetDBSnapshotIdentifier = snapshot_identifier, + Tags = tags) + + return response + + + +def copy_remote(snapshot_identifier, snapshot_object): + client = boto3.client('rds', region_name=_DESTINATION_REGION) + + if snapshot_object['Encrypted']: + logger.info('Copying encrypted snapshot %s to remote region %s' % (snapshot_object['Arn'], _DESTINATION_REGION)) + response = client.copy_db_snapshot( + SourceDBSnapshotIdentifier = snapshot_object['Arn'], + TargetDBSnapshotIdentifier = snapshot_identifier, + KmsKeyId = _KMS_KEY_DEST_REGION, + SourceRegion = _REGION, + CopyTags = True) + + else: + logger.info('Copying snapshot %s to remote region %s' % (snapshot_object['Arn'], _DESTINATION_REGION)) + response = client.copy_db_snapshot( + SourceDBSnapshotIdentifier = snapshot_object['Arn'], + TargetDBSnapshotIdentifier = snapshot_identifier, + SourceRegion = _REGION, + CopyTags = True) + + return response diff --git a/lambda/copy_snapshots_no_x_account_rds/.DS_Store b/lambda/copy_snapshots_no_x_account_rds/.DS_Store new file mode 100644 index 0000000000000000000000000000000000000000..a36d3b6abec65c41b11d47b0b71a19f498362d64 GIT binary patch literal 6148 zcmeH~u}T9$5Qb+pMZhN5Xt`iz6Yver5DN<{A0Xxu#K4&gnt)o}!}wl4iheV@>K#cf zs57whZ+2(?oBbiTHzE>7oB5f@Kt%cs#h{ZNrpr6G_Hyt(7>r{gkFu1htia!4TKkF2 zq^9O)Y1eOqV zMgF~%B{1Y^NtUqssf=mz8;S_UA|kl&Z&28WZW>+G_KJJ#=P?pzz= 0: + return max(timestamps) + + else: + return None + + + +def requires_backup(backup_interval, instance, filtered_snapshots): +# Returns True if latest snapshot is older than INTERVAL + latest = get_latest_snapshot_ts(instance['DBInstanceIdentifier'], filtered_snapshots) + + if latest is not None: + backup_age = datetime.now() - latest + + if backup_age.total_seconds() >= (backup_interval * 60 * 60): + return True + + else: + return False + + elif latest is None: + return True + + +def paginate_api_call(client, api_call, objecttype, *args, **kwargs): +#Takes an RDS boto client and paginates through api_call calls and returns a list of objects of objecttype + response = {} + response[objecttype] = [] + + # Create a paginator + paginator = client.get_paginator(api_call) + + # Create a PageIterator from the Paginator + page_iterator = paginator.paginate(**kwargs) + for page in page_iterator: + for item in page[objecttype]: + response[objecttype].append(item) + + return response + + +def copy_local(snapshot_identifier, snapshot_object): + client = boto3.client('rds', region_name=_REGION) + + tags = [{ + 'Key': 'CopiedBy', + 'Value': 'Snapshot Tool for RDS' + }] + + if snapshot_object['Encrypted']: + logger.info('Copying encrypted snapshot %s locally' % snapshot_identifier) + response = client.copy_db_snapshot( + SourceDBSnapshotIdentifier = snapshot_object['Arn'], + TargetDBSnapshotIdentifier = snapshot_identifier, + KmsKeyId = _KMS_KEY_SOURCE_REGION, + Tags = tags) + + else: + logger.info('Copying snapshot %s locally' %snapshot_identifier) + response = client.copy_db_snapshot( + SourceDBSnapshotIdentifier = snapshot_object['Arn'], + TargetDBSnapshotIdentifier = snapshot_identifier, + Tags = tags) + + return response + + + +def copy_remote(snapshot_identifier, snapshot_object): + client = boto3.client('rds', region_name=_DESTINATION_REGION) + + if snapshot_object['Encrypted']: + logger.info('Copying encrypted snapshot %s to remote region %s' % (snapshot_object['Arn'], _DESTINATION_REGION)) + response = client.copy_db_snapshot( + SourceDBSnapshotIdentifier = snapshot_object['Arn'], + TargetDBSnapshotIdentifier = snapshot_identifier, + KmsKeyId = _KMS_KEY_DEST_REGION, + SourceRegion = _REGION, + CopyTags = True) + + else: + logger.info('Copying snapshot %s to remote region %s' % (snapshot_object['Arn'], _DESTINATION_REGION)) + response = client.copy_db_snapshot( + SourceDBSnapshotIdentifier = snapshot_object['Arn'], + TargetDBSnapshotIdentifier = snapshot_identifier, + SourceRegion = _REGION, + CopyTags = True) + + return response diff --git a/lambda/delete_old_snapshots_dest_rds/.DS_Store b/lambda/delete_old_snapshots_dest_rds/.DS_Store new file mode 100644 index 0000000000000000000000000000000000000000..7c182899b73d888dd9cbac613a057e96cc48a6dc GIT binary patch literal 6148 zcmeH~&u$Yj5XL`4kScnqdVvel@&L`MZ?LU`5E9~2!F9=|OQ}e+QL{lRr+p(H5q@L4 zsA!AAlxtm5UTLf!I@j{uyqh)i zuECofB|k~yUr{JM$Oy~6dB0xumw5Z86J0U3&Gtg2KE)GfO}VAD)P!00q*5QLx6IU9 zXGAf@w&iygdneeE6GI9eOHsbx6EFX-g z51FT39T9=WeXj3Bo!5$S_BUo->kIKN(CpFdD2-Y5t6OV&gx$?GeX+%g{XedGsqtvu z=4jpFExfwrjjGEdGr*l~va_4eb7#OAa0dP`VCO?YGORKV3FFa07M}p%h~_Ai?E}DU zii}mpAt60TNTy&i4f%-?k~!@~)~hlO36r@(etd*Ha>y@~5J$&<5!Dr{gq}MC&cG%E zPg8Zz?*BJ`fB$b9dCeJc2L3As!f0|bIVPlhw{8VzcWpuXOfoUOAz?Hj%hz!%*j4 0: + return max(timestamps) + + else: + return None + + + +def requires_backup(backup_interval, instance, filtered_snapshots): +# Returns True if latest snapshot is older than INTERVAL + latest = get_latest_snapshot_ts(instance['DBInstanceIdentifier'], filtered_snapshots) + + if latest is not None: + backup_age = datetime.now() - latest + + if backup_age.total_seconds() >= (backup_interval * 60 * 60): + return True + + else: + return False + + elif latest is None: + return True + + +def paginate_api_call(client, api_call, objecttype, *args, **kwargs): +#Takes an RDS boto client and paginates through api_call calls and returns a list of objects of objecttype + response = {} + response[objecttype] = [] + + # Create a paginator + paginator = client.get_paginator(api_call) + + # Create a PageIterator from the Paginator + page_iterator = paginator.paginate(**kwargs) + for page in page_iterator: + for item in page[objecttype]: + response[objecttype].append(item) + + return response + + +def copy_local(snapshot_identifier, snapshot_object): + client = boto3.client('rds', region_name=_REGION) + + tags = [{ + 'Key': 'CopiedBy', + 'Value': 'Snapshot Tool for RDS' + }] + + if snapshot_object['Encrypted']: + logger.info('Copying encrypted snapshot %s locally' % snapshot_identifier) + response = client.copy_db_snapshot( + SourceDBSnapshotIdentifier = snapshot_object['Arn'], + TargetDBSnapshotIdentifier = snapshot_identifier, + KmsKeyId = _KMS_KEY_SOURCE_REGION, + Tags = tags) + + else: + logger.info('Copying snapshot %s locally' %snapshot_identifier) + response = client.copy_db_snapshot( + SourceDBSnapshotIdentifier = snapshot_object['Arn'], + TargetDBSnapshotIdentifier = snapshot_identifier, + Tags = tags) + + return response + + + +def copy_remote(snapshot_identifier, snapshot_object): + client = boto3.client('rds', region_name=_DESTINATION_REGION) + + if snapshot_object['Encrypted']: + logger.info('Copying encrypted snapshot %s to remote region %s' % (snapshot_object['Arn'], _DESTINATION_REGION)) + response = client.copy_db_snapshot( + SourceDBSnapshotIdentifier = snapshot_object['Arn'], + TargetDBSnapshotIdentifier = snapshot_identifier, + KmsKeyId = _KMS_KEY_DEST_REGION, + SourceRegion = _REGION, + CopyTags = True) + + else: + logger.info('Copying snapshot %s to remote region %s' % (snapshot_object['Arn'], _DESTINATION_REGION)) + response = client.copy_db_snapshot( + SourceDBSnapshotIdentifier = snapshot_object['Arn'], + TargetDBSnapshotIdentifier = snapshot_identifier, + SourceRegion = _REGION, + CopyTags = True) + + return response diff --git a/lambda/delete_old_snapshots_no_x_account_rds/.DS_Store b/lambda/delete_old_snapshots_no_x_account_rds/.DS_Store new file mode 100644 index 0000000000000000000000000000000000000000..0d3e1147999aebcad7ac8ad0fcef8525cb299260 GIT binary patch literal 6148 zcmeHKu};G<5PdEkidZTZSdhH3RN@apRR(50AS7ujDupJZsRT^<16KYP-kq(gG-X8y z?y9@zUVP8aa}>t_+;lU)0>%JFbiu(f!vT});yoM0%n{KkM~f0EYE+o_dRyW*Dj;ij z$<->kW+islFU1-wWN3&FT+^1AGE*}?(=!zO)z~oC^8XWA)|lWGOEkAtUF21rlU*LI zclz6h8e70AN6YYtoHZG17s^tQXGz8tSsJ|Taa^-*vcqw9aDFZ8<@rm=C~TF^smJCn z_Sj@rW~P8CU<&*f1-NI6jZOkuGzClnQ(&urydOMWFp1a&)K>?C9s!6$rq$S%4?d~! z5tE2rKwhD_P)ZD?E*>#lC}-Zseo4eGU?_)+hYuG=cJYMbbae82?GBd&v}g*L0$&x7 zvmb3N`~S`N=YQ>F!4xnB{-y$MIJ=uod2exV?cJs9wVr-Y7n6Qnz- 0: + return max(timestamps) + + else: + return None + + + +def requires_backup(backup_interval, instance, filtered_snapshots): +# Returns True if latest snapshot is older than INTERVAL + latest = get_latest_snapshot_ts(instance['DBInstanceIdentifier'], filtered_snapshots) + + if latest is not None: + backup_age = datetime.now() - latest + + if backup_age.total_seconds() >= (backup_interval * 60 * 60): + return True + + else: + return False + + elif latest is None: + return True + + +def paginate_api_call(client, api_call, objecttype, *args, **kwargs): +#Takes an RDS boto client and paginates through api_call calls and returns a list of objects of objecttype + response = {} + response[objecttype] = [] + + # Create a paginator + paginator = client.get_paginator(api_call) + + # Create a PageIterator from the Paginator + page_iterator = paginator.paginate(**kwargs) + for page in page_iterator: + for item in page[objecttype]: + response[objecttype].append(item) + + return response + + +def copy_local(snapshot_identifier, snapshot_object): + client = boto3.client('rds', region_name=_REGION) + + tags = [{ + 'Key': 'CopiedBy', + 'Value': 'Snapshot Tool for RDS' + }] + + if snapshot_object['Encrypted']: + logger.info('Copying encrypted snapshot %s locally' % snapshot_identifier) + response = client.copy_db_snapshot( + SourceDBSnapshotIdentifier = snapshot_object['Arn'], + TargetDBSnapshotIdentifier = snapshot_identifier, + KmsKeyId = _KMS_KEY_SOURCE_REGION, + Tags = tags) + + else: + logger.info('Copying snapshot %s locally' %snapshot_identifier) + response = client.copy_db_snapshot( + SourceDBSnapshotIdentifier = snapshot_object['Arn'], + TargetDBSnapshotIdentifier = snapshot_identifier, + Tags = tags) + + return response + + + +def copy_remote(snapshot_identifier, snapshot_object): + client = boto3.client('rds', region_name=_DESTINATION_REGION) + + if snapshot_object['Encrypted']: + logger.info('Copying encrypted snapshot %s to remote region %s' % (snapshot_object['Arn'], _DESTINATION_REGION)) + response = client.copy_db_snapshot( + SourceDBSnapshotIdentifier = snapshot_object['Arn'], + TargetDBSnapshotIdentifier = snapshot_identifier, + KmsKeyId = _KMS_KEY_DEST_REGION, + SourceRegion = _REGION, + CopyTags = True) + + else: + logger.info('Copying snapshot %s to remote region %s' % (snapshot_object['Arn'], _DESTINATION_REGION)) + response = client.copy_db_snapshot( + SourceDBSnapshotIdentifier = snapshot_object['Arn'], + TargetDBSnapshotIdentifier = snapshot_identifier, + SourceRegion = _REGION, + CopyTags = True) + + return response diff --git a/lambda/delete_old_snapshots_rds/.DS_Store b/lambda/delete_old_snapshots_rds/.DS_Store new file mode 100644 index 0000000000000000000000000000000000000000..d86f46b28905aefde1ac0d8df1e1887cf410812a GIT binary patch literal 6148 zcmeHK!AiqG5PhRP6ucBKdQiw06#M`omU{5w#Se%{8?g|ZVAF%1^5^9hjH5+~3^gh&2Z;XSlGc8~YD?B! zVz>Sb8?2F|AwKg=EiuC@V>g)M9V;|bJ(}-O_dv(xR3qTxk*^G1TB{V0Am`3aZa)%}%l^9Ybwipu9 zxt_$hG-4Mpq(fr!A@Rvy&5PL6$)8v_Bn{}^6>tUm3TW?3o9O+2%fHNElW!sJTme_$ zzbYW3`NMq1)5W`W@0@zqCLE6(Y8uz+gvNRABY;2jJ#ucFVL#D1 0: + return max(timestamps) + + else: + return None + + + +def requires_backup(backup_interval, instance, filtered_snapshots): +# Returns True if latest snapshot is older than INTERVAL + latest = get_latest_snapshot_ts(instance['DBInstanceIdentifier'], filtered_snapshots) + + if latest is not None: + backup_age = datetime.now() - latest + + if backup_age.total_seconds() >= (backup_interval * 60 * 60): + return True + + else: + return False + + elif latest is None: + return True + + +def paginate_api_call(client, api_call, objecttype, *args, **kwargs): +#Takes an RDS boto client and paginates through api_call calls and returns a list of objects of objecttype + response = {} + response[objecttype] = [] + + # Create a paginator + paginator = client.get_paginator(api_call) + + # Create a PageIterator from the Paginator + page_iterator = paginator.paginate(**kwargs) + for page in page_iterator: + for item in page[objecttype]: + response[objecttype].append(item) + + return response + + +def copy_local(snapshot_identifier, snapshot_object): + client = boto3.client('rds', region_name=_REGION) + + tags = [{ + 'Key': 'CopiedBy', + 'Value': 'Snapshot Tool for RDS' + }] + + if snapshot_object['Encrypted']: + logger.info('Copying encrypted snapshot %s locally' % snapshot_identifier) + response = client.copy_db_snapshot( + SourceDBSnapshotIdentifier = snapshot_object['Arn'], + TargetDBSnapshotIdentifier = snapshot_identifier, + KmsKeyId = _KMS_KEY_SOURCE_REGION, + Tags = tags) + + else: + logger.info('Copying snapshot %s locally' %snapshot_identifier) + response = client.copy_db_snapshot( + SourceDBSnapshotIdentifier = snapshot_object['Arn'], + TargetDBSnapshotIdentifier = snapshot_identifier, + Tags = tags) + + return response + + + +def copy_remote(snapshot_identifier, snapshot_object): + client = boto3.client('rds', region_name=_DESTINATION_REGION) + + if snapshot_object['Encrypted']: + logger.info('Copying encrypted snapshot %s to remote region %s' % (snapshot_object['Arn'], _DESTINATION_REGION)) + response = client.copy_db_snapshot( + SourceDBSnapshotIdentifier = snapshot_object['Arn'], + TargetDBSnapshotIdentifier = snapshot_identifier, + KmsKeyId = _KMS_KEY_DEST_REGION, + SourceRegion = _REGION, + CopyTags = True) + + else: + logger.info('Copying snapshot %s to remote region %s' % (snapshot_object['Arn'], _DESTINATION_REGION)) + response = client.copy_db_snapshot( + SourceDBSnapshotIdentifier = snapshot_object['Arn'], + TargetDBSnapshotIdentifier = snapshot_identifier, + SourceRegion = _REGION, + CopyTags = True) + + return response diff --git a/lambda/packaged_lambdas/.DS_Store b/lambda/packaged_lambdas/.DS_Store new file mode 100644 index 0000000000000000000000000000000000000000..5a8b4865864d74f475430a9f697c5de3cc364c6c GIT binary patch literal 6148 zcmeHKO-sW-5PhpX6np8>V?26jga05c5fQ{o(Q8cFpvBshCK2_Nf8As=kM0G z#u%jy@vqRNgjz%yH<-EpX?GN24710M{? z{t!?FBae+kyLB+JPXMA^Bdjnl7XYydJVqWHhisu3XG(OY#y>HPGiQC^`$Zlbht3?v zKR%52Z2SwwSnnJ^u-#!Ihngw_%D_4U`>xoP^Z)eg_y4+)mXrZy;9oIdTJb0zG9-Vt xZUiT1Z9u)Dib%i4p*3ORk7JvWqxg_&g>8Xkh>^#}A!{gl5wJ99q73{g1MgH)k5K>s literal 0 HcmV?d00001 diff --git a/lambda/share_snapshots_rds/.DS_Store b/lambda/share_snapshots_rds/.DS_Store new file mode 100644 index 0000000000000000000000000000000000000000..7c049ee9ed5581ce55106dc8105ea0963cf1ed64 GIT binary patch literal 6148 zcmeHKyG{c!5S)b+k!T_cp!636(NLQx6jT&^K*%LQARSzkOA4s&&tZ1$pd5EKLTFdA z$64>rtw)L<0A#XTUI8NjLpDXxL651scj_ny9}~qmmY8FPC+zUnL-ZG2TKg$V*8PeC zhxIS8#S1E~WQ;Z5@5*XXlvTla73+p={AtD!aE=(|ph|<6_1pp|Yso(usZ%FhRU? I1%5$+FU15zFaQ7m literal 0 HcmV?d00001 diff --git a/lambda/share_snapshots_rds/snapshots_tool_utils.py b/lambda/share_snapshots_rds/snapshots_tool_utils.py new file mode 100644 index 0000000..5a797d9 --- /dev/null +++ b/lambda/share_snapshots_rds/snapshots_tool_utils.py @@ -0,0 +1,364 @@ +''' +Copyright 2017 Amazon.com, Inc. or its affiliates. All Rights Reserved. + +Licensed under the Apache License, Version 2.0 (the "License"). You may not use this file except in compliance with the License. A copy of the License is located at + + http://aws.amazon.com/apache2.0/ + +or in the "license" file accompanying this file. This file is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. +''' + + +# snapshots_tool_utils +# Support module for the Snapshot Tool for RDS + +import boto3 +from datetime import datetime, timedelta +import os +import logging +import re + + +# Initialize everything +_LOGLEVEL = os.getenv('LOG_LEVEL', 'ERROR').strip() + +_DESTINATION_REGION = os.getenv( + 'DEST_REGION', os.getenv('AWS_DEFAULT_REGION')).strip() + +_KMS_KEY_DEST_REGION = os.getenv('KMS_KEY_DEST_REGION', 'None').strip() + +_KMS_KEY_SOURCE_REGION = os.getenv('KMS_KEY_SOURCE_REGION', 'None').strip() + +_TIMESTAMP_FORMAT = '%Y-%m-%d-%H-%M' + +if os.getenv('REGION_OVERRIDE', 'NO') != 'NO': + _REGION = os.getenv('REGION_OVERRIDE').strip() +else: + _REGION = os.getenv('AWS_DEFAULT_REGION') + +_SUPPORTED_ENGINES = [ 'mariadb', 'sqlserver-se', 'sqlserver-ee', 'sqlserver-ex', 'sqlserver-web', 'mysql', 'oracle-se', 'oracle-se1', 'oracle-se2', 'oracle-ee', 'postgres' ] + + +logger = logging.getLogger() +logger.setLevel(_LOGLEVEL.upper()) + + +class SnapshotToolException(Exception): + pass + + +def search_tag_copydbsnapshot(response): +# Takes a list_tags_for_resource response and searches for our CopyDBSnapshot tag + try: + + for tag in response['TagList']: + if tag['Key'] == 'CopyDBSnapshot' and tag['Value'] == 'True': return True + + except Exception: return False + + else: return False + + + +def search_tag_created(response): +# Takes a describe_db_snapshots response and searches for our CreatedBy tag + try: + + for tag in response['TagList']: + if tag['Key'] == 'CreatedBy' and tag['Value'] == 'Snapshot Tool for RDS': return True + + except Exception: return False + + else: return False + + + +def search_tag_shared(response): +# Takes a describe_db_snapshots response and searches for our shareAndCopy tag + try: + for tag in response['TagList']: + if tag['Key'] == 'shareAndCopy' and tag['Value'] == 'YES': + for tag2 in response['TagList']: + if tag2['Key'] == 'CreatedBy' and tag2['Value'] == 'Snapshot Tool for RDS': + return True + + except Exception: + return False + + return False + + + +def search_tag_copied(response): +# Search for a tag indicating we copied this snapshot + try: + for tag in response['TagList']: + if tag['Key'] == 'CopiedBy' and tag['Value'] == 'Snapshot Tool for RDS': + return True + + except Exception: + return False + + return False + +def get_own_snapshots_no_x_account(pattern, response, REGION): + # Filters our own snapshots + filtered = {} + for snapshot in response['DBSnapshots']: + + if snapshot['SnapshotType'] == 'manual' and re.search(pattern, snapshot['DBInstanceIdentifier']) and snapshot['Engine'] in _SUPPORTED_ENGINES: + client = boto3.client('rds', region_name=REGION) + response_tags = client.list_tags_for_resource( + ResourceName=snapshot['DBSnapshotArn']) + + if search_tag_created(response_tags): + filtered[snapshot['DBSnapshotIdentifier']] = { + 'Arn': snapshot['DBSnapshotArn'], 'Status': snapshot['Status'], 'DBInstanceIdentifier': snapshot['DBInstanceIdentifier']} + #Changed the next line to search for ALL_CLUSTERS or ALL_SNAPSHOTS so it will work with no-x-account + elif snapshot['SnapshotType'] == 'manual' and pattern == 'ALL_SNAPSHOTS' and snapshot['Engine'] in _SUPPORTED_ENGINES: + client = boto3.client('rds', region_name=REGION) + response_tags = client.list_tags_for_resource( + ResourceName=snapshot['DBSnapshotArn']) + + if search_tag_created(response_tags): + filtered[snapshot['DBSnapshotIdentifier']] = { + 'Arn': snapshot['DBSnapshotArn'], 'Status': snapshot['Status'], 'DBInstanceIdentifier': snapshot['DBInstanceIdentifier']} + + return filtered + + +def get_shared_snapshots(pattern, response): +# Returns a dict with only shared snapshots filtered by pattern, with DBSnapshotIdentifier as key and the response as attribute + filtered = {} + for snapshot in response['DBSnapshots']: + if snapshot['SnapshotType'] == 'shared' and re.search(pattern, snapshot['DBInstanceIdentifier']) and snapshot['Engine'] in _SUPPORTED_ENGINES: + filtered[get_snapshot_identifier(snapshot)] = { + 'Arn': snapshot['DBSnapshotIdentifier'], 'Encrypted': snapshot['Encrypted'], 'DBInstanceIdentifier': snapshot['DBInstanceIdentifier']} + if snapshot['Encrypted'] is True: + filtered[get_snapshot_identifier(snapshot)]['KmsKeyId'] = snapshot['KmsKeyId'] + + elif snapshot['SnapshotType'] == 'shared' and pattern == 'ALL_SNAPSHOTS' and snapshot['Engine'] in _SUPPORTED_ENGINES: + filtered[get_snapshot_identifier(snapshot)] = { + 'Arn': snapshot['DBSnapshotIdentifier'], 'Encrypted': snapshot['Encrypted'], 'DBInstanceIdentifier': snapshot['DBInstanceIdentifier']} + if snapshot['Encrypted'] is True: + filtered[get_snapshot_identifier(snapshot)]['KmsKeyId'] = snapshot['KmsKeyId'] + return filtered + + + +def get_snapshot_identifier(snapshot): +# Function that will return the RDS Snapshot identifier given an ARN + match = re.match('arn:aws:rds:.*:.*:snapshot:(.+)', + snapshot['DBSnapshotArn']) + return match.group(1) + + +def get_own_snapshots_dest(pattern, response): +# Returns a dict with local snapshots, filtered by pattern, with DBSnapshotIdentifier as key and Arn, Status as attributes + filtered = {} + for snapshot in response['DBSnapshots']: + + if snapshot['SnapshotType'] == 'manual' and re.search(pattern, snapshot['DBInstanceIdentifier']) and snapshot['Engine'] in _SUPPORTED_ENGINES: + filtered[snapshot['DBSnapshotIdentifier']] = { + 'Arn': snapshot['DBSnapshotArn'], 'Status': snapshot['Status'], 'Encrypted': snapshot['Encrypted'], 'DBInstanceIdentifier': snapshot['DBInstanceIdentifier']} + + if snapshot['Encrypted'] is True: + filtered[snapshot['DBSnapshotIdentifier']]['KmsKeyId'] = snapshot['KmsKeyId'] + + elif snapshot['SnapshotType'] == 'manual' and pattern == 'ALL_SNAPSHOTS' and snapshot['Engine'] in _SUPPORTED_ENGINES: + filtered[snapshot['DBSnapshotIdentifier']] = { + 'Arn': snapshot['DBSnapshotArn'], 'Status': snapshot['Status'], 'Encrypted': snapshot['Encrypted'], 'DBInstanceIdentifier': snapshot['DBInstanceIdentifier'] } + + if snapshot['Encrypted'] is True: + filtered[snapshot['DBSnapshotIdentifier']]['KmsKeyId'] = snapshot['KmsKeyId'] + + return filtered + +def filter_instances(taggedinstance, pattern, instance_list): +# Takes the response from describe-db-instances and filters according to pattern in DBInstanceIdentifier + filtered_list = [] + + for instance in instance_list['DBInstances']: + + if taggedinstance == 'TRUE': + client = boto3.client('rds', region_name=_REGION) + response = client.list_tags_for_resource(ResourceName=instance['DBInstanceArn']) + + if pattern == 'ALL_INSTANCES' and instance['Engine'] in _SUPPORTED_ENGINES: + if (taggedinstance == 'TRUE' and search_tag_copydbsnapshot(response)) or taggedinstance == 'FALSE': + filtered_list.append(instance) + + else: + match = re.search(pattern, instance['DBInstanceIdentifier']) + + if match and instance['Engine'] in _SUPPORTED_ENGINES: + if (taggedinstance == 'TRUE' and search_tag_copydbsnapshot(response)) or taggedinstance == 'FALSE': + filtered_list.append(instance) + + return filtered_list + + +def get_own_snapshots_source(pattern, response, backup_interval=None): +# Filters our own snapshots + filtered = {} + + for snapshot in response['DBSnapshots']: + + # No need to consider snapshots that are still in progress + if 'SnapshotCreateTime' not in snapshot: + continue + + # No need to get tags for snapshots outside of the backup interval + if backup_interval and snapshot['SnapshotCreateTime'].replace(tzinfo=None) < datetime.utcnow().replace(tzinfo=None) - timedelta(hours=backup_interval): + continue + + if snapshot['SnapshotType'] == 'manual' and re.search(pattern, snapshot['DBInstanceIdentifier']) and snapshot['Engine'] in _SUPPORTED_ENGINES: + client = boto3.client('rds', region_name=_REGION) + response_tags = client.list_tags_for_resource( + ResourceName=snapshot['DBSnapshotArn']) + + if search_tag_created(response_tags): + filtered[snapshot['DBSnapshotIdentifier']] = { + 'Arn': snapshot['DBSnapshotArn'], 'Status': snapshot['Status'], 'DBInstanceIdentifier': snapshot['DBInstanceIdentifier']} + + elif snapshot['SnapshotType'] == 'manual' and (pattern == 'ALL_CLUSTERS' or pattern == 'ALL_SNAPSHOTS' or pattern == 'ALL_INSTANCES') and snapshot['Engine'] in _SUPPORTED_ENGINES: + client = boto3.client('rds', region_name=_REGION) + response_tags = client.list_tags_for_resource( + ResourceName=snapshot['DBSnapshotArn']) + + if search_tag_created(response_tags): + filtered[snapshot['DBSnapshotIdentifier']] = { + 'Arn': snapshot['DBSnapshotArn'], 'Status': snapshot['Status'], 'DBInstanceIdentifier': snapshot['DBInstanceIdentifier']} + + return filtered + + +def get_timestamp_no_minute(snapshot_identifier, snapshot_list): +# Get a timestamp from the name of a snapshot and strip out the minutes + pattern = '%s-(.+)-\d{2}' % snapshot_list[snapshot_identifier]['DBInstanceIdentifier'] + timestamp_format = '%Y-%m-%d-%H' + date_time = re.search(pattern, snapshot_identifier) + + if date_time is not None: + return datetime.strptime(date_time.group(1), timestamp_format) + + +def get_timestamp(snapshot_identifier, snapshot_list): +# Searches for a timestamp on a snapshot name + pattern = '%s-(.+)' % snapshot_list[snapshot_identifier]['DBInstanceIdentifier'] + date_time = re.search(pattern, snapshot_identifier) + + if date_time is not None: + + try: + return datetime.strptime(date_time.group(1), _TIMESTAMP_FORMAT) + + except Exception: + return None + + return None + + + +def get_latest_snapshot_ts(instance_identifier, filtered_snapshots): +# Get latest snapshot for a specific DBInstanceIdentifier + timestamps = [] + + for snapshot,snapshot_object in filtered_snapshots.items(): + + if snapshot_object['DBInstanceIdentifier'] == instance_identifier: + timestamp = get_timestamp_no_minute(snapshot, filtered_snapshots) + + if timestamp is not None: + timestamps.append(timestamp) + + if len(timestamps) > 0: + return max(timestamps) + + else: + return None + + + +def requires_backup(backup_interval, instance, filtered_snapshots): +# Returns True if latest snapshot is older than INTERVAL + latest = get_latest_snapshot_ts(instance['DBInstanceIdentifier'], filtered_snapshots) + + if latest is not None: + backup_age = datetime.now() - latest + + if backup_age.total_seconds() >= (backup_interval * 60 * 60): + return True + + else: + return False + + elif latest is None: + return True + + +def paginate_api_call(client, api_call, objecttype, *args, **kwargs): +#Takes an RDS boto client and paginates through api_call calls and returns a list of objects of objecttype + response = {} + response[objecttype] = [] + + # Create a paginator + paginator = client.get_paginator(api_call) + + # Create a PageIterator from the Paginator + page_iterator = paginator.paginate(**kwargs) + for page in page_iterator: + for item in page[objecttype]: + response[objecttype].append(item) + + return response + + +def copy_local(snapshot_identifier, snapshot_object): + client = boto3.client('rds', region_name=_REGION) + + tags = [{ + 'Key': 'CopiedBy', + 'Value': 'Snapshot Tool for RDS' + }] + + if snapshot_object['Encrypted']: + logger.info('Copying encrypted snapshot %s locally' % snapshot_identifier) + response = client.copy_db_snapshot( + SourceDBSnapshotIdentifier = snapshot_object['Arn'], + TargetDBSnapshotIdentifier = snapshot_identifier, + KmsKeyId = _KMS_KEY_SOURCE_REGION, + Tags = tags) + + else: + logger.info('Copying snapshot %s locally' %snapshot_identifier) + response = client.copy_db_snapshot( + SourceDBSnapshotIdentifier = snapshot_object['Arn'], + TargetDBSnapshotIdentifier = snapshot_identifier, + Tags = tags) + + return response + + + +def copy_remote(snapshot_identifier, snapshot_object): + client = boto3.client('rds', region_name=_DESTINATION_REGION) + + if snapshot_object['Encrypted']: + logger.info('Copying encrypted snapshot %s to remote region %s' % (snapshot_object['Arn'], _DESTINATION_REGION)) + response = client.copy_db_snapshot( + SourceDBSnapshotIdentifier = snapshot_object['Arn'], + TargetDBSnapshotIdentifier = snapshot_identifier, + KmsKeyId = _KMS_KEY_DEST_REGION, + SourceRegion = _REGION, + CopyTags = True) + + else: + logger.info('Copying snapshot %s to remote region %s' % (snapshot_object['Arn'], _DESTINATION_REGION)) + response = client.copy_db_snapshot( + SourceDBSnapshotIdentifier = snapshot_object['Arn'], + TargetDBSnapshotIdentifier = snapshot_identifier, + SourceRegion = _REGION, + CopyTags = True) + + return response diff --git a/lambda/take_snapshots_rds/.DS_Store b/lambda/take_snapshots_rds/.DS_Store new file mode 100644 index 0000000000000000000000000000000000000000..8973dbb88f3983d8a294cdb0d944e22ad8e6680b GIT binary patch literal 6148 zcmeHKJx{|x41ICOa6sj_?u5u-~iamUPD-Dh^Hm0sE5y5E_cv)1)aXWE^MGvEw31I~am@WlZ4Y?a|bMAy!M zGvEw-GN9{2;7}|Qb`fpqpt36faX_~TZ9N`qPDofJ>>_f8A~BU1Qzcz7B&O3JQe2U+ zix|@(>GC1z$sd;&v8S_suy9Bb(X}(+4D=b$yDw#=`~M|Bnd~LsLR>im&cJ_VKnByB z>4a~W_SPTose5hUxaCmMxK0fU?cO1Pe{>x=vB@|e)Mi|fu#2cw)NkoTKL{8huAG5i GVBj4(yF<+Y literal 0 HcmV?d00001 diff --git a/lambda/take_snapshots_rds/snapshots_tool_utils.py b/lambda/take_snapshots_rds/snapshots_tool_utils.py new file mode 100644 index 0000000..5a797d9 --- /dev/null +++ b/lambda/take_snapshots_rds/snapshots_tool_utils.py @@ -0,0 +1,364 @@ +''' +Copyright 2017 Amazon.com, Inc. or its affiliates. All Rights Reserved. + +Licensed under the Apache License, Version 2.0 (the "License"). You may not use this file except in compliance with the License. A copy of the License is located at + + http://aws.amazon.com/apache2.0/ + +or in the "license" file accompanying this file. This file is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. +''' + + +# snapshots_tool_utils +# Support module for the Snapshot Tool for RDS + +import boto3 +from datetime import datetime, timedelta +import os +import logging +import re + + +# Initialize everything +_LOGLEVEL = os.getenv('LOG_LEVEL', 'ERROR').strip() + +_DESTINATION_REGION = os.getenv( + 'DEST_REGION', os.getenv('AWS_DEFAULT_REGION')).strip() + +_KMS_KEY_DEST_REGION = os.getenv('KMS_KEY_DEST_REGION', 'None').strip() + +_KMS_KEY_SOURCE_REGION = os.getenv('KMS_KEY_SOURCE_REGION', 'None').strip() + +_TIMESTAMP_FORMAT = '%Y-%m-%d-%H-%M' + +if os.getenv('REGION_OVERRIDE', 'NO') != 'NO': + _REGION = os.getenv('REGION_OVERRIDE').strip() +else: + _REGION = os.getenv('AWS_DEFAULT_REGION') + +_SUPPORTED_ENGINES = [ 'mariadb', 'sqlserver-se', 'sqlserver-ee', 'sqlserver-ex', 'sqlserver-web', 'mysql', 'oracle-se', 'oracle-se1', 'oracle-se2', 'oracle-ee', 'postgres' ] + + +logger = logging.getLogger() +logger.setLevel(_LOGLEVEL.upper()) + + +class SnapshotToolException(Exception): + pass + + +def search_tag_copydbsnapshot(response): +# Takes a list_tags_for_resource response and searches for our CopyDBSnapshot tag + try: + + for tag in response['TagList']: + if tag['Key'] == 'CopyDBSnapshot' and tag['Value'] == 'True': return True + + except Exception: return False + + else: return False + + + +def search_tag_created(response): +# Takes a describe_db_snapshots response and searches for our CreatedBy tag + try: + + for tag in response['TagList']: + if tag['Key'] == 'CreatedBy' and tag['Value'] == 'Snapshot Tool for RDS': return True + + except Exception: return False + + else: return False + + + +def search_tag_shared(response): +# Takes a describe_db_snapshots response and searches for our shareAndCopy tag + try: + for tag in response['TagList']: + if tag['Key'] == 'shareAndCopy' and tag['Value'] == 'YES': + for tag2 in response['TagList']: + if tag2['Key'] == 'CreatedBy' and tag2['Value'] == 'Snapshot Tool for RDS': + return True + + except Exception: + return False + + return False + + + +def search_tag_copied(response): +# Search for a tag indicating we copied this snapshot + try: + for tag in response['TagList']: + if tag['Key'] == 'CopiedBy' and tag['Value'] == 'Snapshot Tool for RDS': + return True + + except Exception: + return False + + return False + +def get_own_snapshots_no_x_account(pattern, response, REGION): + # Filters our own snapshots + filtered = {} + for snapshot in response['DBSnapshots']: + + if snapshot['SnapshotType'] == 'manual' and re.search(pattern, snapshot['DBInstanceIdentifier']) and snapshot['Engine'] in _SUPPORTED_ENGINES: + client = boto3.client('rds', region_name=REGION) + response_tags = client.list_tags_for_resource( + ResourceName=snapshot['DBSnapshotArn']) + + if search_tag_created(response_tags): + filtered[snapshot['DBSnapshotIdentifier']] = { + 'Arn': snapshot['DBSnapshotArn'], 'Status': snapshot['Status'], 'DBInstanceIdentifier': snapshot['DBInstanceIdentifier']} + #Changed the next line to search for ALL_CLUSTERS or ALL_SNAPSHOTS so it will work with no-x-account + elif snapshot['SnapshotType'] == 'manual' and pattern == 'ALL_SNAPSHOTS' and snapshot['Engine'] in _SUPPORTED_ENGINES: + client = boto3.client('rds', region_name=REGION) + response_tags = client.list_tags_for_resource( + ResourceName=snapshot['DBSnapshotArn']) + + if search_tag_created(response_tags): + filtered[snapshot['DBSnapshotIdentifier']] = { + 'Arn': snapshot['DBSnapshotArn'], 'Status': snapshot['Status'], 'DBInstanceIdentifier': snapshot['DBInstanceIdentifier']} + + return filtered + + +def get_shared_snapshots(pattern, response): +# Returns a dict with only shared snapshots filtered by pattern, with DBSnapshotIdentifier as key and the response as attribute + filtered = {} + for snapshot in response['DBSnapshots']: + if snapshot['SnapshotType'] == 'shared' and re.search(pattern, snapshot['DBInstanceIdentifier']) and snapshot['Engine'] in _SUPPORTED_ENGINES: + filtered[get_snapshot_identifier(snapshot)] = { + 'Arn': snapshot['DBSnapshotIdentifier'], 'Encrypted': snapshot['Encrypted'], 'DBInstanceIdentifier': snapshot['DBInstanceIdentifier']} + if snapshot['Encrypted'] is True: + filtered[get_snapshot_identifier(snapshot)]['KmsKeyId'] = snapshot['KmsKeyId'] + + elif snapshot['SnapshotType'] == 'shared' and pattern == 'ALL_SNAPSHOTS' and snapshot['Engine'] in _SUPPORTED_ENGINES: + filtered[get_snapshot_identifier(snapshot)] = { + 'Arn': snapshot['DBSnapshotIdentifier'], 'Encrypted': snapshot['Encrypted'], 'DBInstanceIdentifier': snapshot['DBInstanceIdentifier']} + if snapshot['Encrypted'] is True: + filtered[get_snapshot_identifier(snapshot)]['KmsKeyId'] = snapshot['KmsKeyId'] + return filtered + + + +def get_snapshot_identifier(snapshot): +# Function that will return the RDS Snapshot identifier given an ARN + match = re.match('arn:aws:rds:.*:.*:snapshot:(.+)', + snapshot['DBSnapshotArn']) + return match.group(1) + + +def get_own_snapshots_dest(pattern, response): +# Returns a dict with local snapshots, filtered by pattern, with DBSnapshotIdentifier as key and Arn, Status as attributes + filtered = {} + for snapshot in response['DBSnapshots']: + + if snapshot['SnapshotType'] == 'manual' and re.search(pattern, snapshot['DBInstanceIdentifier']) and snapshot['Engine'] in _SUPPORTED_ENGINES: + filtered[snapshot['DBSnapshotIdentifier']] = { + 'Arn': snapshot['DBSnapshotArn'], 'Status': snapshot['Status'], 'Encrypted': snapshot['Encrypted'], 'DBInstanceIdentifier': snapshot['DBInstanceIdentifier']} + + if snapshot['Encrypted'] is True: + filtered[snapshot['DBSnapshotIdentifier']]['KmsKeyId'] = snapshot['KmsKeyId'] + + elif snapshot['SnapshotType'] == 'manual' and pattern == 'ALL_SNAPSHOTS' and snapshot['Engine'] in _SUPPORTED_ENGINES: + filtered[snapshot['DBSnapshotIdentifier']] = { + 'Arn': snapshot['DBSnapshotArn'], 'Status': snapshot['Status'], 'Encrypted': snapshot['Encrypted'], 'DBInstanceIdentifier': snapshot['DBInstanceIdentifier'] } + + if snapshot['Encrypted'] is True: + filtered[snapshot['DBSnapshotIdentifier']]['KmsKeyId'] = snapshot['KmsKeyId'] + + return filtered + +def filter_instances(taggedinstance, pattern, instance_list): +# Takes the response from describe-db-instances and filters according to pattern in DBInstanceIdentifier + filtered_list = [] + + for instance in instance_list['DBInstances']: + + if taggedinstance == 'TRUE': + client = boto3.client('rds', region_name=_REGION) + response = client.list_tags_for_resource(ResourceName=instance['DBInstanceArn']) + + if pattern == 'ALL_INSTANCES' and instance['Engine'] in _SUPPORTED_ENGINES: + if (taggedinstance == 'TRUE' and search_tag_copydbsnapshot(response)) or taggedinstance == 'FALSE': + filtered_list.append(instance) + + else: + match = re.search(pattern, instance['DBInstanceIdentifier']) + + if match and instance['Engine'] in _SUPPORTED_ENGINES: + if (taggedinstance == 'TRUE' and search_tag_copydbsnapshot(response)) or taggedinstance == 'FALSE': + filtered_list.append(instance) + + return filtered_list + + +def get_own_snapshots_source(pattern, response, backup_interval=None): +# Filters our own snapshots + filtered = {} + + for snapshot in response['DBSnapshots']: + + # No need to consider snapshots that are still in progress + if 'SnapshotCreateTime' not in snapshot: + continue + + # No need to get tags for snapshots outside of the backup interval + if backup_interval and snapshot['SnapshotCreateTime'].replace(tzinfo=None) < datetime.utcnow().replace(tzinfo=None) - timedelta(hours=backup_interval): + continue + + if snapshot['SnapshotType'] == 'manual' and re.search(pattern, snapshot['DBInstanceIdentifier']) and snapshot['Engine'] in _SUPPORTED_ENGINES: + client = boto3.client('rds', region_name=_REGION) + response_tags = client.list_tags_for_resource( + ResourceName=snapshot['DBSnapshotArn']) + + if search_tag_created(response_tags): + filtered[snapshot['DBSnapshotIdentifier']] = { + 'Arn': snapshot['DBSnapshotArn'], 'Status': snapshot['Status'], 'DBInstanceIdentifier': snapshot['DBInstanceIdentifier']} + + elif snapshot['SnapshotType'] == 'manual' and (pattern == 'ALL_CLUSTERS' or pattern == 'ALL_SNAPSHOTS' or pattern == 'ALL_INSTANCES') and snapshot['Engine'] in _SUPPORTED_ENGINES: + client = boto3.client('rds', region_name=_REGION) + response_tags = client.list_tags_for_resource( + ResourceName=snapshot['DBSnapshotArn']) + + if search_tag_created(response_tags): + filtered[snapshot['DBSnapshotIdentifier']] = { + 'Arn': snapshot['DBSnapshotArn'], 'Status': snapshot['Status'], 'DBInstanceIdentifier': snapshot['DBInstanceIdentifier']} + + return filtered + + +def get_timestamp_no_minute(snapshot_identifier, snapshot_list): +# Get a timestamp from the name of a snapshot and strip out the minutes + pattern = '%s-(.+)-\d{2}' % snapshot_list[snapshot_identifier]['DBInstanceIdentifier'] + timestamp_format = '%Y-%m-%d-%H' + date_time = re.search(pattern, snapshot_identifier) + + if date_time is not None: + return datetime.strptime(date_time.group(1), timestamp_format) + + +def get_timestamp(snapshot_identifier, snapshot_list): +# Searches for a timestamp on a snapshot name + pattern = '%s-(.+)' % snapshot_list[snapshot_identifier]['DBInstanceIdentifier'] + date_time = re.search(pattern, snapshot_identifier) + + if date_time is not None: + + try: + return datetime.strptime(date_time.group(1), _TIMESTAMP_FORMAT) + + except Exception: + return None + + return None + + + +def get_latest_snapshot_ts(instance_identifier, filtered_snapshots): +# Get latest snapshot for a specific DBInstanceIdentifier + timestamps = [] + + for snapshot,snapshot_object in filtered_snapshots.items(): + + if snapshot_object['DBInstanceIdentifier'] == instance_identifier: + timestamp = get_timestamp_no_minute(snapshot, filtered_snapshots) + + if timestamp is not None: + timestamps.append(timestamp) + + if len(timestamps) > 0: + return max(timestamps) + + else: + return None + + + +def requires_backup(backup_interval, instance, filtered_snapshots): +# Returns True if latest snapshot is older than INTERVAL + latest = get_latest_snapshot_ts(instance['DBInstanceIdentifier'], filtered_snapshots) + + if latest is not None: + backup_age = datetime.now() - latest + + if backup_age.total_seconds() >= (backup_interval * 60 * 60): + return True + + else: + return False + + elif latest is None: + return True + + +def paginate_api_call(client, api_call, objecttype, *args, **kwargs): +#Takes an RDS boto client and paginates through api_call calls and returns a list of objects of objecttype + response = {} + response[objecttype] = [] + + # Create a paginator + paginator = client.get_paginator(api_call) + + # Create a PageIterator from the Paginator + page_iterator = paginator.paginate(**kwargs) + for page in page_iterator: + for item in page[objecttype]: + response[objecttype].append(item) + + return response + + +def copy_local(snapshot_identifier, snapshot_object): + client = boto3.client('rds', region_name=_REGION) + + tags = [{ + 'Key': 'CopiedBy', + 'Value': 'Snapshot Tool for RDS' + }] + + if snapshot_object['Encrypted']: + logger.info('Copying encrypted snapshot %s locally' % snapshot_identifier) + response = client.copy_db_snapshot( + SourceDBSnapshotIdentifier = snapshot_object['Arn'], + TargetDBSnapshotIdentifier = snapshot_identifier, + KmsKeyId = _KMS_KEY_SOURCE_REGION, + Tags = tags) + + else: + logger.info('Copying snapshot %s locally' %snapshot_identifier) + response = client.copy_db_snapshot( + SourceDBSnapshotIdentifier = snapshot_object['Arn'], + TargetDBSnapshotIdentifier = snapshot_identifier, + Tags = tags) + + return response + + + +def copy_remote(snapshot_identifier, snapshot_object): + client = boto3.client('rds', region_name=_DESTINATION_REGION) + + if snapshot_object['Encrypted']: + logger.info('Copying encrypted snapshot %s to remote region %s' % (snapshot_object['Arn'], _DESTINATION_REGION)) + response = client.copy_db_snapshot( + SourceDBSnapshotIdentifier = snapshot_object['Arn'], + TargetDBSnapshotIdentifier = snapshot_identifier, + KmsKeyId = _KMS_KEY_DEST_REGION, + SourceRegion = _REGION, + CopyTags = True) + + else: + logger.info('Copying snapshot %s to remote region %s' % (snapshot_object['Arn'], _DESTINATION_REGION)) + response = client.copy_db_snapshot( + SourceDBSnapshotIdentifier = snapshot_object['Arn'], + TargetDBSnapshotIdentifier = snapshot_identifier, + SourceRegion = _REGION, + CopyTags = True) + + return response diff --git a/terraform/modules/rds_source/resource.tf b/terraform/modules/rds_source/resource.tf deleted file mode 100644 index bd79f2a..0000000 --- a/terraform/modules/rds_source/resource.tf +++ /dev/null @@ -1,331 +0,0 @@ -resource "aws_cloudwatch_composite_alarm" "alarmcw_backups_failed" { - actions_enabled = "true" - // CF Property(ComparisonOperator) = "GreaterThanOrEqualToThreshold" - // CF Property(EvaluationPeriods) = "1" - // CF Property(MetricName) = "ExecutionsFailed" - // CF Property(Namespace) = "AWS/States" - // CF Property(Period) = "300" - // CF Property(Statistic) = "Sum" - // CF Property(Threshold) = "1.0" - alarm_actions = [ - aws_sns_topic.topic_backups_failed.id - ] - // CF Property(Dimensions) = [ - // { - // Name = "StateMachineArn" - // Value = aws_ec2_instance_state.state_machine_take_snapshots_rds.id - // } - // ] -} - -resource "aws_cloudwatch_composite_alarm" "alarmcw_share_failed" { - count = locals.Share ? 1 : 0 - actions_enabled = "true" - // CF Property(ComparisonOperator) = "GreaterThanOrEqualToThreshold" - // CF Property(EvaluationPeriods) = "2" - // CF Property(MetricName) = "ExecutionsFailed" - // CF Property(Namespace) = "AWS/States" - // CF Property(Period) = "3600" - // CF Property(Statistic) = "Sum" - // CF Property(Threshold) = "2.0" - alarm_actions = [ - aws_sns_topic.topic_share_failed.id - ] - // CF Property(Dimensions) = [ - // { - // Name = "StateMachineArn" - // Value = aws_ec2_instance_state.statemachine_share_snapshots_rds[0].id - // } - // ] -} - -resource "aws_cloudwatch_composite_alarm" "alarmcw_delete_old_failed" { - count = locals.DeleteOld ? 1 : 0 - actions_enabled = "true" - // CF Property(ComparisonOperator) = "GreaterThanOrEqualToThreshold" - // CF Property(EvaluationPeriods) = "2" - // CF Property(MetricName) = "ExecutionsFailed" - // CF Property(Namespace) = "AWS/States" - // CF Property(Period) = "3600" - // CF Property(Statistic) = "Sum" - // CF Property(Threshold) = "2.0" - alarm_actions = [ - aws_sns_topic.topic_delete_old_failed.id - ] - // CF Property(Dimensions) = [ - // { - // Name = "StateMachineArn" - // Value = aws_ec2_instance_state.statemachine_delete_old_snapshots_rds[0].id - // } - // ] -} - -resource "aws_iam_role" "iamrole_snapshots_rds" { - assume_role_policy = { - Version = "2012-10-17" - Statement = [ - { - Effect = "Allow" - Principal = { - Service = "lambda.amazonaws.com" - } - Action = "sts:AssumeRole" - } - ] - } - force_detach_policies = [ - { - PolicyName = "inline_policy_snapshots_rds_cw_logs" - PolicyDocument = { - Version = "2012-10-17" - Statement = [ - { - Effect = "Allow" - Action = [ - "logs:CreateLogGroup", - "logs:CreateLogStream", - "logs:PutLogEvents" - ] - Resource = "arn:aws:logs:*:*:*" - } - ] - } - }, - { - PolicyName = "inline_policy_snapshots_rds" - PolicyDocument = { - Version = "2012-10-17" - Statement = [ - { - Effect = "Allow" - Action = [ - "rds:CreateDBSnapshot", - "rds:DeleteDBSnapshot", - "rds:DescribeDBInstances", - "rds:DescribeDBSnapshots", - "rds:ModifyDBSnapshotAttribute", - "rds:DescribeDBSnapshotAttributes", - "rds:ListTagsForResource", - "rds:AddTagsToResource" - ] - Resource = "*" - } - ] - } - } - ] -} - -resource "aws_lambda_function" "lambda_take_snapshots_rds" { - code_signing_config_arn = { - S3Bucket = var.code_bucket - S3Key = "take_snapshots_rds.zip" - } - memory_size = 512 - description = "This functions triggers snapshots creation for RDS instances. It checks for existing snapshots following the pattern and interval specified in the environment variables with the following format: -YYYY-MM-DD-HH-MM" - environment { - variables = { - INTERVAL = var.backup_interval - PATTERN = var.instance_name_pattern - LOG_LEVEL = var.log_level - REGION_OVERRIDE = var.source_region_override - TAGGEDINSTANCE = var.tagged_instance - } - } - role = aws_iam_role.iamrole_snapshots_rds.arn - runtime = "python3.7" - handler = "lambda_function.lambda_handler" - timeout = 300 -} - -resource "aws_lambda_function" "lambda_share_snapshots_rds" { - count = locals.Share ? 1 : 0 - code_signing_config_arn = { - S3Bucket = var.code_bucket - S3Key = "share_snapshots_rds.zip" - } - memory_size = 512 - description = "This function shares snapshots created by the take_snapshots_rds function with DEST_ACCOUNT specified in the environment variables. " - environment { - variables = { - DEST_ACCOUNT = var.destination_account - LOG_LEVEL = var.log_level - PATTERN = var.instance_name_pattern - REGION_OVERRIDE = var.source_region_override - } - } - role = aws_iam_role.iamrole_snapshots_rds.arn - runtime = "python3.7" - handler = "lambda_function.lambda_handler" - timeout = 300 -} - -resource "aws_lambda_function" "lambda_delete_old_snapshots_rds" { - count = locals.DeleteOld ? 1 : 0 - code_signing_config_arn = { - S3Bucket = var.code_bucket - S3Key = "delete_old_snapshots_rds.zip" - } - memory_size = 512 - description = "This function deletes snapshots created by the take_snapshots_rds function. " - environment { - variables = { - RETENTION_DAYS = var.retention_days - PATTERN = var.instance_name_pattern - LOG_LEVEL = var.log_level - REGION_OVERRIDE = var.source_region_override - } - } - role = aws_iam_role.iamrole_snapshots_rds.arn - runtime = "python3.7" - handler = "lambda_function.lambda_handler" - timeout = 300 -} - -resource "aws_iam_role" "iamrole_state_execution" { - assume_role_policy = { - Version = "2012-10-17" - Statement = [ - { - Effect = "Allow" - Principal = { - Service = join("", ["states.", data.aws_region.current.name, ".amazonaws.com"]) - } - Action = "sts:AssumeRole" - } - ] - } - force_detach_policies = [ - { - PolicyName = "inline_policy_snapshots_rds" - PolicyDocument = { - Version = "2012-10-17" - Statement = [ - { - Effect = "Allow" - Action = [ - "lambda:InvokeFunction" - ] - Resource = "*" - } - ] - } - } - ] -} - -resource "aws_ec2_instance_state" "state_machine_take_snapshots_rds" { - // CF Property(DefinitionString) = join("", [join(" - // ", [" {"Comment":"Triggers snapshot backup for RDS instances",", " "StartAt":"TakeSnapshots",", " "States":{", " "TakeSnapshots":{", " "Type":"Task",", " "Resource": "]), """, aws_lambda_function.lambda_take_snapshots_rds.arn, "" - // ,", join(" - // ", [" "Retry":[", " {", " "ErrorEquals":[ ", " "SnapshotToolException"", " ],", " "IntervalSeconds":300,", " "MaxAttempts":20,", " "BackoffRate":1", " },", " {", " "ErrorEquals":[ ", " "States.ALL"], ", " "IntervalSeconds": 30,", " "MaxAttempts": 20,", " "BackoffRate": 1", " }", " ],", " "End": true ", " }", " }}"])]) - // CF Property(RoleArn) = aws_iam_role.iamrole_state_execution.arn -} - -resource "aws_ec2_instance_state" "statemachine_share_snapshots_rds" { - count = locals.Share ? 1 : 0 - // CF Property(DefinitionString) = join("", [join(" - // ", [" {"Comment":"Shares snapshots with DEST_ACCOUNT",", " "StartAt":"ShareSnapshots",", " "States":{", " "ShareSnapshots":{", " "Type":"Task",", " "Resource": "]), """, aws_lambda_function.lambda_share_snapshots_rds.arn, "" - // ,", join(" - // ", [" "Retry":[", " {", " "ErrorEquals":[ ", " "SnapshotToolException"", " ],", " "IntervalSeconds":300,", " "MaxAttempts":3,", " "BackoffRate":1", " },", " {", " "ErrorEquals":[ ", " "States.ALL"], ", " "IntervalSeconds": 30,", " "MaxAttempts": 20,", " "BackoffRate": 1", " }", " ],", " "End": true ", " }", " }}"])]) - // CF Property(RoleArn) = aws_iam_role.iamrole_state_execution.arn -} - -resource "aws_ec2_instance_state" "statemachine_delete_old_snapshots_rds" { - count = locals.DeleteOld ? 1 : 0 - // CF Property(DefinitionString) = join("", [join(" - // ", [" {"Comment":"DeleteOld management for RDS snapshots",", " "StartAt":"DeleteOld",", " "States":{", " "DeleteOld":{", " "Type":"Task",", " "Resource": "]), """, aws_lambda_function.lambda_delete_old_snapshots_rds.arn, "" - // ,", join(" - // ", [" "Retry":[", " {", " "ErrorEquals":[ ", " "SnapshotToolException"", " ],", " "IntervalSeconds":300,", " "MaxAttempts":7,", " "BackoffRate":1", " },", " {", " "ErrorEquals":[ ", " "States.ALL"], ", " "IntervalSeconds": 30,", " "MaxAttempts": 20,", " "BackoffRate": 1", " }", " ],", " "End": true ", " }", " }}"])]) - // CF Property(RoleArn) = aws_iam_role.iamrole_state_execution.arn -} - -resource "aws_iam_role" "iamrole_step_invocation" { - assume_role_policy = { - Version = "2012-10-17" - Statement = [ - { - Effect = "Allow" - Principal = { - Service = "events.amazonaws.com" - } - Action = "sts:AssumeRole" - } - ] - } - force_detach_policies = [ - { - PolicyName = "inline_policy_state_invocation" - PolicyDocument = { - Version = "2012-10-17" - Statement = [ - { - Effect = "Allow" - Action = [ - "states:StartExecution" - ] - Resource = "*" - } - ] - } - } - ] -} - -resource "aws_iot_topic_rule_destination" "cw_event_backup_rds" { - // CF Property(Description) = "Triggers the TakeSnapshotsRDS state machine" - // CF Property(ScheduleExpression) = join("", ["cron(", var.backup_schedule, ")"]) - // CF Property(State) = "ENABLED" - // CF Property(Targets) = [ - // { - // Arn = aws_ec2_instance_state.state_machine_take_snapshots_rds.id - // Id = "Target1" - // RoleArn = aws_iam_role.iamrole_step_invocation.arn - // } - // ] -} - -resource "aws_iot_topic_rule_destination" "cw_event_share_snapshots_rds" { - count = locals.Share ? 1 : 0 - // CF Property(Description) = "Triggers the ShareSnapshotsRDS state machine" - // CF Property(ScheduleExpression) = join("", ["cron(", "/10 * * * ? *", ")"]) - // CF Property(State) = "ENABLED" - // CF Property(Targets) = [ - // { - // Arn = aws_ec2_instance_state.statemachine_share_snapshots_rds[0].id - // Id = "Target1" - // RoleArn = aws_iam_role.iamrole_step_invocation.arn - // } - // ] -} - -resource "aws_iot_topic_rule_destination" "cw_event_delete_old_snapshots_rds" { - count = locals.DeleteOld ? 1 : 0 - // CF Property(Description) = "Triggers the DeleteOldSnapshotsRDS state machine" - // CF Property(ScheduleExpression) = join("", ["cron(", "0 /1 * * ? *", ")"]) - // CF Property(State) = "ENABLED" - // CF Property(Targets) = [ - // { - // Arn = aws_ec2_instance_state.statemachine_delete_old_snapshots_rds[0].id - // Id = "Target1" - // RoleArn = aws_iam_role.iamrole_step_invocation.arn - // } - // ] -} - -resource "aws_inspector_resource_group" "cwloggrouplambda_take_snapshots_rds" { - // CF Property(RetentionInDays) = var.lambda_cw_log_retention - // CF Property(LogGroupName) = "/aws/lambda/${aws_lambda_function.lambda_take_snapshots_rds.arn}" -} - -resource "aws_inspector_resource_group" "cwloggrouplambda_share_snapshots_rds" { - count = locals.Share ? 1 : 0 - // CF Property(RetentionInDays) = var.lambda_cw_log_retention - // CF Property(LogGroupName) = "/aws/lambda/${aws_lambda_function.lambda_share_snapshots_rds[0].arn}" -} - -resource "aws_inspector_resource_group" "cwloggrouplambda_delete_old_snapshots_rds" { - // CF Property(RetentionInDays) = var.lambda_cw_log_retention - // CF Property(LogGroupName) = "/aws/lambda/${var.log_group_name}" -} - From d4abb5342473bd251e351989fb173cedd399e90a Mon Sep 17 00:00:00 2001 From: charlie keegan Date: Wed, 8 Feb 2023 17:20:58 +0000 Subject: [PATCH 31/48] update gitignore --- .gitignore | 4 +++- lambda/.DS_Store | Bin 10244 -> 0 bytes 2 files changed, 3 insertions(+), 1 deletion(-) delete mode 100644 lambda/.DS_Store diff --git a/.gitignore b/.gitignore index dc82526..f0c2b8b 100644 --- a/.gitignore +++ b/.gitignore @@ -31,4 +31,6 @@ override.tf.json # Ignore CLI configuration files .terraformrc -terraform.rc \ No newline at end of file +terraform.rc + +.DS_Store \ No newline at end of file diff --git a/lambda/.DS_Store b/lambda/.DS_Store deleted file mode 100644 index c2defc924c9df9d209960bb19b8840bb2b4b1519..0000000000000000000000000000000000000000 GIT binary patch literal 0 HcmV?d00001 literal 10244 zcmeHM&2G~`5T0!VbybieepG~1u!N8}l#)Uf32_M_95`?x2qb=F>?UoZI(8I04Je9~ zJMaqd2;4by;sJOUBqaD|*J)EXRcunBV(m)1v$1!^+3)l2&Wwpjl)JTcB9n*=RF?TO zXeJcC&-GB5i>_RO6woKKC?Jn)YD4Ox5zq)|1T+E~0gb>(L;zUqE-J@Xh720g~Pw1Zu6i<;O4>j!;=IX>ma&!arN4L-+q&~8JE zcTE@l7CGqKaUVnMLjUuak1*sX91EUjC|7wvYaBHkTR zrA8nr0`qF1$@%|>*#3|B{-4yG9;Xq|2>gc#h+L&wDdUfi(p&L4aIW1%eS^w{#|@Q~ yDrjUn9#W>`@q^RxH&A(Ofh8C8OOYHZi7P1o{A0k;c^V!6A9r3<$NwiXcl`%4dM*6` From a6b81c3cee153da315dcfa7f4d9f95038c0ae38b Mon Sep 17 00:00:00 2001 From: charlie keegan Date: Wed, 8 Feb 2023 17:21:28 +0000 Subject: [PATCH 32/48] remove makefile --- terraform/Makefile | 2 -- 1 file changed, 2 deletions(-) delete mode 100644 terraform/Makefile diff --git a/terraform/Makefile b/terraform/Makefile deleted file mode 100644 index 451f93f..0000000 --- a/terraform/Makefile +++ /dev/null @@ -1,2 +0,0 @@ -apply: - terraform apply --auto-approve \ No newline at end of file From f74bd2cec4849b485c47ed7f5cf40fc55ff4a70c Mon Sep 17 00:00:00 2001 From: charlie keegan Date: Thu, 9 Feb 2023 13:57:58 +0000 Subject: [PATCH 33/48] fix cloudwatch rules and logs --- terraform/main.tf | 13 ++++++------- 1 file changed, 6 insertions(+), 7 deletions(-) diff --git a/terraform/main.tf b/terraform/main.tf index 3ea9603..36ce40e 100644 --- a/terraform/main.tf +++ b/terraform/main.tf @@ -1,12 +1,11 @@ -module "test_dbs" { - source = "./modules/test_dbs" -} +# module "test_dbs" { +# source = "./modules/test_dbs" +# } module "snapshots_tool_rds_dest" { source = "./modules/rds_destination" } - -# module "snapshots_tool_rds_source" { -# source = "./modules/snapshots_tool_rds_source" -# } +module "snapshots_tool_rds_source" { + source = "./modules/rds_source" +} From 8f2da9632f187c9700ae35c54679a10e5f629f25 Mon Sep 17 00:00:00 2001 From: charlie keegan Date: Thu, 9 Feb 2023 13:59:28 +0000 Subject: [PATCH 34/48] break out source into separate files --- terraform/modules/rds_source/cloudwatch.tf | 125 +++++++++++++++++++ terraform/modules/rds_source/iam.tf | 121 ++++++++++++++++++ terraform/modules/rds_source/lambda.tf | 66 ++++++++++ terraform/modules/rds_source/stepfunction.tf | 26 ++++ 4 files changed, 338 insertions(+) create mode 100644 terraform/modules/rds_source/cloudwatch.tf create mode 100644 terraform/modules/rds_source/iam.tf create mode 100644 terraform/modules/rds_source/lambda.tf create mode 100644 terraform/modules/rds_source/stepfunction.tf diff --git a/terraform/modules/rds_source/cloudwatch.tf b/terraform/modules/rds_source/cloudwatch.tf new file mode 100644 index 0000000..feddf26 --- /dev/null +++ b/terraform/modules/rds_source/cloudwatch.tf @@ -0,0 +1,125 @@ + +resource "aws_cloudwatch_metric_alarm" "alarmcw_backups_failed" { + + alarm_name = "failed-rds-copy" + comparison_operator = "GreaterThanOrEqualToThreshold" + evaluation_periods = "1" + metric_name = "ExecutionsFailed" + namespace = "AWS/States" + period = "300" + statistic = "Sum" + threshold = "1.0" + + dimensions = { + StateMachineArn = aws_sfn_state_machine.state_machine_take_snapshots_rds.arn + } + + alarm_description = "" + alarm_actions = [ + aws_sns_topic.topic_backups_failed.id + ] +} + +resource "aws_cloudwatch_metric_alarm" "alarmcw_share_failed" { + count = local.Share ? 1 : 0 + alarm_name = "failed-rds-delete-old-snapshot" + comparison_operator = "GreaterThanOrEqualToThreshold" + evaluation_periods = "2" + metric_name = "ExecutionsFailed" + namespace = "AWS/States" + period = "3600" + statistic = "Sum" + threshold = "2.0" + + dimensions = { + StateMachineArn = aws_sfn_state_machine.statemachine_share_snapshots_rds[*].arn + } + + alarm_description = "" + alarm_actions = [ + aws_sns_topic.topic_share_failed.id + ] +} + +resource "aws_cloudwatch_metric_alarm" "alarmcw_delete_old_failed" { + count = local.DeleteOld ? 1 : 0 + alarm_name = "failed-rds-delete-old-snapshot" + comparison_operator = "GreaterThanOrEqualToThreshold" + evaluation_periods = "2" + metric_name = "ExecutionsFailed" + namespace = "AWS/States" + period = "3600" + statistic = "Sum" + threshold = "2.0" + + dimensions = { + StateMachineArn = aws_sfn_state_machine.statemachine_delete_old_snapshots_rds[*].arn + } + + alarm_description = "" + alarm_actions = [ + aws_sns_topic.topic_delete_old_failed.id + ] +} + +resource "aws_cloudwatch_event_rule" "backup_rds" { + name = "trigger-take-snapshot-state-machine" + description = "Triggers the TakeSnapshotsRDS state machine" + schedule_expression = "cron(${var.backup_schedule})" + is_enabled = true +} + +resource "aws_cloudwatch_event_target" "backup_rds" { + rule = aws_cloudwatch_event_rule.backup_rds.name + target_id = "TakeSnapshotTarget1" + arn = aws_sfn_state_machine.state_machine_take_snapshots_rds[0].id + role_arn = aws_iam_role.iamrole_step_invocation.arn +} + + +resource "aws_cloudwatch_event_rule" "share_snapshots_rds" { + count = local.DeleteOld ? 1 : 0 + name = "trigger-share-snapshot-state-machine" + description = "Triggers the ShareSnapshotsRDS state machine" + schedule_expression = "cron(/10 * * * ? *)" + is_enabled = true +} + +resource "aws_cloudwatch_event_target" "share_snapshots_rds" { + count = local.DeleteOld ? 1 : 0 + rule = aws_cloudwatch_event_rule.share_snapshots_rds.name + target_id = "ShareSnapshotTarget1" + arn = aws_sfn_state_machine.statemachine_share_snapshots_rds[0].id + role_arn = aws_iam_role.iamrole_step_invocation.arn +} + +resource "aws_cloudwatch_event_rule" "delete_old_snapshots_rds" { + name = "trigger-delete-snapshot-state-machine" + description = "Triggers the DeleteOldSnapshotsRDS state machine" + schedule_expression = "cron(0 /1 * * ? *)" + is_enabled = true +} + +resource "aws_cloudwatch_event_target" "delete_old_snapshots_rds" { + rule = aws_cloudwatch_event_rule.delete_old_snapshots_rds.name + target_id = "DeleteSnapshotTarget1" + arn = aws_sfn_state_machine.statemachine_delete_old_snapshots_rds[0].id + role_arn = aws_iam_role.iamrole_step_invocation.arn +} + +resource "aws_cloudwatch_log_group" "take_snapshots_rds" { + name = "/aws/lambda/${aws_lambda_function.lambda_take_snapshots_rds.arn}" + retention_in_days = var.lambda_cw_log_retention +} + + +resource "aws_cloudwatch_log_group" "share_snapshots_rds" { + count = local.Share ? 1 : 0 + name = "/aws/lambda/${aws_lambda_function.lambda_share_snapshots_rds[0].arn}" + retention_in_days = var.lambda_cw_log_retention +} + +resource "aws_cloudwatch_log_group" "delete_old_snapshots_rds" { + name = "/aws/lambda/${var.log_group_name}" + retention_in_days = var.lambda_cw_log_retention +} diff --git a/terraform/modules/rds_source/iam.tf b/terraform/modules/rds_source/iam.tf new file mode 100644 index 0000000..73b45e5 --- /dev/null +++ b/terraform/modules/rds_source/iam.tf @@ -0,0 +1,121 @@ + +resource "aws_iam_role" "iamrole_snapshots_rds" { + assume_role_policy = { + Version = "2012-10-17" + Statement = [ + { + Effect = "Allow" + Principal = { + Service = "lambda.amazonaws.com" + } + Action = "sts:AssumeRole" + } + ] + } + force_detach_policies = [ + { + PolicyName = "inline_policy_snapshots_rds_cw_logs" + PolicyDocument = { + Version = "2012-10-17" + Statement = [ + { + Effect = "Allow" + Action = [ + "logs:CreateLogGroup", + "logs:CreateLogStream", + "logs:PutLogEvents" + ] + Resource = "arn:aws:logs:*:*:*" + } + ] + } + }, + { + PolicyName = "inline_policy_snapshots_rds" + PolicyDocument = { + Version = "2012-10-17" + Statement = [ + { + Effect = "Allow" + Action = [ + "rds:CreateDBSnapshot", + "rds:DeleteDBSnapshot", + "rds:DescribeDBInstances", + "rds:DescribeDBSnapshots", + "rds:ModifyDBSnapshotAttribute", + "rds:DescribeDBSnapshotAttributes", + "rds:ListTagsForResource", + "rds:AddTagsToResource" + ] + Resource = "*" + } + ] + } + } + ] +} + + +resource "aws_iam_role" "iamrole_state_execution" { + assume_role_policy = { + Version = "2012-10-17" + Statement = [ + { + Effect = "Allow" + Principal = { + Service = join("", ["states.", data.aws_region.current.name, ".amazonaws.com"]) + } + Action = "sts:AssumeRole" + } + ] + } + force_detach_policies = [ + { + PolicyName = "inline_policy_snapshots_rds" + PolicyDocument = { + Version = "2012-10-17" + Statement = [ + { + Effect = "Allow" + Action = [ + "lambda:InvokeFunction" + ] + Resource = "*" + } + ] + } + } + ] +} + +resource "aws_iam_role" "iamrole_step_invocation" { + assume_role_policy = { + Version = "2012-10-17" + Statement = [ + { + Effect = "Allow" + Principal = { + Service = "events.amazonaws.com" + } + Action = "sts:AssumeRole" + } + ] + } + force_detach_policies = [ + { + PolicyName = "inline_policy_state_invocation" + PolicyDocument = { + Version = "2012-10-17" + Statement = [ + { + Effect = "Allow" + Action = [ + "states:StartExecution" + ] + Resource = "*" + } + ] + } + } + ] +} diff --git a/terraform/modules/rds_source/lambda.tf b/terraform/modules/rds_source/lambda.tf new file mode 100644 index 0000000..1f286d7 --- /dev/null +++ b/terraform/modules/rds_source/lambda.tf @@ -0,0 +1,66 @@ + +resource "aws_lambda_function" "lambda_take_snapshots_rds" { + code_signing_config_arn = { + S3Bucket = var.code_bucket + S3Key = "take_snapshots_rds.zip" + } + memory_size = 512 + description = "This functions triggers snapshots creation for RDS instances. It checks for existing snapshots following the pattern and interval specified in the environment variables with the following format: -YYYY-MM-DD-HH-MM" + environment { + variables = { + INTERVAL = var.backup_interval + PATTERN = var.instance_name_pattern + LOG_LEVEL = var.log_level + REGION_OVERRIDE = var.source_region_override + TAGGEDINSTANCE = var.tagged_instance + } + } + role = aws_iam_role.iamrole_snapshots_rds.arn + runtime = "python3.7" + handler = "lambda_function.lambda_handler" + timeout = 300 +} + +resource "aws_lambda_function" "lambda_share_snapshots_rds" { + count = locals.Share ? 1 : 0 + code_signing_config_arn = { + S3Bucket = var.code_bucket + S3Key = "share_snapshots_rds.zip" + } + memory_size = 512 + description = "This function shares snapshots created by the take_snapshots_rds function with DEST_ACCOUNT specified in the environment variables. " + environment { + variables = { + DEST_ACCOUNT = var.destination_account + LOG_LEVEL = var.log_level + PATTERN = var.instance_name_pattern + REGION_OVERRIDE = var.source_region_override + } + } + role = aws_iam_role.iamrole_snapshots_rds.arn + runtime = "python3.7" + handler = "lambda_function.lambda_handler" + timeout = 300 +} + +resource "aws_lambda_function" "lambda_delete_old_snapshots_rds" { + count = locals.DeleteOld ? 1 : 0 + code_signing_config_arn = { + S3Bucket = var.code_bucket + S3Key = "delete_old_snapshots_rds.zip" + } + memory_size = 512 + description = "This function deletes snapshots created by the take_snapshots_rds function. " + environment { + variables = { + RETENTION_DAYS = var.retention_days + PATTERN = var.instance_name_pattern + LOG_LEVEL = var.log_level + REGION_OVERRIDE = var.source_region_override + } + } + role = aws_iam_role.iamrole_snapshots_rds.arn + runtime = "python3.7" + handler = "lambda_function.lambda_handler" + timeout = 300 +} diff --git a/terraform/modules/rds_source/stepfunction.tf b/terraform/modules/rds_source/stepfunction.tf new file mode 100644 index 0000000..4fcc3a2 --- /dev/null +++ b/terraform/modules/rds_source/stepfunction.tf @@ -0,0 +1,26 @@ + +resource "aws_ec2_instance_state" "state_machine_take_snapshots_rds" { + // CF Property(DefinitionString) = join("", [join(" + // ", [" {"Comment":"Triggers snapshot backup for RDS instances",", " "StartAt":"TakeSnapshots",", " "States":{", " "TakeSnapshots":{", " "Type":"Task",", " "Resource": "]), """, aws_lambda_function.lambda_take_snapshots_rds.arn, "" + // ,", join(" + // ", [" "Retry":[", " {", " "ErrorEquals":[ ", " "SnapshotToolException"", " ],", " "IntervalSeconds":300,", " "MaxAttempts":20,", " "BackoffRate":1", " },", " {", " "ErrorEquals":[ ", " "States.ALL"], ", " "IntervalSeconds": 30,", " "MaxAttempts": 20,", " "BackoffRate": 1", " }", " ],", " "End": true ", " }", " }}"])]) + // CF Property(RoleArn) = aws_iam_role.iamrole_state_execution.arn +} + +resource "aws_ec2_instance_state" "statemachine_share_snapshots_rds" { + count = locals.Share ? 1 : 0 + // CF Property(DefinitionString) = join("", [join(" + // ", [" {"Comment":"Shares snapshots with DEST_ACCOUNT",", " "StartAt":"ShareSnapshots",", " "States":{", " "ShareSnapshots":{", " "Type":"Task",", " "Resource": "]), """, aws_lambda_function.lambda_share_snapshots_rds.arn, "" + // ,", join(" + // ", [" "Retry":[", " {", " "ErrorEquals":[ ", " "SnapshotToolException"", " ],", " "IntervalSeconds":300,", " "MaxAttempts":3,", " "BackoffRate":1", " },", " {", " "ErrorEquals":[ ", " "States.ALL"], ", " "IntervalSeconds": 30,", " "MaxAttempts": 20,", " "BackoffRate": 1", " }", " ],", " "End": true ", " }", " }}"])]) + // CF Property(RoleArn) = aws_iam_role.iamrole_state_execution.arn +} + +resource "aws_ec2_instance_state" "statemachine_delete_old_snapshots_rds" { + count = locals.DeleteOld ? 1 : 0 + // CF Property(DefinitionString) = join("", [join(" + // ", [" {"Comment":"DeleteOld management for RDS snapshots",", " "StartAt":"DeleteOld",", " "States":{", " "DeleteOld":{", " "Type":"Task",", " "Resource": "]), """, aws_lambda_function.lambda_delete_old_snapshots_rds.arn, "" + // ,", join(" + // ", [" "Retry":[", " {", " "ErrorEquals":[ ", " "SnapshotToolException"", " ],", " "IntervalSeconds":300,", " "MaxAttempts":7,", " "BackoffRate":1", " },", " {", " "ErrorEquals":[ ", " "States.ALL"], ", " "IntervalSeconds": 30,", " "MaxAttempts": 20,", " "BackoffRate": 1", " }", " ],", " "End": true ", " }", " }}"])]) + // CF Property(RoleArn) = aws_iam_role.iamrole_state_execution.arn +} From 84f6a7621df90832650d9ce15a8ef5f33e40cadc Mon Sep 17 00:00:00 2001 From: charlie keegan Date: Thu, 9 Feb 2023 16:57:27 +0000 Subject: [PATCH 35/48] source iam fix --- terraform/modules/rds_source/iam.tf | 169 +++++++++++++++------------- 1 file changed, 90 insertions(+), 79 deletions(-) diff --git a/terraform/modules/rds_source/iam.tf b/terraform/modules/rds_source/iam.tf index 73b45e5..925dbaa 100644 --- a/terraform/modules/rds_source/iam.tf +++ b/terraform/modules/rds_source/iam.tf @@ -1,6 +1,5 @@ - -resource "aws_iam_role" "iamrole_snapshots_rds" { - assume_role_policy = { +resource "aws_iam_role" "snapshots_rds" { + assume_role_policy = jsonencode({ Version = "2012-10-17" Statement = [ { @@ -11,53 +10,66 @@ resource "aws_iam_role" "iamrole_snapshots_rds" { Action = "sts:AssumeRole" } ] + }) + force_detach_policies = true +} + +data "aws_iam_policy_document" "snapshot_rds" { + + statement { + sid = "inline_policy_snapshots_rds_cw_logs" + effect = "Allow" + actions = [ + "logs:CreateLogGroup", + "logs:CreateLogStream", + "logs:PutLogEvents" + ] + resources = ["arn:aws:logs:*:*:*"] } - force_detach_policies = [ - { - PolicyName = "inline_policy_snapshots_rds_cw_logs" - PolicyDocument = { - Version = "2012-10-17" - Statement = [ - { - Effect = "Allow" - Action = [ - "logs:CreateLogGroup", - "logs:CreateLogStream", - "logs:PutLogEvents" - ] - Resource = "arn:aws:logs:*:*:*" - } - ] - } - }, - { - PolicyName = "inline_policy_snapshots_rds" - PolicyDocument = { - Version = "2012-10-17" - Statement = [ - { - Effect = "Allow" - Action = [ - "rds:CreateDBSnapshot", - "rds:DeleteDBSnapshot", - "rds:DescribeDBInstances", - "rds:DescribeDBSnapshots", - "rds:ModifyDBSnapshotAttribute", - "rds:DescribeDBSnapshotAttributes", - "rds:ListTagsForResource", - "rds:AddTagsToResource" - ] - Resource = "*" - } - ] + + statement { + sid = "inline_policy_snapshots_rds" + effect = "Allow" + actions = [ + "rds:CreateDBSnapshot", + "rds:DeleteDBSnapshot", + "rds:DescribeDBInstances", + "rds:DescribeDBSnapshots", + "rds:ModifyDBSnapshotAttribute", + "rds:DescribeDBSnapshotAttributes", + "rds:ListTagsForResource", + "rds:AddTagsToResource" + ] + resources = ["*"] + + } + +} + +resource "aws_iam_policy" "snapshot_rds" { + name = "snapshot_rds" + policy = data.aws_iam_policy_document.snapshot_rds.json +} + +resource "aws_iam_role" "state_execution" { + assume_role_policy = jsonencode({ + Version = "2012-10-17" + Statement = [ + { + Effect = "Allow" + Principal = { + Service = "lambda.amazonaws.com" + } + Action = "sts:AssumeRole" } - } - ] + ] + }) + force_detach_policies = true } resource "aws_iam_role" "iamrole_state_execution" { - assume_role_policy = { + assume_role_policy = jsonencode({ Version = "2012-10-17" Statement = [ { @@ -68,28 +80,28 @@ resource "aws_iam_role" "iamrole_state_execution" { Action = "sts:AssumeRole" } ] + }) + force_detach_policies = true + inline_policy { + name = "inline_policy_rds_snapshot" + policy = jsonencode({ + Version = "2012-10-17" + Statement = [ + { + Effect = "Allow" + Action = [ + "lambda:InvokeFunction" + ] + Resource = "*" + } + ] + }) } - force_detach_policies = [ - { - PolicyName = "inline_policy_snapshots_rds" - PolicyDocument = { - Version = "2012-10-17" - Statement = [ - { - Effect = "Allow" - Action = [ - "lambda:InvokeFunction" - ] - Resource = "*" - } - ] - } - } - ] } resource "aws_iam_role" "iamrole_step_invocation" { - assume_role_policy = { + name = "invoke-state-machines" + assume_role_policy = jsonencode({ Version = "2012-10-17" Statement = [ { @@ -100,22 +112,21 @@ resource "aws_iam_role" "iamrole_step_invocation" { Action = "sts:AssumeRole" } ] + }) + force_detach_policies = true + inline_policy { + name = "inline_policy_state_invocation" + policy = jsonencode({ + Version = "2012-10-17" + Statement = [ + { + Effect = "Allow" + Action = [ + "states:StartExecution" + ] + Resource = "*" + } + ] + }) } - force_detach_policies = [ - { - PolicyName = "inline_policy_state_invocation" - PolicyDocument = { - Version = "2012-10-17" - Statement = [ - { - Effect = "Allow" - Action = [ - "states:StartExecution" - ] - Resource = "*" - } - ] - } - } - ] } From 0188dbf7f64d7fce00a3fbb258ac35b8b450c521 Mon Sep 17 00:00:00 2001 From: charlie keegan Date: Thu, 9 Feb 2023 17:12:12 +0000 Subject: [PATCH 36/48] fix step function defintiions --- terraform/modules/rds_source/stepfunction.tf | 128 ++++++++++++++++--- 1 file changed, 108 insertions(+), 20 deletions(-) diff --git a/terraform/modules/rds_source/stepfunction.tf b/terraform/modules/rds_source/stepfunction.tf index 4fcc3a2..be3f3ac 100644 --- a/terraform/modules/rds_source/stepfunction.tf +++ b/terraform/modules/rds_source/stepfunction.tf @@ -1,26 +1,114 @@ -resource "aws_ec2_instance_state" "state_machine_take_snapshots_rds" { - // CF Property(DefinitionString) = join("", [join(" - // ", [" {"Comment":"Triggers snapshot backup for RDS instances",", " "StartAt":"TakeSnapshots",", " "States":{", " "TakeSnapshots":{", " "Type":"Task",", " "Resource": "]), """, aws_lambda_function.lambda_take_snapshots_rds.arn, "" - // ,", join(" - // ", [" "Retry":[", " {", " "ErrorEquals":[ ", " "SnapshotToolException"", " ],", " "IntervalSeconds":300,", " "MaxAttempts":20,", " "BackoffRate":1", " },", " {", " "ErrorEquals":[ ", " "States.ALL"], ", " "IntervalSeconds": 30,", " "MaxAttempts": 20,", " "BackoffRate": 1", " }", " ],", " "End": true ", " }", " }}"])]) - // CF Property(RoleArn) = aws_iam_role.iamrole_state_execution.arn +resource "aws_sfn_state_machine" "state_machine_take_snapshots_rds" { + name = "take-snapshots-rds" + role_arn = aws_iam_role.iamrole_state_execution.arn + + definition = < Date: Thu, 9 Feb 2023 17:32:23 +0000 Subject: [PATCH 37/48] fix reference issues --- terraform/modules/rds_source/cloudwatch.tf | 10 ++--- terraform/modules/rds_source/lambda.tf | 45 +++++++++----------- terraform/modules/rds_source/output.tf | 8 ++-- terraform/modules/rds_source/stepfunction.tf | 10 ++--- 4 files changed, 35 insertions(+), 38 deletions(-) diff --git a/terraform/modules/rds_source/cloudwatch.tf b/terraform/modules/rds_source/cloudwatch.tf index feddf26..7cf1573 100644 --- a/terraform/modules/rds_source/cloudwatch.tf +++ b/terraform/modules/rds_source/cloudwatch.tf @@ -16,7 +16,7 @@ resource "aws_cloudwatch_metric_alarm" "alarmcw_backups_failed" { alarm_description = "" alarm_actions = [ - aws_sns_topic.topic_backups_failed.id + aws_sns_topic.backups_failed.id ] } @@ -37,7 +37,7 @@ resource "aws_cloudwatch_metric_alarm" "alarmcw_share_failed" { alarm_description = "" alarm_actions = [ - aws_sns_topic.topic_share_failed.id + aws_sns_topic.share_failed.id ] } @@ -58,7 +58,7 @@ resource "aws_cloudwatch_metric_alarm" "alarmcw_delete_old_failed" { alarm_description = "" alarm_actions = [ - aws_sns_topic.topic_delete_old_failed.id + aws_sns_topic.delete_old_failed.id ] } @@ -72,7 +72,7 @@ resource "aws_cloudwatch_event_rule" "backup_rds" { resource "aws_cloudwatch_event_target" "backup_rds" { rule = aws_cloudwatch_event_rule.backup_rds.name target_id = "TakeSnapshotTarget1" - arn = aws_sfn_state_machine.state_machine_take_snapshots_rds[0].id + arn = aws_sfn_state_machine.state_machine_take_snapshots_rds.id role_arn = aws_iam_role.iamrole_step_invocation.arn } @@ -87,7 +87,7 @@ resource "aws_cloudwatch_event_rule" "share_snapshots_rds" { resource "aws_cloudwatch_event_target" "share_snapshots_rds" { count = local.DeleteOld ? 1 : 0 - rule = aws_cloudwatch_event_rule.share_snapshots_rds.name + rule = aws_cloudwatch_event_rule.share_snapshots_rds[*].name target_id = "ShareSnapshotTarget1" arn = aws_sfn_state_machine.statemachine_share_snapshots_rds[0].id role_arn = aws_iam_role.iamrole_step_invocation.arn diff --git a/terraform/modules/rds_source/lambda.tf b/terraform/modules/rds_source/lambda.tf index 1f286d7..f93c5bc 100644 --- a/terraform/modules/rds_source/lambda.tf +++ b/terraform/modules/rds_source/lambda.tf @@ -1,11 +1,10 @@ resource "aws_lambda_function" "lambda_take_snapshots_rds" { - code_signing_config_arn = { - S3Bucket = var.code_bucket - S3Key = "take_snapshots_rds.zip" - } - memory_size = 512 - description = "This functions triggers snapshots creation for RDS instances. It checks for existing snapshots following the pattern and interval specified in the environment variables with the following format: -YYYY-MM-DD-HH-MM" + function_name = "take-rds-snapshots" + s3_bucket = var.code_bucket + s3_key = "take_snapshots_rds.zip" + memory_size = 512 + description = "This functions triggers snapshots creation for RDS instances. It checks for existing snapshots following the pattern and interval specified in the environment variables with the following format: -YYYY-MM-DD-HH-MM" environment { variables = { INTERVAL = var.backup_interval @@ -15,20 +14,19 @@ resource "aws_lambda_function" "lambda_take_snapshots_rds" { TAGGEDINSTANCE = var.tagged_instance } } - role = aws_iam_role.iamrole_snapshots_rds.arn + role = aws_iam_role.snapshots_rds.arn runtime = "python3.7" handler = "lambda_function.lambda_handler" timeout = 300 } resource "aws_lambda_function" "lambda_share_snapshots_rds" { - count = locals.Share ? 1 : 0 - code_signing_config_arn = { - S3Bucket = var.code_bucket - S3Key = "share_snapshots_rds.zip" - } - memory_size = 512 - description = "This function shares snapshots created by the take_snapshots_rds function with DEST_ACCOUNT specified in the environment variables. " + count = local.Share ? 1 : 0 + function_name = "share-rds-snapshot" + s3_bucket = var.code_bucket + s3_key = "share_snapshots_rds.zip" + memory_size = 512 + description = "This function shares snapshots created by the take_snapshots_rds function with DEST_ACCOUNT specified in the environment variables. " environment { variables = { DEST_ACCOUNT = var.destination_account @@ -37,20 +35,19 @@ resource "aws_lambda_function" "lambda_share_snapshots_rds" { REGION_OVERRIDE = var.source_region_override } } - role = aws_iam_role.iamrole_snapshots_rds.arn + role = aws_iam_role.snapshots_rds.arn runtime = "python3.7" handler = "lambda_function.lambda_handler" timeout = 300 } -resource "aws_lambda_function" "lambda_delete_old_snapshots_rds" { - count = locals.DeleteOld ? 1 : 0 - code_signing_config_arn = { - S3Bucket = var.code_bucket - S3Key = "delete_old_snapshots_rds.zip" - } - memory_size = 512 - description = "This function deletes snapshots created by the take_snapshots_rds function. " +resource "aws_lambda_function" "lambda_delete_snapshots_rds" { + count = local.DeleteOld ? 1 : 0 + function_name = "delete-old-rds-snapshots" + s3_bucket = var.code_bucket + s3_key = "delete_old_snapshots_rds.zip" + memory_size = 512 + description = "This function deletes snapshots created by the take_snapshots_rds function. " environment { variables = { RETENTION_DAYS = var.retention_days @@ -59,7 +56,7 @@ resource "aws_lambda_function" "lambda_delete_old_snapshots_rds" { REGION_OVERRIDE = var.source_region_override } } - role = aws_iam_role.iamrole_snapshots_rds.arn + role = aws_iam_role.snapshots_rds.arn runtime = "python3.7" handler = "lambda_function.lambda_handler" timeout = 300 diff --git a/terraform/modules/rds_source/output.tf b/terraform/modules/rds_source/output.tf index 5a4e15a..1670971 100644 --- a/terraform/modules/rds_source/output.tf +++ b/terraform/modules/rds_source/output.tf @@ -1,20 +1,20 @@ output "backup_failed_topic" { description = "Subscribe to this topic to receive alerts of failed backups" - value = aws_sns_topic.topic_backups_failed.id + value = aws_sns_topic.backups_failed.id } output "share_failed_topic" { description = "Subscribe to this topic to receive alerts of failures at sharing snapshots with destination account" - value = aws_sns_topic.topic_share_failed.id + value = aws_sns_topic.share_failed.id } output "delete_old_failed_topic" { description = "Subscribe to this topic to receive alerts of failures at deleting old snapshots" - value = aws_sns_topic.topic_delete_old_failed.id + value = aws_sns_topic.delete_old_failed.id } output "source_url" { description = "For more information and documentation, see the source repository at GitHub." - value = "https://github.com/awslabs/rds-snapshot-tool" + value = "https://github.com/awslabs/rds-snapshot-tool" } diff --git a/terraform/modules/rds_source/stepfunction.tf b/terraform/modules/rds_source/stepfunction.tf index be3f3ac..6727dcd 100644 --- a/terraform/modules/rds_source/stepfunction.tf +++ b/terraform/modules/rds_source/stepfunction.tf @@ -36,8 +36,8 @@ resource "aws_sfn_state_machine" "state_machine_take_snapshots_rds" { EOF } -resource "aws_sfn_state_machine" "lambda_share_snapshots_rds" { - count = locals.Share ? 1 : 0 +resource "aws_sfn_state_machine" "statemachine_share_snapshots_rds" { + count = local.Share ? 1 : 0 name = "share-snapshots-rds" role_arn = aws_iam_role.iamrole_state_execution.arn @@ -48,7 +48,7 @@ resource "aws_sfn_state_machine" "lambda_share_snapshots_rds" { "States": { "ShareSnapshots": { "Type":"Task", - "Resource": "${aws_lambda_function.lambda_share_snapshots_rds.arn}", + "Resource": "${aws_lambda_function.lambda_share_snapshots_rds[*].arn}", "Retry": [ { "ErrorEquals": [ @@ -75,7 +75,7 @@ resource "aws_sfn_state_machine" "lambda_share_snapshots_rds" { EOF } -resource "aws_sfn_state_machine" "statemachine_delete_old_snapshots_dest_rds" { +resource "aws_sfn_state_machine" "statemachine_delete_old_snapshots_rds" { count = local.DeleteOld ? 1 : 0 name = "delete-old-snapshots-source-rds" role_arn = aws_iam_role.iamrole_state_execution.arn @@ -87,7 +87,7 @@ resource "aws_sfn_state_machine" "statemachine_delete_old_snapshots_dest_rds" { "States": { "DeleteOldDestRegion": { "Type": "Task", - "Resource": "${aws_lambda_function.delete_old_dest_rds[*].arn}", + "Resource": "${aws_lambda_function.lambda_delete_snapshots_rds[*].arn}", "Retry": [ { "ErrorEquals": [ From a901f827c2692d27b1518a6daca0a0befe3f99f4 Mon Sep 17 00:00:00 2001 From: charlie keegan Date: Thu, 9 Feb 2023 17:41:00 +0000 Subject: [PATCH 38/48] format sfn statemachine definition --- terraform/modules/rds_source/stepfunction.tf | 106 +++++++++---------- 1 file changed, 52 insertions(+), 54 deletions(-) diff --git a/terraform/modules/rds_source/stepfunction.tf b/terraform/modules/rds_source/stepfunction.tf index 6727dcd..bae749a 100644 --- a/terraform/modules/rds_source/stepfunction.tf +++ b/terraform/modules/rds_source/stepfunction.tf @@ -43,33 +43,32 @@ resource "aws_sfn_state_machine" "statemachine_share_snapshots_rds" { definition = < Date: Fri, 10 Feb 2023 10:23:10 +0000 Subject: [PATCH 39/48] remove counts --- terraform/modules/rds_source/cloudwatch.tf | 17 ++++++----------- terraform/modules/rds_source/lambda.tf | 2 -- terraform/modules/rds_source/stepfunction.tf | 12 ++++++------ 3 files changed, 12 insertions(+), 19 deletions(-) diff --git a/terraform/modules/rds_source/cloudwatch.tf b/terraform/modules/rds_source/cloudwatch.tf index 7cf1573..86ed053 100644 --- a/terraform/modules/rds_source/cloudwatch.tf +++ b/terraform/modules/rds_source/cloudwatch.tf @@ -21,7 +21,6 @@ resource "aws_cloudwatch_metric_alarm" "alarmcw_backups_failed" { } resource "aws_cloudwatch_metric_alarm" "alarmcw_share_failed" { - count = local.Share ? 1 : 0 alarm_name = "failed-rds-delete-old-snapshot" comparison_operator = "GreaterThanOrEqualToThreshold" evaluation_periods = "2" @@ -32,7 +31,7 @@ resource "aws_cloudwatch_metric_alarm" "alarmcw_share_failed" { threshold = "2.0" dimensions = { - StateMachineArn = aws_sfn_state_machine.statemachine_share_snapshots_rds[*].arn + StateMachineArn = aws_sfn_state_machine.statemachine_share_snapshots_rds.arn } alarm_description = "" @@ -42,7 +41,6 @@ resource "aws_cloudwatch_metric_alarm" "alarmcw_share_failed" { } resource "aws_cloudwatch_metric_alarm" "alarmcw_delete_old_failed" { - count = local.DeleteOld ? 1 : 0 alarm_name = "failed-rds-delete-old-snapshot" comparison_operator = "GreaterThanOrEqualToThreshold" evaluation_periods = "2" @@ -53,7 +51,7 @@ resource "aws_cloudwatch_metric_alarm" "alarmcw_delete_old_failed" { threshold = "2.0" dimensions = { - StateMachineArn = aws_sfn_state_machine.statemachine_delete_old_snapshots_rds[*].arn + StateMachineArn = aws_sfn_state_machine.statemachine_share_snapshots_rds.arn } alarm_description = "" @@ -78,7 +76,6 @@ resource "aws_cloudwatch_event_target" "backup_rds" { resource "aws_cloudwatch_event_rule" "share_snapshots_rds" { - count = local.DeleteOld ? 1 : 0 name = "trigger-share-snapshot-state-machine" description = "Triggers the ShareSnapshotsRDS state machine" schedule_expression = "cron(/10 * * * ? *)" @@ -86,10 +83,9 @@ resource "aws_cloudwatch_event_rule" "share_snapshots_rds" { } resource "aws_cloudwatch_event_target" "share_snapshots_rds" { - count = local.DeleteOld ? 1 : 0 - rule = aws_cloudwatch_event_rule.share_snapshots_rds[*].name + rule = aws_cloudwatch_event_rule.share_snapshots_rds.name target_id = "ShareSnapshotTarget1" - arn = aws_sfn_state_machine.statemachine_share_snapshots_rds[0].id + arn = aws_sfn_state_machine.statemachine_share_snapshots_rds.id role_arn = aws_iam_role.iamrole_step_invocation.arn } @@ -103,7 +99,7 @@ resource "aws_cloudwatch_event_rule" "delete_old_snapshots_rds" { resource "aws_cloudwatch_event_target" "delete_old_snapshots_rds" { rule = aws_cloudwatch_event_rule.delete_old_snapshots_rds.name target_id = "DeleteSnapshotTarget1" - arn = aws_sfn_state_machine.statemachine_delete_old_snapshots_rds[0].id + arn = aws_sfn_state_machine.statemachine_delete_old_snapshots_rds.id role_arn = aws_iam_role.iamrole_step_invocation.arn } @@ -114,8 +110,7 @@ resource "aws_cloudwatch_log_group" "take_snapshots_rds" { resource "aws_cloudwatch_log_group" "share_snapshots_rds" { - count = local.Share ? 1 : 0 - name = "/aws/lambda/${aws_lambda_function.lambda_share_snapshots_rds[0].arn}" + name = "/aws/lambda/${aws_lambda_function.lambda_share_snapshots_rds.arn}" retention_in_days = var.lambda_cw_log_retention } diff --git a/terraform/modules/rds_source/lambda.tf b/terraform/modules/rds_source/lambda.tf index f93c5bc..b0902de 100644 --- a/terraform/modules/rds_source/lambda.tf +++ b/terraform/modules/rds_source/lambda.tf @@ -21,7 +21,6 @@ resource "aws_lambda_function" "lambda_take_snapshots_rds" { } resource "aws_lambda_function" "lambda_share_snapshots_rds" { - count = local.Share ? 1 : 0 function_name = "share-rds-snapshot" s3_bucket = var.code_bucket s3_key = "share_snapshots_rds.zip" @@ -42,7 +41,6 @@ resource "aws_lambda_function" "lambda_share_snapshots_rds" { } resource "aws_lambda_function" "lambda_delete_snapshots_rds" { - count = local.DeleteOld ? 1 : 0 function_name = "delete-old-rds-snapshots" s3_bucket = var.code_bucket s3_key = "delete_old_snapshots_rds.zip" diff --git a/terraform/modules/rds_source/stepfunction.tf b/terraform/modules/rds_source/stepfunction.tf index bae749a..b5d168f 100644 --- a/terraform/modules/rds_source/stepfunction.tf +++ b/terraform/modules/rds_source/stepfunction.tf @@ -37,7 +37,6 @@ EOF } resource "aws_sfn_state_machine" "statemachine_share_snapshots_rds" { - count = local.Share ? 1 : 0 name = "share-snapshots-rds" role_arn = aws_iam_role.iamrole_state_execution.arn @@ -48,8 +47,9 @@ resource "aws_sfn_state_machine" "statemachine_share_snapshots_rds" { "States": { "ShareSnapshots": { "Type": "Task", - "Resource": "${aws_lambda_function.lambda_share_snapshots_rds[*].arn}", - "Retry": [{ + "Resource": "${aws_lambda_function.lambda_share_snapshots_rds.arn}", + "Retry": [ + { "ErrorEquals": [ "SnapshotToolException" ], @@ -75,7 +75,6 @@ EOF } resource "aws_sfn_state_machine" "statemachine_delete_old_snapshots_rds" { - count = local.DeleteOld ? 1 : 0 name = "delete-old-snapshots-source-rds" role_arn = aws_iam_role.iamrole_state_execution.arn @@ -86,8 +85,9 @@ resource "aws_sfn_state_machine" "statemachine_delete_old_snapshots_rds" { "States": { "DeleteOldDestRegion": { "Type": "Task", - "Resource": "${aws_lambda_function.lambda_delete_snapshots_rds[*].arn}", - "Retry": [{ + "Resource": "${aws_lambda_function.lambda_delete_snapshots_rds.arn}", + "Retry": [ + { "ErrorEquals": [ "SnapshotToolException" ], From b17ee7018f0c8fe1b51b89e34d3739821f606511 Mon Sep 17 00:00:00 2001 From: charlie keegan Date: Fri, 10 Feb 2023 10:28:16 +0000 Subject: [PATCH 40/48] add test dbs --- terraform/main.tf | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/terraform/main.tf b/terraform/main.tf index 36ce40e..e78d523 100644 --- a/terraform/main.tf +++ b/terraform/main.tf @@ -1,6 +1,6 @@ -# module "test_dbs" { -# source = "./modules/test_dbs" -# } +module "test_dbs" { + source = "./modules/test_dbs" +} module "snapshots_tool_rds_dest" { source = "./modules/rds_destination" From e9fafa93801f2afa4da31fa904618ac3e3ccb96c Mon Sep 17 00:00:00 2001 From: charlie keegan Date: Fri, 10 Feb 2023 10:34:11 +0000 Subject: [PATCH 41/48] fix naming issues --- terraform/modules/rds_source/cloudwatch.tf | 4 +- terraform/modules/rds_source/iam.tf | 6 +-- terraform/modules/rds_source/stepfunction.tf | 53 ++++++++++---------- 3 files changed, 31 insertions(+), 32 deletions(-) diff --git a/terraform/modules/rds_source/cloudwatch.tf b/terraform/modules/rds_source/cloudwatch.tf index 86ed053..e215ced 100644 --- a/terraform/modules/rds_source/cloudwatch.tf +++ b/terraform/modules/rds_source/cloudwatch.tf @@ -104,13 +104,13 @@ resource "aws_cloudwatch_event_target" "delete_old_snapshots_rds" { } resource "aws_cloudwatch_log_group" "take_snapshots_rds" { - name = "/aws/lambda/${aws_lambda_function.lambda_take_snapshots_rds.arn}" + name = "/aws/lambda/${aws_lambda_function.lambda_take_snapshots_rds.function_name}" retention_in_days = var.lambda_cw_log_retention } resource "aws_cloudwatch_log_group" "share_snapshots_rds" { - name = "/aws/lambda/${aws_lambda_function.lambda_share_snapshots_rds.arn}" + name = "/aws/lambda/${aws_lambda_function.lambda_share_snapshots_rds.function_name}" retention_in_days = var.lambda_cw_log_retention } diff --git a/terraform/modules/rds_source/iam.tf b/terraform/modules/rds_source/iam.tf index 925dbaa..5f60f4a 100644 --- a/terraform/modules/rds_source/iam.tf +++ b/terraform/modules/rds_source/iam.tf @@ -17,7 +17,7 @@ resource "aws_iam_role" "snapshots_rds" { data "aws_iam_policy_document" "snapshot_rds" { statement { - sid = "inline_policy_snapshots_rds_cw_logs" + sid = "snapshotsRdsCwLogs" effect = "Allow" actions = [ "logs:CreateLogGroup", @@ -28,7 +28,7 @@ data "aws_iam_policy_document" "snapshot_rds" { } statement { - sid = "inline_policy_snapshots_rds" + sid = "snapshotsRds" effect = "Allow" actions = [ "rds:CreateDBSnapshot", @@ -100,7 +100,7 @@ resource "aws_iam_role" "iamrole_state_execution" { } resource "aws_iam_role" "iamrole_step_invocation" { - name = "invoke-state-machines" + name = "invoke-state-machines-rds-source" assume_role_policy = jsonencode({ Version = "2012-10-17" Statement = [ diff --git a/terraform/modules/rds_source/stepfunction.tf b/terraform/modules/rds_source/stepfunction.tf index b5d168f..8479925 100644 --- a/terraform/modules/rds_source/stepfunction.tf +++ b/terraform/modules/rds_source/stepfunction.tf @@ -5,33 +5,32 @@ resource "aws_sfn_state_machine" "state_machine_take_snapshots_rds" { definition = < Date: Fri, 10 Feb 2023 10:41:19 +0000 Subject: [PATCH 42/48] update spolicy name --- terraform/modules/rds_source/iam.tf | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/terraform/modules/rds_source/iam.tf b/terraform/modules/rds_source/iam.tf index 5f60f4a..6ed6540 100644 --- a/terraform/modules/rds_source/iam.tf +++ b/terraform/modules/rds_source/iam.tf @@ -47,7 +47,7 @@ data "aws_iam_policy_document" "snapshot_rds" { } resource "aws_iam_policy" "snapshot_rds" { - name = "snapshot_rds" + name = "snapshot_rds_source" policy = data.aws_iam_policy_document.snapshot_rds.json } From 93a4810d14f86dc44dde141b219c6f8d1fa522a7 Mon Sep 17 00:00:00 2001 From: charlie keegan Date: Fri, 10 Feb 2023 12:07:26 +0000 Subject: [PATCH 43/48] fix iam policy --- terraform/modules/rds_source/iam.tf | 12 ++++++++---- terraform/modules/rds_source/variable.tf | 2 +- 2 files changed, 9 insertions(+), 5 deletions(-) diff --git a/terraform/modules/rds_source/iam.tf b/terraform/modules/rds_source/iam.tf index 6ed6540..7f426e2 100644 --- a/terraform/modules/rds_source/iam.tf +++ b/terraform/modules/rds_source/iam.tf @@ -1,4 +1,5 @@ resource "aws_iam_role" "snapshots_rds" { + name = "snapshot-rds-source" assume_role_policy = jsonencode({ Version = "2012-10-17" Statement = [ @@ -24,7 +25,7 @@ data "aws_iam_policy_document" "snapshot_rds" { "logs:CreateLogStream", "logs:PutLogEvents" ] - resources = ["arn:aws:logs:*:*:*"] + resources = ["*"] } statement { @@ -46,11 +47,12 @@ data "aws_iam_policy_document" "snapshot_rds" { } -resource "aws_iam_policy" "snapshot_rds" { - name = "snapshot_rds_source" - policy = data.aws_iam_policy_document.snapshot_rds.json +resource "aws_iam_role_policy_attachment" "snapshot_rds" { + role = aws_iam_role.snapshots_rds.name + policy_arn = data.aws_iam_policy_document.snapshot_rds.json } + resource "aws_iam_role" "state_execution" { assume_role_policy = jsonencode({ Version = "2012-10-17" @@ -69,6 +71,7 @@ resource "aws_iam_role" "state_execution" { resource "aws_iam_role" "iamrole_state_execution" { + name = "invoke-state-machine-rds-source" assume_role_policy = jsonencode({ Version = "2012-10-17" Statement = [ @@ -91,6 +94,7 @@ resource "aws_iam_role" "iamrole_state_execution" { Effect = "Allow" Action = [ "lambda:InvokeFunction" + ] Resource = "*" } diff --git a/terraform/modules/rds_source/variable.tf b/terraform/modules/rds_source/variable.tf index e1092a1..9521f88 100644 --- a/terraform/modules/rds_source/variable.tf +++ b/terraform/modules/rds_source/variable.tf @@ -19,7 +19,7 @@ variable "backup_interval" { variable "destination_account" { description = "Destination account with no dashes." type = string - default = "000000000000" + default = "817503874046" } variable "share_snapshots" { From ca07fb6875b27b7becacfc068c4267129c8ff9bb Mon Sep 17 00:00:00 2001 From: charlie keegan Date: Fri, 10 Feb 2023 13:13:04 +0000 Subject: [PATCH 44/48] run format --- terraform/modules/rds_source/iam.tf | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/terraform/modules/rds_source/iam.tf b/terraform/modules/rds_source/iam.tf index 7f426e2..2f7be42 100644 --- a/terraform/modules/rds_source/iam.tf +++ b/terraform/modules/rds_source/iam.tf @@ -47,9 +47,14 @@ data "aws_iam_policy_document" "snapshot_rds" { } +resource "aws_iam_policy" "snapshot_rds" { + name = "snapshot-rds-policy" + policy = data.aws_iam_policy_document.snapshot_rds.json +} + resource "aws_iam_role_policy_attachment" "snapshot_rds" { role = aws_iam_role.snapshots_rds.name - policy_arn = data.aws_iam_policy_document.snapshot_rds.json + policy_arn = aws_iam_policy.snapshot_rds.arn } From 221e7d3c2e5105a3e64f93ff3018fbf4833ddaf3 Mon Sep 17 00:00:00 2001 From: charlie keegan Date: Fri, 10 Feb 2023 13:40:19 +0000 Subject: [PATCH 45/48] generalise variables --- lambda/Makefile | 4 ++-- terraform/modules/rds_destination/variables.tf | 2 +- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/lambda/Makefile b/lambda/Makefile index 48367f5..f284746 100644 --- a/lambda/Makefile +++ b/lambda/Makefile @@ -16,11 +16,11 @@ # Override S3 destination by changing this variable or setting it in the # environment -S3DEST?=code-bucket-816708952522 +S3DEST?=[YOUR BUCKET HERE] # Set these if, for example, you use profiles on the AWS command line # or if your 'aws' executable is in a weird place. -AWSARGS=--region us-east-1 --profile default +AWSARGS=--region [YOUR REGION] --profile [YOUR PROFILE, or 'default', or remove this] AWSCMD=aws ZIPCMD=zip diff --git a/terraform/modules/rds_destination/variables.tf b/terraform/modules/rds_destination/variables.tf index 1981e96..8214895 100644 --- a/terraform/modules/rds_destination/variables.tf +++ b/terraform/modules/rds_destination/variables.tf @@ -1,7 +1,7 @@ variable "code_bucket" { description = "Name of the bucket that contains the lambda functions to deploy." type = string - default = "code-bucket-816708952522" + default = "" } variable "snapshot_pattern" { From cca653d59425c618001aea34bf8903e48d9456ed Mon Sep 17 00:00:00 2001 From: charlie keegan Date: Fri, 10 Feb 2023 13:41:50 +0000 Subject: [PATCH 46/48] remove bucket name --- terraform/modules/rds_source/variable.tf | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/terraform/modules/rds_source/variable.tf b/terraform/modules/rds_source/variable.tf index 9521f88..132fbbe 100644 --- a/terraform/modules/rds_source/variable.tf +++ b/terraform/modules/rds_source/variable.tf @@ -1,7 +1,7 @@ variable "code_bucket" { description = "Name of the bucket that contains the lambda functions to deploy." type = string - default = "code-bucket-816708952522" + default = "" } variable "instance_name_pattern" { From 5cc8b0063398a4ba73c2c62ae32ecfb3df0ab457 Mon Sep 17 00:00:00 2001 From: charlie keegan Date: Wed, 1 Mar 2023 10:29:57 -0500 Subject: [PATCH 47/48] replace with main branch --- LICENSE | 201 ----- LICENSE.txt | 201 ----- NOTICE | 2 - cftemplates/snapshots_tool_rds_dest.json | 588 --------------- cftemplates/snapshots_tool_rds_source.json | 707 ------------------ lambda/.gitignore | 2 - lambda/Makefile | 51 -- lambda_code/.DS_Store | Bin 0 -> 20484 bytes lambda_code/copy_snapshots_dest_rds/.DS_Store | Bin 0 -> 6148 bytes .../lambda_function.py | 232 +++--- .../snapshots_tool_utils.py | 2 +- .../copy_snapshots_no_x_account_rds/.DS_Store | Bin 0 -> 6148 bytes .../lambda_function.py | 2 +- .../snapshots_tool_utils.py | 364 +++++++++ .../delete_old_snapshots_dest_rds/.DS_Store | Bin 0 -> 6148 bytes .../lambda_function.py | 176 ++--- .../snapshots_tool_utils.py | 364 +++++++++ .../.DS_Store | Bin 0 -> 6148 bytes .../lambda_function.py | 2 +- .../snapshots_tool_utils.py | 364 +++++++++ .../lambda_function.py | 159 ++-- .../snapshots_tool_utils.py | 364 +++++++++ lambda_code/share_snapshots_rds/.DS_Store | Bin 0 -> 6148 bytes .../share_snapshots_rds/lambda_function.py | 146 ++-- .../snapshots_tool_utils.py | 364 +++++++++ lambda_code/take_snapshots_rds/.DS_Store | Bin 0 -> 6148 bytes .../take_snapshots_rds/lambda_function.py | 196 ++--- .../snapshots_tool_utils.py | 364 +++++++++ rds_snapshot_tool_destination/.DS_Store | Bin 0 -> 6148 bytes .../README.md | 36 +- rds_snapshot_tool_destination/main.tf | 4 + .../modules/.DS_Store | Bin 0 -> 6148 bytes .../modules/rds_destination/cloudwatch.tf | 94 +++ .../modules/rds_destination/data.tf | 3 + .../modules/rds_destination/iam.tf | 149 ++++ .../modules/rds_destination/lambda.tf | 44 ++ .../modules/rds_destination/locals.tf | 4 + .../modules/rds_destination/outputs.tf | 9 + .../modules/rds_destination/sns.tf | 59 ++ .../modules/rds_destination/stepfunction.tf | 74 ++ .../modules/rds_destination/variables.tf | 71 ++ rds_snapshot_tool_destination/provider.tf | 13 + rds_snapshot_tool_source/.DS_Store | Bin 0 -> 6148 bytes rds_snapshot_tool_source/README.md | 84 +++ rds_snapshot_tool_source/main.tf | 9 + rds_snapshot_tool_source/modules/.DS_Store | Bin 0 -> 6148 bytes .../modules/rds_source/cloudwatch.tf | 121 +++ .../modules/rds_source/data.tf | 4 + .../modules/rds_source/iam.tf | 142 ++++ .../modules/rds_source/lambda.tf | 61 ++ .../modules/rds_source/locals.tf | 5 + .../modules/rds_source/output.tf | 20 + .../modules/rds_source/sns.tf | 62 ++ .../modules/rds_source/stepfunction.tf | 124 +++ .../modules/rds_source/variable.tf | 77 ++ .../modules/test_dbs/main.tf | 13 + rds_snapshot_tool_source/provider.tf | 13 + 57 files changed, 3905 insertions(+), 2241 deletions(-) delete mode 100644 LICENSE delete mode 100644 LICENSE.txt delete mode 100644 NOTICE delete mode 100644 cftemplates/snapshots_tool_rds_dest.json delete mode 100644 cftemplates/snapshots_tool_rds_source.json delete mode 100644 lambda/.gitignore delete mode 100644 lambda/Makefile create mode 100644 lambda_code/.DS_Store create mode 100644 lambda_code/copy_snapshots_dest_rds/.DS_Store rename {lambda => lambda_code}/copy_snapshots_dest_rds/lambda_function.py (97%) rename {lambda => lambda_code/copy_snapshots_dest_rds}/snapshots_tool_utils.py (99%) create mode 100644 lambda_code/copy_snapshots_no_x_account_rds/.DS_Store rename {lambda => lambda_code}/copy_snapshots_no_x_account_rds/lambda_function.py (99%) create mode 100644 lambda_code/copy_snapshots_no_x_account_rds/snapshots_tool_utils.py create mode 100644 lambda_code/delete_old_snapshots_dest_rds/.DS_Store rename {lambda => lambda_code}/delete_old_snapshots_dest_rds/lambda_function.py (96%) create mode 100644 lambda_code/delete_old_snapshots_dest_rds/snapshots_tool_utils.py create mode 100644 lambda_code/delete_old_snapshots_no_x_account_rds/.DS_Store rename {lambda => lambda_code}/delete_old_snapshots_no_x_account_rds/lambda_function.py (99%) create mode 100644 lambda_code/delete_old_snapshots_no_x_account_rds/snapshots_tool_utils.py rename {lambda => lambda_code}/delete_old_snapshots_rds/lambda_function.py (97%) create mode 100644 lambda_code/delete_old_snapshots_rds/snapshots_tool_utils.py create mode 100644 lambda_code/share_snapshots_rds/.DS_Store rename {lambda => lambda_code}/share_snapshots_rds/lambda_function.py (96%) create mode 100644 lambda_code/share_snapshots_rds/snapshots_tool_utils.py create mode 100644 lambda_code/take_snapshots_rds/.DS_Store rename {lambda => lambda_code}/take_snapshots_rds/lambda_function.py (96%) create mode 100644 lambda_code/take_snapshots_rds/snapshots_tool_utils.py create mode 100644 rds_snapshot_tool_destination/.DS_Store rename README.md => rds_snapshot_tool_destination/README.md (84%) create mode 100644 rds_snapshot_tool_destination/main.tf create mode 100644 rds_snapshot_tool_destination/modules/.DS_Store create mode 100644 rds_snapshot_tool_destination/modules/rds_destination/cloudwatch.tf create mode 100644 rds_snapshot_tool_destination/modules/rds_destination/data.tf create mode 100644 rds_snapshot_tool_destination/modules/rds_destination/iam.tf create mode 100644 rds_snapshot_tool_destination/modules/rds_destination/lambda.tf create mode 100644 rds_snapshot_tool_destination/modules/rds_destination/locals.tf create mode 100644 rds_snapshot_tool_destination/modules/rds_destination/outputs.tf create mode 100644 rds_snapshot_tool_destination/modules/rds_destination/sns.tf create mode 100644 rds_snapshot_tool_destination/modules/rds_destination/stepfunction.tf create mode 100644 rds_snapshot_tool_destination/modules/rds_destination/variables.tf create mode 100644 rds_snapshot_tool_destination/provider.tf create mode 100644 rds_snapshot_tool_source/.DS_Store create mode 100644 rds_snapshot_tool_source/README.md create mode 100644 rds_snapshot_tool_source/main.tf create mode 100644 rds_snapshot_tool_source/modules/.DS_Store create mode 100644 rds_snapshot_tool_source/modules/rds_source/cloudwatch.tf create mode 100644 rds_snapshot_tool_source/modules/rds_source/data.tf create mode 100644 rds_snapshot_tool_source/modules/rds_source/iam.tf create mode 100644 rds_snapshot_tool_source/modules/rds_source/lambda.tf create mode 100644 rds_snapshot_tool_source/modules/rds_source/locals.tf create mode 100644 rds_snapshot_tool_source/modules/rds_source/output.tf create mode 100644 rds_snapshot_tool_source/modules/rds_source/sns.tf create mode 100644 rds_snapshot_tool_source/modules/rds_source/stepfunction.tf create mode 100644 rds_snapshot_tool_source/modules/rds_source/variable.tf create mode 100644 rds_snapshot_tool_source/modules/test_dbs/main.tf create mode 100644 rds_snapshot_tool_source/provider.tf diff --git a/LICENSE b/LICENSE deleted file mode 100644 index 8dada3e..0000000 --- a/LICENSE +++ /dev/null @@ -1,201 +0,0 @@ - Apache License - Version 2.0, January 2004 - http://www.apache.org/licenses/ - - TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION - - 1. Definitions. - - "License" shall mean the terms and conditions for use, reproduction, - and distribution as defined by Sections 1 through 9 of this document. - - "Licensor" shall mean the copyright owner or entity authorized by - the copyright owner that is granting the License. - - "Legal Entity" shall mean the union of the acting entity and all - other entities that control, are controlled by, or are under common - control with that entity. For the purposes of this definition, - "control" means (i) the power, direct or indirect, to cause the - direction or management of such entity, whether by contract or - otherwise, or (ii) ownership of fifty percent (50%) or more of the - outstanding shares, or (iii) beneficial ownership of such entity. - - "You" (or "Your") shall mean an individual or Legal Entity - exercising permissions granted by this License. - - "Source" form shall mean the preferred form for making modifications, - including but not limited to software source code, documentation - source, and configuration files. - - "Object" form shall mean any form resulting from mechanical - transformation or translation of a Source form, including but - not limited to compiled object code, generated documentation, - and conversions to other media types. - - "Work" shall mean the work of authorship, whether in Source or - Object form, made available under the License, as indicated by a - copyright notice that is included in or attached to the work - (an example is provided in the Appendix below). - - "Derivative Works" shall mean any work, whether in Source or Object - form, that is based on (or derived from) the Work and for which the - editorial revisions, annotations, elaborations, or other modifications - represent, as a whole, an original work of authorship. For the purposes - of this License, Derivative Works shall not include works that remain - separable from, or merely link (or bind by name) to the interfaces of, - the Work and Derivative Works thereof. - - "Contribution" shall mean any work of authorship, including - the original version of the Work and any modifications or additions - to that Work or Derivative Works thereof, that is intentionally - submitted to Licensor for inclusion in the Work by the copyright owner - or by an individual or Legal Entity authorized to submit on behalf of - the copyright owner. For the purposes of this definition, "submitted" - means any form of electronic, verbal, or written communication sent - to the Licensor or its representatives, including but not limited to - communication on electronic mailing lists, source code control systems, - and issue tracking systems that are managed by, or on behalf of, the - Licensor for the purpose of discussing and improving the Work, but - excluding communication that is conspicuously marked or otherwise - designated in writing by the copyright owner as "Not a Contribution." - - "Contributor" shall mean Licensor and any individual or Legal Entity - on behalf of whom a Contribution has been received by Licensor and - subsequently incorporated within the Work. - - 2. Grant of Copyright License. Subject to the terms and conditions of - this License, each Contributor hereby grants to You a perpetual, - worldwide, non-exclusive, no-charge, royalty-free, irrevocable - copyright license to reproduce, prepare Derivative Works of, - publicly display, publicly perform, sublicense, and distribute the - Work and such Derivative Works in Source or Object form. - - 3. Grant of Patent License. Subject to the terms and conditions of - this License, each Contributor hereby grants to You a perpetual, - worldwide, non-exclusive, no-charge, royalty-free, irrevocable - (except as stated in this section) patent license to make, have made, - use, offer to sell, sell, import, and otherwise transfer the Work, - where such license applies only to those patent claims licensable - by such Contributor that are necessarily infringed by their - Contribution(s) alone or by combination of their Contribution(s) - with the Work to which such Contribution(s) was submitted. If You - institute patent litigation against any entity (including a - cross-claim or counterclaim in a lawsuit) alleging that the Work - or a Contribution incorporated within the Work constitutes direct - or contributory patent infringement, then any patent licenses - granted to You under this License for that Work shall terminate - as of the date such litigation is filed. - - 4. Redistribution. You may reproduce and distribute copies of the - Work or Derivative Works thereof in any medium, with or without - modifications, and in Source or Object form, provided that You - meet the following conditions: - - (a) You must give any other recipients of the Work or - Derivative Works a copy of this License; and - - (b) You must cause any modified files to carry prominent notices - stating that You changed the files; and - - (c) You must retain, in the Source form of any Derivative Works - that You distribute, all copyright, patent, trademark, and - attribution notices from the Source form of the Work, - excluding those notices that do not pertain to any part of - the Derivative Works; and - - (d) If the Work includes a "NOTICE" text file as part of its - distribution, then any Derivative Works that You distribute must - include a readable copy of the attribution notices contained - within such NOTICE file, excluding those notices that do not - pertain to any part of the Derivative Works, in at least one - of the following places: within a NOTICE text file distributed - as part of the Derivative Works; within the Source form or - documentation, if provided along with the Derivative Works; or, - within a display generated by the Derivative Works, if and - wherever such third-party notices normally appear. The contents - of the NOTICE file are for informational purposes only and - do not modify the License. You may add Your own attribution - notices within Derivative Works that You distribute, alongside - or as an addendum to the NOTICE text from the Work, provided - that such additional attribution notices cannot be construed - as modifying the License. - - You may add Your own copyright statement to Your modifications and - may provide additional or different license terms and conditions - for use, reproduction, or distribution of Your modifications, or - for any such Derivative Works as a whole, provided Your use, - reproduction, and distribution of the Work otherwise complies with - the conditions stated in this License. - - 5. Submission of Contributions. Unless You explicitly state otherwise, - any Contribution intentionally submitted for inclusion in the Work - by You to the Licensor shall be under the terms and conditions of - this License, without any additional terms or conditions. - Notwithstanding the above, nothing herein shall supersede or modify - the terms of any separate license agreement you may have executed - with Licensor regarding such Contributions. - - 6. Trademarks. This License does not grant permission to use the trade - names, trademarks, service marks, or product names of the Licensor, - except as required for reasonable and customary use in describing the - origin of the Work and reproducing the content of the NOTICE file. - - 7. Disclaimer of Warranty. Unless required by applicable law or - agreed to in writing, Licensor provides the Work (and each - Contributor provides its Contributions) on an "AS IS" BASIS, - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or - implied, including, without limitation, any warranties or conditions - of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A - PARTICULAR PURPOSE. You are solely responsible for determining the - appropriateness of using or redistributing the Work and assume any - risks associated with Your exercise of permissions under this License. - - 8. Limitation of Liability. In no event and under no legal theory, - whether in tort (including negligence), contract, or otherwise, - unless required by applicable law (such as deliberate and grossly - negligent acts) or agreed to in writing, shall any Contributor be - liable to You for damages, including any direct, indirect, special, - incidental, or consequential damages of any character arising as a - result of this License or out of the use or inability to use the - Work (including but not limited to damages for loss of goodwill, - work stoppage, computer failure or malfunction, or any and all - other commercial damages or losses), even if such Contributor - has been advised of the possibility of such damages. - - 9. Accepting Warranty or Additional Liability. While redistributing - the Work or Derivative Works thereof, You may choose to offer, - and charge a fee for, acceptance of support, warranty, indemnity, - or other liability obligations and/or rights consistent with this - License. However, in accepting such obligations, You may act only - on Your own behalf and on Your sole responsibility, not on behalf - of any other Contributor, and only if You agree to indemnify, - defend, and hold each Contributor harmless for any liability - incurred by, or claims asserted against, such Contributor by reason - of your accepting any such warranty or additional liability. - - END OF TERMS AND CONDITIONS - - APPENDIX: How to apply the Apache License to your work. - - To apply the Apache License to your work, attach the following - boilerplate notice, with the fields enclosed by brackets "{}" - replaced with your own identifying information. (Don't include - the brackets!) The text should be enclosed in the appropriate - comment syntax for the file format. We also recommend that a - file or class name and description of purpose be included on the - same "printed page" as the copyright notice for easier - identification within third-party archives. - - Copyright {yyyy} {name of copyright owner} - - Licensed under the Apache License, Version 2.0 (the "License"); - you may not use this file except in compliance with the License. - You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - - Unless required by applicable law or agreed to in writing, software - distributed under the License is distributed on an "AS IS" BASIS, - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - See the License for the specific language governing permissions and - limitations under the License. diff --git a/LICENSE.txt b/LICENSE.txt deleted file mode 100644 index 8dada3e..0000000 --- a/LICENSE.txt +++ /dev/null @@ -1,201 +0,0 @@ - Apache License - Version 2.0, January 2004 - http://www.apache.org/licenses/ - - TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION - - 1. Definitions. - - "License" shall mean the terms and conditions for use, reproduction, - and distribution as defined by Sections 1 through 9 of this document. - - "Licensor" shall mean the copyright owner or entity authorized by - the copyright owner that is granting the License. - - "Legal Entity" shall mean the union of the acting entity and all - other entities that control, are controlled by, or are under common - control with that entity. For the purposes of this definition, - "control" means (i) the power, direct or indirect, to cause the - direction or management of such entity, whether by contract or - otherwise, or (ii) ownership of fifty percent (50%) or more of the - outstanding shares, or (iii) beneficial ownership of such entity. - - "You" (or "Your") shall mean an individual or Legal Entity - exercising permissions granted by this License. - - "Source" form shall mean the preferred form for making modifications, - including but not limited to software source code, documentation - source, and configuration files. - - "Object" form shall mean any form resulting from mechanical - transformation or translation of a Source form, including but - not limited to compiled object code, generated documentation, - and conversions to other media types. - - "Work" shall mean the work of authorship, whether in Source or - Object form, made available under the License, as indicated by a - copyright notice that is included in or attached to the work - (an example is provided in the Appendix below). - - "Derivative Works" shall mean any work, whether in Source or Object - form, that is based on (or derived from) the Work and for which the - editorial revisions, annotations, elaborations, or other modifications - represent, as a whole, an original work of authorship. For the purposes - of this License, Derivative Works shall not include works that remain - separable from, or merely link (or bind by name) to the interfaces of, - the Work and Derivative Works thereof. - - "Contribution" shall mean any work of authorship, including - the original version of the Work and any modifications or additions - to that Work or Derivative Works thereof, that is intentionally - submitted to Licensor for inclusion in the Work by the copyright owner - or by an individual or Legal Entity authorized to submit on behalf of - the copyright owner. For the purposes of this definition, "submitted" - means any form of electronic, verbal, or written communication sent - to the Licensor or its representatives, including but not limited to - communication on electronic mailing lists, source code control systems, - and issue tracking systems that are managed by, or on behalf of, the - Licensor for the purpose of discussing and improving the Work, but - excluding communication that is conspicuously marked or otherwise - designated in writing by the copyright owner as "Not a Contribution." - - "Contributor" shall mean Licensor and any individual or Legal Entity - on behalf of whom a Contribution has been received by Licensor and - subsequently incorporated within the Work. - - 2. Grant of Copyright License. Subject to the terms and conditions of - this License, each Contributor hereby grants to You a perpetual, - worldwide, non-exclusive, no-charge, royalty-free, irrevocable - copyright license to reproduce, prepare Derivative Works of, - publicly display, publicly perform, sublicense, and distribute the - Work and such Derivative Works in Source or Object form. - - 3. Grant of Patent License. Subject to the terms and conditions of - this License, each Contributor hereby grants to You a perpetual, - worldwide, non-exclusive, no-charge, royalty-free, irrevocable - (except as stated in this section) patent license to make, have made, - use, offer to sell, sell, import, and otherwise transfer the Work, - where such license applies only to those patent claims licensable - by such Contributor that are necessarily infringed by their - Contribution(s) alone or by combination of their Contribution(s) - with the Work to which such Contribution(s) was submitted. If You - institute patent litigation against any entity (including a - cross-claim or counterclaim in a lawsuit) alleging that the Work - or a Contribution incorporated within the Work constitutes direct - or contributory patent infringement, then any patent licenses - granted to You under this License for that Work shall terminate - as of the date such litigation is filed. - - 4. Redistribution. You may reproduce and distribute copies of the - Work or Derivative Works thereof in any medium, with or without - modifications, and in Source or Object form, provided that You - meet the following conditions: - - (a) You must give any other recipients of the Work or - Derivative Works a copy of this License; and - - (b) You must cause any modified files to carry prominent notices - stating that You changed the files; and - - (c) You must retain, in the Source form of any Derivative Works - that You distribute, all copyright, patent, trademark, and - attribution notices from the Source form of the Work, - excluding those notices that do not pertain to any part of - the Derivative Works; and - - (d) If the Work includes a "NOTICE" text file as part of its - distribution, then any Derivative Works that You distribute must - include a readable copy of the attribution notices contained - within such NOTICE file, excluding those notices that do not - pertain to any part of the Derivative Works, in at least one - of the following places: within a NOTICE text file distributed - as part of the Derivative Works; within the Source form or - documentation, if provided along with the Derivative Works; or, - within a display generated by the Derivative Works, if and - wherever such third-party notices normally appear. The contents - of the NOTICE file are for informational purposes only and - do not modify the License. You may add Your own attribution - notices within Derivative Works that You distribute, alongside - or as an addendum to the NOTICE text from the Work, provided - that such additional attribution notices cannot be construed - as modifying the License. - - You may add Your own copyright statement to Your modifications and - may provide additional or different license terms and conditions - for use, reproduction, or distribution of Your modifications, or - for any such Derivative Works as a whole, provided Your use, - reproduction, and distribution of the Work otherwise complies with - the conditions stated in this License. - - 5. Submission of Contributions. Unless You explicitly state otherwise, - any Contribution intentionally submitted for inclusion in the Work - by You to the Licensor shall be under the terms and conditions of - this License, without any additional terms or conditions. - Notwithstanding the above, nothing herein shall supersede or modify - the terms of any separate license agreement you may have executed - with Licensor regarding such Contributions. - - 6. Trademarks. This License does not grant permission to use the trade - names, trademarks, service marks, or product names of the Licensor, - except as required for reasonable and customary use in describing the - origin of the Work and reproducing the content of the NOTICE file. - - 7. Disclaimer of Warranty. Unless required by applicable law or - agreed to in writing, Licensor provides the Work (and each - Contributor provides its Contributions) on an "AS IS" BASIS, - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or - implied, including, without limitation, any warranties or conditions - of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A - PARTICULAR PURPOSE. You are solely responsible for determining the - appropriateness of using or redistributing the Work and assume any - risks associated with Your exercise of permissions under this License. - - 8. Limitation of Liability. In no event and under no legal theory, - whether in tort (including negligence), contract, or otherwise, - unless required by applicable law (such as deliberate and grossly - negligent acts) or agreed to in writing, shall any Contributor be - liable to You for damages, including any direct, indirect, special, - incidental, or consequential damages of any character arising as a - result of this License or out of the use or inability to use the - Work (including but not limited to damages for loss of goodwill, - work stoppage, computer failure or malfunction, or any and all - other commercial damages or losses), even if such Contributor - has been advised of the possibility of such damages. - - 9. Accepting Warranty or Additional Liability. While redistributing - the Work or Derivative Works thereof, You may choose to offer, - and charge a fee for, acceptance of support, warranty, indemnity, - or other liability obligations and/or rights consistent with this - License. However, in accepting such obligations, You may act only - on Your own behalf and on Your sole responsibility, not on behalf - of any other Contributor, and only if You agree to indemnify, - defend, and hold each Contributor harmless for any liability - incurred by, or claims asserted against, such Contributor by reason - of your accepting any such warranty or additional liability. - - END OF TERMS AND CONDITIONS - - APPENDIX: How to apply the Apache License to your work. - - To apply the Apache License to your work, attach the following - boilerplate notice, with the fields enclosed by brackets "{}" - replaced with your own identifying information. (Don't include - the brackets!) The text should be enclosed in the appropriate - comment syntax for the file format. We also recommend that a - file or class name and description of purpose be included on the - same "printed page" as the copyright notice for easier - identification within third-party archives. - - Copyright {yyyy} {name of copyright owner} - - Licensed under the Apache License, Version 2.0 (the "License"); - you may not use this file except in compliance with the License. - You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - - Unless required by applicable law or agreed to in writing, software - distributed under the License is distributed on an "AS IS" BASIS, - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - See the License for the specific language governing permissions and - limitations under the License. diff --git a/NOTICE b/NOTICE deleted file mode 100644 index 65146ef..0000000 --- a/NOTICE +++ /dev/null @@ -1,2 +0,0 @@ -RDS Snapshot Tool -Copyright 2017 Amazon.com, Inc. or its affiliates. All Rights Reserved. diff --git a/cftemplates/snapshots_tool_rds_dest.json b/cftemplates/snapshots_tool_rds_dest.json deleted file mode 100644 index 2a370e1..0000000 --- a/cftemplates/snapshots_tool_rds_dest.json +++ /dev/null @@ -1,588 +0,0 @@ -{ - "AWSTemplateFormatVersion": "2010-09-09", - "Parameters": { - "CodeBucket": { - "Type": "String", - "Description": "Name of the bucket that contains the lambda functions to deploy." - }, - "SnapshotPattern": { - "Type": "String", - "Default": "ALL_SNAPSHOTS", - "Description": "Python regex for matching instance names to backup. Use \"ALL_SNAPSHOTS\" to back up every RDS instance in the region." - }, - "RetentionDays": { - "Type": "Number", - "Default": "7", - "Description": "Number of days to keep snapshots in retention before deleting them" - }, - "DestinationRegion": { - "Type": "String", - "Description": "Destination region for snapshots." - }, - "LogLevel": { - "Type": "String", - "Default": "ERROR", - "Description": "Log level for Lambda functions (DEBUG, INFO, WARN, ERROR, CRITICAL are valid values)." - }, - "LambdaCWLogRetention": { - "Type": "Number", - "Default": "7", - "Description": "Number of days to retain logs from the lambda functions in CloudWatch Logs" - }, - "SourceRegionOverride": { - "Type": "String", - "Default": "NO", - "Description": "Set to the region where your RDS instances run, only if such region does not support Step Functions. Leave as NO otherwise" - }, - "KmsKeyDestination": { - "Type": "String", - "Default": "None", - "Description": "Set to the ARN for the KMS key in the destination region to re-encrypt encrypted snapshots. Leave None if you are not using encryption" - }, - "KmsKeySource": { - "Type": "String", - "Default": "None", - "Description": "Set to the ARN for the KMS key in the SOURCE region to re-encrypt encrypted snapshots. Leave None if you are not using encryption" - }, - "DeleteOldSnapshots": { - "Type": "String", - "Default": "TRUE", - "Description": "Set to TRUE to enable deletion of snapshot based on RetentionDays. Set to FALSE to disable", - "AllowedValues": ["TRUE", "FALSE"] - }, - "CrossAccountCopy": { - "Type": "String", - "AllowedValues": ["TRUE", "FALSE"], - "Default": "TRUE", - "Description": "Enable copying snapshots across accounts. Set to FALSE if your source snapshosts are not on a different account" - }, - "LogGroupName": { - "Type": "String", - "Default": "lambdaDeleteOldSnapshotsRDS-dest", - "Description": "Name for RDS snapshot log group." - } - }, - "Conditions": { - "DeleteOld": { - "Fn::Equals": [{ - "Ref": "DeleteOldSnapshots" - }, "TRUE"] - }, - "CrossAccount": { - "Fn::Equals": [{ - "Ref": "CrossAccountCopy" - }, "TRUE" ] - } - }, - "Resources": { - "topicCopyFailedDest": { - "Type": "AWS::SNS::Topic", - "Properties": { - "DisplayName": "copies_failed_dest_rds" - } - }, - "topicDeleteOldFailedDest": { - "Type": "AWS::SNS::Topic", - "Properties": { - "DisplayName": "delete_old_failed_dest_rds" - } - }, - "snspolicyCopyFailedDest": { - "Type": "AWS::SNS::TopicPolicy", - "Properties": { - "Topics": [{ - "Ref": "topicCopyFailedDest" - }, { - "Ref": "topicDeleteOldFailedDest" - }], - "PolicyDocument": { - "Version": "2008-10-17", - "Id": "__default_policy_ID", - "Statement": [{ - "Sid": "__default_statement_ID", - "Effect": "Allow", - "Principal": { - "AWS": "*" - }, - "Action": [ - "SNS:GetTopicAttributes", - "SNS:SetTopicAttributes", - "SNS:AddPermission", - "SNS:RemovePermission", - "SNS:DeleteTopic", - "SNS:Subscribe", - "SNS:ListSubscriptionsByTopic", - "SNS:Publish", - "SNS:Receive" - ], - "Resource": "*", - "Condition": { - "StringEquals": { - "AWS:SourceOwner": { - "Ref": "AWS::AccountId" - } - } - } - }] - } - } - }, - "alarmcwCopyFailedDest": { - "Type": "AWS::CloudWatch::Alarm", - "Properties": { - "ActionsEnabled": "true", - "ComparisonOperator": "GreaterThanOrEqualToThreshold", - "EvaluationPeriods": "1", - "MetricName": "ExecutionsFailed", - "Namespace": "AWS/States", - "Period": "300", - "Statistic": "Sum", - "Threshold": "1.0", - "AlarmActions": [{ - "Ref": "topicCopyFailedDest" - }], - "Dimensions": [{ - "Name": "StateMachineArn", - "Value": { - "Ref": "statemachineCopySnapshotsDestRDS" - } - }] - } - }, - "alarmcwDeleteOldFailedDest": { - "Type": "AWS::CloudWatch::Alarm", - "Condition": "DeleteOld", - "Properties": { - "ActionsEnabled": "true", - "ComparisonOperator": "GreaterThanOrEqualToThreshold", - "EvaluationPeriods": "2", - "MetricName": "ExecutionsFailed", - "Namespace": "AWS/States", - "Period": "3600", - "Statistic": "Sum", - "Threshold": "2.0", - "AlarmActions": [{ - "Ref": "topicDeleteOldFailedDest" - }], - "Dimensions": [{ - "Name": "StateMachineArn", - "Value": { - "Ref": "statemachineDeleteOldSnapshotsDestRDS" - } - }] - } - }, - "iamroleSnapshotsRDS": { - "Type": "AWS::IAM::Role", - "Properties": { - "AssumeRolePolicyDocument": { - "Version": "2012-10-17", - "Statement": [{ - "Effect": "Allow", - "Principal": { - "Service": "lambda.amazonaws.com" - }, - "Action": "sts:AssumeRole" - }] - }, - "Policies": [{ - "PolicyName": "inline_policy_snapshots_rds_cw_logs", - "PolicyDocument": { - "Version": "2012-10-17", - "Statement": [{ - "Effect": "Allow", - "Action": [ - "logs:CreateLogGroup", - "logs:CreateLogStream", - "logs:PutLogEvents" - ], - "Resource": "arn:aws:logs:*:*:*" - }] - } - }, - { - "PolicyName": "inline_policy_snapshots_rds", - "PolicyDocument": { - "Version": "2012-10-17", - "Statement": [{ - "Effect": "Allow", - "Action": [ - "rds:CreateDBSnapshot", - "rds:DeleteDBSnapshot", - "rds:DescribeDBInstances", - "rds:DescribeDBSnapshots", - "rds:ModifyDBSnapshotAttribute", - "rds:DescribeDBSnapshotAttributes", - "rds:CopyDBSnapshot", - "rds:ListTagsForResource", - "rds:AddTagsToResource" - ], - "Resource": "*" - }] - } - - }, - { - "PolicyName": "inline_policy_snapshot_rds_kms_access", - "PolicyDocument": { - "Version": "2012-10-17", - "Statement": [{ - "Sid": "AllowUseOfTheKey", - "Effect": "Allow", - "Action": [ - "kms:Encrypt", - "kms:Decrypt", - "kms:ReEncrypt*", - "kms:GenerateDataKey*", - "kms:DescribeKey" - ], - "Resource": [ - "*" - ] - }, - { - "Sid": "AllowAttachmentOfPersistentResources", - "Effect": "Allow", - "Action": [ - "kms:CreateGrant", - "kms:ListGrants", - "kms:RevokeGrant" - ], - "Resource": [ - "*" - ], - "Condition": { - "Bool": { - "kms:GrantIsForAWSResource": true - } - } - } - ] - } - } - ] - } - }, - "lambdaCopySnapshotsRDS": { - "Type": "AWS::Lambda::Function", - "Properties": { - "Code": { - "S3Bucket": { - "Ref": "CodeBucket" - }, - "S3Key": { "Fn::If" : [ "CrossAccount", "copy_snapshots_dest_rds.zip", "copy_snapshots_no_x_account_rds.zip" ]} - }, - "MemorySize" : 512, - "Description": "This functions copies snapshots for RDS Instances shared with this account. It checks for existing snapshots following the pattern specified in the environment variables with the following format: -YYYY-MM-DD-HH-MM", - "Environment": { - "Variables": { - "SNAPSHOT_PATTERN": { - "Ref": "SnapshotPattern" - }, - "DEST_REGION": { - "Ref": "DestinationRegion" - }, - "LOG_LEVEL": { - "Ref": "LogLevel" - }, - "REGION_OVERRIDE": { - "Ref": "SourceRegionOverride" - }, - "KMS_KEY_DEST_REGION": { - "Ref": "KmsKeyDestination" - }, - "KMS_KEY_SOURCE_REGION": { - "Ref": "KmsKeySource" - }, - "RETENTION_DAYS": { - "Ref": "RetentionDays" - } - } - }, - "Role": { - "Fn::GetAtt": ["iamroleSnapshotsRDS", "Arn"] - }, - "Runtime": "python3.7", - "Handler": "lambda_function.lambda_handler", - "Timeout": 300 - } - }, - "lambdaDeleteOldDestRDS": { - "Type": "AWS::Lambda::Function", - "Condition": "DeleteOld", - "Properties": { - "Code": { - "S3Bucket": { - "Ref": "CodeBucket" - }, - "S3Key": {"Fn::If" : [ "CrossAccount", "delete_old_snapshots_dest_rds.zip", "delete_old_snapshots_no_x_account_rds.zip" ]} - }, - "MemorySize" : 512, - "Description": "This function enforces retention on the snapshots shared with the destination account. ", - "Environment": { - "Variables": { - "SNAPSHOT_PATTERN": { - "Ref": "SnapshotPattern" - }, - "DEST_REGION": { - "Ref": "DestinationRegion" - }, - "RETENTION_DAYS": { - "Ref": "RetentionDays" - }, - "LOG_LEVEL": { - "Ref": "LogLevel" - } - } - }, - "Role": { - "Fn::GetAtt": ["iamroleSnapshotsRDS", "Arn"] - }, - "Runtime": "python3.7", - "Handler": "lambda_function.lambda_handler", - "Timeout": 300 - } - }, - "iamroleStateExecution": { - "Type": "AWS::IAM::Role", - "Properties": { - "AssumeRolePolicyDocument": { - "Version": "2012-10-17", - "Statement": [{ - "Effect": "Allow", - "Principal": { - "Service": { - "Fn::Join": ["", ["states.", { - "Ref": "AWS::Region" - }, ".amazonaws.com"]] - } - }, - "Action": "sts:AssumeRole" - }] - }, - "Policies": [{ - "PolicyName": "inline_policy_rds_snapshot", - "PolicyDocument": { - "Version": "2012-10-17", - "Statement": [{ - "Effect": "Allow", - "Action": [ - "lambda:InvokeFunction" - ], - "Resource": "*" - }] - } - }] - } - }, - "statemachineCopySnapshotsDestRDS": { - "Type": "AWS::StepFunctions::StateMachine", - "Properties": { - "DefinitionString": { - "Fn::Join": ["", [{ - "Fn::Join": ["\n", [ - " {\"Comment\":\"Copies snapshots locally and then to DEST_REGION\",", - " \"StartAt\":\"CopySnapshots\",", - " \"States\":{", - " \"CopySnapshots\":{", - " \"Type\":\"Task\",", - " \"Resource\": " - ]] - }, - "\"", - { - "Fn::GetAtt": ["lambdaCopySnapshotsRDS", "Arn"] - }, "\"\n,", - { - "Fn::Join": ["\n", [ - " \"Retry\":[", - " {", - " \"ErrorEquals\":[ ", - " \"SnapshotToolException\"", - " ],", - " \"IntervalSeconds\":300,", - " \"MaxAttempts\":5,", - " \"BackoffRate\":1", - " },", - " {", - " \"ErrorEquals\":[ ", - " \"States.ALL\"], ", - " \"IntervalSeconds\": 30,", - " \"MaxAttempts\": 20,", - " \"BackoffRate\": 1", - " }", - " ],", - " \"End\": true ", - " }", - " }}" - ]] - } - ]] - }, - "RoleArn": { - "Fn::GetAtt": ["iamroleStateExecution", "Arn"] - } - } - }, - "statemachineDeleteOldSnapshotsDestRDS": { - "Type": "AWS::StepFunctions::StateMachine", - "Condition": "DeleteOld", - "Properties": { - "DefinitionString": { - "Fn::Join": ["", [{ - "Fn::Join": ["\n", [ - " {\"Comment\":\"DeleteOld for RDS snapshots in destination region\",", - " \"StartAt\":\"DeleteOldDestRegion\",", - " \"States\":{", - " \"DeleteOldDestRegion\":{", - " \"Type\":\"Task\",", - " \"Resource\": " - ]] - }, - "\"", - { - "Fn::GetAtt": ["lambdaDeleteOldDestRDS", "Arn"] - }, "\"\n,", - { - "Fn::Join": ["\n", [ - " \"Retry\":[", - " {", - " \"ErrorEquals\":[ ", - " \"SnapshotToolException\"", - " ],", - " \"IntervalSeconds\":600,", - " \"MaxAttempts\":5,", - " \"BackoffRate\":1", - " },", - " {", - " \"ErrorEquals\":[ ", - " \"States.ALL\"], ", - " \"IntervalSeconds\": 30,", - " \"MaxAttempts\": 20,", - " \"BackoffRate\": 1", - " }", - " ],", - " \"End\": true ", - " }", - " }}" - ]] - } - ]] - }, - "RoleArn": { - "Fn::GetAtt": ["iamroleStateExecution", "Arn"] - } - } - }, - "iamroleStepInvocation": { - "Type": "AWS::IAM::Role", - "Properties": { - "AssumeRolePolicyDocument": { - "Version": "2012-10-17", - "Statement": [{ - "Effect": "Allow", - "Principal": { - "Service": "events.amazonaws.com" - }, - "Action": "sts:AssumeRole" - }] - }, - "Policies": [{ - "PolicyName": "inline_policy_state_invocation", - "PolicyDocument": { - "Version": "2012-10-17", - "Statement": [{ - "Effect": "Allow", - "Action": [ - "states:StartExecution" - ], - "Resource": "*" - }] - } - }] - } - }, - "cwEventCopySnapshotsRDS": { - "Type": "AWS::Events::Rule", - "Properties": { - "Description": "Triggers the RDS Copy state machine in the destination account", - "ScheduleExpression": { - "Fn::Join": ["", ["cron(", "/30 * * * ? *", ")"]] - }, - "State": "ENABLED", - "Targets": [{ - "Arn": { - "Ref": "statemachineCopySnapshotsDestRDS" - }, - "Id": "Target1", - "RoleArn": { - "Fn::GetAtt": ["iamroleStepInvocation", "Arn"] - } - }] - } - }, - "cwEventDeleteOldSnapshotsRDS": { - "Type": "AWS::Events::Rule", - "Condition": "DeleteOld", - "Properties": { - "Description": "Triggers the RDS DeleteOld state machine in the destination account", - "ScheduleExpression": { - "Fn::Join": ["", ["cron(", "0 /1 * * ? *", ")"]] - }, - "State": "ENABLED", - "Targets": [{ - "Arn": { - "Ref": "statemachineDeleteOldSnapshotsDestRDS" - }, - "Id": "Target1", - "RoleArn": { - "Fn::GetAtt": ["iamroleStepInvocation", "Arn"] - } - }] - } - }, - "cwloggroupDeleteOldSnapshotsDestRDS":{ - "Type": "AWS::Logs::LogGroup", - "Description": "Log group for the lambdaCopySnapshotsRDS function's logs", - "Condition": "DeleteOld", - "DependsOn": "lambdaDeleteOldDestRDS", - "Properties": { - "RetentionInDays": { "Ref": "LambdaCWLogRetention" }, - "LogGroupName": { - "Fn::Sub": [ "/aws/lambda/${func}", { "func": { "Ref" : "LogGroupName" } } ] - } - } - }, - "cwloggrouplambdaCopySnapshotsRDS":{ - "Type": "AWS::Logs::LogGroup", - "Description": "Log group for the lambdaCopySnapshotsRDS function's logs", - "DependsOn": "lambdaCopySnapshotsRDS", - "Properties": { - "RetentionInDays": { "Ref": "LambdaCWLogRetention" }, - "LogGroupName": { - "Fn::Sub": [ "/aws/lambda/${func}", { "func": { "Ref" : "lambdaCopySnapshotsRDS" } } ] - } - } - } - }, - "Outputs": { - "CopyFailedTopic": { - "Description": "Subscribe to this topic to receive alerts of failed copies", - "Value": { - "Ref": "topicCopyFailedDest" - } - }, - "DeleteOldFailedTopic": { - "Condition": "DeleteOld", - "Description": "Subscribe to this topic to receive alerts of failures at deleting old snapshots", - "Value": { - "Ref": "topicDeleteOldFailedDest" - } - }, - "SourceURL": { - "Description": "For more information and documentation, see the source repository at GitHub.", - "Value": "https://github.com/awslabs/rds-snapshot-tool" - } - }, - "Description": "Snapshots Tool for RDS cross-region and cross-account (destination account stack)" -} diff --git a/cftemplates/snapshots_tool_rds_source.json b/cftemplates/snapshots_tool_rds_source.json deleted file mode 100644 index a0a08f3..0000000 --- a/cftemplates/snapshots_tool_rds_source.json +++ /dev/null @@ -1,707 +0,0 @@ -{ - "AWSTemplateFormatVersion": "2010-09-09", - "Parameters": { - "CodeBucket": { - "Type": "String", - "Description": "Name of the bucket that contains the lambda functions to deploy." - }, - "InstanceNamePattern": { - "Type": "String", - "Default": "ALL_INSTANCES", - "Description": "Python regex for matching cluster identifiers to backup. Use \"ALL_INSTANCES\" to back up every RDS instance in the region." - }, - "BackupInterval": { - "Type": "Number", - "Default": "24", - "Description": "Interval for backups in hours. Default is 24" - }, - "DestinationAccount": { - "Type": "Number", - "Default": "000000000000", - "Description": "Destination account with no dashes." - }, - "ShareSnapshots": { - "Type": "String", - "Default": "TRUE", - "AllowedValues": ["TRUE", "FALSE"] - }, - "BackupSchedule": { - "Type": "String", - "Default": "0 1 * * ? *", - "Description": "Backup schedule in Cloudwatch Event cron format. Needs to run at least once for every Interval. The default value runs once every at 1AM UTC. More information: http://docs.aws.amazon.com/AmazonCloudWatch/latest/events/ScheduledEvents.html" - }, - "RetentionDays": { - "Type": "Number", - "Default": "7", - "Description": "Number of days to keep snapshots in retention before deleting them" - }, - "LogLevel": { - "Type": "String", - "Default": "ERROR", - "Description": "Log level for Lambda functions (DEBUG, INFO, WARN, ERROR, CRITICAL are valid values)." - }, - "LambdaCWLogRetention": { - "Type": "Number", - "Default": "7", - "Description": "Number of days to retain logs from the lambda functions in CloudWatch Logs" - }, - "SourceRegionOverride": { - "Type": "String", - "Default": "NO", - "Description": "Set to the region where your RDS instances run, only if such region does not support Step Functions. Leave as NO otherwise" - }, - "DeleteOldSnapshots": { - "Type": "String", - "Default": "TRUE", - "Description": "Set to TRUE to enable deletion of snapshot based on RetentionDays. Set to FALSE to disable", - "AllowedValues": ["TRUE", "FALSE"] - }, - "TaggedInstance": { - "Type": "String", - "Default": "FALSE", - "Description": "Set to TRUE to filter instances that have tag CopyDBSnapshot set to True. Set to FALSE to disable", - "AllowedValues": ["TRUE", "FALSE"] - }, - "LogGroupName": { - "Type": "String", - "Default": "lambdaDeleteOldSnapshotsRDS-source", - "Description": "Name for RDS snapshot log group." - } - }, - "Conditions": { - "Share": { - "Fn::Equals": [{ - "Ref": "ShareSnapshots" - }, "TRUE"] - }, - "DeleteOld": { - "Fn::Equals": [{ - "Ref": "DeleteOldSnapshots" - }, "TRUE"] - } - }, - "Resources": { - "topicBackupsFailed": { - "Type": "AWS::SNS::Topic", - "Properties": { - "DisplayName": "backups_failed_rds" - } - }, - "topicShareFailed": { - "Type": "AWS::SNS::Topic", - "Properties": { - "DisplayName": "share_failed_rds" - } - }, - "topicDeleteOldFailed": { - "Type": "AWS::SNS::Topic", - "Properties": { - "DisplayName": "delete_old_failed_rds" - } - }, - "snspolicySnapshotsRDS": { - "Type": "AWS::SNS::TopicPolicy", - "Properties": { - "Topics": [{ - "Ref": "topicBackupsFailed" - }, - { - "Ref": "topicShareFailed" - }, { - "Ref": "topicDeleteOldFailed" - } - ], - "PolicyDocument": { - "Version": "2008-10-17", - "Id": "__default_policy_ID", - "Statement": [{ - "Sid": "__default_statement_ID", - "Effect": "Allow", - "Principal": { - "AWS": "*" - }, - "Action": [ - "SNS:GetTopicAttributes", - "SNS:SetTopicAttributes", - "SNS:AddPermission", - "SNS:RemovePermission", - "SNS:DeleteTopic", - "SNS:Subscribe", - "SNS:ListSubscriptionsByTopic", - "SNS:Publish", - "SNS:Receive" - ], - "Resource": "*", - "Condition": { - "StringEquals": { - "AWS:SourceOwner": { - "Ref": "AWS::AccountId" - } - } - } - }] - } - } - }, - "alarmcwBackupsFailed": { - "Type": "AWS::CloudWatch::Alarm", - "Properties": { - "ActionsEnabled": "true", - "ComparisonOperator": "GreaterThanOrEqualToThreshold", - "EvaluationPeriods": "1", - "MetricName": "ExecutionsFailed", - "Namespace": "AWS/States", - "Period": "300", - "Statistic": "Sum", - "Threshold": "1.0", - "AlarmActions": [{ - "Ref": "topicBackupsFailed" - }], - "Dimensions": [{ - "Name": "StateMachineArn", - "Value": { - "Ref": "stateMachineTakeSnapshotsRDS" - } - }] - } - }, - "alarmcwShareFailed": { - "Condition": "Share", - "Type": "AWS::CloudWatch::Alarm", - "Properties": { - "ActionsEnabled": "true", - "ComparisonOperator": "GreaterThanOrEqualToThreshold", - "EvaluationPeriods": "2", - "MetricName": "ExecutionsFailed", - "Namespace": "AWS/States", - "Period": "3600", - "Statistic": "Sum", - "Threshold": "2.0", - "AlarmActions": [{ - "Ref": "topicShareFailed" - }], - "Dimensions": [{ - "Name": "StateMachineArn", - "Value": { - "Ref": "statemachineShareSnapshotsRDS" - } - }] - } - }, - "alarmcwDeleteOldFailed": { - "Condition": "DeleteOld", - "Type": "AWS::CloudWatch::Alarm", - "Properties": { - "ActionsEnabled": "true", - "ComparisonOperator": "GreaterThanOrEqualToThreshold", - "EvaluationPeriods": "2", - "MetricName": "ExecutionsFailed", - "Namespace": "AWS/States", - "Period": "3600", - "Statistic": "Sum", - "Threshold": "2.0", - "AlarmActions": [{ - "Ref": "topicDeleteOldFailed" - }], - "Dimensions": [{ - "Name": "StateMachineArn", - "Value": { - "Ref": "statemachineDeleteOldSnapshotsRDS" - } - }] - } - }, - "iamroleSnapshotsRDS": { - "Type": "AWS::IAM::Role", - "Properties": { - "AssumeRolePolicyDocument": { - "Version": "2012-10-17", - "Statement": [{ - "Effect": "Allow", - "Principal": { - "Service": "lambda.amazonaws.com" - }, - "Action": "sts:AssumeRole" - }] - }, - "Policies": [{ - "PolicyName": "inline_policy_snapshots_rds_cw_logs", - "PolicyDocument": { - "Version": "2012-10-17", - "Statement": [{ - "Effect": "Allow", - "Action": [ - "logs:CreateLogGroup", - "logs:CreateLogStream", - "logs:PutLogEvents" - ], - "Resource": "arn:aws:logs:*:*:*" - }] - } - }, - { - "PolicyName": "inline_policy_snapshots_rds", - "PolicyDocument": { - "Version": "2012-10-17", - "Statement": [{ - "Effect": "Allow", - "Action": [ - "rds:CreateDBSnapshot", - "rds:DeleteDBSnapshot", - "rds:DescribeDBInstances", - "rds:DescribeDBSnapshots", - "rds:ModifyDBSnapshotAttribute", - "rds:DescribeDBSnapshotAttributes", - "rds:ListTagsForResource", - "rds:AddTagsToResource" - ], - "Resource": "*" - }] - } - - } - ] - } - }, - "lambdaTakeSnapshotsRDS": { - "Type": "AWS::Lambda::Function", - "Properties": { - "Code": { - "S3Bucket": { - "Ref": "CodeBucket" - }, - "S3Key": "take_snapshots_rds.zip" - }, - "MemorySize" : 512, - "Description": "This functions triggers snapshots creation for RDS instances. It checks for existing snapshots following the pattern and interval specified in the environment variables with the following format: -YYYY-MM-DD-HH-MM", - "Environment": { - "Variables": { - "INTERVAL": { - "Ref": "BackupInterval" - }, - "PATTERN": { - "Ref": "InstanceNamePattern" - }, - "LOG_LEVEL": { - "Ref": "LogLevel" - }, - "REGION_OVERRIDE": { - "Ref": "SourceRegionOverride" - }, - "TAGGEDINSTANCE": { - "Ref": "TaggedInstance" - } - } - }, - "Role": { - "Fn::GetAtt": ["iamroleSnapshotsRDS", "Arn"] - }, - "Runtime": "python3.7", - "Handler": "lambda_function.lambda_handler", - "Timeout": 300 - } - }, - "lambdaShareSnapshotsRDS": { - "Type": "AWS::Lambda::Function", - "Condition": "Share", - "Properties": { - "Code": { - "S3Bucket": { - "Ref": "CodeBucket" - }, - "S3Key": "share_snapshots_rds.zip" - }, - "MemorySize" : 512, - "Description": "This function shares snapshots created by the take_snapshots_rds function with DEST_ACCOUNT specified in the environment variables. ", - "Environment": { - "Variables": { - "DEST_ACCOUNT": { - "Ref": "DestinationAccount" - }, - "LOG_LEVEL": { - "Ref": "LogLevel" - }, - "PATTERN": { - "Ref": "InstanceNamePattern" - }, - "REGION_OVERRIDE": { - "Ref": "SourceRegionOverride" - } - } - }, - "Role": { - "Fn::GetAtt": ["iamroleSnapshotsRDS", "Arn"] - }, - "Runtime": "python3.7", - "Handler": "lambda_function.lambda_handler", - "Timeout": 300 - } - }, - "lambdaDeleteOldSnapshotsRDS": { - "Type": "AWS::Lambda::Function", - "Condition": "DeleteOld", - "Properties": { - "Code": { - "S3Bucket": { - "Ref": "CodeBucket" - }, - "S3Key": "delete_old_snapshots_rds.zip" - }, - "MemorySize" : 512, - "Description": "This function deletes snapshots created by the take_snapshots_rds function. ", - "Environment": { - "Variables": { - "RETENTION_DAYS": { - "Ref": "RetentionDays" - }, - "PATTERN": { - "Ref": "InstanceNamePattern" - }, - "LOG_LEVEL": { - "Ref": "LogLevel" - }, - "REGION_OVERRIDE": { - "Ref": "SourceRegionOverride" - } - } - }, - "Role": { - "Fn::GetAtt": ["iamroleSnapshotsRDS", "Arn"] - }, - "Runtime": "python3.7", - "Handler": "lambda_function.lambda_handler", - "Timeout": 300 - } - }, - "iamroleStateExecution": { - "Type": "AWS::IAM::Role", - "Properties": { - "AssumeRolePolicyDocument": { - "Version": "2012-10-17", - "Statement": [{ - "Effect": "Allow", - "Principal": { - "Service": { - "Fn::Join": ["", ["states.", { - "Ref": "AWS::Region" - }, ".amazonaws.com"]] - } - }, - "Action": "sts:AssumeRole" - }] - }, - "Policies": [{ - "PolicyName": "inline_policy_snapshots_rds", - "PolicyDocument": { - "Version": "2012-10-17", - "Statement": [{ - "Effect": "Allow", - "Action": [ - "lambda:InvokeFunction" - ], - "Resource": "*" - }] - } - }] - } - }, - "stateMachineTakeSnapshotsRDS": { - "Type": "AWS::StepFunctions::StateMachine", - "Properties": { - "DefinitionString": { - "Fn::Join": ["", [{ - "Fn::Join": ["\n", [ - " {\"Comment\":\"Triggers snapshot backup for RDS instances\",", - " \"StartAt\":\"TakeSnapshots\",", - " \"States\":{", - " \"TakeSnapshots\":{", - " \"Type\":\"Task\",", - " \"Resource\": " - ]] - }, - "\"", - { - "Fn::GetAtt": ["lambdaTakeSnapshotsRDS", "Arn"] - }, "\"\n,", - { - "Fn::Join": ["\n", [ - " \"Retry\":[", - " {", - " \"ErrorEquals\":[ ", - " \"SnapshotToolException\"", - " ],", - " \"IntervalSeconds\":300,", - " \"MaxAttempts\":20,", - " \"BackoffRate\":1", - " },", - " {", - " \"ErrorEquals\":[ ", - " \"States.ALL\"], ", - " \"IntervalSeconds\": 30,", - " \"MaxAttempts\": 20,", - " \"BackoffRate\": 1", - " }", - " ],", - " \"End\": true ", - " }", - " }}" - ]] - } - ]] - }, - "RoleArn": { - "Fn::GetAtt": ["iamroleStateExecution", "Arn"] - } - } - }, - "statemachineShareSnapshotsRDS": { - "Type": "AWS::StepFunctions::StateMachine", - "Condition": "Share", - "Properties": { - "DefinitionString": { - "Fn::Join": ["", [{ - "Fn::Join": ["\n", [ - " {\"Comment\":\"Shares snapshots with DEST_ACCOUNT\",", - " \"StartAt\":\"ShareSnapshots\",", - " \"States\":{", - " \"ShareSnapshots\":{", - " \"Type\":\"Task\",", - " \"Resource\": " - ]] - }, - "\"", - { - "Fn::GetAtt": ["lambdaShareSnapshotsRDS", "Arn"] - }, "\"\n,", - { - "Fn::Join": ["\n", [ - " \"Retry\":[", - " {", - " \"ErrorEquals\":[ ", - " \"SnapshotToolException\"", - " ],", - " \"IntervalSeconds\":300,", - " \"MaxAttempts\":3,", - " \"BackoffRate\":1", - " },", - " {", - " \"ErrorEquals\":[ ", - " \"States.ALL\"], ", - " \"IntervalSeconds\": 30,", - " \"MaxAttempts\": 20,", - " \"BackoffRate\": 1", - " }", - " ],", - " \"End\": true ", - " }", - " }}" - ]] - } - ]] - }, - "RoleArn": { - "Fn::GetAtt": ["iamroleStateExecution", "Arn"] - } - } - }, - "statemachineDeleteOldSnapshotsRDS": { - "Type": "AWS::StepFunctions::StateMachine", - "Condition": "DeleteOld", - "Properties": { - "DefinitionString": { - "Fn::Join": ["", [{ - "Fn::Join": ["\n", [ - " {\"Comment\":\"DeleteOld management for RDS snapshots\",", - " \"StartAt\":\"DeleteOld\",", - " \"States\":{", - " \"DeleteOld\":{", - " \"Type\":\"Task\",", - " \"Resource\": " - ]] - }, - "\"", - { - "Fn::GetAtt": ["lambdaDeleteOldSnapshotsRDS", "Arn"] - }, "\"\n,", - { - "Fn::Join": ["\n", [ - " \"Retry\":[", - " {", - " \"ErrorEquals\":[ ", - " \"SnapshotToolException\"", - " ],", - " \"IntervalSeconds\":300,", - " \"MaxAttempts\":7,", - " \"BackoffRate\":1", - " },", - " {", - " \"ErrorEquals\":[ ", - " \"States.ALL\"], ", - " \"IntervalSeconds\": 30,", - " \"MaxAttempts\": 20,", - " \"BackoffRate\": 1", - " }", - " ],", - " \"End\": true ", - " }", - " }}" - ]] - } - ]] - }, - "RoleArn": { - "Fn::GetAtt": ["iamroleStateExecution", "Arn"] - } - } - }, - "iamroleStepInvocation": { - "Type": "AWS::IAM::Role", - "Properties": { - "AssumeRolePolicyDocument": { - "Version": "2012-10-17", - "Statement": [{ - "Effect": "Allow", - "Principal": { - "Service": "events.amazonaws.com" - }, - "Action": "sts:AssumeRole" - }] - }, - "Policies": [{ - "PolicyName": "inline_policy_state_invocation", - "PolicyDocument": { - "Version": "2012-10-17", - "Statement": [{ - "Effect": "Allow", - "Action": [ - "states:StartExecution" - ], - "Resource": "*" - }] - } - }] - } - }, - "cwEventBackupRDS": { - "Type": "AWS::Events::Rule", - "Properties": { - "Description": "Triggers the TakeSnapshotsRDS state machine", - "ScheduleExpression": { - "Fn::Join": ["", ["cron(", { - "Ref": "BackupSchedule" - }, ")"]] - }, - "State": "ENABLED", - "Targets": [{ - "Arn": { - "Ref": "stateMachineTakeSnapshotsRDS" - }, - "Id": "Target1", - "RoleArn": { - "Fn::GetAtt": ["iamroleStepInvocation", "Arn"] - } - }] - } - }, - "cwEventShareSnapshotsRDS": { - "Type": "AWS::Events::Rule", - "Condition": "Share", - "Properties": { - "Description": "Triggers the ShareSnapshotsRDS state machine", - "ScheduleExpression": { - "Fn::Join": ["", ["cron(", "/10 * * * ? *", ")"]] - }, - "State": "ENABLED", - "Targets": [{ - "Arn": { - "Ref": "statemachineShareSnapshotsRDS" - }, - "Id": "Target1", - "RoleArn": { - "Fn::GetAtt": ["iamroleStepInvocation", "Arn"] - } - }] - } - }, - "cwEventDeleteOldSnapshotsRDS": { - "Type": "AWS::Events::Rule", - "Condition": "DeleteOld", - "Properties": { - "Description": "Triggers the DeleteOldSnapshotsRDS state machine", - "ScheduleExpression": { - "Fn::Join": ["", ["cron(", "0 /1 * * ? *", ")"]] - }, - "State": "ENABLED", - "Targets": [{ - "Arn": { - "Ref": "statemachineDeleteOldSnapshotsRDS" - }, - "Id": "Target1", - "RoleArn": { - "Fn::GetAtt": ["iamroleStepInvocation", "Arn"] - } - }] - } - }, - "cwloggrouplambdaTakeSnapshotsRDS":{ - "Type": "AWS::Logs::LogGroup", - "Description": "Log group for the lambdaTakeSnapshotsRDS function's logs", - "DependsOn": "lambdaTakeSnapshotsRDS", - "Properties": { - "RetentionInDays": { "Ref": "LambdaCWLogRetention" }, - "LogGroupName": { - "Fn::Sub": [ "/aws/lambda/${func}", { "func": { "Ref" : "lambdaTakeSnapshotsRDS" } } ] - } - } - }, - "cwloggrouplambdaShareSnapshotsRDS":{ - "Condition": "Share", - "Type": "AWS::Logs::LogGroup", - "Description": "Log group for the lambdaShareSnapshotsRDS function's logs", - "DependsOn": "lambdaShareSnapshotsRDS", - "Properties": { - "RetentionInDays": { "Ref": "LambdaCWLogRetention" }, - "LogGroupName": { - "Fn::Sub": [ "/aws/lambda/${func}", { "func": { "Ref" : "lambdaShareSnapshotsRDS" } } ] - } - } - }, - "cwloggrouplambdaDeleteOldSnapshotsRDS":{ - "Type": "AWS::Logs::LogGroup", - "Description": "Log group for the lambdaDeleteOldSnapshotsRDS function's logs", - "Properties": { - "RetentionInDays": { "Ref": "LambdaCWLogRetention" }, - "LogGroupName": { - "Fn::Sub": [ "/aws/lambda/${func}", { "func": { "Ref" : "LogGroupName" } } ] - } - } - } - }, - "Outputs": { - "BackupFailedTopic": { - "Description": "Subscribe to this topic to receive alerts of failed backups", - "Value": { - "Ref": "topicBackupsFailed" - } - }, - "ShareFailedTopic": { - "Condition": "Share", - "Description": "Subscribe to this topic to receive alerts of failures at sharing snapshots with destination account", - "Value": { - "Ref": "topicShareFailed" - } - }, - "DeleteOldFailedTopic": { - "Condition": "DeleteOld", - "Description": "Subscribe to this topic to receive alerts of failures at deleting old snapshots", - "Value": { - "Ref": "topicDeleteOldFailed" - } - }, - "SourceURL": { - "Description": "For more information and documentation, see the source repository at GitHub.", - "Value": "https://github.com/awslabs/rds-snapshot-tool" - } - }, - "Description": "Snapshots Tool for RDS cross-region and cross-account (source account stack)" -} diff --git a/lambda/.gitignore b/lambda/.gitignore deleted file mode 100644 index b7b8780..0000000 --- a/lambda/.gitignore +++ /dev/null @@ -1,2 +0,0 @@ -._* -*.zip diff --git a/lambda/Makefile b/lambda/Makefile deleted file mode 100644 index f284746..0000000 --- a/lambda/Makefile +++ /dev/null @@ -1,51 +0,0 @@ -# -# Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved. -# Licensed under the Apache License, Version 2.0 (the "License"). You may not use this file except in compliance with the License. A copy of the License is located at -# http://aws.amazon.com/apache2.0/ -# or in the "license" file accompanying this file. This file is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. - -# Makefile for generating zip files for lambda functions and then copying them -# to S3 for deployment. This Makefile will NOT WORK unless you fill in the S3DEST -# and AWSARGS variables below. Once those parameters are established, simply type -# 'make' or 'gmake' (depending on your UNIX-like OS) and it will build. -# -# Behaviour: -# Creates a file named ._foo.whatever based on foo.whatever.Uploads foo.whatever to -# the S3 bucket. The ._ file is a hack to figure out whether the file has -# been modified since the last time we uploaded to s3. - -# Override S3 destination by changing this variable or setting it in the -# environment -S3DEST?=[YOUR BUCKET HERE] - -# Set these if, for example, you use profiles on the AWS command line -# or if your 'aws' executable is in a weird place. -AWSARGS=--region [YOUR REGION] --profile [YOUR PROFILE, or 'default', or remove this] -AWSCMD=aws -ZIPCMD=zip - -# disable all implicit make rules -.SUFFIXES: - -# if you define "._foo" as a file on this line, then it will zip up a -# folder called foo, adding a standard file into it to make foo.zip. -all: ._copy_snapshots_dest_rds \ - ._copy_snapshots_no_x_account_rds \ - ._delete_old_snapshots_dest_rds \ - ._delete_old_snapshots_no_x_account_rds \ - ._delete_old_snapshots_rds \ - ._share_snapshots_rds \ - ._take_snapshots_rds - -clean: - rm -f ._* - -._%: %.zip - "$(AWSCMD)" $(AWSARGS) s3 cp "$<" "s3://$(S3DEST)" \ - --grants read=uri=http://acs.amazonaws.com/groups/global/AllUsers - cp "$<" "$@" - -# This rule is a BSD make style rule that says "to make foo.zip, call -# 'zip -jqr foo snapshot_tool_utils.py'" -%.zip: % - $(ZIPCMD) -jqr "$@" "$<" snapshots_tool_utils.py \ No newline at end of file diff --git a/lambda_code/.DS_Store b/lambda_code/.DS_Store new file mode 100644 index 0000000000000000000000000000000000000000..f8ff79eee12f2fe76637f675b14d51e094b2a098 GIT binary patch literal 20484 zcmeHPYitx%6uxJ9%?__A&+aO8X@eRl&}~{Mpt60Syi6;l6v|_n?TqcnbY|I^ZP5*; zF+LK|_)C2M&;%1B;fIM4OpL}Dqlr%>1XN6nCK4slm?%Hg=(%%e>2z6OmswiWJI$Rl z_s%_MX3l>1-tXL*TS5plrIjclYX~6%CMu;#gv2Hi57~=q^EaJ+TD+if$R8hxKMafjb4KaYJV|j#DM;UYFAd^g; z08=O6UK!j71)Qs+Uc{+RAjTx$a}RJ2To3|L9m!*3>%x@OrmH76l3X6(4}6NCxEL}5pJ zuQ||e%XZpol2hAgZj)*1iR`hQ*sQ4oacM@gY1p!6s3}K5Q&g<7y;Cz3bD+sg8;a%V zwzCsLPzc7Qa3&L7zO=SF(onyzI+BUj)Ge)!MC%)B_w5S`Gpm-Y+!)`j_ZjA6%y-~+ z)?~8Bm-O&)OhrGrVs+~Yd=hG)Xl1?Uq93(r8(wgC)cr7jJp=bu*cpPbA3J$~TJ*i- zz;U|M*kx?Yv?E{>dBZz^{}u=fd|of~3RKAhxB@l3Fx#`ZYMu+Tg0+XhRlIf{NELY9 z93V}@F+nD~3x?6L5%0@Fnar}~b=8sD=w-XI1GKxrzWP#K8SUx{dRQ-p3R1{}pu+U5 z{$@5pjG2}7>Z$Aq5fmqeHg}3!w`kp_(XQRE#-*ti4iRfH+m=%MacJo5HuZF$Va27g zZdvb6>$0sj>N>4#ZL!t8Ha+UH%#_{iRE$ef+EO>FSWoL$Td+2&PNSAuleT6W)@C(j zp{$D4IIbkYBa>#%sk~`nP5p{hEo(NkotQjjs#G>j6uWS&F>LJ~O-*$SXo}stRkpfS zL(z<$&VJ3(61uv{ko(j!hzElwX3Z|2914eS%$~|s5u_?%OFE(LNo)3f&Nb#%gv9L% z-r!xPLa(AouM0uvOjSiFw2hu=PN}$81i>l3`IeBl$->8#WVd4VB2nD58wW|r!dsv> z9<3Ea9lD%Qbx8>_wWZ6%P-j1TYH7V&xbjvpv;}!ZA3ma02GwqHtT7bYNw2&?w(QoV zYT#Xc;9h<;p6jxD<&@ewfaA1Vx)w{FUk%Ht(rQJsO?uLCNJXes5Vlf-jf$cwCnnD> zpLO>}sv&7czlOX3e+otqq~n4hRK*ssU&Zlka0bg<*weTyL@G%gxs|k$t;kje$YbP5 za*!M%N60bqHu;czPEL~3^F3 zU_ESr4(NjIumg5N5A=ZvN!SDT!-Mb;JPeP(6Yw-V2hYO`@G`s(Z@^JF2Jgbh@Ckeh zr{G8U3C_W9@H?CjKtK%42~-3kfq8+-z`{UH0N-oTEs$-pT`Zs72b%Dx(3a$Kt-zjq zEh&7*o#NV2ZAqsmv9>dP#>}}@w=Ak%x-xbFV+@p89D331Kzdk1a3JjRy2YTAILWN5o12+k!NXW#@}G3)l*z@hrO_TrFU`lLAgwToR53v2{sf zSaw0UJ}5}QG7aL2Aht7UEX&>|t_sqoFA*mnZVI;u*j6MTAby3sMvjtq$VUj`Z^(D# zEIE(B6<`XK!(0R}h3gL4K~6ygz-*@gADh=ZqUJi z0oaRB&cHqd@}saH4#4AZ5T1f(;8_>aufkyl#|t1`mxU*j@K&K9_0ok0KI{17>~SaL zDhS|x<`7vZRyPk-DCn9wL}B=2^V|q~fJSDsojBh6?tVDsr?UgrP>k=1K#yj7#Sx1@ z%QKnU06Pid5B!G7XAm6?hL1N4%Oq7?kXu=@okEeyX%eI zo%#QeyCUK0rFq9*?6KUL^`fl|5x%Rpl85>gc^^3>u;K~~H zv5>b)CG?1w4VBPr9(Ghp>zpldw#3=e<=N7;3+IcnOqbBmM3>O`jDmHmpOZLF;zo_c zjY9~Rb`;}q=5QVzIR6_zr}Yt1@S#ItpRm-C8oK^}^`-y+k5h^7xCgig z{+k|vu`TT_bWuch!9?z3(jXy%X%{BC(RzbSIx4uMRcN`h93Q2#93N$yTfU1*x?$NM ilX!&Ni2ox1wtD3{{_vlV|25v_YJ8E6(Q^$J+57`w+oH1@V-^m;4Wg<&0T*E43hX&L&p$$qDprKhvt+--jT7}7np#A3 zem<@ulZcFPQ@L2!n>{z**++&mCkOWA81W14cNZlEfg7;MkzE(HCqgga^y>{tEnwC%0;vJ&^%eQ zLs35+`xjp>T0 0: - log_message = 'Copies pending: %s. Needs retrying' % pending_copies - logger.error(log_message) - raise SnapshotToolException(log_message) - - -if __name__ == '__main__': - lambda_handler(None, None) +''' +Copyright 2017 Amazon.com, Inc. or its affiliates. All Rights Reserved. + +Licensed under the Apache License, Version 2.0 (the "License"). You may not use this file except in compliance with the License. A copy of the License is located at + + http://aws.amazon.com/apache2.0/ + +or in the "license" file accompanying this file. This file is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. +''' + +# copy_snapshots_dest_rds +# This lambda function will copy shared RDS snapshots that match the regex specified in the environment variable SNAPSHOT_PATTERN, into the account where it runs. If the snapshot is shared and exists in the local region, it will copy it to the region specified in the environment variable DEST_REGION. If it finds that the snapshots are shared, exist in the local and destination regions, it will delete them from the local region. Copying snapshots cross-account and cross-region need to be separate operations. This function will need to run as many times necessary for the workflow to complete. +# Set SNAPSHOT_PATTERN to a regex that matches your RDS Instance identifiers +# Set DEST_REGION to the destination AWS region +import boto3 +from datetime import datetime +import time +import os +import logging +import re +from snapshots_tool_utils import * + +# Initialize everything +LOGLEVEL = os.getenv('LOG_LEVEL', 'ERROR').strip() +PATTERN = os.getenv('SNAPSHOT_PATTERN', 'ALL_SNAPSHOTS') +DESTINATION_REGION = os.getenv('DEST_REGION').strip() +RETENTION_DAYS = int(os.getenv('RETENTION_DAYS')) + +if os.getenv('REGION_OVERRIDE', 'NO') != 'NO': + REGION = os.getenv('REGION_OVERRIDE').strip() +else: + REGION = os.getenv('AWS_DEFAULT_REGION') + + +logger = logging.getLogger() +logger.setLevel(LOGLEVEL.upper()) + + +def lambda_handler(event, context): + # Describe all snapshots + pending_copies = 0 + client = boto3.client('rds', region_name=REGION) + response = paginate_api_call(client, 'describe_db_snapshots', 'DBSnapshots', IncludeShared=True) + + shared_snapshots = get_shared_snapshots(PATTERN, response) + own_snapshots = get_own_snapshots_dest(PATTERN, response) + + # Get list of snapshots in DEST_REGION + client_dest = boto3.client('rds', region_name=DESTINATION_REGION) + response_dest = paginate_api_call(client_dest, 'describe_db_snapshots', 'DBSnapshots') + own_dest_snapshots = get_own_snapshots_dest(PATTERN, response_dest) + + for shared_identifier, shared_attributes in shared_snapshots.items(): + + if shared_identifier not in own_snapshots.keys() and shared_identifier not in own_dest_snapshots.keys(): + # Check date + creation_date = get_timestamp(shared_identifier, shared_snapshots) + if creation_date: + time_difference = datetime.now() - creation_date + days_difference = time_difference.total_seconds() / 3600 / 24 + + # Only copy if it's newer than RETENTION_DAYS + if days_difference < RETENTION_DAYS: + + # Copy to own account + try: + copy_local(shared_identifier, shared_attributes) + + except Exception as e: + pending_copies += 1 + logger.error('Local copy pending: %s (%s)' % (shared_identifier, e)) + + else: + if REGION != DESTINATION_REGION: + pending_copies += 1 + logger.error('Remote copy pending: %s' % shared_identifier) + + else: + logger.info('Not copying %s locally. Older than %s days' % (shared_identifier, RETENTION_DAYS)) + + else: + logger.info('Not copying %s locally. No valid timestamp' % shared_identifier) + + + # Copy to DESTINATION_REGION + elif shared_identifier not in own_dest_snapshots.keys() and shared_identifier in own_snapshots.keys() and REGION != DESTINATION_REGION: + if own_snapshots[shared_identifier]['Status'] == 'available': + try: + copy_remote(shared_identifier, own_snapshots[shared_identifier]) + + except Exception as e: + pending_copies += 1 + logger.error('Remote copy pending: %s: %s (%s)' % ( + shared_identifier, own_snapshots[shared_identifier]['Arn'], e)) + else: + pending_copies += 1 + logger.error('Remote copy pending: %s: %s' % ( + shared_identifier, own_snapshots[shared_identifier]['Arn'])) + + # Delete local snapshots + elif shared_identifier in own_dest_snapshots.keys() and shared_identifier in own_snapshots.keys() and own_dest_snapshots[shared_identifier]['Status'] == 'available' and REGION != DESTINATION_REGION: + + response = client.delete_db_snapshot( + DBSnapshotIdentifier=shared_identifier + ) + + logger.info('Deleting local snapshot: %s' % shared_identifier) + + if pending_copies > 0: + log_message = 'Copies pending: %s. Needs retrying' % pending_copies + logger.error(log_message) + raise SnapshotToolException(log_message) + + +if __name__ == '__main__': + lambda_handler(None, None) \ No newline at end of file diff --git a/lambda/snapshots_tool_utils.py b/lambda_code/copy_snapshots_dest_rds/snapshots_tool_utils.py similarity index 99% rename from lambda/snapshots_tool_utils.py rename to lambda_code/copy_snapshots_dest_rds/snapshots_tool_utils.py index 5a797d9..eeaeee8 100644 --- a/lambda/snapshots_tool_utils.py +++ b/lambda_code/copy_snapshots_dest_rds/snapshots_tool_utils.py @@ -361,4 +361,4 @@ def copy_remote(snapshot_identifier, snapshot_object): SourceRegion = _REGION, CopyTags = True) - return response + return response \ No newline at end of file diff --git a/lambda_code/copy_snapshots_no_x_account_rds/.DS_Store b/lambda_code/copy_snapshots_no_x_account_rds/.DS_Store new file mode 100644 index 0000000000000000000000000000000000000000..5008ddfcf53c02e82d7eee2e57c38e5672ef89f6 GIT binary patch literal 6148 zcmeH~Jr2S!425mzP>H1@V-^m;4Wg<&0T*E43hX&L&p$$qDprKhvt+--jT7}7np#A3 zem<@ulZcFPQ@L2!n>{z**++&mCkOWA81W14cNZlEfg7;MkzE(HCqgga^y>{tEnwC%0;vJ&^%eQ zLs35+`xjp>T0 0: + return max(timestamps) + + else: + return None + + + +def requires_backup(backup_interval, instance, filtered_snapshots): +# Returns True if latest snapshot is older than INTERVAL + latest = get_latest_snapshot_ts(instance['DBInstanceIdentifier'], filtered_snapshots) + + if latest is not None: + backup_age = datetime.now() - latest + + if backup_age.total_seconds() >= (backup_interval * 60 * 60): + return True + + else: + return False + + elif latest is None: + return True + + +def paginate_api_call(client, api_call, objecttype, *args, **kwargs): +#Takes an RDS boto client and paginates through api_call calls and returns a list of objects of objecttype + response = {} + response[objecttype] = [] + + # Create a paginator + paginator = client.get_paginator(api_call) + + # Create a PageIterator from the Paginator + page_iterator = paginator.paginate(**kwargs) + for page in page_iterator: + for item in page[objecttype]: + response[objecttype].append(item) + + return response + + +def copy_local(snapshot_identifier, snapshot_object): + client = boto3.client('rds', region_name=_REGION) + + tags = [{ + 'Key': 'CopiedBy', + 'Value': 'Snapshot Tool for RDS' + }] + + if snapshot_object['Encrypted']: + logger.info('Copying encrypted snapshot %s locally' % snapshot_identifier) + response = client.copy_db_snapshot( + SourceDBSnapshotIdentifier = snapshot_object['Arn'], + TargetDBSnapshotIdentifier = snapshot_identifier, + KmsKeyId = _KMS_KEY_SOURCE_REGION, + Tags = tags) + + else: + logger.info('Copying snapshot %s locally' %snapshot_identifier) + response = client.copy_db_snapshot( + SourceDBSnapshotIdentifier = snapshot_object['Arn'], + TargetDBSnapshotIdentifier = snapshot_identifier, + Tags = tags) + + return response + + + +def copy_remote(snapshot_identifier, snapshot_object): + client = boto3.client('rds', region_name=_DESTINATION_REGION) + + if snapshot_object['Encrypted']: + logger.info('Copying encrypted snapshot %s to remote region %s' % (snapshot_object['Arn'], _DESTINATION_REGION)) + response = client.copy_db_snapshot( + SourceDBSnapshotIdentifier = snapshot_object['Arn'], + TargetDBSnapshotIdentifier = snapshot_identifier, + KmsKeyId = _KMS_KEY_DEST_REGION, + SourceRegion = _REGION, + CopyTags = True) + + else: + logger.info('Copying snapshot %s to remote region %s' % (snapshot_object['Arn'], _DESTINATION_REGION)) + response = client.copy_db_snapshot( + SourceDBSnapshotIdentifier = snapshot_object['Arn'], + TargetDBSnapshotIdentifier = snapshot_identifier, + SourceRegion = _REGION, + CopyTags = True) + + return response \ No newline at end of file diff --git a/lambda_code/delete_old_snapshots_dest_rds/.DS_Store b/lambda_code/delete_old_snapshots_dest_rds/.DS_Store new file mode 100644 index 0000000000000000000000000000000000000000..5008ddfcf53c02e82d7eee2e57c38e5672ef89f6 GIT binary patch literal 6148 zcmeH~Jr2S!425mzP>H1@V-^m;4Wg<&0T*E43hX&L&p$$qDprKhvt+--jT7}7np#A3 zem<@ulZcFPQ@L2!n>{z**++&mCkOWA81W14cNZlEfg7;MkzE(HCqgga^y>{tEnwC%0;vJ&^%eQ zLs35+`xjp>T0 RETENTION_DAYS: - # delete it - logger.info('Deleting %s. %s days old' % - (snapshot, days_difference)) - - try: - client.delete_db_snapshot( - DBSnapshotIdentifier=snapshot) - - except Exception as e: - delete_pending += 1 - logger.info('Could not delete %s (%s)' % (snapshot, e)) - - else: - logger.info('Not deleting %s. Only %s days old' % - (snapshot, days_difference)) - - else: - logger.info( - 'Not deleting %s. Did not find correct tag' % snapshot) - - if delete_pending > 0: - log_message = 'Snapshots pending delete: %s' % delete_pending - logger.error(log_message) - raise SnapshotToolException(log_message) - - -if __name__ == '__main__': - lambda_handler(None, None) +''' +Copyright 2017 Amazon.com, Inc. or its affiliates. All Rights Reserved. + +Licensed under the Apache License, Version 2.0 (the "License"). You may not use this file except in compliance with the License. A copy of the License is located at + + http://aws.amazon.com/apache2.0/ + +or in the "license" file accompanying this file. This file is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. +''' + +# delete_old_snapshots_dest_rds +# This lambda function will delete manual RDS snapshots that have expired in the region specified in the environment variable DEST_REGION, and according to the environment variables SNAPSHOT_PATTERN and RETENTION_DAYS. +# Set SNAPSHOT_PATTERN to a regex that matches your RDS Instance identifiers +# Set DEST_REGION to the destination AWS region +# Set RETENTION_DAYS to the amount of days snapshots need to be kept before deleting +import boto3 +import time +import os +import logging +from datetime import datetime +import re +from snapshots_tool_utils import * + + +# Initialize everything +DEST_REGION = os.getenv('DEST_REGION', os.getenv('AWS_DEFAULT_REGION')).strip() +LOGLEVEL = os.getenv('LOG_LEVEL', 'ERROR').strip() +PATTERN = os.getenv('SNAPSHOT_PATTERN', 'ALL_SNAPSHOTS') +RETENTION_DAYS = int(os.getenv('RETENTION_DAYS')) +TIMESTAMP_FORMAT = '%Y-%m-%d-%H-%M' + +logger = logging.getLogger() +logger.setLevel(LOGLEVEL.upper()) + + + +def lambda_handler(event, context): + delete_pending = 0 + # Search for all snapshots + client = boto3.client('rds', region_name=DEST_REGION) + response = paginate_api_call(client, 'describe_db_snapshots', 'DBSnapshots') + + # Filter out the ones not created automatically or with other methods + filtered_list = get_own_snapshots_dest(PATTERN, response) + + for snapshot in filtered_list.keys(): + + creation_date = get_timestamp(snapshot, filtered_list) + + if creation_date: + snapshot_arn = filtered_list[snapshot]['Arn'] + response_tags = client.list_tags_for_resource( + ResourceName=snapshot_arn) + + if search_tag_copied(response_tags): + difference = datetime.now() - creation_date + days_difference = difference.total_seconds() / 3600 / 24 + + # if we are past RETENTION_DAYS + if days_difference > RETENTION_DAYS: + # delete it + logger.info('Deleting %s. %s days old' % + (snapshot, days_difference)) + + try: + client.delete_db_snapshot( + DBSnapshotIdentifier=snapshot) + + except Exception as e: + delete_pending += 1 + logger.info('Could not delete %s (%s)' % (snapshot, e)) + + else: + logger.info('Not deleting %s. Only %s days old' % + (snapshot, days_difference)) + + else: + logger.info( + 'Not deleting %s. Did not find correct tag' % snapshot) + + if delete_pending > 0: + log_message = 'Snapshots pending delete: %s' % delete_pending + logger.error(log_message) + raise SnapshotToolException(log_message) + + +if __name__ == '__main__': + lambda_handler(None, None) \ No newline at end of file diff --git a/lambda_code/delete_old_snapshots_dest_rds/snapshots_tool_utils.py b/lambda_code/delete_old_snapshots_dest_rds/snapshots_tool_utils.py new file mode 100644 index 0000000..eeaeee8 --- /dev/null +++ b/lambda_code/delete_old_snapshots_dest_rds/snapshots_tool_utils.py @@ -0,0 +1,364 @@ +''' +Copyright 2017 Amazon.com, Inc. or its affiliates. All Rights Reserved. + +Licensed under the Apache License, Version 2.0 (the "License"). You may not use this file except in compliance with the License. A copy of the License is located at + + http://aws.amazon.com/apache2.0/ + +or in the "license" file accompanying this file. This file is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. +''' + + +# snapshots_tool_utils +# Support module for the Snapshot Tool for RDS + +import boto3 +from datetime import datetime, timedelta +import os +import logging +import re + + +# Initialize everything +_LOGLEVEL = os.getenv('LOG_LEVEL', 'ERROR').strip() + +_DESTINATION_REGION = os.getenv( + 'DEST_REGION', os.getenv('AWS_DEFAULT_REGION')).strip() + +_KMS_KEY_DEST_REGION = os.getenv('KMS_KEY_DEST_REGION', 'None').strip() + +_KMS_KEY_SOURCE_REGION = os.getenv('KMS_KEY_SOURCE_REGION', 'None').strip() + +_TIMESTAMP_FORMAT = '%Y-%m-%d-%H-%M' + +if os.getenv('REGION_OVERRIDE', 'NO') != 'NO': + _REGION = os.getenv('REGION_OVERRIDE').strip() +else: + _REGION = os.getenv('AWS_DEFAULT_REGION') + +_SUPPORTED_ENGINES = [ 'mariadb', 'sqlserver-se', 'sqlserver-ee', 'sqlserver-ex', 'sqlserver-web', 'mysql', 'oracle-se', 'oracle-se1', 'oracle-se2', 'oracle-ee', 'postgres' ] + + +logger = logging.getLogger() +logger.setLevel(_LOGLEVEL.upper()) + + +class SnapshotToolException(Exception): + pass + + +def search_tag_copydbsnapshot(response): +# Takes a list_tags_for_resource response and searches for our CopyDBSnapshot tag + try: + + for tag in response['TagList']: + if tag['Key'] == 'CopyDBSnapshot' and tag['Value'] == 'True': return True + + except Exception: return False + + else: return False + + + +def search_tag_created(response): +# Takes a describe_db_snapshots response and searches for our CreatedBy tag + try: + + for tag in response['TagList']: + if tag['Key'] == 'CreatedBy' and tag['Value'] == 'Snapshot Tool for RDS': return True + + except Exception: return False + + else: return False + + + +def search_tag_shared(response): +# Takes a describe_db_snapshots response and searches for our shareAndCopy tag + try: + for tag in response['TagList']: + if tag['Key'] == 'shareAndCopy' and tag['Value'] == 'YES': + for tag2 in response['TagList']: + if tag2['Key'] == 'CreatedBy' and tag2['Value'] == 'Snapshot Tool for RDS': + return True + + except Exception: + return False + + return False + + + +def search_tag_copied(response): +# Search for a tag indicating we copied this snapshot + try: + for tag in response['TagList']: + if tag['Key'] == 'CopiedBy' and tag['Value'] == 'Snapshot Tool for RDS': + return True + + except Exception: + return False + + return False + +def get_own_snapshots_no_x_account(pattern, response, REGION): + # Filters our own snapshots + filtered = {} + for snapshot in response['DBSnapshots']: + + if snapshot['SnapshotType'] == 'manual' and re.search(pattern, snapshot['DBInstanceIdentifier']) and snapshot['Engine'] in _SUPPORTED_ENGINES: + client = boto3.client('rds', region_name=REGION) + response_tags = client.list_tags_for_resource( + ResourceName=snapshot['DBSnapshotArn']) + + if search_tag_created(response_tags): + filtered[snapshot['DBSnapshotIdentifier']] = { + 'Arn': snapshot['DBSnapshotArn'], 'Status': snapshot['Status'], 'DBInstanceIdentifier': snapshot['DBInstanceIdentifier']} + #Changed the next line to search for ALL_CLUSTERS or ALL_SNAPSHOTS so it will work with no-x-account + elif snapshot['SnapshotType'] == 'manual' and pattern == 'ALL_SNAPSHOTS' and snapshot['Engine'] in _SUPPORTED_ENGINES: + client = boto3.client('rds', region_name=REGION) + response_tags = client.list_tags_for_resource( + ResourceName=snapshot['DBSnapshotArn']) + + if search_tag_created(response_tags): + filtered[snapshot['DBSnapshotIdentifier']] = { + 'Arn': snapshot['DBSnapshotArn'], 'Status': snapshot['Status'], 'DBInstanceIdentifier': snapshot['DBInstanceIdentifier']} + + return filtered + + +def get_shared_snapshots(pattern, response): +# Returns a dict with only shared snapshots filtered by pattern, with DBSnapshotIdentifier as key and the response as attribute + filtered = {} + for snapshot in response['DBSnapshots']: + if snapshot['SnapshotType'] == 'shared' and re.search(pattern, snapshot['DBInstanceIdentifier']) and snapshot['Engine'] in _SUPPORTED_ENGINES: + filtered[get_snapshot_identifier(snapshot)] = { + 'Arn': snapshot['DBSnapshotIdentifier'], 'Encrypted': snapshot['Encrypted'], 'DBInstanceIdentifier': snapshot['DBInstanceIdentifier']} + if snapshot['Encrypted'] is True: + filtered[get_snapshot_identifier(snapshot)]['KmsKeyId'] = snapshot['KmsKeyId'] + + elif snapshot['SnapshotType'] == 'shared' and pattern == 'ALL_SNAPSHOTS' and snapshot['Engine'] in _SUPPORTED_ENGINES: + filtered[get_snapshot_identifier(snapshot)] = { + 'Arn': snapshot['DBSnapshotIdentifier'], 'Encrypted': snapshot['Encrypted'], 'DBInstanceIdentifier': snapshot['DBInstanceIdentifier']} + if snapshot['Encrypted'] is True: + filtered[get_snapshot_identifier(snapshot)]['KmsKeyId'] = snapshot['KmsKeyId'] + return filtered + + + +def get_snapshot_identifier(snapshot): +# Function that will return the RDS Snapshot identifier given an ARN + match = re.match('arn:aws:rds:.*:.*:snapshot:(.+)', + snapshot['DBSnapshotArn']) + return match.group(1) + + +def get_own_snapshots_dest(pattern, response): +# Returns a dict with local snapshots, filtered by pattern, with DBSnapshotIdentifier as key and Arn, Status as attributes + filtered = {} + for snapshot in response['DBSnapshots']: + + if snapshot['SnapshotType'] == 'manual' and re.search(pattern, snapshot['DBInstanceIdentifier']) and snapshot['Engine'] in _SUPPORTED_ENGINES: + filtered[snapshot['DBSnapshotIdentifier']] = { + 'Arn': snapshot['DBSnapshotArn'], 'Status': snapshot['Status'], 'Encrypted': snapshot['Encrypted'], 'DBInstanceIdentifier': snapshot['DBInstanceIdentifier']} + + if snapshot['Encrypted'] is True: + filtered[snapshot['DBSnapshotIdentifier']]['KmsKeyId'] = snapshot['KmsKeyId'] + + elif snapshot['SnapshotType'] == 'manual' and pattern == 'ALL_SNAPSHOTS' and snapshot['Engine'] in _SUPPORTED_ENGINES: + filtered[snapshot['DBSnapshotIdentifier']] = { + 'Arn': snapshot['DBSnapshotArn'], 'Status': snapshot['Status'], 'Encrypted': snapshot['Encrypted'], 'DBInstanceIdentifier': snapshot['DBInstanceIdentifier'] } + + if snapshot['Encrypted'] is True: + filtered[snapshot['DBSnapshotIdentifier']]['KmsKeyId'] = snapshot['KmsKeyId'] + + return filtered + +def filter_instances(taggedinstance, pattern, instance_list): +# Takes the response from describe-db-instances and filters according to pattern in DBInstanceIdentifier + filtered_list = [] + + for instance in instance_list['DBInstances']: + + if taggedinstance == 'TRUE': + client = boto3.client('rds', region_name=_REGION) + response = client.list_tags_for_resource(ResourceName=instance['DBInstanceArn']) + + if pattern == 'ALL_INSTANCES' and instance['Engine'] in _SUPPORTED_ENGINES: + if (taggedinstance == 'TRUE' and search_tag_copydbsnapshot(response)) or taggedinstance == 'FALSE': + filtered_list.append(instance) + + else: + match = re.search(pattern, instance['DBInstanceIdentifier']) + + if match and instance['Engine'] in _SUPPORTED_ENGINES: + if (taggedinstance == 'TRUE' and search_tag_copydbsnapshot(response)) or taggedinstance == 'FALSE': + filtered_list.append(instance) + + return filtered_list + + +def get_own_snapshots_source(pattern, response, backup_interval=None): +# Filters our own snapshots + filtered = {} + + for snapshot in response['DBSnapshots']: + + # No need to consider snapshots that are still in progress + if 'SnapshotCreateTime' not in snapshot: + continue + + # No need to get tags for snapshots outside of the backup interval + if backup_interval and snapshot['SnapshotCreateTime'].replace(tzinfo=None) < datetime.utcnow().replace(tzinfo=None) - timedelta(hours=backup_interval): + continue + + if snapshot['SnapshotType'] == 'manual' and re.search(pattern, snapshot['DBInstanceIdentifier']) and snapshot['Engine'] in _SUPPORTED_ENGINES: + client = boto3.client('rds', region_name=_REGION) + response_tags = client.list_tags_for_resource( + ResourceName=snapshot['DBSnapshotArn']) + + if search_tag_created(response_tags): + filtered[snapshot['DBSnapshotIdentifier']] = { + 'Arn': snapshot['DBSnapshotArn'], 'Status': snapshot['Status'], 'DBInstanceIdentifier': snapshot['DBInstanceIdentifier']} + + elif snapshot['SnapshotType'] == 'manual' and (pattern == 'ALL_CLUSTERS' or pattern == 'ALL_SNAPSHOTS' or pattern == 'ALL_INSTANCES') and snapshot['Engine'] in _SUPPORTED_ENGINES: + client = boto3.client('rds', region_name=_REGION) + response_tags = client.list_tags_for_resource( + ResourceName=snapshot['DBSnapshotArn']) + + if search_tag_created(response_tags): + filtered[snapshot['DBSnapshotIdentifier']] = { + 'Arn': snapshot['DBSnapshotArn'], 'Status': snapshot['Status'], 'DBInstanceIdentifier': snapshot['DBInstanceIdentifier']} + + return filtered + + +def get_timestamp_no_minute(snapshot_identifier, snapshot_list): +# Get a timestamp from the name of a snapshot and strip out the minutes + pattern = '%s-(.+)-\d{2}' % snapshot_list[snapshot_identifier]['DBInstanceIdentifier'] + timestamp_format = '%Y-%m-%d-%H' + date_time = re.search(pattern, snapshot_identifier) + + if date_time is not None: + return datetime.strptime(date_time.group(1), timestamp_format) + + +def get_timestamp(snapshot_identifier, snapshot_list): +# Searches for a timestamp on a snapshot name + pattern = '%s-(.+)' % snapshot_list[snapshot_identifier]['DBInstanceIdentifier'] + date_time = re.search(pattern, snapshot_identifier) + + if date_time is not None: + + try: + return datetime.strptime(date_time.group(1), _TIMESTAMP_FORMAT) + + except Exception: + return None + + return None + + + +def get_latest_snapshot_ts(instance_identifier, filtered_snapshots): +# Get latest snapshot for a specific DBInstanceIdentifier + timestamps = [] + + for snapshot,snapshot_object in filtered_snapshots.items(): + + if snapshot_object['DBInstanceIdentifier'] == instance_identifier: + timestamp = get_timestamp_no_minute(snapshot, filtered_snapshots) + + if timestamp is not None: + timestamps.append(timestamp) + + if len(timestamps) > 0: + return max(timestamps) + + else: + return None + + + +def requires_backup(backup_interval, instance, filtered_snapshots): +# Returns True if latest snapshot is older than INTERVAL + latest = get_latest_snapshot_ts(instance['DBInstanceIdentifier'], filtered_snapshots) + + if latest is not None: + backup_age = datetime.now() - latest + + if backup_age.total_seconds() >= (backup_interval * 60 * 60): + return True + + else: + return False + + elif latest is None: + return True + + +def paginate_api_call(client, api_call, objecttype, *args, **kwargs): +#Takes an RDS boto client and paginates through api_call calls and returns a list of objects of objecttype + response = {} + response[objecttype] = [] + + # Create a paginator + paginator = client.get_paginator(api_call) + + # Create a PageIterator from the Paginator + page_iterator = paginator.paginate(**kwargs) + for page in page_iterator: + for item in page[objecttype]: + response[objecttype].append(item) + + return response + + +def copy_local(snapshot_identifier, snapshot_object): + client = boto3.client('rds', region_name=_REGION) + + tags = [{ + 'Key': 'CopiedBy', + 'Value': 'Snapshot Tool for RDS' + }] + + if snapshot_object['Encrypted']: + logger.info('Copying encrypted snapshot %s locally' % snapshot_identifier) + response = client.copy_db_snapshot( + SourceDBSnapshotIdentifier = snapshot_object['Arn'], + TargetDBSnapshotIdentifier = snapshot_identifier, + KmsKeyId = _KMS_KEY_SOURCE_REGION, + Tags = tags) + + else: + logger.info('Copying snapshot %s locally' %snapshot_identifier) + response = client.copy_db_snapshot( + SourceDBSnapshotIdentifier = snapshot_object['Arn'], + TargetDBSnapshotIdentifier = snapshot_identifier, + Tags = tags) + + return response + + + +def copy_remote(snapshot_identifier, snapshot_object): + client = boto3.client('rds', region_name=_DESTINATION_REGION) + + if snapshot_object['Encrypted']: + logger.info('Copying encrypted snapshot %s to remote region %s' % (snapshot_object['Arn'], _DESTINATION_REGION)) + response = client.copy_db_snapshot( + SourceDBSnapshotIdentifier = snapshot_object['Arn'], + TargetDBSnapshotIdentifier = snapshot_identifier, + KmsKeyId = _KMS_KEY_DEST_REGION, + SourceRegion = _REGION, + CopyTags = True) + + else: + logger.info('Copying snapshot %s to remote region %s' % (snapshot_object['Arn'], _DESTINATION_REGION)) + response = client.copy_db_snapshot( + SourceDBSnapshotIdentifier = snapshot_object['Arn'], + TargetDBSnapshotIdentifier = snapshot_identifier, + SourceRegion = _REGION, + CopyTags = True) + + return response \ No newline at end of file diff --git a/lambda_code/delete_old_snapshots_no_x_account_rds/.DS_Store b/lambda_code/delete_old_snapshots_no_x_account_rds/.DS_Store new file mode 100644 index 0000000000000000000000000000000000000000..5008ddfcf53c02e82d7eee2e57c38e5672ef89f6 GIT binary patch literal 6148 zcmeH~Jr2S!425mzP>H1@V-^m;4Wg<&0T*E43hX&L&p$$qDprKhvt+--jT7}7np#A3 zem<@ulZcFPQ@L2!n>{z**++&mCkOWA81W14cNZlEfg7;MkzE(HCqgga^y>{tEnwC%0;vJ&^%eQ zLs35+`xjp>T0 0: + return max(timestamps) + + else: + return None + + + +def requires_backup(backup_interval, instance, filtered_snapshots): +# Returns True if latest snapshot is older than INTERVAL + latest = get_latest_snapshot_ts(instance['DBInstanceIdentifier'], filtered_snapshots) + + if latest is not None: + backup_age = datetime.now() - latest + + if backup_age.total_seconds() >= (backup_interval * 60 * 60): + return True + + else: + return False + + elif latest is None: + return True + + +def paginate_api_call(client, api_call, objecttype, *args, **kwargs): +#Takes an RDS boto client and paginates through api_call calls and returns a list of objects of objecttype + response = {} + response[objecttype] = [] + + # Create a paginator + paginator = client.get_paginator(api_call) + + # Create a PageIterator from the Paginator + page_iterator = paginator.paginate(**kwargs) + for page in page_iterator: + for item in page[objecttype]: + response[objecttype].append(item) + + return response + + +def copy_local(snapshot_identifier, snapshot_object): + client = boto3.client('rds', region_name=_REGION) + + tags = [{ + 'Key': 'CopiedBy', + 'Value': 'Snapshot Tool for RDS' + }] + + if snapshot_object['Encrypted']: + logger.info('Copying encrypted snapshot %s locally' % snapshot_identifier) + response = client.copy_db_snapshot( + SourceDBSnapshotIdentifier = snapshot_object['Arn'], + TargetDBSnapshotIdentifier = snapshot_identifier, + KmsKeyId = _KMS_KEY_SOURCE_REGION, + Tags = tags) + + else: + logger.info('Copying snapshot %s locally' %snapshot_identifier) + response = client.copy_db_snapshot( + SourceDBSnapshotIdentifier = snapshot_object['Arn'], + TargetDBSnapshotIdentifier = snapshot_identifier, + Tags = tags) + + return response + + + +def copy_remote(snapshot_identifier, snapshot_object): + client = boto3.client('rds', region_name=_DESTINATION_REGION) + + if snapshot_object['Encrypted']: + logger.info('Copying encrypted snapshot %s to remote region %s' % (snapshot_object['Arn'], _DESTINATION_REGION)) + response = client.copy_db_snapshot( + SourceDBSnapshotIdentifier = snapshot_object['Arn'], + TargetDBSnapshotIdentifier = snapshot_identifier, + KmsKeyId = _KMS_KEY_DEST_REGION, + SourceRegion = _REGION, + CopyTags = True) + + else: + logger.info('Copying snapshot %s to remote region %s' % (snapshot_object['Arn'], _DESTINATION_REGION)) + response = client.copy_db_snapshot( + SourceDBSnapshotIdentifier = snapshot_object['Arn'], + TargetDBSnapshotIdentifier = snapshot_identifier, + SourceRegion = _REGION, + CopyTags = True) + + return response \ No newline at end of file diff --git a/lambda/delete_old_snapshots_rds/lambda_function.py b/lambda_code/delete_old_snapshots_rds/lambda_function.py similarity index 97% rename from lambda/delete_old_snapshots_rds/lambda_function.py rename to lambda_code/delete_old_snapshots_rds/lambda_function.py index d6aee96..a6c51ec 100644 --- a/lambda/delete_old_snapshots_rds/lambda_function.py +++ b/lambda_code/delete_old_snapshots_rds/lambda_function.py @@ -1,80 +1,79 @@ -''' -Copyright 2017 Amazon.com, Inc. or its affiliates. All Rights Reserved. - -Licensed under the Apache License, Version 2.0 (the "License"). You may not use this file except in compliance with the License. A copy of the License is located at - - http://aws.amazon.com/apache2.0/ - -or in the "license" file accompanying this file. This file is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. -''' - -# delete_old_snapshots_rds -# This Lambda function will delete snapshots that have expired and match the regex set in the PATTERN environment variable. It will also look for a matching timestamp in the following format: YYYY-MM-DD-HH-mm -# Set PATTERN to a regex that matches your RDS Instance identifiers -import boto3 -from datetime import datetime -import time -import os -import logging -import re -from snapshots_tool_utils import * - -LOGLEVEL = os.getenv('LOG_LEVEL', 'ERROR').strip() -PATTERN = os.getenv('PATTERN', 'ALL_INSTANCES') -RETENTION_DAYS = int(os.getenv('RETENTION_DAYS', '7')) -TIMESTAMP_FORMAT = '%Y-%m-%d-%H-%M' - -if os.getenv('REGION_OVERRIDE', 'NO') != 'NO': - REGION = os.getenv('REGION_OVERRIDE').strip() -else: - REGION = os.getenv('AWS_DEFAULT_REGION') - - -logger = logging.getLogger() -logger.setLevel(LOGLEVEL.upper()) - - -def lambda_handler(event, context): - pending_delete = 0 - client = boto3.client('rds', region_name=REGION) - response = paginate_api_call(client, 'describe_db_snapshots', 'DBSnapshots') - - filtered_list = get_own_snapshots_source(PATTERN, response) - - for snapshot in filtered_list.keys(): - - creation_date = get_timestamp(snapshot, filtered_list) - - if creation_date: - difference = datetime.now() - creation_date - days_difference = difference.total_seconds() / 3600 / 24 - logger.debug('%s created %s days ago' % - (snapshot, days_difference)) - - # if we are past RETENTION_DAYS - if days_difference > RETENTION_DAYS: - # delete it - logger.info('Deleting %s' % snapshot) - - try: - client.delete_db_snapshot( - DBSnapshotIdentifier=snapshot) - - except Exception as e: - pending_delete += 1 - logger.info('Could not delete %s (%s)' % (snapshot, e)) - - else: - logger.info('Not deleting %s. Created only %s' % (snapshot, days_difference)) - - - if pending_delete > 0: - message = 'Snapshots pending delete: %s' % pending_delete - logger.error(message) - raise SnapshotToolException(message) - - -if __name__ == '__main__': - lambda_handler(None, None) - - +''' +Copyright 2017 Amazon.com, Inc. or its affiliates. All Rights Reserved. + +Licensed under the Apache License, Version 2.0 (the "License"). You may not use this file except in compliance with the License. A copy of the License is located at + + http://aws.amazon.com/apache2.0/ + +or in the "license" file accompanying this file. This file is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. +''' + +# delete_old_snapshots_rds +# This Lambda function will delete snapshots that have expired and match the regex set in the PATTERN environment variable. It will also look for a matching timestamp in the following format: YYYY-MM-DD-HH-mm +# Set PATTERN to a regex that matches your RDS Instance identifiers +import boto3 +from datetime import datetime +import time +import os +import logging +import re +from snapshots_tool_utils import * + +LOGLEVEL = os.getenv('LOG_LEVEL', 'ERROR').strip() +PATTERN = os.getenv('PATTERN', 'ALL_INSTANCES') +RETENTION_DAYS = int(os.getenv('RETENTION_DAYS', '7')) +TIMESTAMP_FORMAT = '%Y-%m-%d-%H-%M' + +if os.getenv('REGION_OVERRIDE', 'NO') != 'NO': + REGION = os.getenv('REGION_OVERRIDE').strip() +else: + REGION = os.getenv('AWS_DEFAULT_REGION') + + +logger = logging.getLogger() +logger.setLevel(LOGLEVEL.upper()) + + +def lambda_handler(event, context): + pending_delete = 0 + client = boto3.client('rds', region_name=REGION) + response = paginate_api_call(client, 'describe_db_snapshots', 'DBSnapshots') + + filtered_list = get_own_snapshots_source(PATTERN, response) + + for snapshot in filtered_list.keys(): + + creation_date = get_timestamp(snapshot, filtered_list) + + if creation_date: + difference = datetime.now() - creation_date + days_difference = difference.total_seconds() / 3600 / 24 + logger.debug('%s created %s days ago' % + (snapshot, days_difference)) + + # if we are past RETENTION_DAYS + if days_difference > RETENTION_DAYS: + # delete it + logger.info('Deleting %s' % snapshot) + + try: + client.delete_db_snapshot( + DBSnapshotIdentifier=snapshot) + + except Exception as e: + pending_delete += 1 + logger.info('Could not delete %s (%s)' % (snapshot, e)) + + else: + logger.info('Not deleting %s. Created only %s' % (snapshot, days_difference)) + + + if pending_delete > 0: + message = 'Snapshots pending delete: %s' % pending_delete + logger.error(message) + raise SnapshotToolException(message) + + +if __name__ == '__main__': + lambda_handler(None, None) + diff --git a/lambda_code/delete_old_snapshots_rds/snapshots_tool_utils.py b/lambda_code/delete_old_snapshots_rds/snapshots_tool_utils.py new file mode 100644 index 0000000..eeaeee8 --- /dev/null +++ b/lambda_code/delete_old_snapshots_rds/snapshots_tool_utils.py @@ -0,0 +1,364 @@ +''' +Copyright 2017 Amazon.com, Inc. or its affiliates. All Rights Reserved. + +Licensed under the Apache License, Version 2.0 (the "License"). You may not use this file except in compliance with the License. A copy of the License is located at + + http://aws.amazon.com/apache2.0/ + +or in the "license" file accompanying this file. This file is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. +''' + + +# snapshots_tool_utils +# Support module for the Snapshot Tool for RDS + +import boto3 +from datetime import datetime, timedelta +import os +import logging +import re + + +# Initialize everything +_LOGLEVEL = os.getenv('LOG_LEVEL', 'ERROR').strip() + +_DESTINATION_REGION = os.getenv( + 'DEST_REGION', os.getenv('AWS_DEFAULT_REGION')).strip() + +_KMS_KEY_DEST_REGION = os.getenv('KMS_KEY_DEST_REGION', 'None').strip() + +_KMS_KEY_SOURCE_REGION = os.getenv('KMS_KEY_SOURCE_REGION', 'None').strip() + +_TIMESTAMP_FORMAT = '%Y-%m-%d-%H-%M' + +if os.getenv('REGION_OVERRIDE', 'NO') != 'NO': + _REGION = os.getenv('REGION_OVERRIDE').strip() +else: + _REGION = os.getenv('AWS_DEFAULT_REGION') + +_SUPPORTED_ENGINES = [ 'mariadb', 'sqlserver-se', 'sqlserver-ee', 'sqlserver-ex', 'sqlserver-web', 'mysql', 'oracle-se', 'oracle-se1', 'oracle-se2', 'oracle-ee', 'postgres' ] + + +logger = logging.getLogger() +logger.setLevel(_LOGLEVEL.upper()) + + +class SnapshotToolException(Exception): + pass + + +def search_tag_copydbsnapshot(response): +# Takes a list_tags_for_resource response and searches for our CopyDBSnapshot tag + try: + + for tag in response['TagList']: + if tag['Key'] == 'CopyDBSnapshot' and tag['Value'] == 'True': return True + + except Exception: return False + + else: return False + + + +def search_tag_created(response): +# Takes a describe_db_snapshots response and searches for our CreatedBy tag + try: + + for tag in response['TagList']: + if tag['Key'] == 'CreatedBy' and tag['Value'] == 'Snapshot Tool for RDS': return True + + except Exception: return False + + else: return False + + + +def search_tag_shared(response): +# Takes a describe_db_snapshots response and searches for our shareAndCopy tag + try: + for tag in response['TagList']: + if tag['Key'] == 'shareAndCopy' and tag['Value'] == 'YES': + for tag2 in response['TagList']: + if tag2['Key'] == 'CreatedBy' and tag2['Value'] == 'Snapshot Tool for RDS': + return True + + except Exception: + return False + + return False + + + +def search_tag_copied(response): +# Search for a tag indicating we copied this snapshot + try: + for tag in response['TagList']: + if tag['Key'] == 'CopiedBy' and tag['Value'] == 'Snapshot Tool for RDS': + return True + + except Exception: + return False + + return False + +def get_own_snapshots_no_x_account(pattern, response, REGION): + # Filters our own snapshots + filtered = {} + for snapshot in response['DBSnapshots']: + + if snapshot['SnapshotType'] == 'manual' and re.search(pattern, snapshot['DBInstanceIdentifier']) and snapshot['Engine'] in _SUPPORTED_ENGINES: + client = boto3.client('rds', region_name=REGION) + response_tags = client.list_tags_for_resource( + ResourceName=snapshot['DBSnapshotArn']) + + if search_tag_created(response_tags): + filtered[snapshot['DBSnapshotIdentifier']] = { + 'Arn': snapshot['DBSnapshotArn'], 'Status': snapshot['Status'], 'DBInstanceIdentifier': snapshot['DBInstanceIdentifier']} + #Changed the next line to search for ALL_CLUSTERS or ALL_SNAPSHOTS so it will work with no-x-account + elif snapshot['SnapshotType'] == 'manual' and pattern == 'ALL_SNAPSHOTS' and snapshot['Engine'] in _SUPPORTED_ENGINES: + client = boto3.client('rds', region_name=REGION) + response_tags = client.list_tags_for_resource( + ResourceName=snapshot['DBSnapshotArn']) + + if search_tag_created(response_tags): + filtered[snapshot['DBSnapshotIdentifier']] = { + 'Arn': snapshot['DBSnapshotArn'], 'Status': snapshot['Status'], 'DBInstanceIdentifier': snapshot['DBInstanceIdentifier']} + + return filtered + + +def get_shared_snapshots(pattern, response): +# Returns a dict with only shared snapshots filtered by pattern, with DBSnapshotIdentifier as key and the response as attribute + filtered = {} + for snapshot in response['DBSnapshots']: + if snapshot['SnapshotType'] == 'shared' and re.search(pattern, snapshot['DBInstanceIdentifier']) and snapshot['Engine'] in _SUPPORTED_ENGINES: + filtered[get_snapshot_identifier(snapshot)] = { + 'Arn': snapshot['DBSnapshotIdentifier'], 'Encrypted': snapshot['Encrypted'], 'DBInstanceIdentifier': snapshot['DBInstanceIdentifier']} + if snapshot['Encrypted'] is True: + filtered[get_snapshot_identifier(snapshot)]['KmsKeyId'] = snapshot['KmsKeyId'] + + elif snapshot['SnapshotType'] == 'shared' and pattern == 'ALL_SNAPSHOTS' and snapshot['Engine'] in _SUPPORTED_ENGINES: + filtered[get_snapshot_identifier(snapshot)] = { + 'Arn': snapshot['DBSnapshotIdentifier'], 'Encrypted': snapshot['Encrypted'], 'DBInstanceIdentifier': snapshot['DBInstanceIdentifier']} + if snapshot['Encrypted'] is True: + filtered[get_snapshot_identifier(snapshot)]['KmsKeyId'] = snapshot['KmsKeyId'] + return filtered + + + +def get_snapshot_identifier(snapshot): +# Function that will return the RDS Snapshot identifier given an ARN + match = re.match('arn:aws:rds:.*:.*:snapshot:(.+)', + snapshot['DBSnapshotArn']) + return match.group(1) + + +def get_own_snapshots_dest(pattern, response): +# Returns a dict with local snapshots, filtered by pattern, with DBSnapshotIdentifier as key and Arn, Status as attributes + filtered = {} + for snapshot in response['DBSnapshots']: + + if snapshot['SnapshotType'] == 'manual' and re.search(pattern, snapshot['DBInstanceIdentifier']) and snapshot['Engine'] in _SUPPORTED_ENGINES: + filtered[snapshot['DBSnapshotIdentifier']] = { + 'Arn': snapshot['DBSnapshotArn'], 'Status': snapshot['Status'], 'Encrypted': snapshot['Encrypted'], 'DBInstanceIdentifier': snapshot['DBInstanceIdentifier']} + + if snapshot['Encrypted'] is True: + filtered[snapshot['DBSnapshotIdentifier']]['KmsKeyId'] = snapshot['KmsKeyId'] + + elif snapshot['SnapshotType'] == 'manual' and pattern == 'ALL_SNAPSHOTS' and snapshot['Engine'] in _SUPPORTED_ENGINES: + filtered[snapshot['DBSnapshotIdentifier']] = { + 'Arn': snapshot['DBSnapshotArn'], 'Status': snapshot['Status'], 'Encrypted': snapshot['Encrypted'], 'DBInstanceIdentifier': snapshot['DBInstanceIdentifier'] } + + if snapshot['Encrypted'] is True: + filtered[snapshot['DBSnapshotIdentifier']]['KmsKeyId'] = snapshot['KmsKeyId'] + + return filtered + +def filter_instances(taggedinstance, pattern, instance_list): +# Takes the response from describe-db-instances and filters according to pattern in DBInstanceIdentifier + filtered_list = [] + + for instance in instance_list['DBInstances']: + + if taggedinstance == 'TRUE': + client = boto3.client('rds', region_name=_REGION) + response = client.list_tags_for_resource(ResourceName=instance['DBInstanceArn']) + + if pattern == 'ALL_INSTANCES' and instance['Engine'] in _SUPPORTED_ENGINES: + if (taggedinstance == 'TRUE' and search_tag_copydbsnapshot(response)) or taggedinstance == 'FALSE': + filtered_list.append(instance) + + else: + match = re.search(pattern, instance['DBInstanceIdentifier']) + + if match and instance['Engine'] in _SUPPORTED_ENGINES: + if (taggedinstance == 'TRUE' and search_tag_copydbsnapshot(response)) or taggedinstance == 'FALSE': + filtered_list.append(instance) + + return filtered_list + + +def get_own_snapshots_source(pattern, response, backup_interval=None): +# Filters our own snapshots + filtered = {} + + for snapshot in response['DBSnapshots']: + + # No need to consider snapshots that are still in progress + if 'SnapshotCreateTime' not in snapshot: + continue + + # No need to get tags for snapshots outside of the backup interval + if backup_interval and snapshot['SnapshotCreateTime'].replace(tzinfo=None) < datetime.utcnow().replace(tzinfo=None) - timedelta(hours=backup_interval): + continue + + if snapshot['SnapshotType'] == 'manual' and re.search(pattern, snapshot['DBInstanceIdentifier']) and snapshot['Engine'] in _SUPPORTED_ENGINES: + client = boto3.client('rds', region_name=_REGION) + response_tags = client.list_tags_for_resource( + ResourceName=snapshot['DBSnapshotArn']) + + if search_tag_created(response_tags): + filtered[snapshot['DBSnapshotIdentifier']] = { + 'Arn': snapshot['DBSnapshotArn'], 'Status': snapshot['Status'], 'DBInstanceIdentifier': snapshot['DBInstanceIdentifier']} + + elif snapshot['SnapshotType'] == 'manual' and (pattern == 'ALL_CLUSTERS' or pattern == 'ALL_SNAPSHOTS' or pattern == 'ALL_INSTANCES') and snapshot['Engine'] in _SUPPORTED_ENGINES: + client = boto3.client('rds', region_name=_REGION) + response_tags = client.list_tags_for_resource( + ResourceName=snapshot['DBSnapshotArn']) + + if search_tag_created(response_tags): + filtered[snapshot['DBSnapshotIdentifier']] = { + 'Arn': snapshot['DBSnapshotArn'], 'Status': snapshot['Status'], 'DBInstanceIdentifier': snapshot['DBInstanceIdentifier']} + + return filtered + + +def get_timestamp_no_minute(snapshot_identifier, snapshot_list): +# Get a timestamp from the name of a snapshot and strip out the minutes + pattern = '%s-(.+)-\d{2}' % snapshot_list[snapshot_identifier]['DBInstanceIdentifier'] + timestamp_format = '%Y-%m-%d-%H' + date_time = re.search(pattern, snapshot_identifier) + + if date_time is not None: + return datetime.strptime(date_time.group(1), timestamp_format) + + +def get_timestamp(snapshot_identifier, snapshot_list): +# Searches for a timestamp on a snapshot name + pattern = '%s-(.+)' % snapshot_list[snapshot_identifier]['DBInstanceIdentifier'] + date_time = re.search(pattern, snapshot_identifier) + + if date_time is not None: + + try: + return datetime.strptime(date_time.group(1), _TIMESTAMP_FORMAT) + + except Exception: + return None + + return None + + + +def get_latest_snapshot_ts(instance_identifier, filtered_snapshots): +# Get latest snapshot for a specific DBInstanceIdentifier + timestamps = [] + + for snapshot,snapshot_object in filtered_snapshots.items(): + + if snapshot_object['DBInstanceIdentifier'] == instance_identifier: + timestamp = get_timestamp_no_minute(snapshot, filtered_snapshots) + + if timestamp is not None: + timestamps.append(timestamp) + + if len(timestamps) > 0: + return max(timestamps) + + else: + return None + + + +def requires_backup(backup_interval, instance, filtered_snapshots): +# Returns True if latest snapshot is older than INTERVAL + latest = get_latest_snapshot_ts(instance['DBInstanceIdentifier'], filtered_snapshots) + + if latest is not None: + backup_age = datetime.now() - latest + + if backup_age.total_seconds() >= (backup_interval * 60 * 60): + return True + + else: + return False + + elif latest is None: + return True + + +def paginate_api_call(client, api_call, objecttype, *args, **kwargs): +#Takes an RDS boto client and paginates through api_call calls and returns a list of objects of objecttype + response = {} + response[objecttype] = [] + + # Create a paginator + paginator = client.get_paginator(api_call) + + # Create a PageIterator from the Paginator + page_iterator = paginator.paginate(**kwargs) + for page in page_iterator: + for item in page[objecttype]: + response[objecttype].append(item) + + return response + + +def copy_local(snapshot_identifier, snapshot_object): + client = boto3.client('rds', region_name=_REGION) + + tags = [{ + 'Key': 'CopiedBy', + 'Value': 'Snapshot Tool for RDS' + }] + + if snapshot_object['Encrypted']: + logger.info('Copying encrypted snapshot %s locally' % snapshot_identifier) + response = client.copy_db_snapshot( + SourceDBSnapshotIdentifier = snapshot_object['Arn'], + TargetDBSnapshotIdentifier = snapshot_identifier, + KmsKeyId = _KMS_KEY_SOURCE_REGION, + Tags = tags) + + else: + logger.info('Copying snapshot %s locally' %snapshot_identifier) + response = client.copy_db_snapshot( + SourceDBSnapshotIdentifier = snapshot_object['Arn'], + TargetDBSnapshotIdentifier = snapshot_identifier, + Tags = tags) + + return response + + + +def copy_remote(snapshot_identifier, snapshot_object): + client = boto3.client('rds', region_name=_DESTINATION_REGION) + + if snapshot_object['Encrypted']: + logger.info('Copying encrypted snapshot %s to remote region %s' % (snapshot_object['Arn'], _DESTINATION_REGION)) + response = client.copy_db_snapshot( + SourceDBSnapshotIdentifier = snapshot_object['Arn'], + TargetDBSnapshotIdentifier = snapshot_identifier, + KmsKeyId = _KMS_KEY_DEST_REGION, + SourceRegion = _REGION, + CopyTags = True) + + else: + logger.info('Copying snapshot %s to remote region %s' % (snapshot_object['Arn'], _DESTINATION_REGION)) + response = client.copy_db_snapshot( + SourceDBSnapshotIdentifier = snapshot_object['Arn'], + TargetDBSnapshotIdentifier = snapshot_identifier, + SourceRegion = _REGION, + CopyTags = True) + + return response \ No newline at end of file diff --git a/lambda_code/share_snapshots_rds/.DS_Store b/lambda_code/share_snapshots_rds/.DS_Store new file mode 100644 index 0000000000000000000000000000000000000000..5008ddfcf53c02e82d7eee2e57c38e5672ef89f6 GIT binary patch literal 6148 zcmeH~Jr2S!425mzP>H1@V-^m;4Wg<&0T*E43hX&L&p$$qDprKhvt+--jT7}7np#A3 zem<@ulZcFPQ@L2!n>{z**++&mCkOWA81W14cNZlEfg7;MkzE(HCqgga^y>{tEnwC%0;vJ&^%eQ zLs35+`xjp>T0 0: - log_message = 'Could not share all snapshots. Pending: %s' % pending_snapshots - logger.error(log_message) - raise SnapshotToolException(log_message) - - -if __name__ == '__main__': - lambda_handler(None, None) +''' +Copyright 2017 Amazon.com, Inc. or its affiliates. All Rights Reserved. + +Licensed under the Apache License, Version 2.0 (the "License"). You may not use this file except in compliance with the License. A copy of the License is located at + + http://aws.amazon.com/apache2.0/ + +or in the "license" file accompanying this file. This file is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. +''' + +# share_snapshots_rds +# This Lambda function shares snapshots created by aurora_take_snapshot with the account set in the environment variable DEST_ACCOUNT +# It will only share snapshots tagged with shareAndCopy and a value of YES +import boto3 +from datetime import datetime +import time +import os +import logging +import re +from snapshots_tool_utils import * + + +# Initialize from environment variable +LOGLEVEL = os.getenv('LOG_LEVEL', 'ERROR').strip() +DEST_ACCOUNTID = str(os.getenv('DEST_ACCOUNT')).strip() +PATTERN = os.getenv('PATTERN', 'ALL_INSTANCES') + +if os.getenv('REGION_OVERRIDE', 'NO') != 'NO': + REGION = os.getenv('REGION_OVERRIDE').strip() +else: + REGION = os.getenv('AWS_DEFAULT_REGION') + +SUPPORTED_ENGINES = [ 'mariadb', 'sqlserver-se', 'sqlserver-ee', 'sqlserver-ex', 'sqlserver-web', 'mysql', 'oracle-se', 'oracle-se1', 'oracle-se2', 'oracle-ee', 'postgres' ] + +logger = logging.getLogger() +logger.setLevel(LOGLEVEL.upper()) + + + +def lambda_handler(event, context): + pending_snapshots = 0 + client = boto3.client('rds', region_name=REGION) + response = paginate_api_call(client, 'describe_db_snapshots', 'DBSnapshots', SnapshotType='manual') + filtered = get_own_snapshots_source(PATTERN, response) + + # Search all snapshots for the correct tag + for snapshot_identifier,snapshot_object in filtered.items(): + snapshot_arn = snapshot_object['Arn'] + response_tags = client.list_tags_for_resource( + ResourceName=snapshot_arn) + + if snapshot_object['Status'].lower() == 'available' and search_tag_shared(response_tags): + try: + # Share snapshot with dest_account + response_modify = client.modify_db_snapshot_attribute( + DBSnapshotIdentifier=snapshot_identifier, + AttributeName='restore', + ValuesToAdd=[ + DEST_ACCOUNTID + ] + ) + except Exception as e: + logger.error('Exception sharing %s (%s)' % (snapshot_identifier, e)) + pending_snapshots += 1 + + if pending_snapshots > 0: + log_message = 'Could not share all snapshots. Pending: %s' % pending_snapshots + logger.error(log_message) + raise SnapshotToolException(log_message) + + +if __name__ == '__main__': + lambda_handler(None, None) \ No newline at end of file diff --git a/lambda_code/share_snapshots_rds/snapshots_tool_utils.py b/lambda_code/share_snapshots_rds/snapshots_tool_utils.py new file mode 100644 index 0000000..eeaeee8 --- /dev/null +++ b/lambda_code/share_snapshots_rds/snapshots_tool_utils.py @@ -0,0 +1,364 @@ +''' +Copyright 2017 Amazon.com, Inc. or its affiliates. All Rights Reserved. + +Licensed under the Apache License, Version 2.0 (the "License"). You may not use this file except in compliance with the License. A copy of the License is located at + + http://aws.amazon.com/apache2.0/ + +or in the "license" file accompanying this file. This file is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. +''' + + +# snapshots_tool_utils +# Support module for the Snapshot Tool for RDS + +import boto3 +from datetime import datetime, timedelta +import os +import logging +import re + + +# Initialize everything +_LOGLEVEL = os.getenv('LOG_LEVEL', 'ERROR').strip() + +_DESTINATION_REGION = os.getenv( + 'DEST_REGION', os.getenv('AWS_DEFAULT_REGION')).strip() + +_KMS_KEY_DEST_REGION = os.getenv('KMS_KEY_DEST_REGION', 'None').strip() + +_KMS_KEY_SOURCE_REGION = os.getenv('KMS_KEY_SOURCE_REGION', 'None').strip() + +_TIMESTAMP_FORMAT = '%Y-%m-%d-%H-%M' + +if os.getenv('REGION_OVERRIDE', 'NO') != 'NO': + _REGION = os.getenv('REGION_OVERRIDE').strip() +else: + _REGION = os.getenv('AWS_DEFAULT_REGION') + +_SUPPORTED_ENGINES = [ 'mariadb', 'sqlserver-se', 'sqlserver-ee', 'sqlserver-ex', 'sqlserver-web', 'mysql', 'oracle-se', 'oracle-se1', 'oracle-se2', 'oracle-ee', 'postgres' ] + + +logger = logging.getLogger() +logger.setLevel(_LOGLEVEL.upper()) + + +class SnapshotToolException(Exception): + pass + + +def search_tag_copydbsnapshot(response): +# Takes a list_tags_for_resource response and searches for our CopyDBSnapshot tag + try: + + for tag in response['TagList']: + if tag['Key'] == 'CopyDBSnapshot' and tag['Value'] == 'True': return True + + except Exception: return False + + else: return False + + + +def search_tag_created(response): +# Takes a describe_db_snapshots response and searches for our CreatedBy tag + try: + + for tag in response['TagList']: + if tag['Key'] == 'CreatedBy' and tag['Value'] == 'Snapshot Tool for RDS': return True + + except Exception: return False + + else: return False + + + +def search_tag_shared(response): +# Takes a describe_db_snapshots response and searches for our shareAndCopy tag + try: + for tag in response['TagList']: + if tag['Key'] == 'shareAndCopy' and tag['Value'] == 'YES': + for tag2 in response['TagList']: + if tag2['Key'] == 'CreatedBy' and tag2['Value'] == 'Snapshot Tool for RDS': + return True + + except Exception: + return False + + return False + + + +def search_tag_copied(response): +# Search for a tag indicating we copied this snapshot + try: + for tag in response['TagList']: + if tag['Key'] == 'CopiedBy' and tag['Value'] == 'Snapshot Tool for RDS': + return True + + except Exception: + return False + + return False + +def get_own_snapshots_no_x_account(pattern, response, REGION): + # Filters our own snapshots + filtered = {} + for snapshot in response['DBSnapshots']: + + if snapshot['SnapshotType'] == 'manual' and re.search(pattern, snapshot['DBInstanceIdentifier']) and snapshot['Engine'] in _SUPPORTED_ENGINES: + client = boto3.client('rds', region_name=REGION) + response_tags = client.list_tags_for_resource( + ResourceName=snapshot['DBSnapshotArn']) + + if search_tag_created(response_tags): + filtered[snapshot['DBSnapshotIdentifier']] = { + 'Arn': snapshot['DBSnapshotArn'], 'Status': snapshot['Status'], 'DBInstanceIdentifier': snapshot['DBInstanceIdentifier']} + #Changed the next line to search for ALL_CLUSTERS or ALL_SNAPSHOTS so it will work with no-x-account + elif snapshot['SnapshotType'] == 'manual' and pattern == 'ALL_SNAPSHOTS' and snapshot['Engine'] in _SUPPORTED_ENGINES: + client = boto3.client('rds', region_name=REGION) + response_tags = client.list_tags_for_resource( + ResourceName=snapshot['DBSnapshotArn']) + + if search_tag_created(response_tags): + filtered[snapshot['DBSnapshotIdentifier']] = { + 'Arn': snapshot['DBSnapshotArn'], 'Status': snapshot['Status'], 'DBInstanceIdentifier': snapshot['DBInstanceIdentifier']} + + return filtered + + +def get_shared_snapshots(pattern, response): +# Returns a dict with only shared snapshots filtered by pattern, with DBSnapshotIdentifier as key and the response as attribute + filtered = {} + for snapshot in response['DBSnapshots']: + if snapshot['SnapshotType'] == 'shared' and re.search(pattern, snapshot['DBInstanceIdentifier']) and snapshot['Engine'] in _SUPPORTED_ENGINES: + filtered[get_snapshot_identifier(snapshot)] = { + 'Arn': snapshot['DBSnapshotIdentifier'], 'Encrypted': snapshot['Encrypted'], 'DBInstanceIdentifier': snapshot['DBInstanceIdentifier']} + if snapshot['Encrypted'] is True: + filtered[get_snapshot_identifier(snapshot)]['KmsKeyId'] = snapshot['KmsKeyId'] + + elif snapshot['SnapshotType'] == 'shared' and pattern == 'ALL_SNAPSHOTS' and snapshot['Engine'] in _SUPPORTED_ENGINES: + filtered[get_snapshot_identifier(snapshot)] = { + 'Arn': snapshot['DBSnapshotIdentifier'], 'Encrypted': snapshot['Encrypted'], 'DBInstanceIdentifier': snapshot['DBInstanceIdentifier']} + if snapshot['Encrypted'] is True: + filtered[get_snapshot_identifier(snapshot)]['KmsKeyId'] = snapshot['KmsKeyId'] + return filtered + + + +def get_snapshot_identifier(snapshot): +# Function that will return the RDS Snapshot identifier given an ARN + match = re.match('arn:aws:rds:.*:.*:snapshot:(.+)', + snapshot['DBSnapshotArn']) + return match.group(1) + + +def get_own_snapshots_dest(pattern, response): +# Returns a dict with local snapshots, filtered by pattern, with DBSnapshotIdentifier as key and Arn, Status as attributes + filtered = {} + for snapshot in response['DBSnapshots']: + + if snapshot['SnapshotType'] == 'manual' and re.search(pattern, snapshot['DBInstanceIdentifier']) and snapshot['Engine'] in _SUPPORTED_ENGINES: + filtered[snapshot['DBSnapshotIdentifier']] = { + 'Arn': snapshot['DBSnapshotArn'], 'Status': snapshot['Status'], 'Encrypted': snapshot['Encrypted'], 'DBInstanceIdentifier': snapshot['DBInstanceIdentifier']} + + if snapshot['Encrypted'] is True: + filtered[snapshot['DBSnapshotIdentifier']]['KmsKeyId'] = snapshot['KmsKeyId'] + + elif snapshot['SnapshotType'] == 'manual' and pattern == 'ALL_SNAPSHOTS' and snapshot['Engine'] in _SUPPORTED_ENGINES: + filtered[snapshot['DBSnapshotIdentifier']] = { + 'Arn': snapshot['DBSnapshotArn'], 'Status': snapshot['Status'], 'Encrypted': snapshot['Encrypted'], 'DBInstanceIdentifier': snapshot['DBInstanceIdentifier'] } + + if snapshot['Encrypted'] is True: + filtered[snapshot['DBSnapshotIdentifier']]['KmsKeyId'] = snapshot['KmsKeyId'] + + return filtered + +def filter_instances(taggedinstance, pattern, instance_list): +# Takes the response from describe-db-instances and filters according to pattern in DBInstanceIdentifier + filtered_list = [] + + for instance in instance_list['DBInstances']: + + if taggedinstance == 'TRUE': + client = boto3.client('rds', region_name=_REGION) + response = client.list_tags_for_resource(ResourceName=instance['DBInstanceArn']) + + if pattern == 'ALL_INSTANCES' and instance['Engine'] in _SUPPORTED_ENGINES: + if (taggedinstance == 'TRUE' and search_tag_copydbsnapshot(response)) or taggedinstance == 'FALSE': + filtered_list.append(instance) + + else: + match = re.search(pattern, instance['DBInstanceIdentifier']) + + if match and instance['Engine'] in _SUPPORTED_ENGINES: + if (taggedinstance == 'TRUE' and search_tag_copydbsnapshot(response)) or taggedinstance == 'FALSE': + filtered_list.append(instance) + + return filtered_list + + +def get_own_snapshots_source(pattern, response, backup_interval=None): +# Filters our own snapshots + filtered = {} + + for snapshot in response['DBSnapshots']: + + # No need to consider snapshots that are still in progress + if 'SnapshotCreateTime' not in snapshot: + continue + + # No need to get tags for snapshots outside of the backup interval + if backup_interval and snapshot['SnapshotCreateTime'].replace(tzinfo=None) < datetime.utcnow().replace(tzinfo=None) - timedelta(hours=backup_interval): + continue + + if snapshot['SnapshotType'] == 'manual' and re.search(pattern, snapshot['DBInstanceIdentifier']) and snapshot['Engine'] in _SUPPORTED_ENGINES: + client = boto3.client('rds', region_name=_REGION) + response_tags = client.list_tags_for_resource( + ResourceName=snapshot['DBSnapshotArn']) + + if search_tag_created(response_tags): + filtered[snapshot['DBSnapshotIdentifier']] = { + 'Arn': snapshot['DBSnapshotArn'], 'Status': snapshot['Status'], 'DBInstanceIdentifier': snapshot['DBInstanceIdentifier']} + + elif snapshot['SnapshotType'] == 'manual' and (pattern == 'ALL_CLUSTERS' or pattern == 'ALL_SNAPSHOTS' or pattern == 'ALL_INSTANCES') and snapshot['Engine'] in _SUPPORTED_ENGINES: + client = boto3.client('rds', region_name=_REGION) + response_tags = client.list_tags_for_resource( + ResourceName=snapshot['DBSnapshotArn']) + + if search_tag_created(response_tags): + filtered[snapshot['DBSnapshotIdentifier']] = { + 'Arn': snapshot['DBSnapshotArn'], 'Status': snapshot['Status'], 'DBInstanceIdentifier': snapshot['DBInstanceIdentifier']} + + return filtered + + +def get_timestamp_no_minute(snapshot_identifier, snapshot_list): +# Get a timestamp from the name of a snapshot and strip out the minutes + pattern = '%s-(.+)-\d{2}' % snapshot_list[snapshot_identifier]['DBInstanceIdentifier'] + timestamp_format = '%Y-%m-%d-%H' + date_time = re.search(pattern, snapshot_identifier) + + if date_time is not None: + return datetime.strptime(date_time.group(1), timestamp_format) + + +def get_timestamp(snapshot_identifier, snapshot_list): +# Searches for a timestamp on a snapshot name + pattern = '%s-(.+)' % snapshot_list[snapshot_identifier]['DBInstanceIdentifier'] + date_time = re.search(pattern, snapshot_identifier) + + if date_time is not None: + + try: + return datetime.strptime(date_time.group(1), _TIMESTAMP_FORMAT) + + except Exception: + return None + + return None + + + +def get_latest_snapshot_ts(instance_identifier, filtered_snapshots): +# Get latest snapshot for a specific DBInstanceIdentifier + timestamps = [] + + for snapshot,snapshot_object in filtered_snapshots.items(): + + if snapshot_object['DBInstanceIdentifier'] == instance_identifier: + timestamp = get_timestamp_no_minute(snapshot, filtered_snapshots) + + if timestamp is not None: + timestamps.append(timestamp) + + if len(timestamps) > 0: + return max(timestamps) + + else: + return None + + + +def requires_backup(backup_interval, instance, filtered_snapshots): +# Returns True if latest snapshot is older than INTERVAL + latest = get_latest_snapshot_ts(instance['DBInstanceIdentifier'], filtered_snapshots) + + if latest is not None: + backup_age = datetime.now() - latest + + if backup_age.total_seconds() >= (backup_interval * 60 * 60): + return True + + else: + return False + + elif latest is None: + return True + + +def paginate_api_call(client, api_call, objecttype, *args, **kwargs): +#Takes an RDS boto client and paginates through api_call calls and returns a list of objects of objecttype + response = {} + response[objecttype] = [] + + # Create a paginator + paginator = client.get_paginator(api_call) + + # Create a PageIterator from the Paginator + page_iterator = paginator.paginate(**kwargs) + for page in page_iterator: + for item in page[objecttype]: + response[objecttype].append(item) + + return response + + +def copy_local(snapshot_identifier, snapshot_object): + client = boto3.client('rds', region_name=_REGION) + + tags = [{ + 'Key': 'CopiedBy', + 'Value': 'Snapshot Tool for RDS' + }] + + if snapshot_object['Encrypted']: + logger.info('Copying encrypted snapshot %s locally' % snapshot_identifier) + response = client.copy_db_snapshot( + SourceDBSnapshotIdentifier = snapshot_object['Arn'], + TargetDBSnapshotIdentifier = snapshot_identifier, + KmsKeyId = _KMS_KEY_SOURCE_REGION, + Tags = tags) + + else: + logger.info('Copying snapshot %s locally' %snapshot_identifier) + response = client.copy_db_snapshot( + SourceDBSnapshotIdentifier = snapshot_object['Arn'], + TargetDBSnapshotIdentifier = snapshot_identifier, + Tags = tags) + + return response + + + +def copy_remote(snapshot_identifier, snapshot_object): + client = boto3.client('rds', region_name=_DESTINATION_REGION) + + if snapshot_object['Encrypted']: + logger.info('Copying encrypted snapshot %s to remote region %s' % (snapshot_object['Arn'], _DESTINATION_REGION)) + response = client.copy_db_snapshot( + SourceDBSnapshotIdentifier = snapshot_object['Arn'], + TargetDBSnapshotIdentifier = snapshot_identifier, + KmsKeyId = _KMS_KEY_DEST_REGION, + SourceRegion = _REGION, + CopyTags = True) + + else: + logger.info('Copying snapshot %s to remote region %s' % (snapshot_object['Arn'], _DESTINATION_REGION)) + response = client.copy_db_snapshot( + SourceDBSnapshotIdentifier = snapshot_object['Arn'], + TargetDBSnapshotIdentifier = snapshot_identifier, + SourceRegion = _REGION, + CopyTags = True) + + return response \ No newline at end of file diff --git a/lambda_code/take_snapshots_rds/.DS_Store b/lambda_code/take_snapshots_rds/.DS_Store new file mode 100644 index 0000000000000000000000000000000000000000..5008ddfcf53c02e82d7eee2e57c38e5672ef89f6 GIT binary patch literal 6148 zcmeH~Jr2S!425mzP>H1@V-^m;4Wg<&0T*E43hX&L&p$$qDprKhvt+--jT7}7np#A3 zem<@ulZcFPQ@L2!n>{z**++&mCkOWA81W14cNZlEfg7;MkzE(HCqgga^y>{tEnwC%0;vJ&^%eQ zLs35+`xjp>T0 0: - log_message = 'Could not back up every instance. Backups pending: %s' % pending_backups - logger.error(log_message) - raise SnapshotToolException(log_message) - - -if __name__ == '__main__': - lambda_handler(None, None) +''' +Copyright 2017 Amazon.com, Inc. or its affiliates. All Rights Reserved. + +Licensed under the Apache License, Version 2.0 (the "License"). You may not use this file except in compliance with the License. A copy of the License is located at + + http://aws.amazon.com/apache2.0/ + +or in the "license" file accompanying this file. This file is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. +''' + + +# take_snapshots_rds +# This lambda function takes a snapshot of RDS instances according to the environment variable PATTERN and INTERVAL +# Set PATTERN to a regex that matches your RDS Instance identifiers +# Set INTERVAL to the amount of hours between backups. This function will list available manual snapshots and only trigger a new one if the latest is older than INTERVAL hours +# Set FILTERINSTANCE to True to only take snapshots for RDS Instances with tag CopyDBSnapshot set to True +import boto3 +from datetime import datetime +import time +import os +import logging +import re +from snapshots_tool_utils import * + +# Initialize everything +LOGLEVEL = os.getenv('LOG_LEVEL').strip() +BACKUP_INTERVAL = int(os.getenv('INTERVAL', '24')) +PATTERN = os.getenv('PATTERN', 'ALL_INSTANCES') +TAGGEDINSTANCE = os.getenv('TAGGEDINSTANCE', 'FALSE') + +if os.getenv('REGION_OVERRIDE', 'NO') != 'NO': + REGION = os.getenv('REGION_OVERRIDE').strip() +else: + REGION = os.getenv('AWS_DEFAULT_REGION') + +TIMESTAMP_FORMAT = '%Y-%m-%d-%H-%M' + +logger = logging.getLogger() +logger.setLevel(LOGLEVEL.upper()) + + +def lambda_handler(event, context): + + client = boto3.client('rds', region_name=REGION) + response = paginate_api_call(client, 'describe_db_instances', 'DBInstances') + now = datetime.now() + pending_backups = 0 + filtered_instances = filter_instances(TAGGEDINSTANCE, PATTERN, response) + filtered_snapshots = get_own_snapshots_source(PATTERN, paginate_api_call(client, 'describe_db_snapshots', 'DBSnapshots'), BACKUP_INTERVAL) + + for db_instance in filtered_instances: + + timestamp_format = now.strftime(TIMESTAMP_FORMAT) + + if requires_backup(BACKUP_INTERVAL, db_instance, filtered_snapshots): + + backup_age = get_latest_snapshot_ts( + db_instance['DBInstanceIdentifier'], + filtered_snapshots) + + if backup_age is not None: + logger.info('Backing up %s. Backed up %s minutes ago' % ( + db_instance['DBInstanceIdentifier'], ((now - backup_age).total_seconds() / 60))) + + else: + logger.info('Backing up %s. No previous backup found' % + db_instance['DBInstanceIdentifier']) + + snapshot_identifier = '%s-%s' % ( + db_instance['DBInstanceIdentifier'], timestamp_format) + + try: + response = client.create_db_snapshot( + DBSnapshotIdentifier=snapshot_identifier, + DBInstanceIdentifier=db_instance['DBInstanceIdentifier'], + Tags=[{'Key': 'CreatedBy', 'Value': 'Snapshot Tool for RDS'}, { + 'Key': 'CreatedOn', 'Value': timestamp_format}, {'Key': 'shareAndCopy', 'Value': 'YES'}] + ) + except Exception as e: + pending_backups += 1 + logger.info('Could not create snapshot %s (%s)' % (snapshot_identifier, e)) + else: + + backup_age = get_latest_snapshot_ts( + db_instance['DBInstanceIdentifier'], + filtered_snapshots) + + logger.info('Skipped %s. Does not require backup. Backed up %s minutes ago' % ( + db_instance['DBInstanceIdentifier'], (now - backup_age).total_seconds() / 60)) + + if pending_backups > 0: + log_message = 'Could not back up every instance. Backups pending: %s' % pending_backups + logger.error(log_message) + raise SnapshotToolException(log_message) + + +if __name__ == '__main__': + lambda_handler(None, None) \ No newline at end of file diff --git a/lambda_code/take_snapshots_rds/snapshots_tool_utils.py b/lambda_code/take_snapshots_rds/snapshots_tool_utils.py new file mode 100644 index 0000000..eeaeee8 --- /dev/null +++ b/lambda_code/take_snapshots_rds/snapshots_tool_utils.py @@ -0,0 +1,364 @@ +''' +Copyright 2017 Amazon.com, Inc. or its affiliates. All Rights Reserved. + +Licensed under the Apache License, Version 2.0 (the "License"). You may not use this file except in compliance with the License. A copy of the License is located at + + http://aws.amazon.com/apache2.0/ + +or in the "license" file accompanying this file. This file is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. +''' + + +# snapshots_tool_utils +# Support module for the Snapshot Tool for RDS + +import boto3 +from datetime import datetime, timedelta +import os +import logging +import re + + +# Initialize everything +_LOGLEVEL = os.getenv('LOG_LEVEL', 'ERROR').strip() + +_DESTINATION_REGION = os.getenv( + 'DEST_REGION', os.getenv('AWS_DEFAULT_REGION')).strip() + +_KMS_KEY_DEST_REGION = os.getenv('KMS_KEY_DEST_REGION', 'None').strip() + +_KMS_KEY_SOURCE_REGION = os.getenv('KMS_KEY_SOURCE_REGION', 'None').strip() + +_TIMESTAMP_FORMAT = '%Y-%m-%d-%H-%M' + +if os.getenv('REGION_OVERRIDE', 'NO') != 'NO': + _REGION = os.getenv('REGION_OVERRIDE').strip() +else: + _REGION = os.getenv('AWS_DEFAULT_REGION') + +_SUPPORTED_ENGINES = [ 'mariadb', 'sqlserver-se', 'sqlserver-ee', 'sqlserver-ex', 'sqlserver-web', 'mysql', 'oracle-se', 'oracle-se1', 'oracle-se2', 'oracle-ee', 'postgres' ] + + +logger = logging.getLogger() +logger.setLevel(_LOGLEVEL.upper()) + + +class SnapshotToolException(Exception): + pass + + +def search_tag_copydbsnapshot(response): +# Takes a list_tags_for_resource response and searches for our CopyDBSnapshot tag + try: + + for tag in response['TagList']: + if tag['Key'] == 'CopyDBSnapshot' and tag['Value'] == 'True': return True + + except Exception: return False + + else: return False + + + +def search_tag_created(response): +# Takes a describe_db_snapshots response and searches for our CreatedBy tag + try: + + for tag in response['TagList']: + if tag['Key'] == 'CreatedBy' and tag['Value'] == 'Snapshot Tool for RDS': return True + + except Exception: return False + + else: return False + + + +def search_tag_shared(response): +# Takes a describe_db_snapshots response and searches for our shareAndCopy tag + try: + for tag in response['TagList']: + if tag['Key'] == 'shareAndCopy' and tag['Value'] == 'YES': + for tag2 in response['TagList']: + if tag2['Key'] == 'CreatedBy' and tag2['Value'] == 'Snapshot Tool for RDS': + return True + + except Exception: + return False + + return False + + + +def search_tag_copied(response): +# Search for a tag indicating we copied this snapshot + try: + for tag in response['TagList']: + if tag['Key'] == 'CopiedBy' and tag['Value'] == 'Snapshot Tool for RDS': + return True + + except Exception: + return False + + return False + +def get_own_snapshots_no_x_account(pattern, response, REGION): + # Filters our own snapshots + filtered = {} + for snapshot in response['DBSnapshots']: + + if snapshot['SnapshotType'] == 'manual' and re.search(pattern, snapshot['DBInstanceIdentifier']) and snapshot['Engine'] in _SUPPORTED_ENGINES: + client = boto3.client('rds', region_name=REGION) + response_tags = client.list_tags_for_resource( + ResourceName=snapshot['DBSnapshotArn']) + + if search_tag_created(response_tags): + filtered[snapshot['DBSnapshotIdentifier']] = { + 'Arn': snapshot['DBSnapshotArn'], 'Status': snapshot['Status'], 'DBInstanceIdentifier': snapshot['DBInstanceIdentifier']} + #Changed the next line to search for ALL_CLUSTERS or ALL_SNAPSHOTS so it will work with no-x-account + elif snapshot['SnapshotType'] == 'manual' and pattern == 'ALL_SNAPSHOTS' and snapshot['Engine'] in _SUPPORTED_ENGINES: + client = boto3.client('rds', region_name=REGION) + response_tags = client.list_tags_for_resource( + ResourceName=snapshot['DBSnapshotArn']) + + if search_tag_created(response_tags): + filtered[snapshot['DBSnapshotIdentifier']] = { + 'Arn': snapshot['DBSnapshotArn'], 'Status': snapshot['Status'], 'DBInstanceIdentifier': snapshot['DBInstanceIdentifier']} + + return filtered + + +def get_shared_snapshots(pattern, response): +# Returns a dict with only shared snapshots filtered by pattern, with DBSnapshotIdentifier as key and the response as attribute + filtered = {} + for snapshot in response['DBSnapshots']: + if snapshot['SnapshotType'] == 'shared' and re.search(pattern, snapshot['DBInstanceIdentifier']) and snapshot['Engine'] in _SUPPORTED_ENGINES: + filtered[get_snapshot_identifier(snapshot)] = { + 'Arn': snapshot['DBSnapshotIdentifier'], 'Encrypted': snapshot['Encrypted'], 'DBInstanceIdentifier': snapshot['DBInstanceIdentifier']} + if snapshot['Encrypted'] is True: + filtered[get_snapshot_identifier(snapshot)]['KmsKeyId'] = snapshot['KmsKeyId'] + + elif snapshot['SnapshotType'] == 'shared' and pattern == 'ALL_SNAPSHOTS' and snapshot['Engine'] in _SUPPORTED_ENGINES: + filtered[get_snapshot_identifier(snapshot)] = { + 'Arn': snapshot['DBSnapshotIdentifier'], 'Encrypted': snapshot['Encrypted'], 'DBInstanceIdentifier': snapshot['DBInstanceIdentifier']} + if snapshot['Encrypted'] is True: + filtered[get_snapshot_identifier(snapshot)]['KmsKeyId'] = snapshot['KmsKeyId'] + return filtered + + + +def get_snapshot_identifier(snapshot): +# Function that will return the RDS Snapshot identifier given an ARN + match = re.match('arn:aws:rds:.*:.*:snapshot:(.+)', + snapshot['DBSnapshotArn']) + return match.group(1) + + +def get_own_snapshots_dest(pattern, response): +# Returns a dict with local snapshots, filtered by pattern, with DBSnapshotIdentifier as key and Arn, Status as attributes + filtered = {} + for snapshot in response['DBSnapshots']: + + if snapshot['SnapshotType'] == 'manual' and re.search(pattern, snapshot['DBInstanceIdentifier']) and snapshot['Engine'] in _SUPPORTED_ENGINES: + filtered[snapshot['DBSnapshotIdentifier']] = { + 'Arn': snapshot['DBSnapshotArn'], 'Status': snapshot['Status'], 'Encrypted': snapshot['Encrypted'], 'DBInstanceIdentifier': snapshot['DBInstanceIdentifier']} + + if snapshot['Encrypted'] is True: + filtered[snapshot['DBSnapshotIdentifier']]['KmsKeyId'] = snapshot['KmsKeyId'] + + elif snapshot['SnapshotType'] == 'manual' and pattern == 'ALL_SNAPSHOTS' and snapshot['Engine'] in _SUPPORTED_ENGINES: + filtered[snapshot['DBSnapshotIdentifier']] = { + 'Arn': snapshot['DBSnapshotArn'], 'Status': snapshot['Status'], 'Encrypted': snapshot['Encrypted'], 'DBInstanceIdentifier': snapshot['DBInstanceIdentifier'] } + + if snapshot['Encrypted'] is True: + filtered[snapshot['DBSnapshotIdentifier']]['KmsKeyId'] = snapshot['KmsKeyId'] + + return filtered + +def filter_instances(taggedinstance, pattern, instance_list): +# Takes the response from describe-db-instances and filters according to pattern in DBInstanceIdentifier + filtered_list = [] + + for instance in instance_list['DBInstances']: + + if taggedinstance == 'TRUE': + client = boto3.client('rds', region_name=_REGION) + response = client.list_tags_for_resource(ResourceName=instance['DBInstanceArn']) + + if pattern == 'ALL_INSTANCES' and instance['Engine'] in _SUPPORTED_ENGINES: + if (taggedinstance == 'TRUE' and search_tag_copydbsnapshot(response)) or taggedinstance == 'FALSE': + filtered_list.append(instance) + + else: + match = re.search(pattern, instance['DBInstanceIdentifier']) + + if match and instance['Engine'] in _SUPPORTED_ENGINES: + if (taggedinstance == 'TRUE' and search_tag_copydbsnapshot(response)) or taggedinstance == 'FALSE': + filtered_list.append(instance) + + return filtered_list + + +def get_own_snapshots_source(pattern, response, backup_interval=None): +# Filters our own snapshots + filtered = {} + + for snapshot in response['DBSnapshots']: + + # No need to consider snapshots that are still in progress + if 'SnapshotCreateTime' not in snapshot: + continue + + # No need to get tags for snapshots outside of the backup interval + if backup_interval and snapshot['SnapshotCreateTime'].replace(tzinfo=None) < datetime.utcnow().replace(tzinfo=None) - timedelta(hours=backup_interval): + continue + + if snapshot['SnapshotType'] == 'manual' and re.search(pattern, snapshot['DBInstanceIdentifier']) and snapshot['Engine'] in _SUPPORTED_ENGINES: + client = boto3.client('rds', region_name=_REGION) + response_tags = client.list_tags_for_resource( + ResourceName=snapshot['DBSnapshotArn']) + + if search_tag_created(response_tags): + filtered[snapshot['DBSnapshotIdentifier']] = { + 'Arn': snapshot['DBSnapshotArn'], 'Status': snapshot['Status'], 'DBInstanceIdentifier': snapshot['DBInstanceIdentifier']} + + elif snapshot['SnapshotType'] == 'manual' and (pattern == 'ALL_CLUSTERS' or pattern == 'ALL_SNAPSHOTS' or pattern == 'ALL_INSTANCES') and snapshot['Engine'] in _SUPPORTED_ENGINES: + client = boto3.client('rds', region_name=_REGION) + response_tags = client.list_tags_for_resource( + ResourceName=snapshot['DBSnapshotArn']) + + if search_tag_created(response_tags): + filtered[snapshot['DBSnapshotIdentifier']] = { + 'Arn': snapshot['DBSnapshotArn'], 'Status': snapshot['Status'], 'DBInstanceIdentifier': snapshot['DBInstanceIdentifier']} + + return filtered + + +def get_timestamp_no_minute(snapshot_identifier, snapshot_list): +# Get a timestamp from the name of a snapshot and strip out the minutes + pattern = '%s-(.+)-\d{2}' % snapshot_list[snapshot_identifier]['DBInstanceIdentifier'] + timestamp_format = '%Y-%m-%d-%H' + date_time = re.search(pattern, snapshot_identifier) + + if date_time is not None: + return datetime.strptime(date_time.group(1), timestamp_format) + + +def get_timestamp(snapshot_identifier, snapshot_list): +# Searches for a timestamp on a snapshot name + pattern = '%s-(.+)' % snapshot_list[snapshot_identifier]['DBInstanceIdentifier'] + date_time = re.search(pattern, snapshot_identifier) + + if date_time is not None: + + try: + return datetime.strptime(date_time.group(1), _TIMESTAMP_FORMAT) + + except Exception: + return None + + return None + + + +def get_latest_snapshot_ts(instance_identifier, filtered_snapshots): +# Get latest snapshot for a specific DBInstanceIdentifier + timestamps = [] + + for snapshot,snapshot_object in filtered_snapshots.items(): + + if snapshot_object['DBInstanceIdentifier'] == instance_identifier: + timestamp = get_timestamp_no_minute(snapshot, filtered_snapshots) + + if timestamp is not None: + timestamps.append(timestamp) + + if len(timestamps) > 0: + return max(timestamps) + + else: + return None + + + +def requires_backup(backup_interval, instance, filtered_snapshots): +# Returns True if latest snapshot is older than INTERVAL + latest = get_latest_snapshot_ts(instance['DBInstanceIdentifier'], filtered_snapshots) + + if latest is not None: + backup_age = datetime.now() - latest + + if backup_age.total_seconds() >= (backup_interval * 60 * 60): + return True + + else: + return False + + elif latest is None: + return True + + +def paginate_api_call(client, api_call, objecttype, *args, **kwargs): +#Takes an RDS boto client and paginates through api_call calls and returns a list of objects of objecttype + response = {} + response[objecttype] = [] + + # Create a paginator + paginator = client.get_paginator(api_call) + + # Create a PageIterator from the Paginator + page_iterator = paginator.paginate(**kwargs) + for page in page_iterator: + for item in page[objecttype]: + response[objecttype].append(item) + + return response + + +def copy_local(snapshot_identifier, snapshot_object): + client = boto3.client('rds', region_name=_REGION) + + tags = [{ + 'Key': 'CopiedBy', + 'Value': 'Snapshot Tool for RDS' + }] + + if snapshot_object['Encrypted']: + logger.info('Copying encrypted snapshot %s locally' % snapshot_identifier) + response = client.copy_db_snapshot( + SourceDBSnapshotIdentifier = snapshot_object['Arn'], + TargetDBSnapshotIdentifier = snapshot_identifier, + KmsKeyId = _KMS_KEY_SOURCE_REGION, + Tags = tags) + + else: + logger.info('Copying snapshot %s locally' %snapshot_identifier) + response = client.copy_db_snapshot( + SourceDBSnapshotIdentifier = snapshot_object['Arn'], + TargetDBSnapshotIdentifier = snapshot_identifier, + Tags = tags) + + return response + + + +def copy_remote(snapshot_identifier, snapshot_object): + client = boto3.client('rds', region_name=_DESTINATION_REGION) + + if snapshot_object['Encrypted']: + logger.info('Copying encrypted snapshot %s to remote region %s' % (snapshot_object['Arn'], _DESTINATION_REGION)) + response = client.copy_db_snapshot( + SourceDBSnapshotIdentifier = snapshot_object['Arn'], + TargetDBSnapshotIdentifier = snapshot_identifier, + KmsKeyId = _KMS_KEY_DEST_REGION, + SourceRegion = _REGION, + CopyTags = True) + + else: + logger.info('Copying snapshot %s to remote region %s' % (snapshot_object['Arn'], _DESTINATION_REGION)) + response = client.copy_db_snapshot( + SourceDBSnapshotIdentifier = snapshot_object['Arn'], + TargetDBSnapshotIdentifier = snapshot_identifier, + SourceRegion = _REGION, + CopyTags = True) + + return response \ No newline at end of file diff --git a/rds_snapshot_tool_destination/.DS_Store b/rds_snapshot_tool_destination/.DS_Store new file mode 100644 index 0000000000000000000000000000000000000000..3e897367dd90905fc3ff7a4c4162518906fce331 GIT binary patch literal 6148 zcmeHK%}T>S5T0$TO(;SR7CZ*L7Hk#4;w8lT0!H+pQX5k=m}X1UnnNk%rZ42H_&m<+ zZbVzXcn~QwF#ApCXVd*sb~*qc{8`iiZ~(wUC9K%Ed?OSmU6G3MR1t;RBY`1gVV;e{ zWF?y&|B(TDcXb$nga~Hv>HWzNBd_5&iL+_D{U$2a+Qw$xs#^`~CU}r}Fda;1nLn6Z z(Wi4Mld!Z0;YB>m`_A@>Os9i59Zpn191St$`Z7)L<&_IrYUIVog0tvYY$Nhn-w1>91Qe_emWj9!Id@u&xk{{}? z^n3DVcimFl;G3z-49vXo?2LBxR_uC+NDXJ%fT%}A0hF=Pf%%K@IP01;e546f_8A#X zsHBQ=(reig_!k-Ay*r>OW!Sed?C^>UO>Abv83zH)AM|xl6W!3NhRE=h9?^Qc! z2i@Rnd}HQul~l8Gm`p$OuMfr)>B>*ikNIRi?!CS;x=M0Anc9Lpn;_-Or(9=dJ~T6( zm9{o^13@R~jC+TRMgKUw7%Y#&#l;ES^Rv_CvJ)J>d3*VOd|TXU^XPb0_yn!&dtAX2 zoDHv-mEVgLOXiXdsSoi~7e-$rb2TAXi3kJ2fH1I24EXaa=@Wv#-1Ft8N{_m2WYWD@Q!^d6CE3p(`iln-Aq@ zR=z_~c6K~JwCSWGhhl{RVc<^&xc7tf`2K(Qa{q6W#0Uez!2ijBYK`L25bx&i*0s0f uyEcS=gR*d5?eJF#9D5Zbm#^X@s1?LRc7Tz`+94tk`6FOy5F-q1m4Rmp@>?YU literal 0 HcmV?d00001 diff --git a/rds_snapshot_tool_destination/modules/rds_destination/cloudwatch.tf b/rds_snapshot_tool_destination/modules/rds_destination/cloudwatch.tf new file mode 100644 index 0000000..9b7574e --- /dev/null +++ b/rds_snapshot_tool_destination/modules/rds_destination/cloudwatch.tf @@ -0,0 +1,94 @@ +resource "aws_cloudwatch_metric_alarm" "alarmcw_copy_failed_dest" { + + alarm_name = "failed-rds-copy" + comparison_operator = "GreaterThanOrEqualToThreshold" + evaluation_periods = "1" + metric_name = "ExecutionsFailed" + namespace = "AWS/States" + period = "300" + statistic = "Sum" + threshold = "1.0" + + dimensions = { + StateMachineArn = aws_sfn_state_machine.statemachine_copy_old_snapshots_dest_rds.arn + } + + alarm_description = "This metric monitors state machine failure for copying snapshots" + alarm_actions = [ + aws_sns_topic.copy_failed_dest.id + ] +} + +resource "aws_cloudwatch_metric_alarm" "alarmcw_delete_old_failed_dest" { + alarm_name = "failed-rds-delete-old-snapshot" + comparison_operator = "GreaterThanOrEqualToThreshold" + evaluation_periods = "2" + metric_name = "ExecutionsFailed" + namespace = "AWS/States" + period = "3600" + statistic = "Sum" + threshold = "2.0" + + dimensions = { + StateMachineArn = aws_sfn_state_machine.statemachine_delete_old_snapshots_dest_rds.arn + } + + alarm_description = "This metric monitors state machine failure for deleting old snapshots" + alarm_actions = [ + aws_sns_topic.delete_old_failed_dest.id + ] +} + +resource "aws_cloudwatch_event_rule" "copy_snapshots_rds" { + name = "capture-snapshot-copy-cw-events" + description = "Capture all CW state change events and triggers RDS copy state machine in destination account" + schedule_expression = "cron(30 * * * ? *)" + is_enabled = true + + + event_pattern = <S5T0$TO(;SR7CZ*L7Hk#4;w8lT0!H+pQX5k=m}X1UnnNk%rZ42H_&m<+ zZbVzXcn~QwF#ApCXVd*sb~*qc{8`iiZ~(wUC9K%Ed?OSmU6G3MR1t;RBY`1gVV;e{ zWF?y&|B(TDcXb$nga~Hv>HWzNBd_5&iL+_D{U$2a+Qw$xs#^`~CU}r}Fda;1nLn6Z z(Wi4Mld!Z0;YB>m`_A@>Os9i59Zpn191St$`Z7)|LPXYI(Eo7n&gvxX)` zGn&!`6`gE9;TiA@{BI2K+1}7nQdM^Ar`d5?%#;4Ruc~g+vM#2kpv()ToSl?)uI6Jkt8;B? zV;hh?*-QF^#bUS{j`o+k;bQaw?&nV*m&=|U?CgE{mi(xG*6O!qQsCk-^48%kT*Fzp zv0PtN8Ga}yu1xq9PwPt6mx%M?k{aA*8a4q-8R^%+OlMQ~S?L5HF~ZDwdHD>uqa(Lp z{#N`BJOiGACuV@}hY-pbIczPOs{_Wg0Kfp#M&NTV!8M`7$YE;{5s0v*KwBzYiy>?| z=0lekIczQ3auTlj5bn#ubtuAo9qWfCokV2OThD-J;2{I+cG&0l|M1`a|HC3b@(g$e zUKIl(IEW9%*pho&JDcOTHiUkMvT$B&(Uicr=h!&>6u*ZWK`dkk7&&Y$Vg};>2pAf? J@eDjG12<9Mno$4% literal 0 HcmV?d00001 diff --git a/rds_snapshot_tool_source/modules/rds_source/cloudwatch.tf b/rds_snapshot_tool_source/modules/rds_source/cloudwatch.tf new file mode 100644 index 0000000..ba9e6da --- /dev/null +++ b/rds_snapshot_tool_source/modules/rds_source/cloudwatch.tf @@ -0,0 +1,121 @@ + +resource "aws_cloudwatch_metric_alarm" "alarmcw_backups_failed" { + + alarm_name = "failed-rds-copy" + comparison_operator = "GreaterThanOrEqualToThreshold" + evaluation_periods = "1" + metric_name = "ExecutionsFailed" + namespace = "AWS/States" + period = "300" + statistic = "Sum" + threshold = "1.0" + + dimensions = { + StateMachineArn = aws_sfn_state_machine.state_machine_take_snapshots_rds.arn + } + + alarm_description = "" + alarm_actions = [ + aws_sns_topic.backups_failed.id + ] +} + +resource "aws_cloudwatch_metric_alarm" "alarmcw_share_failed" { + alarm_name = "failed-rds-share-snapshot" + comparison_operator = "GreaterThanOrEqualToThreshold" + evaluation_periods = "2" + metric_name = "ExecutionsFailed" + namespace = "AWS/States" + period = "3600" + statistic = "Sum" + threshold = "2.0" + + dimensions = { + StateMachineArn = aws_sfn_state_machine.statemachine_share_snapshots_rds.arn + } + + alarm_description = "" + alarm_actions = [ + aws_sns_topic.share_failed.id + ] +} + +resource "aws_cloudwatch_metric_alarm" "alarmcw_delete_old_failed" { + alarm_name = "failed-rds-delete-old-snapshot" + comparison_operator = "GreaterThanOrEqualToThreshold" + evaluation_periods = "2" + metric_name = "ExecutionsFailed" + namespace = "AWS/States" + period = "3600" + statistic = "Sum" + threshold = "2.0" + + dimensions = { + StateMachineArn = aws_sfn_state_machine.statemachine_delete_old_snapshots_rds.arn + } + + alarm_description = "" + alarm_actions = [ + aws_sns_topic.delete_old_failed.id + ] +} + +resource "aws_cloudwatch_event_rule" "backup_rds" { + name = "trigger-take-snapshot-state-machine" + description = "Triggers the TakeSnapshotsRDS state machine" + schedule_expression = "cron(${var.backup_schedule})" + is_enabled = true +} + +resource "aws_cloudwatch_event_target" "backup_rds" { + rule = aws_cloudwatch_event_rule.backup_rds.name + target_id = "TakeSnapshotTarget1" + arn = aws_sfn_state_machine.state_machine_take_snapshots_rds.id + role_arn = aws_iam_role.iamrole_step_invocation.arn +} + + +### Uncomment to share the snapshots on a schedule #### +# resource "aws_cloudwatch_event_rule" "share_snapshots_rds" { +# name = "trigger-share-snapshot-state-machine" +# description = "Triggers the ShareSnapshotsRDS state machine" +# schedule_expression = "cron(/10 * * * ? *)" +# is_enabled = true +# } + +# resource "aws_cloudwatch_event_target" "share_snapshots_rds" { +# rule = aws_cloudwatch_event_rule.share_snapshots_rds.name +# target_id = "ShareSnapshotTarget1" +# arn = aws_sfn_state_machine.statemachine_share_snapshots_rds.id +# role_arn = aws_iam_role.iamrole_step_invocation.arn +# } + +resource "aws_cloudwatch_event_rule" "delete_old_snapshots_rds" { + name = "trigger-delete-snapshot-state-machine" + description = "Triggers the DeleteOldSnapshotsRDS state machine" + schedule_expression = "cron(0 /1 * * ? *)" + is_enabled = true +} + +resource "aws_cloudwatch_event_target" "delete_old_snapshots_rds" { + rule = aws_cloudwatch_event_rule.delete_old_snapshots_rds.name + target_id = "DeleteSnapshotTarget1" + arn = aws_sfn_state_machine.statemachine_delete_old_snapshots_rds.id + role_arn = aws_iam_role.iamrole_step_invocation.arn +} + +resource "aws_cloudwatch_log_group" "take_snapshots_rds" { + name = "/aws/lambda/${aws_lambda_function.lambda_take_snapshots_rds.function_name}" + retention_in_days = var.lambda_cw_log_retention +} + + +resource "aws_cloudwatch_log_group" "share_snapshots_rds" { + name = "/aws/lambda/${aws_lambda_function.lambda_share_snapshots_rds.function_name}" + retention_in_days = var.lambda_cw_log_retention +} + +resource "aws_cloudwatch_log_group" "delete_old_snapshots_rds" { + name = "/aws/lambda/${var.log_group_name}" + retention_in_days = var.lambda_cw_log_retention +} diff --git a/rds_snapshot_tool_source/modules/rds_source/data.tf b/rds_snapshot_tool_source/modules/rds_source/data.tf new file mode 100644 index 0000000..cfd929b --- /dev/null +++ b/rds_snapshot_tool_source/modules/rds_source/data.tf @@ -0,0 +1,4 @@ +data "aws_region" "current" {} + +data "aws_caller_identity" "current" {} + diff --git a/rds_snapshot_tool_source/modules/rds_source/iam.tf b/rds_snapshot_tool_source/modules/rds_source/iam.tf new file mode 100644 index 0000000..4dacd6e --- /dev/null +++ b/rds_snapshot_tool_source/modules/rds_source/iam.tf @@ -0,0 +1,142 @@ +resource "aws_iam_role" "snapshots_rds" { + name = "snapshot-rds-source" + assume_role_policy = jsonencode({ + Version = "2012-10-17" + Statement = [ + { + Effect = "Allow" + Principal = { + Service = "lambda.amazonaws.com" + } + Action = "sts:AssumeRole" + } + ] + }) + force_detach_policies = true +} + +data "aws_iam_policy_document" "snapshot_rds" { + + statement { + sid = "snapshotsRdsCwLogs" + effect = "Allow" + actions = [ + "logs:CreateLogGroup", + "logs:CreateLogStream", + "logs:PutLogEvents" + ] + resources = ["*"] + } + + statement { + sid = "snapshotsRds" + effect = "Allow" + actions = [ + "rds:CreateDBSnapshot", + "rds:DeleteDBSnapshot", + "rds:DescribeDBInstances", + "rds:DescribeDBSnapshots", + "rds:ModifyDBSnapshotAttribute", + "rds:DescribeDBSnapshotAttributes", + "rds:ListTagsForResource", + "rds:AddTagsToResource" + ] + resources = ["*"] + + } + +} + +resource "aws_iam_policy" "snapshot_rds" { + name = "snapshot-rds-policy" + policy = data.aws_iam_policy_document.snapshot_rds.json +} + +resource "aws_iam_role_policy_attachment" "snapshot_rds" { + role = aws_iam_role.snapshots_rds.name + policy_arn = aws_iam_policy.snapshot_rds.arn +} + + +resource "aws_iam_role" "state_execution" { + assume_role_policy = jsonencode({ + Version = "2012-10-17" + Statement = [ + { + Effect = "Allow" + Principal = { + Service = "lambda.amazonaws.com" + } + Action = "sts:AssumeRole" + } + ] + }) + force_detach_policies = true +} + + +resource "aws_iam_role" "iamrole_state_execution" { + name = "invoke-state-machine-rds-source" + assume_role_policy = jsonencode({ + Version = "2012-10-17" + Statement = [ + { + Effect = "Allow" + Principal = { + Service = join("", ["states.", data.aws_region.current.name, ".amazonaws.com"]) + } + Action = "sts:AssumeRole" + } + ] + }) + force_detach_policies = true + inline_policy { + name = "inline_policy_rds_snapshot" + policy = jsonencode({ + Version = "2012-10-17" + Statement = [ + { + Effect = "Allow" + Action = [ + "lambda:InvokeFunction", + "states:StartExecution" + + ] + Resource = "*" + } + ] + }) + } +} + +resource "aws_iam_role" "iamrole_step_invocation" { + name = "invoke-state-machines-rds-source" + assume_role_policy = jsonencode({ + Version = "2012-10-17" + Statement = [ + { + Effect = "Allow" + Principal = { + Service = "events.amazonaws.com" + } + Action = "sts:AssumeRole" + } + ] + }) + force_detach_policies = true + inline_policy { + name = "inline_policy_state_invocation" + policy = jsonencode({ + Version = "2012-10-17" + Statement = [ + { + Effect = "Allow" + Action = [ + "states:StartExecution" + ] + Resource = "*" + } + ] + }) + } +} diff --git a/rds_snapshot_tool_source/modules/rds_source/lambda.tf b/rds_snapshot_tool_source/modules/rds_source/lambda.tf new file mode 100644 index 0000000..b0902de --- /dev/null +++ b/rds_snapshot_tool_source/modules/rds_source/lambda.tf @@ -0,0 +1,61 @@ + +resource "aws_lambda_function" "lambda_take_snapshots_rds" { + function_name = "take-rds-snapshots" + s3_bucket = var.code_bucket + s3_key = "take_snapshots_rds.zip" + memory_size = 512 + description = "This functions triggers snapshots creation for RDS instances. It checks for existing snapshots following the pattern and interval specified in the environment variables with the following format: -YYYY-MM-DD-HH-MM" + environment { + variables = { + INTERVAL = var.backup_interval + PATTERN = var.instance_name_pattern + LOG_LEVEL = var.log_level + REGION_OVERRIDE = var.source_region_override + TAGGEDINSTANCE = var.tagged_instance + } + } + role = aws_iam_role.snapshots_rds.arn + runtime = "python3.7" + handler = "lambda_function.lambda_handler" + timeout = 300 +} + +resource "aws_lambda_function" "lambda_share_snapshots_rds" { + function_name = "share-rds-snapshot" + s3_bucket = var.code_bucket + s3_key = "share_snapshots_rds.zip" + memory_size = 512 + description = "This function shares snapshots created by the take_snapshots_rds function with DEST_ACCOUNT specified in the environment variables. " + environment { + variables = { + DEST_ACCOUNT = var.destination_account + LOG_LEVEL = var.log_level + PATTERN = var.instance_name_pattern + REGION_OVERRIDE = var.source_region_override + } + } + role = aws_iam_role.snapshots_rds.arn + runtime = "python3.7" + handler = "lambda_function.lambda_handler" + timeout = 300 +} + +resource "aws_lambda_function" "lambda_delete_snapshots_rds" { + function_name = "delete-old-rds-snapshots" + s3_bucket = var.code_bucket + s3_key = "delete_old_snapshots_rds.zip" + memory_size = 512 + description = "This function deletes snapshots created by the take_snapshots_rds function. " + environment { + variables = { + RETENTION_DAYS = var.retention_days + PATTERN = var.instance_name_pattern + LOG_LEVEL = var.log_level + REGION_OVERRIDE = var.source_region_override + } + } + role = aws_iam_role.snapshots_rds.arn + runtime = "python3.7" + handler = "lambda_function.lambda_handler" + timeout = 300 +} diff --git a/rds_snapshot_tool_source/modules/rds_source/locals.tf b/rds_snapshot_tool_source/modules/rds_source/locals.tf new file mode 100644 index 0000000..283731b --- /dev/null +++ b/rds_snapshot_tool_source/modules/rds_source/locals.tf @@ -0,0 +1,5 @@ +locals { + Share = var.share_snapshots == "TRUE" + DeleteOld = var.delete_old_snapshots == "TRUE" +} + diff --git a/rds_snapshot_tool_source/modules/rds_source/output.tf b/rds_snapshot_tool_source/modules/rds_source/output.tf new file mode 100644 index 0000000..1670971 --- /dev/null +++ b/rds_snapshot_tool_source/modules/rds_source/output.tf @@ -0,0 +1,20 @@ +output "backup_failed_topic" { + description = "Subscribe to this topic to receive alerts of failed backups" + value = aws_sns_topic.backups_failed.id +} + +output "share_failed_topic" { + description = "Subscribe to this topic to receive alerts of failures at sharing snapshots with destination account" + value = aws_sns_topic.share_failed.id +} + +output "delete_old_failed_topic" { + description = "Subscribe to this topic to receive alerts of failures at deleting old snapshots" + value = aws_sns_topic.delete_old_failed.id +} + +output "source_url" { + description = "For more information and documentation, see the source repository at GitHub." + value = "https://github.com/awslabs/rds-snapshot-tool" +} + diff --git a/rds_snapshot_tool_source/modules/rds_source/sns.tf b/rds_snapshot_tool_source/modules/rds_source/sns.tf new file mode 100644 index 0000000..854090a --- /dev/null +++ b/rds_snapshot_tool_source/modules/rds_source/sns.tf @@ -0,0 +1,62 @@ +data "aws_iam_policy_document" "failed_source" { + + statement { + sid = "__default_policy_ID" + principals { + type = "AWS" + identifiers = ["*"] + } + effect = "Allow" + actions = [ + "SNS:GetTopicAttributes", + "SNS:SetTopicAttributes", + "SNS:AddPermission", + "SNS:RemovePermission", + "SNS:DeleteTopic", + "SNS:Subscribe", + "SNS:ListSubscriptionsByTopic", + "SNS:Publish", + "SNS:Receive" + ] + resources = [ + "*" + ] + + condition { + test = "StringEquals" + variable = "AWS:SourceOwner" + values = [data.aws_caller_identity.current.account_id] + } + } +} + + +resource "aws_sns_topic" "backups_failed" { + name = "backups_failed_rds_topic" + display_name = "backups_failed_rds" +} + +resource "aws_sns_topic_policy" "backups_failed" { + policy = data.aws_iam_policy_document.failed_source.json + arn = aws_sns_topic.backups_failed.arn +} + +resource "aws_sns_topic" "share_failed" { + name = "share_failed_rds_topic" + display_name = "share_failed_rds" +} + +resource "aws_sns_topic_policy" "share_failed" { + policy = data.aws_iam_policy_document.failed_source.json + arn = aws_sns_topic.share_failed.arn +} + +resource "aws_sns_topic" "delete_old_failed" { + name = "delete_old_failed_rds_topic" + display_name = "delete_old_failed_rds" +} + +resource "aws_sns_topic_policy" "topic_delete_old_failed" { + policy = data.aws_iam_policy_document.failed_source.json + arn = aws_sns_topic.delete_old_failed.arn +} diff --git a/rds_snapshot_tool_source/modules/rds_source/stepfunction.tf b/rds_snapshot_tool_source/modules/rds_source/stepfunction.tf new file mode 100644 index 0000000..cda3af4 --- /dev/null +++ b/rds_snapshot_tool_source/modules/rds_source/stepfunction.tf @@ -0,0 +1,124 @@ + +resource "aws_sfn_state_machine" "state_machine_take_snapshots_rds" { + name = "take-snapshots-rds" + role_arn = aws_iam_role.iamrole_state_execution.arn + + definition = < Date: Wed, 1 Mar 2023 10:30:46 -0500 Subject: [PATCH 48/48] add readme --- README.md | 128 ++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 128 insertions(+) create mode 100644 README.md diff --git a/README.md b/README.md new file mode 100644 index 0000000..1a0674d --- /dev/null +++ b/README.md @@ -0,0 +1,128 @@ + +# Snapshot Tool for Amazon RDS +​ +The Snapshot Tool for RDS automates the task of creating manual snapshots, copying them into a different account and a different region, and deleting them after a specified number of days. It also allows you to specify the backup schedule (at what times and how often) and a retention period in days. This version will work with all Amazon RDS instances except Amazon Aurora. For a version that works with Amazon Aurora, please visit the [Snapshot Tool for Amazon Aurora](https://github.com/awslabs/aurora-snapshot-tool). +​ +**IMPORTANT** Run the tf in the same **region** where your RDS instances run (both in the source and destination accounts). If that is not possible because AWS Step Functions is not available, you will need to use the **SourceRegionOverride** parameter explained below. +​ +​ +## Getting Started +​ +​ +### Building From Source and Deploying +​ +1. Create an S3 bucket to hold the Lambda function zip files. The bucket must be in the same region where the Lambda functions will run and the Lambda functions must run in the same region as the RDS instances. If you are using a secondary AWS account to copy the snapshots to, create the S3 bucket in the secondary account as well +2. Clone the repository +3. Create Lambda Zip using the 'Lambda Code' folder; the subfolders represent the lambda functions. While zipping, only zip the python files from each of the subfolders. Do not zip the folder itself. +4. Upload the Lambda zip files to S3 Bucket(s). S3 Buckets should have the following files: + * take_snapshots_rds.zip + * share_snapshots_rds.zip + * delete_old_snapshots_rds.zip + * copy_snapshots_dest_rds.zip + * delete_old_snapshots_dest_rds.zip + * copy_snapshots_no_x_account_rds.zip + * delete_old_snapshots_no_x_account_rds.zip +5. Refer the variables.tf file to identify input parameters based on the requirements. +6. For the deployment in source account, the ‘code_bucket’ variable should refer to the Bucket name where the Source Lambda file are uploaded. And the ‘destination_account’ variable should refer to the AWS Account number of the target account to share the Snapshots with. +7. For the deployment in destination account, the ‘code_bucket’ variable should refer to the Bucket name where the Destination Lambda file are uploaded. And ‘destination_region’ variable should refer to the AWS Region where the snapshot to be copied over. +8. If you only need to copy snapshots across regions and not to a different account, set the ‘CrossAccountCopy’ variable to FALSE. When set to false, the no-x-account version of the Lambda functions will be deployed and will expect snapshots to be in the same account as they run. +9. Run ```terraform apply --auto-approve``` but be sure to use the correct bucket name in the `CodeBucket` parameter when applying terraform +​ +​ +### Source Account +#### Components +The following components will be created in the source account: +* 3 Lambda functions (TakeSnapshotsRDS, ShareSnapshotsRDS, DeleteOldSnapshotsRDS) +* 3 State Machines (Amazon Step Functions) to trigger execution of each Lambda function (stateMachineTakeSnapshotRDS, stateMachineShareSnapshotRDS, stateMachineDeleteOldSnapshotsRDS) +* 3 Cloudwatch Event Rules to trigger the state functions +* 3 Cloudwatch Alarms and associated SNS Topics to alert on State Machines failures + +​ +#### Installing in the source account +Deploy terraform code from the 'rds_snapshot_tool_source' directory +You will need to specify the different input parameters based on your requirements. The default values will back up all RDS instances in the region at 1AM UTC, once a day. +If your instances are encrypted, you will need to provide access to the KMS Key to the destination account. You can read more on how to do that here: https://aws.amazon.com/premiumsupport/knowledge-center/share-cmk-account/ +​ +Here is a break down of each parameter for the source template: +​ +* **BackupInterval** - how many hours between backup +* **BackupSchedule** - at what times and how often to run backups. Set in accordance with **BackupInterval**. For example, set **BackupInterval** to 8 hours and **BackupSchedule** 0 0,8,16 * * ? * if you want backups to run at 0, 8 and 16 UTC. If your backups run more often than **BackupInterval**, snapshots will only be created when the latest snapshot is older than **BackupInterval**. If you set BackupInterval to more than once a day, make sure to adjust BackupSchedule accordingly or backups will only be taken at the times specified in the CRON expression. +* **InstanceNamePattern** - set to the names of the instances you want this tool to back up. You can use a Python regex that will be searched in the instance identifier. For example, if your instances are named *prod-01*, *prod-02*, etc, you can set **InstanceNamePattern** to *prod*. The string you specify will be searched anywhere in the name unless you use an anchor such as ^ or $. In most cases, a simple name like "prod" or "dev" will suffice. More information on Python regular expressions here: https://docs.python.org/2/howto/regex.html +* **DestinationAccount** - the account where you want snapshots to be copied to +* **LogLevel** - The log level you want as output to the Lambda functions. ERROR is usually enough. You can increase to INFO or DEBUG. +* **RetentionDays** - the amount of days you want your snapshots to be kept. Snapshots created more than **RetentionDays** ago will be automatically deleted (only if they contain a tag with Key: CreatedBy, Value: Snapshot Tool for RDS) +* **ShareSnapshots** - Set to TRUE if you are sharing snapshots with a different account. If you set to FALSE, StateMachine, Lambda functions and associated Cloudwatch Alarms related to sharing across accounts will not be created. It is useful if you only want to take backups and manage the retention, but do not need to copy them across accounts or regions. +* **SourceRegionOverride** - if you are running RDS on a region where Step Functions is not available, this parameter will allow you to override the source region. For example, at the time of this writing, you may be running RDS in Northern California (us-west-1) and would like to copy your snapshots to Montreal (ca-central-1). Neither region supports Step Functions at the time of this writing so deploying this tool there will not work. The solution is to run this template in a region that supports Step Functions (such as North Virginia or Ohio) and set **SourceRegionOverride** to *us-west-1*. +**IMPORTANT**: deploy to the closest regions for best results. +​ +* **CodeBucket** - this parameter specifies the bucket where the code for the Lambda functions is located. The Lambda function code is located in the ```lambda``` directory in zip format. These files need to be on the **root* of the bucket or the CloudFormation templates will fail. Please follow the instructions to build source (earlier on this README file) +* **DeleteOldSnapshots** - Set to TRUE to enable functionality that will delete snapshots after **RetentionDays**. Set to FALSE if you want to disable this functionality completely. (Associated Lambda and State Machine resources will not be created in the account). **WARNING** If you decide to enable this functionality later on, bear in mind it will delete **all snapshots**, older than **RetentionDays**, created by this tool; not just the ones created after **DeleteOldSnapshots** is set to TRUE. +* **TaggedInstance** - Set to TRUE to enable functionality that will only take snapshots for RDS Instances with tag CopyDBSnapshot set to True. The settings in InstanceNamePattern and TaggedInstance both need to evaluate successfully for a snapshot to be created (logical AND). +​ +### Destination Account +#### Components +The following components will be created in the destination account: +* 2 Lambda functions (CopySnapshotsDestRDS, DeleteOldSnapshotsDestRDS) +* 2 State Machines (Amazon Step Functions) to trigger execution of each Lambda function (stateMachineCopySnapshotsDestRDS, stateMachineDeleteOldSnapshotsDestRDS) +* 2 Cloudwatch Event Rules to trigger the state functions +* 2 Cloudwatch Alarms and associated SNS Topics to alert on State Machines failures +​ +​ +On your destination account, you will need to deploy terraform code "rds_snapshot_tool_destination". As before, you will need to run it in a region where Step Functions is available. +You will need to specify the different input parameters based on your requirements. The following parameters are available: +​ +* **DestinationRegion** - the region where you want your snapshots to be copied. If you set it to the same as the source region, the snapshots will be copied from the source account but will be kept in the source region. This is useful if you would like to keep a copy of your snapshots in a different account but would prefer not to copy them to a different region. +* **SnapshotPattern** - similar to InstanceNamePattern. See above +* **DeleteOldSnapshots** - Set to TRUE to enable functionanility that will delete snapshots after **RetentionDays**. Set to FALSE if you want to disable this functionality completely. (Associated Lambda and State Machine resources will not be created in the account). **WARNING** If you decide to enable this functionality later on, bear in mind it will delete ALL SNAPSHOTS older than RetentionDays created by this tool, not just the ones created after **DeleteOldSnapshots** is set to TRUE. +* **CrossAccountCopy** - if you only need to copy snapshots across regions and not to a different account, set this to FALSE. When set to false, the no-x-account version of the Lambda functions will be deployed and will expect snapshots to be in the same account as they run. +* **KmsKeySource** KMS Key to be used for copying encrypted snapshots on the source region. If you are copying to a different region, you will also need to provide a second key in the destination region. +* **KmsKeyDestination** KMS Key to be used for copying encrypted snapshots to the destination region. If you are not copying to a different region, this parameter is not necessary. +* **RetentionDays** - as in the source account, the amount of days you want your snapshots to be kept. **Do not set this parameter to a value lower than the source account.** Snapshots created more than **RetentionDays** ago will be automatically deleted (only if they contain a tag with Key: CopiedBy, Value: Snapshot Tool for RDS) +​ +## How it Works +​ +There are two sets of Lambda Step Functions that take regular snapshots and copy them across. Snapshots can take time, and they do not signal when they're complete. Snapshots are scheduled to *begin* at a certain time using CloudWatch Events. Then different Lambda Step Functions run periodically to look for new snapshots. When they find new snapshots, they do the sharing and the copying functions. +​ +### In the Source Account +​ +A CloudWatch Event is scheduled to trigger Lambda Step Function State Machine named `stateMachineTakeSnapshotsRDS`. That state machine invokes a function named `lambdaTakeSnapshotsRDS`. That function triggers a snapshot and applies some standard tags. It matches RDS instances using a regular expression on their names. +​ +There are two other state machines and lambda functions. The `statemachineShareSnapshotsRDS` looks for new snapshots created by the `lambdaTakeSnapshotsRDS` function. When it finds them, it shares them with the destination account. This state machine is, by default, run every 10 minutes. (To change it, you need to change the `ScheduleExpression` property of the `cwEventShareSnapshotsRDS` resource in `snapshots_tool_rds_source.json`). If it finds a new snapshot that is intended to be shared, it shares the snapshot. +​ +The other state machine is the `statemachineDeleteOldSnapshotsRDS` and it calls `lambdaDeleteOldSnapshotsRDS` to delete snapshots according to the `RetentionDays` parameter when the stack is launched. This state machine is, by default, run once each hour. (To change it, you need to change the `ScheduleExpression` property of the `cwEventDeleteOldSnapshotsRDS` resource in `snapshots_tool_rds_source.json`). If it finds a snapshot that is older than the retention time, it deletes the snapshot. +​ +### In the Destination Account +​ +There are two state machines and corresponding lambda functions. The `statemachineCopySnapshotsDestRDS` looks for new snapshots that have been shared but have not yet been copied. When it finds them, it creates a copy in the destination account, encrypted with the KMS key that has been stipulated. This state machine is, by default, run every 10 minutes. (To change it, you need to change the `ScheduleExpression` property of the `cwEventCopySnapshotsRDS` resource in `snapshots_tool_rds_dest.json`). +​ +The other state machine is just like the corresponding state machine and function in the source account. The state machine is `statemachineDeleteOldSnapshotsRDS` and it calls `lambdaDeleteOldSnapshotsRDS` to delete snapshots according to the `RetentionDays` parameter when the stack is launched. This state machine is, by default, run once each hour. (To change it, you need to change the `ScheduleExpression` property of the `cwEventDeleteOldSnapshotsRDS` resource in `snapshots_tool_rds_source.json`). If it finds a snapshot that is older than the retention time, it deletes the snapshot. +Collapse + + + + + + + + + + + + + + + + + + + +Message Praneth Meas, Prasuna Sangela + + + + + + + + +Shift + Return to add a new line \ No newline at end of file