terraform-aws-arc-cicd

Overview

For more information about this repository and its usage, please see Terraform AWS ARC GitHub CICD Module Usage Guide.

Introduction

SourceFuse's AWS Reference Architecture (ARC) Terraform module automates the creation of AWS CodePipeline and CodeBuild projects, facilitating the build and deployment of both application code and Terraform modules. By defining reusable CodeBuild projects, it ensures consistent and efficient build processes that can be shared across multiple CodePipelines. This approach promotes standardization and reduces redundancy in the CI/CD pipeline configuration.

Prerequisites

Before using this module, ensure you have the following:

• AWS credentials configured.
• Terraform installed.
• A working knowledge of Terraform.

Getting Started

1. Define the Module

Initially, it's essential to define a Terraform module, which is organized as a distinct directory encompassing Terraform configuration files. Within this module directory, input variables and output values must be defined in the variables.tf and outputs.tf files, respectively. The following illustrates an example directory structure:

billing/
|-- main.tf
|-- variables.tf
|-- outputs.tf

2. Define Input Variables

Inside the variables.tf or in *.tfvars file, you should define values for the variables that the module requires.

3. Use the Module in Your Main Configuration In your main Terraform configuration file (e.g., main.tf), you can use the module. Specify the source of the module, and version, For Example

module "pipelines" {
  source = "sourcefuse/arc-cicd/aws"

  artifacts_bucket    = local.artifacts_bucket
  codestar_connection = local.codestar_connection

  role_data          = local.role_data
  codebuild_projects = local.codebuild_projects
  codepipelines      = local.codepipeline_data
  chatbot_data       = local.chatbot_data

  tags = module.tags.tags
}

4. Output Values

Inside the outputs.tf file of the module, you can define output values that can be referenced in the main configuration. For example:

output "chatbot_sns_arns" {
  description = "SNS topics created by AWS Chatbot"
  value       = module.example.chatbot_sns_arns
}

5. .tfvars

Inside the .tfvars file of the module, you can provide desired values that can be referenced in the main configuration. For example:

Edit the locals.tf file and provide desired values.

artifacts_bucket - S3 Bucket name where artifacts are stored

codestar_connection - Codestar connection for authenticating to Github

role_data - Details about Roles to be created for Codepipeline and Codebuild projects

codebuild_projects - List of Codebuild projects to be created

codepipelines - Codepipelines to be created

chatbot_data - local.chatbot_data

locals {

  environment_role = {
    dev = "arn:aws:iam::xxxx:role/example-dev-cicd-role"
  }

  branch_map = {
    dev = {
      terraform = "dev"
    }
    poc = {
      terraform = "stg"
    }
  }

  prefix              = "${var.namespace}-${var.environment}"
  codestar_connection = "Github-Connection"
  artifacts_bucket    = "${local.prefix}-pipeline-artifacts"

  policies = [{
    policy_document = data.aws_iam_policy_document.pipeline.json
    policy_name     = "pipeline-policy-to-reject"
  }]

  chatbot_data = {
    name                     = "${var.namespace}-slack"
    slack_channel_id         = "C0xxxxxxx5"
    slack_workspace_id       = "T0xxxxxxRT"
    managed_policy_arns      = ["arn:aws:iam::aws:policy/AWSCodePipeline_FullAccess"]
    guardrail_policies       = ["arn:aws:iam::aws:policy/AWSCodePipeline_FullAccess"]
    role_polices             = local.policies
    enable_slack_integration = true
  }

  notification_event_and_type = {
    event_type_ids = [
      "codepipeline-pipeline-pipeline-execution-failed",
      "codepipeline-pipeline-pipeline-execution-canceled",
      "codepipeline-pipeline-pipeline-execution-started",
      "codepipeline-pipeline-pipeline-execution-resumed",
      "codepipeline-pipeline-pipeline-execution-succeeded",
      "codepipeline-pipeline-pipeline-execution-superseded",
      "codepipeline-pipeline-manual-approval-failed",
      "codepipeline-pipeline-manual-approval-needed"
    ]
    targets = [{
      address = "arn:aws:chatbot::${data.aws_caller_identity.current.account_id}:chat-configuration/slack-channel/${var.namespace}-slack" // it should match chatbot_data.name
      type    = "AWSChatbotSlack"                                                                                                         // Type can be "SNS" , AWSChatbotSlack etc
    }]
  }

  // IAM roles has to be created before creating Codebuild project and Codepipeline
  role_data = {
    "${local.prefix}-codepipeline-role" = {
      pipeline_service                    = "codepipeline"
      assume_role_arns                    = []
      github_secret_arn                   = null
      terraform_state_s3_bucket           = null
      dynamodb_lock_table                 = null
      additional_iam_policy_doc_json_list = []
    },
    "${local.prefix}-codebuild-terraform" = {
      pipeline_service                    = "codebuild"
      assume_role_arns                    = [local.environment_role[var.environment], "arn:aws:iam::1111xxxx1111:role/example-management-mrr-role"]
      github_secret_arn                   = null
      terraform_state_s3_bucket           = "example-shared-services-terraform-state"
      dynamodb_lock_table                 = "example-shared-services-terraform-state-lock"
      additional_iam_policy_doc_json_list = []
    }
  }

  // Codebuild projects have to be created before creating Codepipelines
  codebuild_projects = {
    "${local.prefix}-terraform-plan" = {
      description       = "Codebuild project for Terraform Plan"
      build_type        = "Terraform"
      terraform_version = "terraform-1.8.3-1.x86_64"
      buildspec_file    = null
      role_data = {
        name = "${local.prefix}-codebuild-terraform"
      }
      artifacts_bucket    = local.artifacts_bucket
      buildspec_file_name = "buildspec-tf-apply"
    },
    "${local.prefix}-terraform-apply" = {
      description       = "Codebuild project for Terraform Apply"
      build_type        = "Terraform"
      terraform_version = "terraform-1.8.3-1.x86_64"
      buildspec_file    = null
      role_data = {
        name = "${local.prefix}-codebuild-terraform"
      }
      artifacts_bucket    = local.artifacts_bucket
      buildspec_file_name = "buildspec-tf-apply"
    }
  }


  codepipeline_data = {
    "${local.prefix}-terrafomr-module" = {
      codestar_connection       = local.codestar_connection
      artifacts_bucket          = local.artifacts_bucket
      artifact_store_s3_kms_arn = null
      auto_trigger              = false

      source_repositories = [
        {
          name              = "TF-Source"
          output_artifacts  = ["tf_source_output"]
          github_repository = "githuborg/tf-mono-infra"
          github_branch     = local.branch_map[var.environment].terraform
          auto_trigger      = false
        }
      ]


      pipeline_stages = [
        {
          stage_name       = "Terraform-Plan"
          name             = "Terraform-Plan"
          input_artifacts  = ["tf_source_output"]
          output_artifacts = ["tf_plan_output"]
          version          = "1"
          project_name     = "${local.prefix}-terraform-plan" # This has to match the Codebuild project name
          environment_variables = [
            {
              name  = "ENVIRONMENT",
              value = var.environment
            },
            {
              name  = "TF_VAR_FILE",
              value = "tfvars/${var.environment}.tfvars"
            },
            {
              name  = "WORKING_DIR",
              value = "terraform/example-module"
            },
            {
              name  = "BACKEND_CONFIG_FILE",
              value = "backend/config.shared-services.hcl"
            },
            {
              name  = "WORKSPACE",
              value = var.environment
            }
          ]
        },
        {
          stage_name = "Approval"
          name       = "Approval"
          category   = "Approval"
          provider   = "Manual"
          version    = "1"
        },
        {
          stage_name       = "Terraform-Apply"
          name             = "Terraform-Apply"
          input_artifacts  = ["tf_plan_output"]
          output_artifacts = ["tf_apply_output"]
          version          = "1"
          project_name     = "${local.prefix}-terraform-apply" # This has to match the Codebuild project name
          environment_variables = [
            {
              name  = "ENVIRONMENT",
              value = var.environment
            },
            {
              name  = "TF_VAR_FILE",
              value = "tfvars/${var.environment}.tfvars"
            },
            {
              name  = "WORKING_DIR",
              value = "terraform/example-module"
            },
            {
              name  = "BACKEND_CONFIG_FILE",
              value = "backend/config.shared-services.hcl"
            },
            {
              name  = "WORKSPACE",
              value = var.environment
            }
          ]
        }
      ]
      role_data = {
        name = "${local.prefix}-codepipeline-role"
      }
      notification_data = {
        "${local.prefix}--api-notification" = local.notification_event_and_type // "${local.prefix}--api-notification" name has to be unique for each pipeline
      }
    }
  }

}

First Time Usage

uncomment the backend block in main.tf

terraform init -backend-config=config.dev.hcl

If testing locally, terraform init should be fine

Create a dev workspace

terraform workspace new dev

Plan Terraform

terraform plan -var-file dev.tfvars

Apply Terraform

terraform apply -var-file dev.tfvars

Production Setup

terraform init -backend-config=config.prod.hcl

Create a prod workspace

terraform workspace new prod

Plan Terraform

terraform plan -var-file prod.tfvars

Apply Terraform

terraform apply -var-file prod.tfvars  

Cleanup

Destroy Terraform

terraform destroy -var-file dev.tfvars

Requirements

Name	Version
terraform	>= 1.5.0
aws	~> 5.0

Providers

Name	Version
aws	5.53.0

Modules

Name	Source	Version
chatbot	./modules/chatbot	n/a
codebuild	./modules/codebuild	n/a
codepipeline	./modules/codepipeline	n/a
role	./modules/iam-role	n/a

Resources

Name	Type
aws_s3_bucket.artifact	data source

Inputs

Name	Description	Type	Default	Required
artifacts_bucket	s3 bucket used for codepipeline artifacts	string	n/a	yes
chatbot_data	(optional) Chatbot details to create integration	object({ name = string slack_channel_id = string slack_workspace_id = string guardrail_policies = optional(list(string), ["arn:aws:iam::aws:policy/AWSAccountManagementReadOnlyAccess"]) enable_slack_integration = bool role_polices = optional(list(object({ policy_document = any policy_name = string })), []) managed_policy_arns = optional(list(string), ["arn:aws:iam::aws:policy/AWSResourceExplorerReadOnlyAccess"]) })	null	no
codebuild_projects	Values to create Codebuild project	map(object({ description = optional(string, "") build_timeout = optional(number, 15) queued_timeout = optional(number, 15) compute_type = optional(string, "BUILD_GENERAL1_SMALL") compute_image = optional(string, "aws/codebuild/amazonlinux2-x86_64-standard:5.0") compute_type_container = optional(string, "LINUX_CONTAINER") image_pull_credentials_type = optional(string, "CODEBUILD") privileged_mode = optional(bool, false) build_type = string buildspec_file_name = optional(string, null) buildspec_file = optional(string, null) terraform_version = optional(string, "terraform-1.5.0-1.x86_64") create_role = optional(bool, false) role_data = optional(object({ name = string pipeline_service = optional(string, null) assume_role_arns = optional(list(string), null) github_secret_arn = optional(string, null) terraform_state_s3_bucket = optional(string, null) dynamodb_lock_table = optional(string, null) additional_iam_policy_doc_json_list = optional(list(any), []) }), null) }))	null	no
codepipelines	Codepipeline data to create pipeline and stages	map(object({ artifact_store_s3_kms_arn = string source_repositories = list(object({ name = string output_artifacts = optional(list(string), ["source_output"]) github_repository = string github_branch = string auto_trigger = optional(bool, true) })) pipeline_stages = list(object({ stage_name = string name = string category = optional(string, "Build") provider = optional(string, "CodeBuild") input_artifacts = optional(list(string), []) output_artifacts = optional(list(string), []) version = string project_name = optional(string, null) environment_variables = optional(list(object({ name = string value = string type = optional(string, "PLAINTEXT") })), [] ) })) create_role = optional(bool, false) role_data = optional(object({ name = string github_secret_arn = optional(string, null) additional_iam_policy_doc_json_list = optional(list(any), []) }), null) trigger = optional(list(object({ source_action_name = string push = list(object({ branches = object({ includes = list(string) excludes = list(string) }) file_paths = object({ includes = list(string) excludes = list(string) }) }) ) pull_request = list(object({ events = list(string) filter = list(object({ branches = object({ includes = list(string) excludes = list(string) }) file_paths = object({ includes = list(string) excludes = list(string) }) }) ) })) })), []) notification_data = optional(map(object({ detail_type = optional(string, "FULL") event_type_ids = optional(list(string), [ "codepipeline-pipeline-pipeline-execution-failed", "codepipeline-pipeline-pipeline-execution-canceled", "codepipeline-pipeline-pipeline-execution-started", "codepipeline-pipeline-pipeline-execution-resumed", "codepipeline-pipeline-pipeline-execution-succeeded", "codepipeline-pipeline-pipeline-execution-superseded", "codepipeline-pipeline-manual-approval-failed", "codepipeline-pipeline-manual-approval-needed" ]) targets = list(object({ address = string // eg SNS arn type = optional(string, "SNS") // Type can be "SNS" , AWSChatbotSlack etc })) })), null) }))	{}	no
codestar_connection	codestar connection arn for github repository	string	n/a	yes
role_data	Roles to be created	map(object({ pipeline_service = string assume_role_arns = optional(list(string), null) github_secret_arn = optional(string, null) terraform_state_s3_bucket = optional(string, null) dynamodb_lock_table = optional(string, null) additional_iam_policy_doc_json_list = optional(list(any), []) }))	{}	no
tags	Tags for AWS resources	map(string)	n/a	yes

Outputs

Name	Description
chatbot_sns_arns	SNS topic integrated to AWS Chatbot

Git commits

while Contributing or doing git commit please specify the breaking change in your commit message whether its major,minor or patch

For Example

git commit -m "your commit message #major"

By specifying this , it will bump the version and if you dont specify this in your commit message then by default it will consider patch and will bump that accordingly

Development

Prerequisites

• terraform
• terraform-docs
• pre-commit
• golang
• golint

Configurations

• Configure pre-commit hooks

  pre-commit install

Tests

• Tests are available in test directory
• Configure the dependencies

  cd test/
  go mod init github.com/sourcefuse/terraform-aws-refarch-<module_name>
  go get github.com/gruntwork-io/terratest/modules/terraform

• Now execute the test

  go test -timeout  30m

Authors

This project is authored by:

• SourceFuse