diff --git a/README.md b/README.md index f73f98d..0f82cff 100644 --- a/README.md +++ b/README.md @@ -1,15 +1,12 @@ -# Terraform Modules for Cloudera on Premise Community Edition Infrastructure Creation on +# Terraform Modules for Cloudera on Premise Community Edition Infrastructure Creation on AWS -This repository contains a number of Terraform modules for creation of infrastructure resources for Cloudera on premise clusters on . These modules support deployment of the Community Edition reference cluster. +This repository contains a number of Terraform modules for creation of infrastructure resources for Cloudera on premise clusters on AWS. These modules support deployment of the Community Edition reference cluster. ## Modules | Module name | Description | | ----------- | ----------- | -| [hosts](modules/hosts/README.md) | | +| [network](modules/network/README.md) | Creates and manages AWS networking infrastructure with public/private subnets, NAT gateways, route tables, and security groups for Cloudera on premise deployments. | +| [hosts](modules/hosts/README.md) | Provisions and manages AWS EC2 instances with flexible configuration for compute resources, storage volumes, and networking designed for Cloudera on premise deployments. | Each module contains Terraform resource configuration and example variable definition files. - -## Usage - -The [TODO: CE Public Repo](https://github.com/cloudera-labs/community-edition) repository demonstrates how to use the modules together to deploy the infrastructure for a Cloudera on premise cluster. diff --git a/modules/hosts/README.md b/modules/hosts/README.md index be4693d..2503df2 100644 --- a/modules/hosts/README.md +++ b/modules/hosts/README.md @@ -1,30 +1,37 @@ -# Terraform Module for Hosts on ENTER\_INFRA\_PROVIDER +# Terraform Module for AWS Hosts -> [!IMPORTANT] -> This readme is automation generated using the [`terraform-docs`](https://terraform-docs.io/) command with the static contenta taken from [doc\_fragments/header.md](doc\_fragments/header.md) - -The `hosts` module contains resource files to provision and manage instances with flexible configuration options for compute resources, storage volumes, and networking. This module is designed for Cloudera on premise infrastructure deployments on . +The `hosts` module contains resource files to provision and manage AWS EC2 instances with flexible configuration options for compute resources, storage volumes, and networking. This module is designed for Cloudera on premise infrastructure deployments on AWS IaaS but can be used for any AWS EC2 instance provisioning needs. ## Key Features - +- **Flexible Instance Provisioning**: Create single instances or multiple identical nodes with sequential naming +- **Custom Storage Configuration**: Attach and manage additional EBS volumes with configurable size, type, and mount points +- **Network Integration**: Place instances in specific subnets with optional public IP assignment +- **Security Management**: Apply security groups to control access to instances +- **Resource Tagging**: Add custom tags to all provisioned resources for better organization and cost management +- **IMDSv2 Support**: Automatic detection and configuration of IMDSv2 tokens based on AMI capabilities ## Usage The [examples](./examples) directory has example of using this module: -* `ex01-minimal_inputs` demonstrates how this module can be used to create a number of hosts. +* `ex01-minimal_inputs` demonstrates how this module can be used to create a number of EC2 instances. The [terraform-aws-network](../terraform-aws-network/README.md) module is also used as part of this example. The sample `terraform.tfvars.sample` describes the required inputs for the example. ## Requirements -No requirements. +| Name | Version | +|------|---------| +| [terraform](#requirement\_terraform) | >= 0.13 | +| [aws](#requirement\_aws) | >= 4.60.0 | ## Providers -No providers. +| Name | Version | +|------|---------| +| [aws](#provider\_aws) | >= 4.60.0 | ## Modules @@ -32,13 +39,34 @@ No modules. ## Resources -No resources. +| Name | Type | +|------|------| +| [aws_ebs_volume.inventory](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/ebs_volume) | resource | +| [aws_instance.pvc_base](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/instance) | resource | +| [aws_volume_attachment.inventory](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/volume_attachment) | resource | +| [aws_ami.pvc_base](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/ami) | data source | ## Inputs -No inputs. +| Name | Description | Type | Default | Required | +|------|-------------|------|---------|:--------:| +| [image\_id](#input\_image\_id) | AMI image ID for the hosts | `string` | n/a | yes | +| [name](#input\_name) | Instance name. If 'quantity' is set, name will be -NN. | `string` | n/a | yes | +| [security\_groups](#input\_security\_groups) | List of security group IDs to attach to the instances | `list(string)` | n/a | yes | +| [ssh\_key\_pair](#input\_ssh\_key\_pair) | SSH key pair name | `string` | n/a | yes | +| [subnet\_ids](#input\_subnet\_ids) | List of subnet IDs to provision the instances | `list(string)` | n/a | yes | +| [instance\_type](#input\_instance\_type) | Instance type name for the hosts | `string` | `"t2.micro"` | no | +| [offset](#input\_offset) | Number offset for instance name | `number` | `0` | no | +| [public\_ip](#input\_public\_ip) | Flag to assign public IP addresses to the hosts | `bool` | `false` | no | +| [quantity](#input\_quantity) | Number of instances. Defaults to a single instance without numbering (bare name). | `number` | `0` | no | +| [root\_volume](#input\_root\_volume) | Root volume details | 
object({
    delete_on_termination = optional(bool, true)
    volume_size           = optional(number, 100)
    volume_type           = optional(string)
  })
| `{}` | no | +| [tags](#input\_tags) | Map of tags applied to all cloud-provider assets. | `map(any)` | `{}` | no | +| [volumes](#input\_volumes) | Additional storage volumes to attach to the hosts. Each volume is defined by a device name, mount point, size, type, and optional tags. | 
list(object({
    device_name = string
    mount       = string
    volume_size = optional(number, 100)
    volume_type = optional(string, "gp2")
    tags        = optional(map(string), {})
    }
    )
  )
| `null` | no | ## Outputs -No outputs. +| Name | Description | +|------|-------------| +| [hosts](#output\_hosts) | Hosts | +| [storage\_volumes](#output\_storage\_volumes) | Additional Storage Volumes | \ No newline at end of file diff --git a/modules/hosts/doc_fragments/header.md b/modules/hosts/doc_fragments/header.md index a748ec8..f67dd8c 100644 --- a/modules/hosts/doc_fragments/header.md +++ b/modules/hosts/doc_fragments/header.md @@ -1,18 +1,20 @@ -# Terraform Module for Hosts on ENTER_INFRA_PROVIDER +# Terraform Module for AWS Hosts -> [!IMPORTANT] -> This readme is automation generated using the [`terraform-docs`](https://terraform-docs.io/) command with the static contenta taken from [doc_fragments/header.md](doc_fragments/header.md) - -The `hosts` module contains resource files to provision and manage instances with flexible configuration options for compute resources, storage volumes, and networking. This module is designed for Cloudera on premise infrastructure deployments on . +The `hosts` module contains resource files to provision and manage AWS EC2 instances with flexible configuration options for compute resources, storage volumes, and networking. This module is designed for Cloudera on premise infrastructure deployments on AWS IaaS but can be used for any AWS EC2 instance provisioning needs. ## Key Features - +- **Flexible Instance Provisioning**: Create single instances or multiple identical nodes with sequential naming +- **Custom Storage Configuration**: Attach and manage additional EBS volumes with configurable size, type, and mount points +- **Network Integration**: Place instances in specific subnets with optional public IP assignment +- **Security Management**: Apply security groups to control access to instances +- **Resource Tagging**: Add custom tags to all provisioned resources for better organization and cost management +- **IMDSv2 Support**: Automatic detection and configuration of IMDSv2 tokens based on AMI capabilities ## Usage The [examples](./examples) directory has example of using this module: -* `ex01-minimal_inputs` demonstrates how this module can be used to create a number of hosts. +* `ex01-minimal_inputs` demonstrates how this module can be used to create a number of EC2 instances. The [terraform-aws-network](../terraform-aws-network/README.md) module is also used as part of this example. The sample `terraform.tfvars.sample` describes the required inputs for the example. diff --git a/modules/hosts/examples/ex01-minimal_inputs/main.tf b/modules/hosts/examples/ex01-minimal_inputs/main.tf index 3fd26fd..cf78359 100644 --- a/modules/hosts/examples/ex01-minimal_inputs/main.tf +++ b/modules/hosts/examples/ex01-minimal_inputs/main.tf @@ -13,12 +13,117 @@ # limitations under the License. provider "aws" { - region = var.aws_region + region = var.region + default_tags { + tags = var.asset_tags + } } +# ------- VPC ------- + +resource "aws_vpc" "ex01" { + cidr_block = var.vpc_cidr + tags = { Name = "${var.prefix}-vpc" } + instance_tenancy = "default" + enable_dns_support = true + enable_dns_hostnames = true +} + +resource "aws_internet_gateway" "ex01" { + vpc_id = aws_vpc.ex01.id + tags = { Name = "${var.prefix}-igw" } +} + +resource "aws_vpc_dhcp_options" "ex01" { + domain_name = "${var.prefix}.cldr.internal" + domain_name_servers = ["AmazonProvidedDNS"] + + tags = { Name = "${var.prefix}-vpc-dhcp-options" } +} + +resource "aws_vpc_dhcp_options_association" "ex01" { + vpc_id = aws_vpc.ex01.id + dhcp_options_id = aws_vpc_dhcp_options.ex01.id +} + +# ------- Network Module ------- +module "ex01_network" { + source = "../../../network" + + region = var.region + prefix = var.prefix + vpc_id = aws_vpc.ex01.id + +} + +resource "aws_security_group" "ssh" { + vpc_id = aws_vpc.ex01.id + name = "${var.prefix}-sg-ssh" + description = "SSH traffic [${var.prefix}]" + tags = { Name = "${var.prefix}-sg-ssh" } +} + +resource "aws_vpc_security_group_ingress_rule" "ssh" { + for_each = toset(var.ingress_ssh_cidrs) + + security_group_id = aws_security_group.ssh.id + description = "SSH traffic from ${each.value}" + cidr_ipv4 = each.value + from_port = 22 + ip_protocol = "tcp" + to_port = 22 + tags = { Name = "${var.prefix}-ssh-${index(var.ingress_ssh_cidrs, each.value)}" } +} + +# ------- Host Module ------- module "ex01_hosts" { - source = "../.." + source = "../.." + depends_on = [aws_key_pair.ex01, data.aws_ami.ex01] - # + name = "${var.prefix}-host" + quantity = 2 + image_id = data.aws_ami.ex01.image_id + instance_type = "t2.micro" + subnet_ids = module.ex01_network.public_subnets[*].id # Will use the first public subnet + ssh_key_pair = aws_key_pair.ex01.key_name + security_groups = [ + module.ex01_network.intra_cluster_security_group.id, + aws_security_group.ssh.id + ] + public_ip = true +} +# ------- AMI ------- +locals { + ami_owners = ["amazon"] + ami_filters = { + name = ["al2023-ami-2023*"] + architecture = ["x86_64"] + } } + +data "aws_ami" "ex01" { + owners = local.ami_owners + most_recent = true + + dynamic "filter" { + for_each = local.ami_filters + + content { + name = filter.key + values = filter.value + } + } +} + +# ------- SSH ------- + +data "tls_public_key" "ex01" { + private_key_openssh = file(var.ssh_private_key_file) +} + +resource "aws_key_pair" "ex01" { + key_name = "${var.prefix}-key" + public_key = trimspace(data.tls_public_key.ex01.public_key_openssh) +} + diff --git a/modules/hosts/examples/ex01-minimal_inputs/outputs.tf b/modules/hosts/examples/ex01-minimal_inputs/outputs.tf new file mode 100644 index 0000000..ec35328 --- /dev/null +++ b/modules/hosts/examples/ex01-minimal_inputs/outputs.tf @@ -0,0 +1,34 @@ +# Copyright 2025 Cloudera, Inc. All Rights Reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +output "ssh_key_pair" { + value = { + name = aws_key_pair.ex01.key_name + public_key = trimspace(data.tls_public_key.ex01.public_key_openssh) + type = aws_key_pair.ex01.key_type + fingerprint = aws_key_pair.ex01.fingerprint + } + description = "SSH public key" +} + +output "vpc" { + value = aws_vpc.ex01.id + description = "AWS VPC" +} + +output "nodes" { + value = values(module.ex01_hosts) + + description = "Node information including IDs, names, private and public IPs" +} \ No newline at end of file diff --git a/modules/hosts/examples/ex01-minimal_inputs/terraform.tfvars.sample b/modules/hosts/examples/ex01-minimal_inputs/terraform.tfvars.sample index c89861f..5b56f78 100644 --- a/modules/hosts/examples/ex01-minimal_inputs/terraform.tfvars.sample +++ b/modules/hosts/examples/ex01-minimal_inputs/terraform.tfvars.sample @@ -12,9 +12,20 @@ # See the License for the specific language governing permissions and # limitations under the License. -# - # ------- Global Settings ------- -name_prefix = "" +prefix = "ex01" + +region = "" # Change this to specify Cloud Provider region, e.g. eu-west-1 + +asset_tags = { + "Environment" = "Development" +} + +# ------- Network settings ------- +ingress_ssh_cidrs = [ + "", + "" +] -# ------- Cloud Settings ------- +# ------- SSH settings ------- +ssh_private_key_file = "" # Path to the SSH \ No newline at end of file diff --git a/modules/hosts/examples/ex01-minimal_inputs/variables.tf b/modules/hosts/examples/ex01-minimal_inputs/variables.tf index 7d9614b..b314e6f 100644 --- a/modules/hosts/examples/ex01-minimal_inputs/variables.tf +++ b/modules/hosts/examples/ex01-minimal_inputs/variables.tf @@ -12,12 +12,46 @@ # See the License for the specific language governing permissions and # limitations under the License. -# +# ------- General and Provider Resources ------- -# ------- Global settings ------- -variable "name_prefix" { +variable "prefix" { type = string - description = "Shorthand name to use when naming resources." + description = "Deployment prefix for all cloud-provider assets" + + validation { + condition = length(var.prefix) < 8 || length(var.prefix) > 4 + error_message = "Valid length for prefix is between 4-7 characters." + } +} + +variable "region" { + type = string + description = "AWS region" } -# ------- Cloud Settings ------- +variable "asset_tags" { + type = map(string) + default = {} + description = "Map of tags applied to all cloud-provider assets" +} + +# ------- Network Resources ------- +variable "vpc_cidr" { + type = string + description = "VPC CIDR Block (primary)" + default = "10.10.0.0/16" +} + +variable "ingress_ssh_cidrs" { + type = list(string) + description = "List of CIDRs to add to ingress SSH inbound rules." + + default = [] +} + +# ------- SSH Resources ------- + +variable "ssh_private_key_file" { + type = string + description = "Local SSH private key file" +} diff --git a/modules/hosts/main.tf b/modules/hosts/main.tf index 5811fb5..c74ad2d 100644 --- a/modules/hosts/main.tf +++ b/modules/hosts/main.tf @@ -11,3 +11,47 @@ # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. + +terraform { + required_version = ">= 0.13" + required_providers { + aws = { + source = "hashicorp/aws", + version = ">= 4.60.0", + } + } +} + +data "aws_ami" "pvc_base" { + filter { + name = "image-id" + values = [var.image_id] + } +} + +resource "aws_instance" "pvc_base" { + count = var.quantity == 0 ? 1 : var.quantity + + key_name = var.ssh_key_pair + instance_type = var.instance_type + ami = data.aws_ami.pvc_base.id + + subnet_id = var.subnet_ids[count.index % length(var.subnet_ids)] + associate_public_ip_address = var.public_ip + + vpc_security_group_ids = var.security_groups + + root_block_device { + delete_on_termination = var.root_volume.delete_on_termination + volume_size = var.root_volume.volume_size + volume_type = var.root_volume.volume_type + } + + metadata_options { + http_tokens = data.aws_ami.pvc_base.imds_support == "v2.0" ? "required" : "optional" + } + + tags = merge(var.tags, { + Name = var.quantity == 0 ? var.name : format("%s-%02d", var.name, count.index + var.offset + 1) + }) +} diff --git a/modules/hosts/outputs.tf b/modules/hosts/outputs.tf index 5811fb5..7e93fec 100644 --- a/modules/hosts/outputs.tf +++ b/modules/hosts/outputs.tf @@ -4,10 +4,20 @@ # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # -# http://www.apache.org/licenses/LICENSE-2.0 +# https://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. + +output "hosts" { + value = aws_instance.pvc_base + description = "Hosts" +} + +output "storage_volumes" { + value = local.attached_volumes_by_instance + description = "Additional Storage Volumes" +} \ No newline at end of file diff --git a/modules/hosts/variables.tf b/modules/hosts/variables.tf index 5811fb5..cb8ab78 100644 --- a/modules/hosts/variables.tf +++ b/modules/hosts/variables.tf @@ -4,10 +4,104 @@ # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # -# http://www.apache.org/licenses/LICENSE-2.0 +# https://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. + +variable "tags" { + type = map(any) + default = {} + description = "Map of tags applied to all cloud-provider assets." +} + +variable "ssh_key_pair" { + type = string + description = "SSH key pair name" +} + +# variable "prefix" { +# type = string +# description = "Deployment prefix for all cloud-provider assets." + +# validation { +# condition = length(var.prefix) < 8 || length(var.prefix) > 4 +# error_message = "Valid length for prefix is between 4-7 characters." +# } +# } + +# ------- Instances ------- + +variable "name" { + type = string + description = "Instance name. If 'quantity' is set, name will be -NN." +} + +variable "quantity" { + type = number + description = "Number of instances. Defaults to a single instance without numbering (bare name)." + default = 0 +} + +variable "offset" { + type = number + description = "Number offset for instance name" + default = 0 +} + +variable "subnet_ids" { + type = list(string) + description = "List of subnet IDs to provision the instances" +} + +variable "security_groups" { + type = list(string) + description = "List of security group IDs to attach to the instances" +} + +variable "public_ip" { + type = bool + description = "Flag to assign public IP addresses to the hosts" + default = false +} + +variable "instance_type" { + type = string + description = "Instance type name for the hosts" + default = "t2.micro" +} + +variable "image_id" { + type = string + description = "AMI image ID for the hosts" +} + +variable "root_volume" { + type = object({ + delete_on_termination = optional(bool, true) + volume_size = optional(number, 100) + volume_type = optional(string) + }) + description = "Root volume details" + default = {} +} + +# ------- Additional Storage Volumes ------- +variable "volumes" { + type = list(object({ + device_name = string + mount = string + volume_size = optional(number, 100) + volume_type = optional(string, "gp2") + tags = optional(map(string), {}) + } + ) + ) + + default = null + + description = "Additional storage volumes to attach to the hosts. Each volume is defined by a device name, mount point, size, type, and optional tags." +} diff --git a/modules/hosts/volumes.tf b/modules/hosts/volumes.tf new file mode 100644 index 0000000..f3d3293 --- /dev/null +++ b/modules/hosts/volumes.tf @@ -0,0 +1,81 @@ +# Copyright 2025 Cloudera, Inc. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# https://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +locals { + + volumes = flatten([ + for idx in range(0, var.quantity) : + [ + for node_vol in coalesce(var.volumes, []) : + { + node_index = idx + name = aws_instance.pvc_base[idx].tags.Name + az = aws_instance.pvc_base[idx].availability_zone + device = node_vol.device_name + mount = node_vol.mount + size = node_vol.volume_size + type = node_vol.volume_type + subnet_index = aws_instance.pvc_base[idx].subnet_id + } + ] + ]) +} + +resource "aws_ebs_volume" "inventory" { + for_each = { for idx, volume in local.volumes : idx => volume } + + availability_zone = each.value.az + size = each.value.size + type = each.value.type + tags = { + Name = "${each.value.name}: ${each.value.device}" + mount = each.value.mount + } + #encrypted ... +} + +resource "aws_volume_attachment" "inventory" { + for_each = { for idx, volume in local.volumes : idx => volume } + + device_name = each.value.device + volume_id = aws_ebs_volume.inventory[index(local.volumes, each.value)].id + instance_id = aws_instance.pvc_base[each.value.node_index].id +} + +locals { + # Details for all attached volumes + attached_volumes = [ + for idx, volume in local.volumes : + { + "vol_name" = aws_ebs_volume.inventory[index(local.volumes, volume)].tags["Name"] + "vol_id" = aws_volume_attachment.inventory[index(local.volumes, volume)].volume_id + "instance" = aws_volume_attachment.inventory[index(local.volumes, volume)].instance_id + "device" = aws_volume_attachment.inventory[index(local.volumes, volume)].device_name + "mount" = aws_ebs_volume.inventory[index(local.volumes, volume)].tags["mount"] + } + if length(aws_ebs_volume.inventory) > 0 + ] + + # Attached volume details grouped by instance + attached_volumes_by_instance = { + for vol in local.attached_volumes : + vol.instance => + { + vol_name = vol.vol_name + vol_id = vol.vol_id + device = vol.device + mount = vol.mount + }... + } +} diff --git a/modules/network/.terraform-docs.yaml b/modules/network/.terraform-docs.yaml new file mode 100644 index 0000000..0936036 --- /dev/null +++ b/modules/network/.terraform-docs.yaml @@ -0,0 +1,21 @@ +formatter: markdown +header-from: doc_fragments/header.md +settings: + anchor: true + color: true + default: true + escape: true + html: true + indent: 2 + required: true + sensitive: true + type: true + + +sort: + enabled: true + by: required + +output: + file: README.md + mode: replace \ No newline at end of file diff --git a/modules/network/README.md b/modules/network/README.md new file mode 100644 index 0000000..a4170f2 --- /dev/null +++ b/modules/network/README.md @@ -0,0 +1,86 @@ + +# Terraform Module for AWS Network + +The `network` module contains resource files for creating and managing AWS networking infrastructure required for Cloudera on premise deployments on AWS IaaS. This module simplifies the process of setting up secure, scalable network topologies with public and private subnets, NAT gateways, route tables, and security groups. + +## Key Features + +- **Flexible Subnet Configuration**: Create custom public and private subnets across multiple availability zones +- **Automatic NAT Gateway Setup**: Deploy NAT gateways in public subnets to provide internet access for private resources +- **Route Table Management**: Configure and associate route tables for both public and private subnets +- **Security Group Provisioning**: Create pre-configured security groups for intra-cluster communication and ACME TLS challenges +- **VPC Integration**: Works with existing VPCs to extend your current network architecture +- **Naming Consistency**: Enforces consistent naming conventions with customizable prefixes for all network resources + +## Usage + +The [examples](./examples) directory has example of using this module: + +* `ex01-minimal_inputs` demonstrates how this module can be used to create the a network infrastructure on AWS. + +The sample `terraform.tfvars.sample` describes the required inputs for the example. + +## Requirements + +| Name | Version | +|------|---------| +| [terraform](#requirement\_terraform) | >= 0.13 | +| [aws](#requirement\_aws) | >= 4.60.0 | + +## Providers + +| Name | Version | +|------|---------| +| [aws](#provider\_aws) | >= 4.60.0 | + +## Modules + +No modules. + +## Resources + +| Name | Type | +|------|------| +| [aws_eip.pvc_base](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/eip) | resource | +| [aws_nat_gateway.pvc_base](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/nat_gateway) | resource | +| [aws_route_table.pvc_base_private](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/route_table) | resource | +| [aws_route_table.pvc_base_public](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/route_table) | resource | +| [aws_route_table_association.pvc_base_private](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/route_table_association) | resource | +| [aws_route_table_association.pvc_base_public](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/route_table_association) | resource | +| [aws_security_group.acme_tls](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/security_group) | resource | +| [aws_security_group.pvc_base](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/security_group) | resource | +| [aws_subnet.pvc_base_private](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/subnet) | resource | +| [aws_subnet.pvc_base_public](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/subnet) | resource | +| [aws_vpc_security_group_egress_rule.pvc_base](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/vpc_security_group_egress_rule) | resource | +| [aws_vpc_security_group_ingress_rule.acme_tls](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/vpc_security_group_ingress_rule) | resource | +| [aws_vpc_security_group_ingress_rule.pvc_base](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/vpc_security_group_ingress_rule) | resource | +| [aws_availability_zones.pvc_base](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/availability_zones) | data source | +| [aws_internet_gateway.pvc_base](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/internet_gateway) | data source | +| [aws_vpc.pvc_base](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/vpc) | data source | + +## Inputs + +| Name | Description | Type | Default | Required | +|------|-------------|------|---------|:--------:| +| [prefix](#input\_prefix) | Deployment prefix for all cloud-provider assets | `string` | n/a | yes | +| [vpc\_id](#input\_vpc\_id) | VPC ID | `string` | n/a | yes | +| [nat\_gateway\_name](#input\_nat\_gateway\_name) | NAT gateway name | `string` | `""` | no | +| [private\_route\_table\_name](#input\_private\_route\_table\_name) | Private Route Table name prefix | `string` | `""` | no | +| [private\_subnet\_name](#input\_private\_subnet\_name) | Private Subnet name prefix | `string` | `""` | no | +| [private\_subnets](#input\_private\_subnets) | List of Private Subnet details (name, CIDR, AZ, add'l tags) | 
list(object({
    name = string
    cidr = string
    az   = string
    tags = map(string)
  }))
| `[]` | no | +| [public\_route\_table\_name](#input\_public\_route\_table\_name) | Public Route Table name prefix | `string` | `""` | no | +| [public\_subnet\_name](#input\_public\_subnet\_name) | Public Subnet name prefix | `string` | `""` | no | +| [public\_subnets](#input\_public\_subnets) | List of Public Subnet details (name, CIDR, AZ, add'l tags) | 
list(object({
    name = string
    cidr = string
    az   = string
    tags = map(string)
  }))
| `[]` | no | +| [security\_group\_acme\_name](#input\_security\_group\_acme\_name) | Security Group for ACME Directory communication (e.g. Let's Encrypt) | `string` | `""` | no | +| [security\_group\_intra\_name](#input\_security\_group\_intra\_name) | Security Group for intra-cluster communication | `string` | `""` | no | + +## Outputs + +| Name | Description | +|------|-------------| +| [acme\_tls\_security\_group](#output\_acme\_tls\_security\_group) | ACME Directory traffic Security Group | +| [availability\_zones](#output\_availability\_zones) | AWS Availability Zones | +| [intra\_cluster\_security\_group](#output\_intra\_cluster\_security\_group) | Intra-cluster traffic Security Group | +| [private\_subnets](#output\_private\_subnets) | Cluster private subnets | +| [public\_subnets](#output\_public\_subnets) | Cluster public subnets | + \ No newline at end of file diff --git a/modules/network/doc_fragments/header.md b/modules/network/doc_fragments/header.md new file mode 100644 index 0000000..705fc5a --- /dev/null +++ b/modules/network/doc_fragments/header.md @@ -0,0 +1,20 @@ +# Terraform Module for AWS Network + +The `network` module contains resource files for creating and managing AWS networking infrastructure required for Cloudera on premise deployments on AWS IaaS. This module simplifies the process of setting up secure, scalable network topologies with public and private subnets, NAT gateways, route tables, and security groups. + +## Key Features + +- **Flexible Subnet Configuration**: Create custom public and private subnets across multiple availability zones +- **Automatic NAT Gateway Setup**: Deploy NAT gateways in public subnets to provide internet access for private resources +- **Route Table Management**: Configure and associate route tables for both public and private subnets +- **Security Group Provisioning**: Create pre-configured security groups for intra-cluster communication and ACME TLS challenges +- **VPC Integration**: Works with existing VPCs to extend your current network architecture +- **Naming Consistency**: Enforces consistent naming conventions with customizable prefixes for all network resources + +## Usage + +The [examples](./examples) directory has example of using this module: + +* `ex01-minimal_inputs` demonstrates how this module can be used to create the a network infrastructure on AWS. + +The sample `terraform.tfvars.sample` describes the required inputs for the example. diff --git a/modules/network/examples/ex01-minimal_inputs/main.tf b/modules/network/examples/ex01-minimal_inputs/main.tf new file mode 100644 index 0000000..4da10bb --- /dev/null +++ b/modules/network/examples/ex01-minimal_inputs/main.tf @@ -0,0 +1,57 @@ +# Copyright 2025 Cloudera, Inc. All Rights Reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +provider "aws" { + region = var.region + default_tags { + tags = var.asset_tags + } +} + +# ------- VPC ------- + +resource "aws_vpc" "base" { + cidr_block = var.vpc_cidr + tags = { Name = "${var.prefix}-vpc" } + instance_tenancy = "default" + enable_dns_support = true + enable_dns_hostnames = true +} + +resource "aws_internet_gateway" "base" { + vpc_id = aws_vpc.base.id + tags = { Name = "${var.prefix}-igw" } +} + +resource "aws_vpc_dhcp_options" "base" { + domain_name = "${var.prefix}.cldr.internal" + domain_name_servers = ["AmazonProvidedDNS"] + + tags = { Name = "${var.prefix}-vpc-dhcp-options" } +} + +resource "aws_vpc_dhcp_options_association" "base" { + vpc_id = aws_vpc.base.id + dhcp_options_id = aws_vpc_dhcp_options.base.id +} + +# ------- Network Module ------- +module "ex01_network" { + source = "../.." + + region = var.region + prefix = var.prefix + vpc_id = aws_vpc.base.id + +} diff --git a/modules/network/examples/ex01-minimal_inputs/terraform.tfvars.sample b/modules/network/examples/ex01-minimal_inputs/terraform.tfvars.sample new file mode 100644 index 0000000..ef3bf32 --- /dev/null +++ b/modules/network/examples/ex01-minimal_inputs/terraform.tfvars.sample @@ -0,0 +1,22 @@ +# Copyright 2025 Cloudera, Inc. All Rights Reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# ------- General and Provider Resources ------- +prefix = "ex01" + +region = "" # Change this to specify Cloud Provider region, e.g. eu-west-1 + +asset_tags = { + "Environment" = "Development" +} diff --git a/modules/network/examples/ex01-minimal_inputs/variables.tf b/modules/network/examples/ex01-minimal_inputs/variables.tf new file mode 100644 index 0000000..7c08f37 --- /dev/null +++ b/modules/network/examples/ex01-minimal_inputs/variables.tf @@ -0,0 +1,43 @@ +# Copyright 2025 Cloudera, Inc. All Rights Reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# ------- General and Provider Resources ------- + +variable "prefix" { + type = string + description = "Deployment prefix for all cloud-provider assets" + + validation { + condition = length(var.prefix) < 8 || length(var.prefix) > 4 + error_message = "Valid length for prefix is between 4-7 characters." + } +} + +variable "region" { + type = string + description = "AWS region" +} + +variable "asset_tags" { + type = map(string) + default = {} + description = "Map of tags applied to all cloud-provider assets" +} + +# ------- Network Resources ------- +variable "vpc_cidr" { + type = string + description = "VPC CIDR Block (primary)" + default = "10.10.0.0/16" +} \ No newline at end of file diff --git a/modules/network/main.tf b/modules/network/main.tf new file mode 100644 index 0000000..ee16087 --- /dev/null +++ b/modules/network/main.tf @@ -0,0 +1,192 @@ +# Copyright 2025 Cloudera, Inc. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +terraform { + required_version = ">= 0.13" + required_providers { + aws = { + source = "hashicorp/aws", + version = ">= 4.60.0", + } + } +} + +data "aws_availability_zones" "pvc_base" { + state = "available" +} + +data "aws_vpc" "pvc_base" { + id = var.vpc_id +} + +data "aws_internet_gateway" "pvc_base" { + filter { + name = "attachment.vpc-id" + values = [data.aws_vpc.pvc_base.id] + } +} + +locals { + nat_name = var.nat_gateway_name != "" ? var.nat_gateway_name : "${var.prefix}-pvc-base-nat" + rt_public_name = var.public_route_table_name != "" ? var.public_route_table_name : "${var.prefix}-pvc-base-public" + rt_private_name = var.private_route_table_name != "" ? var.private_route_table_name : "${var.prefix}-pvc-base-private" + public_subnet_name = var.public_subnet_name != "" ? var.public_subnet_name : "${var.prefix}-pvc-base-public" + public_subnets = length(var.public_subnets) > 0 ? var.public_subnets : tolist([{ + name = "${local.public_subnet_name}-01", + cidr = cidrsubnet(data.aws_vpc.pvc_base.cidr_block, 8, 0), + az = data.aws_availability_zones.pvc_base.names[0], + tags = {} + }]) + private_subnet_name = var.private_subnet_name != "" ? var.private_subnet_name : "${var.prefix}-pvc-base-private" + private_subnets = length(var.private_subnets) > 0 ? var.private_subnets : tolist([{ + name = "${local.private_subnet_name}-01", + cidr = cidrsubnet(data.aws_vpc.pvc_base.cidr_block, 8, 1), + az = data.aws_availability_zones.pvc_base.names[0], + tags = {} + }]) + sg_intra_name = var.security_group_intra_name != "" ? var.security_group_intra_name : "${var.prefix}-pvc-base-intra" + sg_acme_name = var.security_group_acme_name != "" ? var.security_group_acme_name : "${var.prefix}-pvc-base-acme-tls" +} + +# ------- AWS Public Network infrastructure ------- + +# TODO Calculate a local for a single default public subnet or defer to the root tf_proxied_cluter module +# TODO Lookup AZ id by input (name) and if not found, use AZ value as ID + +# Public Subnets +resource "aws_subnet" "pvc_base_public" { + for_each = { for idx, subnet in local.public_subnets : idx => subnet } + + vpc_id = data.aws_vpc.pvc_base.id + cidr_block = each.value.cidr + map_public_ip_on_launch = true + availability_zone = each.value.az + tags = merge(each.value.tags, { Name = each.value.name }) +} + +resource "aws_route_table" "pvc_base_public" { + for_each = { for idx, subnet in local.public_subnets : idx => subnet } + + vpc_id = data.aws_vpc.pvc_base.id + + tags = { Name = format("%s-%02d", local.rt_public_name, index(local.public_subnets, each.value) + 1) } + + route { + cidr_block = "0.0.0.0/0" + gateway_id = data.aws_internet_gateway.pvc_base.id + } +} + +# Public Route Table Associations +resource "aws_route_table_association" "pvc_base_public" { + for_each = { for idx, subnet in aws_subnet.pvc_base_public : idx => subnet } + + subnet_id = each.value.id + route_table_id = aws_route_table.pvc_base_public[each.key].id +} + +# ------- AWS Private Networking infrastructure ------- + +# Network Gateways (NAT) +resource "aws_eip" "pvc_base" { + count = length(aws_subnet.pvc_base_public) + + tags = { Name = format("%s-%02d", local.nat_name, count.index + 1) } +} + +resource "aws_nat_gateway" "pvc_base" { + for_each = { for idx, subnet in aws_subnet.pvc_base_public : idx => subnet } + + subnet_id = each.value.id + allocation_id = aws_eip.pvc_base[each.key].id + connectivity_type = "public" + tags = { Name = format("%s-%02d", local.nat_name, each.key + 1) } +} + +# Private Subnets +resource "aws_subnet" "pvc_base_private" { + for_each = { for idx, subnet in local.private_subnets : idx => subnet } + + vpc_id = data.aws_vpc.pvc_base.id + cidr_block = each.value.cidr + map_public_ip_on_launch = false + availability_zone = each.value.az + tags = merge(each.value.tags, { Name = each.value.name }) +} + +# Private Route Tables +resource "aws_route_table" "pvc_base_private" { + for_each = { for idx, subnet in local.private_subnets : idx => subnet } + + vpc_id = data.aws_vpc.pvc_base.id + + tags = { Name = format("%s-%02d", local.rt_private_name, index(local.private_subnets, each.value) + 1) } + + route { + cidr_block = "0.0.0.0/0" + nat_gateway_id = aws_nat_gateway.pvc_base[(index(local.private_subnets, each.value) % length(aws_nat_gateway.pvc_base))].id + } +} + +# Private Route Table Associations +resource "aws_route_table_association" "pvc_base_private" { + for_each = { for idx, subnet in aws_subnet.pvc_base_private : idx => subnet } + + subnet_id = each.value.id + route_table_id = aws_route_table.pvc_base_private[each.key].id +} + +# ------- Security Groups ------- + +# Intra-cluster traffic +resource "aws_security_group" "pvc_base" { + vpc_id = data.aws_vpc.pvc_base.id + name = local.sg_intra_name + description = "Intra-cluster traffic [${var.prefix}]" + tags = { Name = local.sg_intra_name } +} + +resource "aws_vpc_security_group_ingress_rule" "pvc_base" { + security_group_id = aws_security_group.pvc_base.id + description = "Self-reference ingress rule" + ip_protocol = -1 + referenced_security_group_id = aws_security_group.pvc_base.id + tags = { Name = "${var.prefix}-pvc-base-intra" } +} + +resource "aws_vpc_security_group_egress_rule" "pvc_base" { + security_group_id = aws_security_group.pvc_base.id + description = "Self-reference egress rule" + ip_protocol = -1 + referenced_security_group_id = aws_security_group.pvc_base.id + tags = { Name = "${var.prefix}-pvc-base-intra" } +} + +# ACME Directory Challenge +resource "aws_security_group" "acme_tls" { + vpc_id = data.aws_vpc.pvc_base.id + name = local.sg_acme_name + description = "ACME Directory challenge traffic [${var.prefix}]" + tags = { Name = local.sg_acme_name } +} + +resource "aws_vpc_security_group_ingress_rule" "acme_tls" { + security_group_id = aws_security_group.acme_tls.id + description = "ACME http-01 challenge" + cidr_ipv4 = "0.0.0.0/0" + from_port = 80 + to_port = 80 + ip_protocol = "tcp" + tags = { Name = "${var.prefix}-pvc-base-acme-challenge" } +} diff --git a/modules/network/outputs.tf b/modules/network/outputs.tf new file mode 100644 index 0000000..f1ed915 --- /dev/null +++ b/modules/network/outputs.tf @@ -0,0 +1,38 @@ +# Copyright 2025 Cloudera, Inc. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# https://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +output "availability_zones" { + value = data.aws_availability_zones.pvc_base + description = "AWS Availability Zones" +} + +output "public_subnets" { + value = values(aws_subnet.pvc_base_public) + description = "Cluster public subnets" +} + +output "private_subnets" { + value = values(aws_subnet.pvc_base_private) + description = "Cluster private subnets" +} + +output "intra_cluster_security_group" { + value = aws_security_group.pvc_base + description = "Intra-cluster traffic Security Group" +} + +output "acme_tls_security_group" { + value = aws_security_group.acme_tls + description = "ACME Directory traffic Security Group" +} diff --git a/modules/network/variables.tf b/modules/network/variables.tf new file mode 100644 index 0000000..280a6a1 --- /dev/null +++ b/modules/network/variables.tf @@ -0,0 +1,100 @@ +# Copyright 2025 Cloudera, Inc. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# ------- Required ------- +variable "prefix" { + type = string + description = "Deployment prefix for all cloud-provider assets" + + validation { + condition = length(var.prefix) < 8 || length(var.prefix) > 4 + error_message = "Valid length for prefix is between 4-7 characters." + } +} + +variable "vpc_id" { + type = string + description = "VPC ID" +} + +# ------- Network Resources ------- + +# Public Network infrastructure +variable "public_subnet_name" { + type = string + description = "Public Subnet name prefix" + default = "" +} + +variable "public_subnets" { + type = list(object({ + name = string + cidr = string + az = string + tags = map(string) + })) + + description = "List of Public Subnet details (name, CIDR, AZ, add'l tags)" + default = [] +} + +variable "public_route_table_name" { + type = string + description = "Public Route Table name prefix" + default = "" +} + +# Private Network infrastructure +variable "private_subnet_name" { + type = string + description = "Private Subnet name prefix" + default = "" +} + +variable "private_subnets" { + type = list(object({ + name = string + cidr = string + az = string + tags = map(string) + })) + + description = "List of Private Subnet details (name, CIDR, AZ, add'l tags)" + default = [] +} + +variable "nat_gateway_name" { + type = string + description = "NAT gateway name" + default = "" +} + +variable "private_route_table_name" { + type = string + description = "Private Route Table name prefix" + default = "" +} + +# Security Groups +variable "security_group_intra_name" { + type = string + description = "Security Group for intra-cluster communication" + default = "" +} + +variable "security_group_acme_name" { + type = string + description = "Security Group for ACME Directory communication (e.g. Let's Encrypt)" + default = "" +}