Terraform Agent Observability

This repository provides demo code on how to visualize Terraform agent telemetry. It includes Terraform code to deploy the agent pool, set the organization's default settings to use the agent pool. It also includes a docker-compose.yml to launch containers for the Agent, OTel collector, Prometheus, and Grafana. There is Terraform code for creating test workspaces to generate telemetry for visualization.

Read the accompanying Medium blog post or Substack blog post for more details about the integration and additional screenshots.

1. Architecture

2. Deployment

2.1 Agent pool and organization default settings

Step 1: Configure HCP Terraform credentials. Refer to the tfe_provider authentication docs for the various token options and guidance. For example:

export TFE_TOKEN=example

Step 2: In tf-agent directory, run an apply, review the plan output, and approve the plan accordingly. The apply outputs the commands to run the Terraform agent. This includes the agent token.

Caution

In a live environment it is not good practice to output the Terraform agent token. The token is output in this repo purely for demo purposes, such that readers can easily pass the token to the Terraform agent.

terraform init
terraform apply

2.2 Start the containers

Step 1: Run the following commands in the root directory to start up the containers (replace the agent token with the output from the previous step)

export TFC_AGENT_TOKEN=example
export TFC_AGENT_NAME=demo-agent-pool
docker compose up

3. Verify deployment

3.1 Terraform agent pool

Terraform agent pool created with an idle agent

Terraform org settings default execution mode shows Agent

3.2 Prometheus

Visit localhost:9090 and choose Explore metrics

View tfc_agent prefixed metrics

3.3 Grafana

Visit localhost:3000 and login with

• Username: admin
• Password: admin

Initial Terraform Agent Dashboard. This is configured from Metrics-Dashboard.json

3.4 Jaeger

Visit localhost:16686. There are no traces yet since there are no workspace runs.

4. Testing

4.1 Create workspaces for testing

Step 1: In the tf/test directory, copy tf-test/terraform.tfvars.example to terraform.tfvars and change the environment variables accordingly. GitHub credentials can use a personal access tokens. This token needs sufficient permissions to create, delete repositories, and write files to the repository.

Caution

In a live environment, it is not good practice to directly pass the GitHub token. Instead, sensitive credentials should be securely stored and accessed using solutions like HashiCorp Vault, which provides encrypted storage and access controls capabilities.

Step 3: In the tf/test-auto-scaling directory, run an apply, review the plan output, and approve the plan accordingly.

terraform init
terraform apply

4.2 GitHub repo created for testing

GitHub repository created with simple Terraform resources.

4.3 First workspace plan

Agent processes workspace runs one at a time. Agent is Busy

Dashboard shows data about the first run

4.4 Workspaces applied

4.4.1 HCP TF view

All workspaces are eventually applied

Agent transitions to Idle

4.4.2 Grafana dashboard

Dashboard with metrics across all the workspace runs

Zoomed in view for various dashboard sections - Job and workspace performance

Resource utilization (Pool-wide)

Runtime metrics (Pool-wide)

Individual agent details

4.4.3 Prometheus metrics

Some metrics are available during runs. For example

• tfc_agent_core_profiler_cpu_busy_percent
• tfc_agent_core_profiler_memory_used_percent

tfc_agent_core_profiler_memory_used_percent

tfc_agent_core_profiler_cpu_busy_percent

4.4.4 Jaeger

Jaeger UI shows 10 traces. Each workspace has 2 traces - 1 for plan, and 1 for apply.

Example of a plan trace

This can be drilled down to the span information

Example of an apply trace

This can be drilled down to the span information

5. Cleanup

Step 1: Run docker compose down -v

Step 2: In the tf-test directory, run destroy. Review the destroy output before approving.

terraform destroy

Step 3: In the tf-agent directory, run destroy. Review the destroy output before approving.

terraform destroy