Bash environment variable renames cause `unbound variable` errors when updating using a custom AMI

[n-tucker]

Describe the bug
When upgrading our elastic stack version from v6.42.0 to 6.43.0, our agents started failing, with the following log lines in /buildkite/elastic-stack:

/usr/local/bin/bk-install-elastic-stack.sh: line 276: BUILDKITE_AGENT_SIGNING_FAILURE_BEHAVIOR: unbound variable

This appears to have come from this change, where BUILDKITE_AGENT_SIGNING_FAILURE_BEHAVIOR was renamed to BUILDKITE_AGENT_JOB_VERIFICATION_NO_SIGNATURE_BEHAVIOR

For context, we maintain our own Buildkite AMI and traditionally have updated this after an elastic stack update. We do this because from previous experience the elastic stack sometimes adds upstream environment variables that the AMI expects, which in the past has meant the AMI needs to be updated last to avoid unbound variable errors.

Is it possible for future bash environment variable renames to instead use a new variable, deprecate the old one, and remove the deprecated variable in a major release? This would make managing the elastic stack with a custom AMI much easier!

Expected behavior
Elastic stack update with a custom AMI should roll out smoothly, when the elastic stack is updated first following an AMI change

Actual behaviour
Updating this version of the elastic stack caused issues with our agents, as none of them were able to start properly and accept jobs. We needed to update both the elastic stack and AMI in lockstep which adds complexity.

Stack parameters (please complete the following information):

• AWS Region: us-west-2
• Version v6.42.0

Additional context
We suspect this is only an issue for folks using a custom Buildkite AMI

[jamestelfer]

This is effectively a semantic versioning issue. When the parameters change in a backwards-incompatible fashion, it causes a variety of potential failure cases on upgrade.

It would be a lot easier for us to understand at release time if API changes were:

1. documented as a major version and/or
2. had compatibility concerns called out explicitly in the release documentation

[jasonwbarnett]

FYI: We solved this problem by version-coupling our custom AMI with the elastic stack at build time, so they can never drift apart. Here's the approach:

1. Use the upstream Buildkite AMI as the base for your custom AMI

We hardcode the source AMI IDs from the CloudFormation template directly in our Packer config. These are the same AMI IDs listed in the AWSRegion2AMI mapping in the template for a given version:

variable "buildkite_stack_version" {
  type        = string
  description = "Must match the CloudFormation template version"
  validation {
    condition     = can(regex("^v\\d+\\.\\d+\\.\\d+$", var.buildkite_stack_version))
    error_message = "Must be a semantic version prefixed with 'v' (e.g. v6.58.4)."
  }
}

variable "buildkite_source_ami" {
  type = map(map(string))
  default = {
    # These AMI IDs come directly from the CloudFormation template's AWSRegion2AMI mapping
    us-east-1 = { linuxamd64 = "ami-0532f...", linuxarm64 = "ami-01843..." }
    us-west-2 = { linuxamd64 = "ami-00dca...", linuxarm64 = "ami-0b1de..." }
    # ... other regions
  }
}

source "amazon-ebs" "buildkite_agent" {
  source_ami = var.buildkite_source_ami["us-east-1"]["linuxamd64"]
  # ... customize the image with your own provisioners ...

  tags = {
    BuildkiteStackVersion = var.buildkite_stack_version
  }
}

2. Filter for the matching AMI in Terraform using the version tag

A single variable drives both the CloudFormation template URL and the AMI lookup:

variable "buildkite_stack_version" {
  type    = string
  default = "v6.58.4"
}

data "aws_ami" "buildkite_agent" {
  most_recent = true
  filter {
    name   = "name"
    values = ["buildkite-agent-amd64-*"]
  }
  # Version-couple AMI with CloudFormation stack template
  filter {
    name   = "tag:BuildkiteStackVersion"
    values = [var.buildkite_stack_version]
  }
  owners = ["self"]
}

resource "aws_cloudformation_stack" "buildkite" {
  template_url = "https://s3.amazonaws.com/buildkite-aws-stack/${var.buildkite_stack_version}/aws-stack.yml"
  parameters = {
    ImageId = data.aws_ami.buildkite_agent.id
  }
}

3. CI validation prevents mismatches

At AMI build time, a CI step fetches the CloudFormation template for the configured version, extracts the AWSRegion2AMI mapping, and compares it against the hardcoded source AMIs in the Packer config. If they don't match, the build fails before any image is published.

Why this works: Because the source AMI IDs are copied from the CloudFormation template and the version tag ties everything together, it's impossible for the custom AMI to be built from one version's base image while being used with a different version's CloudFormation template. When upgrading, you bump the version, update the source AMI IDs from the new template, and rebuild — the Terraform AMI filter won't resolve until the matching custom AMI exists.

This doesn't address the root semver/deprecation issue raised here, but it does eliminate the class of breakage where custom AMI and stack versions get out of sync.