Skip to content

no-jira: Validate AWS resource tag keys with aws: prefix #2183

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 4 commits into
base: master
Choose a base branch
from
Open

no-jira: Validate AWS resource tag keys with aws: prefix #2183

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
e5aa2b8
Validate aws resource tag keys with aws prefix & immutable
patrickdillon Jan 30, 2025
0eddb96
Generate CRDs: for AWS prefix validation
patrickdillon Oct 9, 2025
40d7615
Regenerate CRDS
patrickdillon Oct 21, 2025
d582954
refactor validation and add ratcheting validation
patrickdillon Oct 21, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
193 changes: 193 additions & 0 deletions config/v1/tests/infrastructures.config.openshift.io/AAA_ungated.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1837,3 +1837,196 @@ tests:
value: value*
type: AWS
expectedStatusError: "invalid AWS resource tag value. The string can contain only the set of alphanumeric characters, space (' '), '_', '.', '/', '=', '+', '-', ':', '@'"
- name: Should not be able to create an aws resourcetag with aws prefix in key
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please also add a ratcheting test (see readme in the tests dir) to prove that existing values are not affected by this change. You should show that

  • Existing entries persist when a new item is added to the list
  • Additional bad entries are not allowed when there is an existing bad entry
  • The bad entry can be updated to a valid entry

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, I added the ratcheting validation tests, but am stuck on the first one (the other two work ok). In order to allow existing bad values, I updated the validation in 34f8d72 but apparently this is invalid, as the tests fail with:

Invalid value: "self == oldSelf || !self.startsWith('aws:')": oldSelf cannot be used on the uncorrelatable portion of the schema within spec.validation.openAPIV3Schema.properties[status].properties[platformStatus].properties[aws].properties[resourceTags]

I see in these k8s docs that this means the rule cannot be applied:

Errors will be generated on CRD writes if a schema node contains a transition rule that can never be applied, e.g. "oldSelf cannot be used on the uncorrelatable portion of the schema within path".

I'm not sure how to resolve this issue. Any insight?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@JoelSpeed claude helped me figure out that I needed to change ResourceTags to listType=map, but now the actual validation of the aws: prefix does not seem to be working

 [config.openshift.io/v1, Resource=infrastructures][ClusterProfiles=Hypershift,SelfManagedHA][FeatureSet="Default"][FeatureGate=-AWSClusterHostedDNSInstall][File=0000_10_config-operator_01_infrastructures-Default.crd.yaml] Infrastructure On Update [It] Should not be able to create an aws resourcetag with aws prefix in key
/Users/padillon/go/src/github.com/openshift/api/tests/generator.go:345

  [FAILED] Expected an error, got nil
  In [It] at: /Users/padillon/go/src/github.com/openshift/api/tests/generator.go:321 @ 10/14/25 22:00:52.111

Stuck on this.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I went ahead and side-stepped the issue by validating for immutability. IMHO that makes more sense, as post-install updates are not supported. On the other hand, I'm not sure if there are issues with making a field immutable post-GA; the intention was always for these fields to be immutable, it just wasn't validated.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We still need a ratcheting test to show that anyone who already has a value with aws: in the prefix won't be broken by this change

Copy link
Contributor Author

@patrickdillon patrickdillon Oct 21, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh right, that makes sense: so we don't brick the rest of the aws platform status from being updated. Because I made resourceTags immutable, we can't cover the three expected cases in your original message. Does the check added in d582954 look good/sufficient? Or what else would you have in mind.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yep, that covers it, thanks

initial: |
apiVersion: config.openshift.io/v1
kind: Infrastructure
spec:
platformSpec:
aws: {}
type: AWS
updated: |
apiVersion: config.openshift.io/v1
kind: Infrastructure
spec:
platformSpec:
aws: {}
type: AWS
status:
controlPlaneTopology: HighlyAvailable
cpuPartitioning: None
infrastructureTopology: HighlyAvailable
platform: AWS
platformStatus:
aws:
region: us-east-1
resourceTags:
- key: aws:key
value: value with space
type: AWS
expectedStatusError: "the prefix 'aws:' is reserved for AWS system usage and cannot be used at the beginning of a key"
- name: Should not be able to modify an existing AWS ResourceTags Tag
initial: |
apiVersion: config.openshift.io/v1
kind: Infrastructure
spec: {}
status:
controlPlaneTopology: "HighlyAvailable"
infrastructureTopology: "HighlyAvailable"
platform: AWS
platformStatus:
type: AWS
aws:
resourceTags:
- {key: "key", value: "value"}
updated: |
apiVersion: config.openshift.io/v1
kind: Infrastructure
spec: {}
status:
platform: AWS
platformStatus:
type: AWS
aws:
resourceTags:
- {key: "key", value: "changed"}
expectedStatusError: "status.platformStatus.aws.resourceTags: Invalid value: \"array\": resourceTags are immutable and may only be configured during installation"
- name: Should not be able to add a Tag to an existing AWS ResourceTags
initial: |
apiVersion: config.openshift.io/v1
kind: Infrastructure
spec: {}
status:
controlPlaneTopology: "HighlyAvailable"
infrastructureTopology: "HighlyAvailable"
platform: AWS
platformStatus:
type: AWS
aws:
resourceTags:
- {key: "key", value: "value"}
updated: |
apiVersion: config.openshift.io/v1
kind: Infrastructure
spec: {}
status:
platform: AWS
platformStatus:
type: AWS
aws:
resourceTags:
- {key: "key", value: "value"}
- {key: "new", value: "entry"}
expectedStatusError: "status.platformStatus.aws.resourceTags: Invalid value: \"array\": resourceTags are immutable and may only be configured during installation"
- name: Should not be able to remove a Tag from an existing AWS ResourceTags
initial: |
apiVersion: config.openshift.io/v1
kind: Infrastructure
spec: {}
status:
platform: AWS
platformStatus:
type: AWS
aws:
resourceTags:
- {key: "key", value: "value"}
- {key: "new", value: "entry"}
updated: |
apiVersion: config.openshift.io/v1
kind: Infrastructure
spec: {}
status:
platform: AWS
platformStatus:
type: AWS
aws:
resourceTags:
- {key: "key", value: "value"}
expectedStatusError: "status.platformStatus.aws.resourceTags: Invalid value: \"array\": resourceTags are immutable and may only be configured during installation"
- name: Should not be able to add AWS ResourceTags to an empty platformStatus.aws
initial: |
apiVersion: config.openshift.io/v1
kind: Infrastructure
spec: {}
status:
platform: AWS
platformStatus:
type: AWS
aws: {}
updated: |
apiVersion: config.openshift.io/v1
kind: Infrastructure
spec: {}
status:
platform: AWS
platformStatus:
aws:
resourceTags:
- {key: "key", value: "value"}
expectedStatusError: "status.platformStatus.aws: Invalid value: \"object\": resourceTags may only be configured during installation"
- name: Should not be able to remove AWS ResourceTags from platformStatus.aws
initial: |
apiVersion: config.openshift.io/v1
kind: Infrastructure
spec: {}
status:
platform: AWS
platformStatus:
type: AWS
aws:
resourceTags:
- {key: "key", value: "value"}
updated: |
apiVersion: config.openshift.io/v1
kind: Infrastructure
spec: {}
status:
platform: AWS
platformStatus:
type: AWS
aws: {}
expectedStatusError: "status.platformStatus.aws: Invalid value: \"object\": resourceTags may only be configured during installation"
- name: Ratcheting validation for AWS prefix ResourceTags should not break updates
initialCRDPatches:
- op: remove
path: /spec/versions/0/schema/openAPIV3Schema/properties/status/properties/platformStatus/properties/aws/properties/resourceTags/items/properties/key/x-kubernetes-validations
initial: |
apiVersion: config.openshift.io/v1
kind: Infrastructure
spec: {}
status:
platform: AWS
platformStatus:
type: AWS
aws:
resourceTags:
- {key: "aws:invalidKey", value: "value"}
updated: |
apiVersion: config.openshift.io/v1
kind: Infrastructure
spec: {}
status:
platform: AWS
platformStatus:
type: AWS
aws:
resourceTags:
- {key: "aws:invalidKey", value: "value"}
serviceEndpoints:
- {name: "newendpoint", url: "https://www.awsendpoint.com"}
expected: |
apiVersion: config.openshift.io/v1
kind: Infrastructure
spec: {}
status:
platform: AWS
controlPlaneTopology: HighlyAvailable
cpuPartitioning: None
infrastructureTopology: HighlyAvailable
platformStatus:
type: AWS
aws:
resourceTags:
- {key: "aws:invalidKey", value: "value"}
serviceEndpoints:
- {name: "newendpoint", url: "https://www.awsendpoint.com"}
4 changes: 4 additions & 0 deletions config/v1/types_infrastructure.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -503,6 +503,7 @@ type AWSPlatformSpec struct {
}

// AWSPlatformStatus holds the current status of the Amazon Web Services infrastructure provider.
// +kubebuilder:validation:XValidation:rule="has(oldSelf.resourceTags) == has(self.resourceTags)",message="resourceTags may only be configured during installation"
type AWSPlatformStatus struct {
// region holds the default AWS region for new AWS resources created by the cluster.
Region string `json:"region"`
Expand All @@ -519,6 +520,7 @@ type AWSPlatformStatus struct {
// AWS supports a maximum of 50 tags per resource. OpenShift reserves 25 tags for its use, leaving 25 tags
// available for the user.
// +kubebuilder:validation:MaxItems=25
// +kubebuilder:validation:XValidation:rule="self.all(x, x in oldSelf) && oldSelf.all(x, x in self)",message="resourceTags are immutable and may only be configured during installation"
// +listType=atomic
// +optional
ResourceTags []AWSResourceTag `json:"resourceTags,omitempty"`
Expand All @@ -543,9 +545,11 @@ type AWSResourceTag struct {
// key sets the key of the AWS resource tag key-value pair. Key is required when defining an AWS resource tag.
// Key should consist of between 1 and 128 characters, and may
// contain only the set of alphanumeric characters, space (' '), '_', '.', '/', '=', '+', '-', ':', and '@'.
// Key must not be start with 'aws:'.
// +kubebuilder:validation:MinLength=1
// +kubebuilder:validation:MaxLength=128
// +kubebuilder:validation:XValidation:rule=`self.matches('^[0-9A-Za-z_.:/=+-@ ]+$')`,message="invalid AWS resource tag key. The string can contain only the set of alphanumeric characters, space (' '), '_', '.', '/', '=', '+', '-', ':', '@'"
// +kubebuilder:validation:XValidation:rule=`!self.startsWith('aws:')`,message="the prefix 'aws:' is reserved for AWS system usage and cannot be used at the beginning of a key"
// +required
Key string `json:"key"`
// value sets the value of the AWS resource tag key-value pair. Value is required when defining an AWS resource tag.
Expand Down
11 changes: 11 additions & 0 deletions ...nerated.crd-manifests/0000_10_config-operator_01_infrastructures-CustomNoUpgrade.crd.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1372,6 +1372,7 @@ spec:
key sets the key of the AWS resource tag key-value pair. Key is required when defining an AWS resource tag.
Key should consist of between 1 and 128 characters, and may
contain only the set of alphanumeric characters, space (' '), '_', '.', '/', '=', '+', '-', ':', and '@'.
Key must not be start with 'aws:'.
maxLength: 128
minLength: 1
type: string
Expand All @@ -1381,6 +1382,9 @@ spec:
space (' '), '_', '.', '/', '=', '+', '-', ':',
'@'
rule: self.matches('^[0-9A-Za-z_.:/=+-@ ]+$')
- message: the prefix 'aws:' is reserved for AWS system
usage and cannot be used at the beginning of a key
rule: '!self.startsWith(''aws:'')'
value:
description: |-
value sets the value of the AWS resource tag key-value pair. Value is required when defining an AWS resource tag.
Expand All @@ -1404,6 +1408,10 @@ spec:
maxItems: 25
type: array
x-kubernetes-list-type: atomic
x-kubernetes-validations:
- message: resourceTags are immutable and may only be configured
during installation
rule: self.all(x, x in oldSelf) && oldSelf.all(x, x in self)
serviceEndpoints:
description: |-
serviceEndpoints list contains custom endpoints which will override default
Expand Down Expand Up @@ -1432,6 +1440,9 @@ spec:
type: array
x-kubernetes-list-type: atomic
type: object
x-kubernetes-validations:
- message: resourceTags may only be configured during installation
rule: has(oldSelf.resourceTags) == has(self.resourceTags)
azure:
description: azure contains settings specific to the Azure infrastructure
provider.
Expand Down
11 changes: 11 additions & 0 deletions ...v1/zz_generated.crd-manifests/0000_10_config-operator_01_infrastructures-Default.crd.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1190,6 +1190,7 @@ spec:
key sets the key of the AWS resource tag key-value pair. Key is required when defining an AWS resource tag.
Key should consist of between 1 and 128 characters, and may
contain only the set of alphanumeric characters, space (' '), '_', '.', '/', '=', '+', '-', ':', and '@'.
Key must not be start with 'aws:'.
maxLength: 128
minLength: 1
type: string
Expand All @@ -1199,6 +1200,9 @@ spec:
space (' '), '_', '.', '/', '=', '+', '-', ':',
'@'
rule: self.matches('^[0-9A-Za-z_.:/=+-@ ]+$')
- message: the prefix 'aws:' is reserved for AWS system
usage and cannot be used at the beginning of a key
rule: '!self.startsWith(''aws:'')'
value:
description: |-
value sets the value of the AWS resource tag key-value pair. Value is required when defining an AWS resource tag.
Expand All @@ -1222,6 +1226,10 @@ spec:
maxItems: 25
type: array
x-kubernetes-list-type: atomic
x-kubernetes-validations:
- message: resourceTags are immutable and may only be configured
during installation
rule: self.all(x, x in oldSelf) && oldSelf.all(x, x in self)
serviceEndpoints:
description: |-
serviceEndpoints list contains custom endpoints which will override default
Expand Down Expand Up @@ -1250,6 +1258,9 @@ spec:
type: array
x-kubernetes-list-type: atomic
type: object
x-kubernetes-validations:
- message: resourceTags may only be configured during installation
rule: has(oldSelf.resourceTags) == has(self.resourceTags)
azure:
description: azure contains settings specific to the Azure infrastructure
provider.
Expand Down
11 changes: 11 additions & 0 deletions ...ted.crd-manifests/0000_10_config-operator_01_infrastructures-DevPreviewNoUpgrade.crd.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1372,6 +1372,7 @@ spec:
key sets the key of the AWS resource tag key-value pair. Key is required when defining an AWS resource tag.
Key should consist of between 1 and 128 characters, and may
contain only the set of alphanumeric characters, space (' '), '_', '.', '/', '=', '+', '-', ':', and '@'.
Key must not be start with 'aws:'.
maxLength: 128
minLength: 1
type: string
Expand All @@ -1381,6 +1382,9 @@ spec:
space (' '), '_', '.', '/', '=', '+', '-', ':',
'@'
rule: self.matches('^[0-9A-Za-z_.:/=+-@ ]+$')
- message: the prefix 'aws:' is reserved for AWS system
usage and cannot be used at the beginning of a key
rule: '!self.startsWith(''aws:'')'
value:
description: |-
value sets the value of the AWS resource tag key-value pair. Value is required when defining an AWS resource tag.
Expand All @@ -1404,6 +1408,10 @@ spec:
maxItems: 25
type: array
x-kubernetes-list-type: atomic
x-kubernetes-validations:
- message: resourceTags are immutable and may only be configured
during installation
rule: self.all(x, x in oldSelf) && oldSelf.all(x, x in self)
serviceEndpoints:
description: |-
serviceEndpoints list contains custom endpoints which will override default
Expand Down Expand Up @@ -1432,6 +1440,9 @@ spec:
type: array
x-kubernetes-list-type: atomic
type: object
x-kubernetes-validations:
- message: resourceTags may only be configured during installation
rule: has(oldSelf.resourceTags) == has(self.resourceTags)
azure:
description: azure contains settings specific to the Azure infrastructure
provider.
Expand Down
11 changes: 11 additions & 0 deletions ...ed.crd-manifests/0000_10_config-operator_01_infrastructures-TechPreviewNoUpgrade.crd.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1372,6 +1372,7 @@ spec:
key sets the key of the AWS resource tag key-value pair. Key is required when defining an AWS resource tag.
Key should consist of between 1 and 128 characters, and may
contain only the set of alphanumeric characters, space (' '), '_', '.', '/', '=', '+', '-', ':', and '@'.
Key must not be start with 'aws:'.
maxLength: 128
minLength: 1
type: string
Expand All @@ -1381,6 +1382,9 @@ spec:
space (' '), '_', '.', '/', '=', '+', '-', ':',
'@'
rule: self.matches('^[0-9A-Za-z_.:/=+-@ ]+$')
- message: the prefix 'aws:' is reserved for AWS system
usage and cannot be used at the beginning of a key
rule: '!self.startsWith(''aws:'')'
value:
description: |-
value sets the value of the AWS resource tag key-value pair. Value is required when defining an AWS resource tag.
Expand All @@ -1404,6 +1408,10 @@ spec:
maxItems: 25
type: array
x-kubernetes-list-type: atomic
x-kubernetes-validations:
- message: resourceTags are immutable and may only be configured
during installation
rule: self.all(x, x in oldSelf) && oldSelf.all(x, x in self)
serviceEndpoints:
description: |-
serviceEndpoints list contains custom endpoints which will override default
Expand Down Expand Up @@ -1432,6 +1440,9 @@ spec:
type: array
x-kubernetes-list-type: atomic
type: object
x-kubernetes-validations:
- message: resourceTags may only be configured during installation
rule: has(oldSelf.resourceTags) == has(self.resourceTags)
azure:
description: azure contains settings specific to the Azure infrastructure
provider.
Expand Down
Loading