Support AMD SEV-SNP on AWS #2424

fangge1212 · 2025-07-28T08:49:35Z

AMD SEV-SNP is one of the confidential computing technologies. This commit adds support for AMD SEV-SNP on AWS, so users can utilize the confidential computing on the cluster nodes.

Upstream CAPA PR: kubernetes-sigs/cluster-api-provider-aws#5598

openshift-ci · 2025-07-28T08:50:00Z

Hello @fangge1212! Some important instructions when contributing to openshift/api:
API design plays an important part in the user experience of OpenShift and as such API PRs are subject to a high level of scrutiny to ensure they follow our best practices. If you haven't already done so, please review the OpenShift API Conventions and ensure that your proposed changes are compliant. Following these conventions will help expedite the api review process for your PR.

openshift-ci · 2025-07-28T08:53:30Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: fangge1212
Once this PR has been reviewed and has the lgtm label, please assign deads2k for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

machine/v1beta1/types_awsprovider.go

everettraven · 2025-07-31T19:45:51Z

machine/v1beta1/types_awsprovider.go

+)
+
+// CpuOptions defines the cpu options for the instance.
+type CpuOptions struct {


A couple questions to help me better understand the direction this should take:

Because all fields are optional here, what happens if this is set to the zero value of the struct {}?

Because there is no constraints here as to how many properties can be specified at once, what would it mean if in the future there are multiple options to configure and I configure more than one of them?

If it is an empty struct, cpuOptions=nil will be passed to the AWS API

I don't get your question. If there are multiple options to configure in the future, then we can configure multiple options.

I don't get your question. If there are multiple options to configure in the future, then we can configure multiple options

What I was trying to get at is similar to this question upstream: kubernetes-sigs/cluster-api-provider-aws#5598 (comment)

If you are only ever going to have mutually exclusive CPU options then using an enum for those options likely makes sense.

If you are going to have a subset of mutually exclusive options but the rest can be specified together, then this structure is generally OK, but I'd think an enum is still reasonable for a single mutual exclusivity field.

Looking at https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instance-specify-cpu-options.html I'd expect something like:

cpuOptions: cores: 4 threadsPerCore: 1 confidentialComputing: AmdSevSnp

to be reasonable (assuming that something like confidentialComputing would be mutually exclusive so you can't set something like AmdSevSnp and TDX if it were to be added in the future)

Updated to:

cpuOptions: confidentialComputing: AMDSevSnp

everettraven · 2025-08-07T17:13:02Z

machine/v1beta1/types_awsprovider.go

+type CPUOptions struct {
+	// confidentialCompute specifies whether confidential computing should be enabled for the instance,
+	// and, if so, which confidential computing technology to use.
+	// If set to Disabled, the instance will not use confidential computing.


"Disabled" is often times an overloaded term and we generally try to avoid using it where we can.

For scenarios like this, I generally go for "None" and would encourage that usage here as well.

IMO, confidentialCompute: None reads more intuitively than confidentialCompute: Disabled.

Updated back to "Disabled" to align with GCP

everettraven · 2025-08-07T17:15:54Z

machine/v1beta1/types_awsprovider.go

+	// If set to Disabled, the instance will not use confidential computing.
+	// If set to AMDSevSnp, the instance will be configured with AMD SEV-SNP.
+	// In this case, ensure the following conditions are met:
+	// 1) The selected instance type supports AMD SEV-SNP.
+	// 2) The selected AWS region supports AMD SEV-SNP.
+	// 3) The selected AMI supports AMD SEV-SNP.
+	// More details can be checked at https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/sev-snp.html
+	// If omitted, the platform will apply a default value — currently Disabled, but this may change over time.


Suggested change

// If set to Disabled, the instance will not use confidential computing.

// If set to AMDSevSnp, the instance will be configured with AMD SEV-SNP.

// In this case, ensure the following conditions are met:

// 1) The selected instance type supports AMD SEV-SNP.

// 2) The selected AWS region supports AMD SEV-SNP.

// 3) The selected AMI supports AMD SEV-SNP.

// More details can be checked at https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/sev-snp.html

// If omitted, the platform will apply a default value — currently Disabled, but this may change over time.

// When set to Disabled, the instance will not use confidential computing.

// When set to AMDSevSnp, the instance will be configured with AMD SEV-SNP.

// In this case, ensure the following conditions are met:

// 1) The selected instance type supports AMD SEV-SNP.

// 2) The selected AWS region supports AMD SEV-SNP.

// 3) The selected AMI supports AMD SEV-SNP.

// More details can be checked at https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/sev-snp.html

// When omitted, this means no opinion and the platform is left to choose a reasonable default, which is subject to change without notice. The current default is Disabled.

everettraven · 2025-08-07T17:43:49Z

machine/v1beta1/types_awsprovider.go

+)
+
+// CPUOptions defines the cpu options for the instance.
+type CPUOptions struct {


We try to avoid things like cpuOptions: {} being valid inputs because it is usually semantically the same as just not specifying the field altogether.

Add the +kubebuilder:validation:MinProperties=1 marker so that it is required for at least one property to be specified, making {} an invalid input.

When I add +kubebuilder:validation:MinProperties=1, kubeapilinter asked me to add tag omitzero and don't use make CPUOptions a pointer. I followed its prompt, but this makes many tests in cluster-api-provider-aws failed:

--- FAIL: TestSSHKeyName/SSH_key_name_is_nil_is_valid (0.01s) sshkeyname_test.go:89: ValidateCreate() error = AWSMachine.infrastructure.cluster.x-k8 "machine-9zsqb" is invalid: spec.cpuOptions: Invalid value: 0: spec.cpuOptions in body should have at least 1 properties, wantErr false

I see. I just took a look and it looks like cluster-api-provider-aws is on Go 1.23.1 which does not contain the omitzero behavior that kube-api-linter is asking you to add. That support was added in Go 1.24.0

Is it possible for you to bump to Go 1.24.z in cluster-api-provider-aws?

Looks like that is in progress here: kubernetes-sigs/cluster-api-provider-aws#5624

everettraven · 2025-08-07T17:45:29Z

machine/v1beta1/types_awsprovider.go

@@ -17,6 +17,9 @@ type AWSMachineProviderConfig struct {
 	AMI AWSResourceReference `json:"ami"`
 	// instanceType is the type of instance to create. Example: m4.xlarge
 	InstanceType string `json:"instanceType"`
+	// cpuOptions is the set of cpu options for the instance.
+	// +optional


What happens if this field is not specified by a user?

If unset, no CPU options are passed to the AWS platform and AWS default values are used.

everettraven · 2025-08-07T17:48:48Z

machine/v1beta1/types_awsprovider.go

@@ -17,6 +17,9 @@ type AWSMachineProviderConfig struct {
 	AMI AWSResourceReference `json:"ami"`
 	// instanceType is the type of instance to create. Example: m4.xlarge
 	InstanceType string `json:"instanceType"`
+	// cpuOptions is the set of cpu options for the instance.


I think we could improve the GoDoc here to be more descriptive.

https://github.com/openshift/enhancements/blob/master/dev-guide/api-conventions.md#write-user-readable-documentation-in-godoc is a good starting point for the things that you should take into consideration and include in the documentation for a field.

Thanks. I updated the doc, please have a look again.

fangge1212 · 2025-08-08T09:11:42Z

/retest-required

fangge1212 · 2025-08-11T02:31:21Z

/retest

damdo · 2025-08-11T16:01:10Z

machine/v1beta1/types_awsprovider.go

+const (
+	// AWSConfidentialComputePolicyNone disables confidential computing for the instance.
+	AWSConfidentialComputePolicyNone AWSConfidentialComputePolicy = "None"
+	// AWSConfidentialComputePolicySEVSNP enables AMD SEV-SNP as the confidential computing technology for the instance.
+	AWSConfidentialComputePolicySEVSNP AWSConfidentialComputePolicy = "AmdSevSnp"
+)


Let's follow the same approach we took for the GCP types: https://github.com/openshift/api/pull/2165/files#diff-9bd66f01f63b7ce0455fe993aa03dfe93964ac62b31278a6dbf3db2fcb7f792eR66-R71

In detail:

Let's use Enabled/Disabled (rather than None)

Let's fully spell out the AmdSevSnp to AMDEncryptedVirtualizationNestedPaging

I updated to Disabled;AMDEncryptedVirtualizationNestedPaging
I don't think Enabled is a meaningfull value, because all the other values than Disabled implies Enabled

Enabled is normally for the default value (so in this case OpenShift engineers choose the default for the user) as you can see here on the GCP one:

// When set to Enabled, the machine will be configured as a confidential computing instance with no preference on the confidential compute policy used. In this mode, the platform chooses a default that is subject to change over time. Currently, the default is to use AMD Secure Encrypted Virtualization.

Ideally we should try and keep consistency across providers for the same feature, so I'd be happy to see it added

Hi @damdo, @fangge1212!

The problem I see in the AWS case is that AFAICT, it doesn't support the default GCP case: AMD SEV. It only supports AMD SEV-SNP. In that case, we would end up defining Enabled as a valid case for AWS, but would have a different behavior than the one from GCP, configuring AMD SEV-SNP machines instead. In my opinion this would be misleading.

This is another discussion, but at the moment the Enabled option for GCP is redundant, as AMDEncryptedVirtualization achieves the same result. In my opinion, Deprecating Enabled in the GCP provider in the future would help getting rid of that redundancy and also having consistency across providers for the same feature, as you mention.

If we are thinking about deprecating Enabled then it is probably ok to leave it out from this one.
cc. @JoelSpeed

We can also use the omission of the parent field to do defaulting behavior as well, so just omitting the cpuOptions field altogether could mean "no opinion on CPU options, platform defaults", which is the path I would prefer if that is semantically equivalent to either of the examples above.

The omission of the cpuOptions field or confidentialCompute field will be equivalent to ConfidentialComputeDisabled.
The current code doesn't support this option: "I don't care, I just want confidential computing to be enabled, I'll let OpenShift default it to something". Do we need this?

The current code doesn't support this option: "I don't care, I just want confidential computing to be enabled, I'll let OpenShift default it to something". Do we need this?

Is that behavior you want your users to be able to do? I wouldn't say this is a requirement on my end, but I do think that we should be clear on the intended user workflows to be able to design this API appropriately.

I don’t think PlatformDefault is very useful in practice. When a user installs a cluster, they already choose the instance type and region, which directly constrains the available confidential computing technologies. That means the platform can’t meaningfully pick a “default” technology on the user’s behalf.

Enabled was likely a mistake in the previous API. I'm ok with the following states for this API

Omitted - means its disabled

Disabled - I've explicitly disabled this as I don't want anyone else defaulting it for me

AMDEncryptedVirtualizationNestedPaging - I specifically want this kind of option

everettraven · 2025-08-29T19:08:36Z

machine/v1beta1/types_awsprovider.go

+	// cpuOptions defines CPU-related settings for the instance, including the confidential computing policy.
+	// If unset, no CPU options will be passed to the AWS platform and AWS default CPU options will be applied.


How do I know what the "AWS default CPU options" that will be applied are? Are these literally defaults AWS imposes on requests that don't specify these options, or are these defaulted elsewhere?

Thanks for the comment! To clarify: OpenShift does not set defaults for cpuOptions. If the field is unset, the RunInstances request is sent without a CpuOptions block, and AWS applies its own defaults for the chosen instance type.
I’ll update the field description to make that clear.

Updated the description, please take a look again.

AMD SEV-SNP is one of the confidential computing technologies. This commit adds support for AMD SEV-SNP on AWS, so users can utilize the confidential computing on the cluster nodes. Signed-off-by: Fangge Jin <[email protected]>

openshift-ci · 2025-09-01T08:00:04Z

@fangge1212: all tests passed!

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

JoelSpeed · 2025-09-02T09:45:17Z

machine/v1beta1/types_awsprovider.go

+	// cpuOptions defines CPU-related settings for the instance, including the confidential computing policy.
+	// If unset, no cpuOptions will be included in the API request to AWS, and the instance will use the default CPU options
+	// applied by AWS for the selected intance type.
+	// +kubebuilder:validation:MinProperties=1


Pop this one on the struct rather than the field please

JoelSpeed · 2025-09-02T09:45:54Z

machine/v1beta1/types_awsprovider.go

+	// More details can be checked at https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/sev-snp.html
+	// When omitted, this means no opinion and the AWS platform is left to choose a reasonable default,
+	// which is subject to change without notice. The current default is Disabled.
+	// +kubebuilder:validation:Enum=Disabled;AMDEncrytedVirtualizationNestedPaging


Any reason to put this here, vs on the type definition on L119?

openshift-ci bot added the size/M Denotes a PR that changes 30-99 lines, ignoring generated files. label Jul 28, 2025

openshift-ci bot requested review from everettraven and mandre July 28, 2025 08:53

fangge1212 force-pushed the aws_amd_sev_snp branch 2 times, most recently from 1271931 to a6478c1 Compare July 29, 2025 08:18

bgartzi reviewed Jul 29, 2025

View reviewed changes

machine/v1beta1/types_awsprovider.go Show resolved Hide resolved

everettraven reviewed Jul 31, 2025

View reviewed changes

fangge1212 force-pushed the aws_amd_sev_snp branch 3 times, most recently from 82e877d to 1df992a Compare August 6, 2025 22:44

everettraven reviewed Aug 7, 2025

View reviewed changes

fangge1212 force-pushed the aws_amd_sev_snp branch 5 times, most recently from bbef962 to e48669f Compare August 8, 2025 06:42

fangge1212 mentioned this pull request Aug 11, 2025

✨ Add support for AMD SEV-SNP instances kubernetes-sigs/cluster-api-provider-aws#5605

Open

5 tasks

damdo reviewed Aug 11, 2025

View reviewed changes

fangge1212 force-pushed the aws_amd_sev_snp branch 3 times, most recently from f28b17a to 520141d Compare August 12, 2025 11:09

everettraven reviewed Aug 29, 2025

View reviewed changes

Support AMD SEV-SNP on AWS

d34306b

AMD SEV-SNP is one of the confidential computing technologies. This commit adds support for AMD SEV-SNP on AWS, so users can utilize the confidential computing on the cluster nodes. Signed-off-by: Fangge Jin <[email protected]>

fangge1212 force-pushed the aws_amd_sev_snp branch from 520141d to d34306b Compare September 1, 2025 07:24

JoelSpeed reviewed Sep 2, 2025

View reviewed changes

		// cpuOptions defines CPU-related settings for the instance, including the confidential computing policy.
		// If unset, no CPU options will be passed to the AWS platform and AWS default CPU options will be applied.

Support AMD SEV-SNP on AWS #2424

Are you sure you want to change the base?

Support AMD SEV-SNP on AWS #2424

Conversation

fangge1212 commented Jul 28, 2025

Uh oh!

openshift-ci bot commented Jul 28, 2025

Uh oh!

openshift-ci bot commented Jul 28, 2025

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

fangge1212 commented Aug 8, 2025

Uh oh!

fangge1212 commented Aug 11, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

damdo Aug 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

everettraven Aug 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

fangge1212 Aug 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

damdo Aug 12, 2025 •

edited

Loading

everettraven Aug 21, 2025 •

edited

Loading

fangge1212 Aug 26, 2025 •

edited

Loading