Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Set up pod identity association in E2E tests #329

Merged
merged 4 commits into from
Feb 5, 2025

Conversation

junpengdev
Copy link
Contributor

Issue #, if available:
This is part 2 of setting up E2E tests for pod identity. Part 1 PR

Description of changes:
This PR covers pod identity association setup.

(Since this PR already contains enough changes, I will create another PR to actually deploy pods and verify EKS Pod Identity volume is correctly attached.)

I followed this public document to set up pod identity association.

In summary, the following is required in order to set up pod identity association.

  1. Create IAM role with trust relationship for Pod Identity role. The role should also be able to access some AWS resource.
  2. Create a service account dedicated for Pod Identity
  3. Associate the IAM role to the service account

The creation of service account and pod identity association can only be done after the EKS cluster is provisioned successfully. We can put this into either

  1. the creation of Pod Identity add-on, or
  2. during the add-on e2e tests.

Here I choose to put it into creation of pod identity add-on. The reason for it is we don't need to check or set up this association in every OS/auth provider combination. It should be a one-time setup for the whole EKS cluster.


The creation of IAM role of pod identity can be put into

  1. nodeadm-stack.ts to provision one IAM role for all kube/CNI combination
  2. setup-cfn.yaml to provision one IAM role for all OS/Auth provider under one kube/CNI combination
  3. during e2e test to check or provision one IAM role for each OS/Auth provider

Option 3 is not applicable for two reasons.

  • pod identity association is now done during creation of the add-on, which requires IAM ARN to complete the process
  • it's very inefficient to check/provision the IAM role for each OS/Auth provider

Option 1 and 2 are both working but I choose option 2 because it makes more sense as the role is tied to one cluster, which the lifecycle is managed by CloudFormation.


Testing (if applicable):

  1. Run e2e test locally
  2. Deploy the change to personal dev stack in CodePipeline

Documentation added/planned (if applicable):

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

@junpengdev junpengdev requested review from g-gaston and jaxesn January 29, 2025 23:19
test/e2e/addon/podidentityverify.go Outdated Show resolved Hide resolved
test/e2e/addon/podidentityverify.go Outdated Show resolved Hide resolved
test/e2e/cluster/create.go Outdated Show resolved Hide resolved
test/e2e/cluster/create.go Outdated Show resolved Hide resolved
test/e2e/kubernetes/kubernetes.go Outdated Show resolved Hide resolved
@junpengdev junpengdev requested a review from g-gaston February 4, 2025 17:37
test/e2e/addon/podidentityaddon.go Outdated Show resolved Hide resolved
test/e2e/addon/podidentityverify.go Outdated Show resolved Hide resolved
namespace = "default"
)

func NewPodIdentityAddon(cluster, name, roleArn string) PodIdentityAddon {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Isn't the addon always the same? If so, why does the caller need to provide? This constructor is already exclusive for the pod identity addon, so i think it encapsulate de knowlege of the addon name and hide that detail from the caller

Wdyt?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

good suggestion. I will add it to my part3 PR.

},
}

if _, err := k8s.CoreV1().ServiceAccounts(namespace).Create(ctx, serviceAccount, metav1.CreateOptions{}); err != nil {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we probably should probably retry this

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's keep it this way for now. I'd like to understand what errors we may have and what errors are retryable.

@junpengdev junpengdev merged commit fbcabca into aws:main Feb 5, 2025
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants