super wip br

[jaxesn]

Issue #, if available:
Description of changes:
This is to show an idea of what it would take to make a br node join as a hybrid node. This adds super rough support to our ec2 based e2e tests for BR nodes, obviously the actual use case here would be to support the BR ova in vsphere environments.

iam-ra:

Current BR issues:

• aws-iam-auth exec's aws_signing_helper, both of which already exist in BR, note aws-iam-auth is not included in the ova variant but would need to b. When iam-auth exec's the signing-helper, it tries to do it thru sh which does not exist on the host. Workaround: i have used an overlay mount to "add" sh to the host to get past this. The actual fix is probably to patch that out in the BR build of iam-auth
• There is no way to add files via the settings.toml/user-data. This uses a admin-container to read the user data yaml and create the two cert files for aws-signing-helper
• Our implementation of the pod-identity addon is specific for hybrid and would likely need changes to support BR (i have not gone through testing that)

User-data

[settings.kubernetes]
cluster-name = "nodeadm-e2e-tests-1-31"
api-server = "https://CE117CE52F637469ED7F21BF3B6D5051.gr7.us-west-2.eks.amazonaws.com"
cluster-certificate = "<cluster-ca>"

authentication-mode = "aws"
cloud-provider = ""
hostname-override = "simpleflow-node-iam-ra-br-amd64"

# this is not strictly required but we do set this via nodeadm for other OSs
provider-id = "eks-hybrid:///us-west-2/nodeadm-e2e-tests-1-31/simpleflow-node-iam-ra-br-amd64"

[settings.network]
hostname = "simpleflow-node-iam-ra-br-amd64"

[settings.aws]
config = "<base64 encode of the following>"
	[default]
		region = us-west-2
		credential_process = aws_signing_helper credential-process --certificate /var/lib/eks-hybrid/roles-anywhere/pki/node.crt --private-key /var/lib/eks-hybrid/roles-anywhere/pki/node.key --profile-arn <profile-arn> --role-arn <role-arn> --trust-anchor-arn <trust-anchor> --role-session-name simpleflow-node-iam-ra-br-amd64

[settings.kubernetes.node-labels]
"eks.amazonaws.com/compute-type" = "hybrid"
# this is not strictly required but we do set this via nodeadm for other OSs
"eks.amazonaws.com/hybrid-credential-provider" = "iam-ra"

[settings.host-containers.hybrid-setup]
superpowered = true
enabled = true
source = "public.ecr.aws/k1e6s8o8/br-test:1"
user-data = "<base64 encode of the following>"

iamra: true
write_files:
  - content: |
      <iam-ra-crt>      
    path: /etc/roles-anywhere/pki/node.crt

  - content: |
      <iam-ra-key>            
    path: /etc/roles-anywhere/pki/node.key


[settings.host-containers.admin]
enabled = true
source = "public.ecr.aws/bottlerocket/bottlerocket-admin:v0.11.16"
user-data = "<base64 encode of the BR ssh keys json>"

[settings.host-containers.control]
enabled = true
source = "public.ecr.aws/bottlerocket/bottlerocket-control:v0.7.20"

ssm:
Current BR issues:

• ssm activation runs in the control admin container and the generated creds are not on the host where kubelet get access them. I manually set the creds using the apiclient after the activation is finished. This allows the kubelet via iam-auth to use these creds to join to the cluster
• for ssm activations the hostname/node name is not configurable and not known until the activation is successful. We need to set the kubernetes-hostname-override and networking.hostname based on this after the activation. After activation, we grab this via the registration file which ssm creates in the control container and set these via apiclient.
• provider-id requires the hostname so this is also set after activation

User-data

[settings.kubernetes]
cluster-name = "nodeadm-e2e-tests-1-31"
api-server = "https://CE117CE52F637469ED7F21BF3B6D5051.gr7.us-west-2.eks.amazonaws.com"
cluster-certificate = "<cluster-ca>"

authentication-mode = "aws"
cloud-provider = ""

// this is set to something fake because it seems like without it BR will not boot
// but this is changed after ssm activation
hostname-override = "foo"
provider-id = "eks-hybrid:///us-west-2/nodeadm-e2e-tests-1-31/foo"

[settings.network]
hostname = "foo"

[settings.aws]
region = "us-west-2"


[settings.kubernetes.node-labels]
"eks.amazonaws.com/compute-type" = "hybrid"
"eks.amazonaws.com/hybrid-credential-provider" = "ssm"

[settings.host-containers.hybrid-setup]
superpowered = true
enabled = true
source = "public.ecr.aws/k1e6s8o8/br-test:1"


[settings.host-containers.admin]
enabled = true
source = "public.ecr.aws/bottlerocket/bottlerocket-admin:v0.11.16"
user-data = user-data = "<base64 encode of the BR ssh keys json>"

[settings.host-containers.control]
enabled = true
source = "public.ecr.aws/bottlerocket/bottlerocket-control:v0.7.20"
user-data =  "<base64 encode of the BR ssm json example following>"
//{"ssm":{"activation-code":"<code>","activation-id":"<id>","region":"us-west-2"}}

Testing (if applicable):

Documentation added/planned (if applicable):

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.