AWS S3 Role Authentication

[quentin-roche]

Hello,

I'm trying to setup Terrakube on AWS EKS cluster. I created the necessary service account and added AwsEnableRoleAuth: "true" to my config.

When the api tries to access S3, I get this error

software.amazon.awssdk.core.exception.SdkClientException: 
Unable to load credentials from any of the providers in the chain 
AwsCredentialsProviderChain(credentialsProviders=[
    SystemPropertyCredentialsProvider(),
    EnvironmentVariableCredentialsProvider(),
    WebIdentityTokenCredentialsProvider(),
    ProfileCredentialsProvider(profileName=default, profileFile=ProfileFile(sections=[])),
    ContainerCredentialsProvider(),
    InstanceProfileCredentialsProvider()
]) : [
    SystemPropertyCredentialsProvider():
        Unable to load credentials from system settings. Access key must be. specified either
        via environment variable (AWS_ACCESS_KEY_ID) or system property (aws.accessKeyId)., 
    EnvironmentVariableCredentialsProvider():
        Unable to load credentials from system settings. Access key must be specified either
        via environment variable (AWS_ACCESS_KEY_ID) or system property (aws.accessKeyId).,
    WebIdentityTokenCredentialsProvider():
        To use web identity tokens, the 'sts' service module must be on the class path., 
    ProfileCredentialsProvider(profileName=default, profileFile=ProfileFile(sections=[])):
        Profile file contained no credentials for profile 'default': ProfileFile(sections=[]),
    ContainerCredentialsProvider():
        Cannot fetch credentials from container - neither AWS_CONTAINER_CREDENTIALS_FULL_URI or
        AWS_CONTAINER_CREDENTIALS_RELATIVE_URI environment variables are set., 
    InstanceProfileCredentialsProvider():
        Failed to load credentials from IMDS.
]

My pod desciption

kubectl -n terrakube describe pod terrakube-api-69df89d8c5-wk8ck
[...]
    Environment:
      TerrakubeRedisUsername:            default
      TerrakubeRedisSSL:                 true
      TerrakubeRedisTruststorePath:      /layers/paketo-buildpacks_bellsoft-liberica/jre/lib/security/cacerts
      TerrakubeRedisTruststorePassword:  changeit
      AwsEnableRoleAuth:                 true
      AWS_STS_REGIONAL_ENDPOINTS:        regional
      AWS_DEFAULT_REGION:                eu-west-2
      AWS_REGION:                        eu-west-2
      AWS_ROLE_ARN:                      arn:aws:iam::442042504377:role/terrakube
      AWS_WEB_IDENTITY_TOKEN_FILE:       /var/run/secrets/eks.amazonaws.com/serviceaccount/token

I verified that the token is available at /var/run/secrets/eks.amazonaws.com/serviceaccount/token

I tried to lookup the error To use web identity tokens, the 'sts' service module must be on the class path and it was suggested to add to api/pom.xml :

<dependency>
    <groupId>software.amazon.awssdk</groupId>
    <artifactId>sts</artifactId>
</dependency>

It seems to work for other people without adding it. Is someone able to confirm that it should work for the helm version 3.24.0 ?

[alfespa17]

Check this

#848 (comment)

[quentin-roche]

Hello,

I look it up before. According to documentation:

AWS credentials provider chain that looks for credentials in this order:

1 - Java System Properties - aws.accessKeyId and aws.secretAccessKey
2 - Environment Variables - AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY
3 - Web Identity Token credentials from system properties or environment variables
4 - Credential profiles file at the default location (~/.aws/credentials) shared by all AWS SDKs and the AWS CLI
5 - Credentials delivered through the Amazon EC2 container service if AWS_CONTAINER_CREDENTIALS_RELATIVE_URI" environment variable is set and security manager has permission to access the variable,
6 - Instance profile credentials delivered through the Amazon EC2 metadata service

The one that will be used for kubernetes service account is Web Identity Token credentials from system properties or environment variables and the documentation says:

A credential provider that will read web identity token file path, aws role arn and aws session name from system properties or environment variables for using web identity token credentials with STS.
Use of this credentials provider requires the 'sts' module to be on the classpath.

StsWebIdentityTokenFileCredentialsProvider in sts package can be used instead of this class if any one of following is required

Pass a custom StsClient to the provider.
Periodically update credentials

I have the warning in the logs:

2025-01-26T14:05:31.197Z  WARN 1 --- [nio-8080-exec-7] s.a.a.a.c.i.WebIdentityCredentialsUtils  : To use web identity tokens, the 'sts' service module must be on the class path.

I have never done Java, so I'm not sure exactly sure, but the code suggest a missing dependency:

<dependency>
    <groupId>software.amazon.awssdk</groupId>
    <artifactId>sts</artifactId>
</dependency>

My config (Terraform template)

``` ## Global Name name: "terrakube"

## Terrakube Security
security:
  adminGroup: "${cognito_admin_group_name}" # The value should be a cognito group (example: TERRAKUBE_ADMIN)
  patSecret: "${path_secret}"  # Sample Key 32 characters z6QHX!y@Nep2QDT!53vgH43^PjRXyC3X
  internalSecret: "${internal_secret}" # Sample Key 32 characters Kb^8cMerPNZV6hS!9!kcD*KuUPUBa^B3
  dexClientId: "cognito"
  dexClientScope: "email openid profile offline_access groups"
  useOpenLDAP: false

## Terraform Storage
storage:
  defaultStorage: false
  aws:
    #accessKey: "${aws_access_key}"
    #secretKey: "${aws_secret_key}"
    accessKey: "12345"
    secretKey: "12345"
    bucketName: "${bucket_name}"
    region: "${bucket_region}"

## Dex
dex:
  config:
    issuer: "https://${domains.api}/dex"
    storage:
      type: memory
    oauth2:
      responseTypes: ["code", "token", "id_token"]
      skipApprovalScreen: true
    web:
      allowedOrigins: ["*"]

    staticClients:
    - id: cognito
      redirectURIs:
      - "https://${domains.ui}"
      - "http://localhost:3000"
      - "http://localhost:10001/login"
      - "http://localhost:10000/login"
      - "/device/callback"
      name: "cognito"
      public: true

    connectors:
    - type: oidc
      id: cognito
      name: cognito
      config:
        issuer: "${cognito_issuer}"
        clientID: "${cognito_client_id}"
        clientSecret: "${cognito_client_secret}"
        redirectURI: "https://${domains.api}/dex/callback"
        scopes:
          - openid
          - email
          - profile
        insecureSkipEmailVerified: true
        insecureEnableGroups: true
        userNameKey: "cognito:username"
        claimMapping:
          groups: "cognito:groups"

## API properties
api:
  version: "2.24.0-beta.3"
  defaultDatabase: false
  defaultRedis: false
  loadSampleData: false
  enabled: true
  replicaCount: "1"
  serviceType: "ClusterIP"
  serviceAccountName: "${service_account_name}"
  properties:
    databaseType: "POSTGRESQL"
    databaseHostname: "${postgres_host}"
    databaseName: "${postgres_database_name}"
    databaseUser: "${postgres_user}"
    databasePassword: "${postgres_password}"
    databaseSslMode: "require"
    databasePort: "5432"


    redisHostname: "${redis_host}"
    redisPassword: "${redis_password}"
    redisPort: "6379"

  env:
  - name: TerrakubeRedisUsername
    value: "${redis_user}"
    - name: TerrakubeRedisSSL
      value: "true"
    - name: TerrakubeRedisTruststorePath
      value: /layers/paketo-buildpacks_bellsoft-liberica/jre/lib/security/cacerts
    - name: TerrakubeRedisTruststorePassword
      value: changeit
    - name: AwsEnableRoleAuth
      value: "true"

executor:
  env:
  - name: TerrakubeRedisUsername
    value: "${redis_user}"
    - name: TerrakubeRedisSSL
      value: "true"
    - name: TerrakubeRedisTruststorePath
      value: /layers/paketo-buildpacks_bellsoft-liberica/jre/lib/security/cacerts
    - name: TerrakubeRedisTruststorePassword
      value: changeit
    - name: AwsIncludeBackendKeys
      value: "false"
    - name: AwsEnableRoleAuth
      value: "true"

registry:
  env:
  - name: AwsEnableRoleAuth
    value: "true"

ingress:
  useTls: true
  includeTlsHosts: true
  ui:
    enabled: false
    domain: "${domains.ui}"
  api:
    enabled: false
    domain: "${domains.api}"
  registry:
    enabled: false
    domain: "${domains.registry}"
  dex:
    enabled: false
```

[alfespa17]

You can create a custom image to test, just add the dependency here

terrakube/api/pom.xml

Line 252 in fd06d6b

<groupId>software.amazon.awssdk</groupId>

And you can build a custom image using the script
https://github.com/AzBuilder/terrakube/blob/main/scripts/build/terrakubeBuild.sh

You only need to install maven, docker and Java sdk

[quentin-roche]

I made an image based on branch release/2.24.X modifying
api/pom.xml, executor/pom.xml and registry/pom.xml by adding:

<dependency>
    <groupId>software.amazon.awssdk</groupId>
    <artifactId>sts</artifactId>
</dependency>

It works now but I was not able to compile on the main branch.

[alfespa17]

I made an image based on branch release/2.24.X modifying api/pom.xml, executor/pom.xml and registry/pom.xml by adding:

<dependency>
    <groupId>software.amazon.awssdk</groupId>
    <artifactId>sts</artifactId>
</dependency>

It works now but I was not able to compile on the main branch.

Can you create a pull request to the main branch adding the missing dependency?

[quentin-roche]

Created the PR #1736