Skip to content

Improvements in cloud provider detection mechanism #83

Description

@m-barczyk

The binary's getCloudProvider() function probes AWS, Azure, and GCP metadata endpoints
to detect which cloud the node runs on:

func getCloudProvider(svc *service.Service, ctx context.Context, logs *logger.Logger) string {
cloudProvider := utils.GetEnvs(logs, "cloud_provider", "")
logs.Info("cloud_provider from env:" + cloudProvider)
if cloudProvider == "" {
// if cloud_provider is not set, check if the cloud provider is AWS, Azure, or Google
isAWS, errAWS := handlers.CheckIfAWS(svc, ctx)
if isAWS {
cloudProvider = utils.CloudProviderAWS
}
isAzure, errAzure := handlers.CheckIfAzure(svc, ctx)
if isAzure {
cloudProvider = utils.CloudProviderAzure
}
isGoogle, errGoogle := handlers.CheckIfGoogle(svc, ctx)
if isGoogle {
cloudProvider = utils.CloudProviderGoogle
}
if errAWS != nil && errAzure != nil && errGoogle != nil {
logs.Exit("ERROR in JFrog Credentials provider, could not check if cloud provider is AWS, Azure, or Google", 1)
}
}
return cloudProvider
}

However, the Helm chart never sets cloud_provider in the provider config env array, even though the cloud is already known at template time (the user explicitly enables aws.enabled, gcp.enabled, or azure.enabled).
cloudProvider := utils.GetEnvs(logs, "cloud_provider", "")
Never sets cloudProvider because there is not ENV for it. Auto detection mechanism must always run.

I suggest to fix it in: #82

But there is another issue.

On AWS EKS auto-detection causes a 30-second timeout during bootstrapping.

func CheckIfAWS(s *service.Service, ctx context.Context) (bool, error) {
s.Logger.Info("Checking if cloud provider is AWS")
token, err := getToken(s, ctx)
if err != nil {
return false, fmt.Errorf("Error getting aws token: %v", err)
}
req, err := http.NewRequestWithContext(ctx, "GET", METADATA_URL, nil)
req.Header.Add("X-aws-ec2-metadata-token", token)
if err != nil {
return false, fmt.Errorf("Error creating request to check if cloud provider is AWS: %v", err)
}

CheckIfAWS() does two IMDS calls:

  1. getToken() — PUT http://169.254.169.254/latest/api/token (IMDSv2)
  2. GET http://169.254.169.254/latest/meta-data/ (with token)

The add-provider-config subcommand runs via nsenter -t 1 -m -p which
enters host mount/PID namespace but NOT the network namespace (-n flag
is absent).

nsenter -t 1 -m -p -- ${JFROG_CREDENTIAL_PROVIDER_BINARY_DIR}/{{ .name }} add-provider-config --yaml --provider-home "${KUBELET_CREDENTIAL_PROVIDER_CONFIG_DIR}" --provider-config "${KUBELET_CREDENTIAL_PROVIDER_CONFIG_FILE_NAME}"
{{- else if eq $cloudProvider "aws" }}
echo "Copying the /etc/${JFROG_CONFIG_FILE}.json configuration file to ${KUBELET_MOUNT_PATH}${KUBELET_CREDENTIAL_PROVIDER_CONFIG_DIR}/${JFROG_CONFIG_FILE}.json"
cp -f "/etc/${JFROG_CONFIG_FILE}.json" "${KUBELET_MOUNT_PATH}${KUBELET_CREDENTIAL_PROVIDER_CONFIG_DIR}/${JFROG_CONFIG_FILE}.json"
sleep 2 # Wait a bit to ensure the file is copied before proceeding
nsenter -t 1 -m -p -- ${JFROG_CREDENTIAL_PROVIDER_BINARY_DIR}/{{ .name }} add-provider-config --provider-home "${KUBELET_CREDENTIAL_PROVIDER_CONFIG_DIR}" --provider-config "${KUBELET_CREDENTIAL_PROVIDER_CONFIG_FILE_NAME}"

The binary inherits the container's network namespace where
IMDSv2 PUT requests are blocked by http_put_response_hop_limit=1 on
the instance. The first call (getToken) always hangs for 30s and times out.

However, CheckIfAWS still returns (true, nil) because the second call
to the metadata endpoint succeeds (GET requests to 169.254.169.254 work
without a token on some configurations, or the fallback IMDSv1 path is
used). So AWS IS detected, but only after a 30s penalty from the failed
token request.

I think nsenter should use -n to hook to pid 1 network as well.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions