What happened: The kubectl commands in ClusterLoader2 do not explicitly specify a namespace and will inherit the namespace if set by the kube context. However, this secret is hard coded to be created in the default namespace, and if we are using a different on in the kube context, it will look for the clutser-loader secret in the wrong namespace. This wasn't an issue in existing CI tests b/c they already use the default namespace automatically when looking at the kube context:

CURRENT   NAME                                           CLUSTER                                        AUTHINFO                                       NAMESPACE
*         k8s-infra-e2e-boskos-scale-29_e2e-3101-62db2   k8s-infra-e2e-boskos-scale-29_e2e-3101-62db2   k8s-infra-e2e-boskos-scale-29_e2e-3101-62db2  

For my use case, I was trying to build a cluster with Azure, and I ran into an issue saying that error: failed to create serviceaccount: namespaces "test-pods" not found in these logs. However, that namespace isn't being passed anywhere to clusterloader2 and it seems to be implicitly picking it up.

Environment:

• Kubernetes version (use kubectl version):
• Cloud provider or hardware configuration:
• OS (e.g: cat /etc/os-release):
• Kernel (e.g. uname -a):
• Install tools:
• Network plugin and version (if this is a network-related bug):
• Others: