Flexvolume plugin conflict in GKE with default agent.data_home value

[twilfong]

The Scalr agent K8s Helm chart creates a DaemonSet in the worker template that makes use of a hostPath directory that is set based on the value in agent.data_home. The default value for this is currently "/home/kubernetes/flexvolume/agent-k8s", which is a directory that the GKE distribution of Kubernetes uses as its Flexvolume plugin directory.

GKE changes the default Flexvolume plugin directory from /var/lib/kubelet/volumeplugins to /home/kubernetes/flexvolume, in its Kubelet configuration. (Flexvolume is deprecated but still supported.) If this directory exists, Kubelet automatically scans it for new custom volume driver plugins, which causes (non-critical) errors to be constantly logged by the kubelet on every node in the cluster where this chart is installed.

The default value for this directory should be changed to something that no service running on the host should expect to be used for any other purpose. A longer-term fix might be to move away from using a hostPath directly.

Also note that (at least with GKE) most volumes on a worker node are mounted with noexec, and /home/kubernetes/flexvolume was likely chosen because it is not mounted with noexec. A suggested new default for agent.data_home would be "/home/kubernetes/bin/scalr/agent-k8s" or another similar directory that is not mounted with the noexec flag, and is also not reserved for some other expected purpose.

See also: #32

[yurii-kryvosheia]

We also have difficulties finding the right agent.data_home for bottlerocket OS. Setting home i.e. to /var/scalr leads to intermittent errors like Failed to instantiate provider "aws" to obtain schema: fork/exec during multiple parallel runs.