RayOnSpark would fail when running k8s from remote

[hkvision]

Each k8s pods has a virtual ip, when spark gets the ip of each executor, it gets ['172.30.22.6', '172.30.1.6', '172.30.1.7', '172.30.45.6', '172.30.1.7']
and when launching ray on the driver, the command is

Executing command: ray start --address 172.30.22.7:10937 --redis-password 123456 --num-cpus 0 --node-ip-address 172.16.0.188

Since a remote node can't recognize this virtual ip within the k8s cluster, the launch ray process gets stuck.

Any ideas @Le-Zheng @glorysdj
cc @jason-dai

[hkvision]

The root cause is our get_node_ip (copied from ray) wouldn't get the actual ip of k8s nodes.

[glorysdj]

The root cause is our get_node_ip (copied from ray) wouldn't get the actual ip of k8s nodes.

even if you can get the node ip, when your different spark executors are scheduled on the same kube-noe, different raylets need to start on same host.

k8s uses virtual network to assign to each pod, can we make each raylet start as companied container in the spark executor pod.

[glorysdj]

if the issue is the remote driver can not access to raymater on k8s, you can port-forward the raymaster with command like below:
kubectl port-forward [raymasterservice 10937]:10027 --address 0.0.0.0
and let driver to connect to hostip:10027 for raymaster