Add ipsec connect wait service

When node goes for a reboot on an IPsec enabled cluster, once it comes up,
libreswan parses /etc/ipsec.d/openshift.conf file and establishes SAs with
peers and it may be still in progress even after kubelet is started, pod
scheduled on this node would fail communicating with other pods until IPsec
tunnels are established.
So this commit adds wait-for-ipsec-connect.service systemd service which depends
on ipsecenabler.service created by IPsec machine config. This new service loads
existing connections into libreswan with auto=start option for every connection
and waits upto 3 minutes until IPsec tunnels are established.
This service is added into the base template to avoid two reboots during upgrade if it
goes into IPsec machine configs rendered by CNO.

TODO: observe ipsec-upgrade behavior with this in CI and need to revisit the logic
as it needs to be enabled only on IPsec enabled clusters.

Signed-off-by: Periyasamy Palanisamy <[email protected]>

templates/common/_base/files/wait-for-ipsec-connect.yaml

@@ -0,0 +1,43 @@
+mode: 0755
+path: "/usr/local/bin/ipsec-connect-wait.sh"
+contents:
+  inline: |
+    #!/bin/bash
+    set -x
+
+    if [ ! -e "/etc/ipsec.d/openshift.conf" ]; then
+      exit 0
+    fi
+
+    #
+    if ! grep -q "auto=start" /etc/ipsec.d/openshift.conf; then
+      sed -i '/^.*conn ovn.*$/a\    auto=start' /etc/ipsec.d/openshift.conf
+    fi
+
+    cat /etc/ipsec.d/openshift.conf
+
+    chroot /proc/1/root ipsec restart
+
+    timeout=180
+    elapsed=0
+    desiredconn=""
+    establishedsa=""
+
+    while [[ $elapsed -lt $timeout ]]; do
+      desiredconn=$(grep -E '^\s*conn\s+' /etc/ipsec.d/openshift.conf | grep -v '%default' | awk '{print $2}' | tr ' ' '\n' | sort | tr '\n' ' ')
+      establishedsa=$(ipsec showstates | grep STATE_V2_ESTABLISHED_CHILD_SA | grep -o '"[^"]*"' | sed 's/"//g' | tr ' ' '\n' | sort | uniq | tr '\n' ' ')
+      if [ "$desiredconn" == "$establishedsa" ]; then
+        echo "IPsec SAs are established for desired connections"
+        break
+      else
+        echo "IPsec SAs are not established yet, waiting"
+        sleep 2s
+        fi
+      elapsed=$((elapsed + 2))
+    done
+
+    if [[ $elapsed -ge $timeout ]]; then
+        echo "Timed out waiting, some connections are not established, desired conns $desiredconn, established conns $establishedsa"
+    fi
+
+    ipsec status

templates/common/_base/units/wait-for-ipsec-connect.yaml

@@ -0,0 +1,16 @@
+name: wait-for-ipsec-connect.service
+enabled: true
+contents: |
+  [Unit]
+  Description=Ensure IKE SA established for existing IPsec connections.
+  After=ipsec.service
+  Before=kubelet-dependencies.target node-valid-hostname.service
+
+  [Service]
+  Type=oneshot
+  ExecStart=/usr/local/bin/ipsec-connect-wait.sh
+  StandardOutput=journal+console
+  StandardError=journal+console
+
+  [Install]
+  WantedBy=ipsec.service