Skip to content

Commit 3958020

Browse files
anjannathpraveenkumar
anjannath
authored and
praveenkumar
committed
add systemd services for configuration after start
the services does the various needed tasks to setup the ocp or microshift cluster, these systemd units runs small shell scripts which are based on: https://github.com/crc-org/crc-cloud/blob/main/pkg/bundle/setup/clustersetup.sh and does the following tasks: - creates crc specific configurations for dnsmasq - sets a new uuid as cluster id - creates the pod for routes-controller - tries to grow the disk and filesystem - checks if the cluster operators are ready - adds the pull secret to the cluster - sets kubeadmin and developer user passwords - sets a custom ca for authentication - sets custom nip.io cluster domain
1 parent 6ba1600 commit 3958020

25 files changed

+549
-0
lines changed

‎.gitignore

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -11,3 +11,4 @@ podman-remote/
1111
.sw[a-p]
1212
crc-cluster-kube-apiserver-operator
1313
crc-cluster-kube-controller-manager-operator
14+
systemd/crc-dnsmasq.sh

‎createdisk-library.sh

Lines changed: 35 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -223,6 +223,7 @@ function prepare_hyperV() {
223223
echo 'CONST{virt}=="microsoft", RUN{builtin}+="kmod load hv_sock"' > /etc/udev/rules.d/90-crc-vsock.rules
224224
EOF
225225
}
226+
226227
function prepare_qemu_guest_agent() {
227228
local vm_ip=$1
228229

@@ -392,3 +393,37 @@ function remove_pull_secret_from_disk() {
392393
esac
393394
}
394395

396+
function copy_systemd_units() {
397+
case "${BUNDLE_TYPE}" in
398+
"snc"|"okd")
399+
export APPS_DOMAIN="apps-crc.testing"
400+
envsubst '${APPS_DOMAIN}' < systemd/dnsmasq.sh.template > systemd/crc-dnsmasq.sh
401+
unset APPS_DOMAIN
402+
;;
403+
"microshift")
404+
export APPS_DOMAIN="apps.crc.testing"
405+
envsubst '${APPS_DOMAIN}' < systemd/dnsmasq.sh.template > systemd/crc-dnsmasq.sh
406+
unset APPS_DOMAIN
407+
;;
408+
esac
409+
410+
${SSH} core@${VM_IP} -- 'mkdir -p /home/core/systemd-units && mkdir -p /home/core/systemd-scripts'
411+
${SCP} systemd/crc-*.service core@${VM_IP}:/home/core/systemd-units/
412+
${SCP} systemd/crc-*.sh core@${VM_IP}:/home/core/systemd-scripts/
413+
414+
case "${BUNDLE_TYPE}" in
415+
"snc"|"okd")
416+
${SCP} systemd/ocp-*.service core@${VM_IP}:/home/core/systemd-units/
417+
${SCP} systemd/ocp-*.sh core@${VM_IP}:/home/core/systemd-scripts/
418+
;;
419+
esac
420+
421+
${SSH} core@${VM_IP} -- 'sudo cp /home/core/systemd-units/* /etc/systemd/system/ && sudo cp /home/core/systemd-scripts/* /usr/local/bin/'
422+
${SSH} core@${VM_IP} -- 'ls /home/core/systemd-scripts/ | xargs -t -I % sudo chmod +x /usr/local/bin/%'
423+
${SSH} core@${VM_IP} -- 'sudo restorecon -rv /usr/local/bin'
424+
425+
# enable only the .path units
426+
${SSH} core@${VM_IP} -- 'ls /home/core/systemd-units/*.service | xargs basename -a | xargs sudo systemctl enable'
427+
428+
${SSH} core@${VM_IP} -- 'rm -rf /home/core/systemd-units /home/core/systemd-scripts'
429+
}

‎createdisk.sh

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -163,6 +163,8 @@ fi
163163

164164
# Beyond this point, packages added to the ADDITIONAL_PACKAGES variable won’t be installed in the guest
165165
install_additional_packages ${VM_IP}
166+
copy_systemd_units
167+
166168
cleanup_vm_image ${VM_NAME} ${VM_IP}
167169

168170
# Enable cloud-init service
@@ -185,6 +187,17 @@ fi
185187

186188
podman_version=$(${SSH} core@${VM_IP} -- 'rpm -q --qf %{version} podman')
187189

190+
# Disable cloud-init network config
191+
${SSH} core@${VM_IP} 'sudo bash -x -s' << EOF
192+
cat << EFF > /etc/cloud/cloud.cfg.d/05_disable-network.cfg
193+
network:
194+
config: disabled
195+
EFF
196+
EOF
197+
198+
# Disable cloud-init hostname update
199+
${SSH} core@${VM_IP} -- 'sudo sed -i "s/^preserve_hostname: false$/preserve_hostname: true/" /etc/cloud/cloud.cfg'
200+
188201
# Cleanup cloud-init config
189202
${SSH} core@${VM_IP} -- "sudo cloud-init clean --logs"
190203

‎docs/self-sufficient-bundle.md

Lines changed: 34 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,34 @@
1+
# Self sufficient bundles
2+
3+
Since release 4.19.0 of OpenShift Local, the bundles generated by `snc` contain additional systemd services to provision the cluster and remove the need for
4+
an outside entity to provision the cluster, although an outside process needs to create some files on pre-defined locations inside the VM for the systemd
5+
services to do their work.
6+
7+
## The following table lists the systemd services and the location of files they need to provision the cluster, users of SNC need to create those files
8+
9+
| Systemd unit | Runs for (ocp, MicroShift, both) | Input files location | Marker env variables |
10+
| :----------------------------: | :------------------------------: | :----------------------------------: | :------------------: |
11+
| `crc-cluster-status.service` | both | none | none |
12+
| `crc-pullsecret.service` | both | /opt/crc/pull-secret | none |
13+
| `crc-dnsmasq.service` | both | none | none |
14+
| `crc-routes-controller.service`| both | none | none |
15+
| `ocp-cluster-ca.service` | ocp | /opt/crc/custom-ca.crt | CRC_CLOUD=1 |
16+
| `ocp-clusterid.service` | ocp | none | none |
17+
| `ocp-custom-domain.service` | ocp | none | CRC_CLOUD=1 |
18+
| `ocp-growfs.service` | ocp | none | none |
19+
| `ocp-userpasswords.service` | ocp | /opt/crc/pass_{kubeadmin, developer} | none |
20+
21+
In addition to the above services we have `ocp-cluster-ca.path`, `crc-pullsecret.path` and `ocp-userpasswords.path` that monitors the filesystem paths
22+
related to their `*.service` counterparts and starts the service when the paths become available.
23+
24+
> [!NOTE]
25+
> "Marker env variable" is set using an env file, if the required env variable is not set then unit is skipped
26+
> some units are run only when CRC_CLOUD=1 is set, these are only needed when using the bundles with crc-cloud
27+
28+
The systemd services are heavily based on the [`clustersetup.sh`](https://github.com/crc-org/crc-cloud/blob/main/pkg/bundle/setup/clustersetup.sh) script found in the `crc-cloud` project.
29+
30+
## Naming convention for the systemd unit files
31+
32+
Systemd units that are needed for both 'OpenShift' and 'MicroShift' are named as `crc-*.service`, units that are needed only for 'OpenShift' are named
33+
as `ocp-*.service` and when we add units that are only needed for 'MicroShift' they should be named as `ucp-*.service`
34+

‎systemd/crc-cluster-status.service

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,13 @@
1+
[Unit]
2+
Description=CRC Unit checking if cluster is ready
3+
After=kubelet.service ocp-clusterid.service ocp-cluster-ca.service ocp-custom-domain.service
4+
After=crc-pullsecret.service
5+
6+
[Service]
7+
Type=oneshot
8+
Restart=on-failure
9+
ExecStart=/usr/local/bin/crc-cluster-status.sh
10+
RemainAfterExit=true
11+
12+
[Install]
13+
WantedBy=multi-user.target

‎systemd/crc-cluster-status.sh

Lines changed: 43 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,43 @@
1+
#!/bin/bash
2+
3+
set -x
4+
5+
export KUBECONFIG=/opt/kubeconfig
6+
7+
function check_cluster_healthy() {
8+
WAIT="authentication|console|etcd|ingress|openshift-apiserver"
9+
10+
until `oc get co > /dev/null 2>&1`
11+
do
12+
sleep 2
13+
done
14+
15+
for i in $(oc get co | grep -P "$WAIT" | awk '{ print $3 }')
16+
do
17+
if [[ $i == "False" ]]
18+
then
19+
return 1
20+
fi
21+
done
22+
return 0
23+
}
24+
25+
rm -rf /tmp/.crc-cluster-ready
26+
27+
COUNTER=0
28+
CLUSTER_HEALTH_SLEEP=8
29+
CLUSTER_HEALTH_RETRIES=500
30+
31+
while ! check_cluster_healthy
32+
do
33+
sleep $CLUSTER_HEALTH_SLEEP
34+
if [[ $COUNTER == $CLUSTER_HEALTH_RETRIES ]]
35+
then
36+
return 1
37+
fi
38+
((COUNTER++))
39+
done
40+
41+
# need to set a marker to let `crc` know the cluster is ready
42+
touch /tmp/.crc-cluster-ready
43+

‎systemd/crc-dnsmasq.service

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,18 @@
1+
[Unit]
2+
Description=CRC Unit for configuring dnsmasq
3+
Wants=ovs-configuration.service
4+
After=ovs-configuration.service
5+
Before=kubelet-dependencies.target
6+
StartLimitIntervalSec=30
7+
8+
[Service]
9+
Type=oneshot
10+
Restart=on-failure
11+
EnvironmentFile=/etc/systemd/system/crc-env
12+
ExecStartPre=/bin/systemctl start ovs-configuration.service
13+
ExecStart=/usr/local/bin/crc-dnsmasq.sh
14+
ExecStartPost=/usr/bin/systemctl restart NetworkManager.service
15+
ExecStartPost=/usr/bin/systemctl restart dnsmasq.service
16+
17+
[Install]
18+
WantedBy=kubelet-dependencies.target

‎systemd/crc-pullsecret.service

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,12 @@
1+
[Unit]
2+
Description=CRC Unit for adding pull secret to cluster
3+
After=kubelet.service
4+
StartLimitIntervalSec=90sec
5+
6+
[Service]
7+
Type=oneshot
8+
Restart=on-failure
9+
ExecStart=/usr/local/bin/crc-pullsecret.sh
10+
11+
[Install]
12+
WantedBy=multi-user.target

‎systemd/crc-pullsecret.sh

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
#!/bin/bash
2+
3+
set -x
4+
5+
source /usr/local/bin/crc-systemd-common.sh
6+
export KUBECONFIG="/opt/kubeconfig"
7+
8+
wait_for_resource secret
9+
10+
# check if existing pull-secret is valid if not add the one from /opt/crc/pull-secret
11+
existingPsB64=$(oc get secret pull-secret -n openshift-config -o jsonpath="{['data']['\.dockerconfigjson']}")
12+
existingPs=$(echo "${existingPsB64}" | base64 -d)
13+
14+
echo "${existingPs}" | jq -e '.auths'
15+
16+
if [[ $? != 0 ]]; then
17+
pullSecretB64=$(cat /opt/crc/pull-secret | base64 -w0)
18+
oc patch secret pull-secret -n openshift-config --type merge -p "{\"data\":{\".dockerconfigjson\":\"${pullSecretB64}\"}}"
19+
rm -f /opt/crc/pull-secret
20+
fi
21+

‎systemd/crc-routes-controller.service

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,12 @@
1+
[Unit]
2+
Description=CRC Unit starting routes controller
3+
Wants=network-online.target gvisor-tap-vsock.service sys-class-net-tap0.device
4+
After=sys-class-net-tap0.device network-online.target kubelet.service gvisor-tap-vsock.service
5+
6+
[Service]
7+
Type=oneshot
8+
EnvironmentFile=/etc/systemd/system/crc-env
9+
ExecStart=/usr/local/bin/crc-routes-controller.sh
10+
11+
[Install]
12+
WantedBy=multi-user.target

‎systemd/crc-routes-controller.sh

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,16 @@
1+
#!/bin/bash
2+
3+
set -x
4+
5+
if [[ ${CRC_NETWORK_MODE_USER} -eq 0 ]]; then
6+
echo -n "network-mode 'system' detected: skipping routes-controller pod deployment"
7+
exit 0
8+
fi
9+
10+
source /usr/local/bin/crc-systemd-common.sh
11+
export KUBECONFIG=/opt/kubeconfig
12+
13+
wait_for_resource pods
14+
15+
oc apply -f /opt/crc/routes-controller.yaml
16+

‎systemd/crc-systemd-common.sh

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,12 @@
1+
# $1 is the resource to check
2+
# $2 is an optional maximum retry count; default 20
3+
function wait_for_resource() {
4+
local retry=0
5+
local max_retry=${2:-20}
6+
until `oc get "$1" > /dev/null 2>&1`
7+
do
8+
[ $retry == $max_retry ] && exit 1
9+
sleep 5
10+
((retry++))
11+
done
12+
}

‎systemd/dnsmasq.sh.template

Lines changed: 26 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,26 @@
1+
#!/bin/bash
2+
3+
set -x
4+
5+
if [[ ${CRC_NETWORK_MODE_USER} -eq 1 ]]; then
6+
echo -n "network-mode 'user' detected: skipping dnsmasq configuration"
7+
exit 0
8+
fi
9+
10+
hostName=$(hostname)
11+
hostIp=$(hostname --all-ip-addresses | awk '{print $1}')
12+
13+
cat << EOF > /etc/dnsmasq.d/crc-dnsmasq.conf
14+
listen-address=$hostIp
15+
expand-hosts
16+
log-queries
17+
local=/crc.testing/
18+
domain=crc.testing
19+
address=/${APPS_DOMAIN}/$hostIp
20+
address=/api.crc.testing/$hostIp
21+
address=/api-int.crc.testing/$hostIp
22+
address=/$hostName.crc.testing/$hostIp
23+
EOF
24+
25+
/bin/systemctl enable --now dnsmasq.service
26+
/bin/nmcli conn modify --temporary ovs-if-br-ex ipv4.dns $hostIp,1.1.1.1

‎systemd/ocp-cluster-ca.service

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,11 @@
1+
[Unit]
2+
Description=CRC Unit setting custom cluster ca
3+
After=kubelet.service ocp-clusterid.service
4+
5+
[Service]
6+
Type=oneshot
7+
Restart=on-failure
8+
ExecStart=/usr/local/bin/ocp-cluster-ca.sh
9+
10+
[Install]
11+
WantedBy=multi-user.target

‎systemd/ocp-cluster-ca.sh

Lines changed: 91 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,91 @@
1+
#!/bin/bash
2+
3+
# The steps followed to generate CA and replace system:admin cert are from:
4+
# https://access.redhat.com/solutions/5286371
5+
# https://access.redhat.com/solutions/6054981
6+
7+
set -x
8+
9+
source /usr/local/bin/crc-systemd-common.sh
10+
export KUBECONFIG="/opt/kubeconfig"
11+
12+
wait_for_resource configmap
13+
14+
custom_ca_path=/opt/crc/custom-ca.crt
15+
external_ip_path=/opt/crc/eip
16+
17+
if [ ! -f ${custom_ca_path} ]; then
18+
echo "Cert bundle /opt/crc/custom-ca.crt not found, generating one..."
19+
# generate a ca bundle and use it, overwrite custom_ca_path
20+
CA_SUBJ="/OU=openshift/CN=admin-kubeconfig-signer-custom"
21+
openssl genrsa -out /tmp/custom-ca.key 4096
22+
openssl req -x509 -new -nodes -key /tmp/custom-ca.key -sha256 -days 365 -out "${custom_ca_path}" -subj "${CA_SUBJ}"
23+
fi
24+
25+
if [ ! -f /opt/crc/pass_kubeadmin ]; then
26+
echo "kubeadmin password file not found"
27+
exit 1
28+
fi
29+
30+
PASS_KUBEADMIN="$(cat /opt/crc/pass_kubeadmin)"
31+
oc create configmap client-ca-custom -n openshift-config --from-file=ca-bundle.crt=${custom_ca_path}
32+
oc patch apiserver cluster --type=merge -p '{"spec": {"clientCA": {"name": "client-ca-custom"}}}'
33+
oc create configmap admin-kubeconfig-client-ca -n openshift-config --from-file=ca-bundle.crt=${custom_ca_path} \
34+
--dry-run=client -o yaml | oc replace -f -
35+
36+
rm -f /opt/crc/custom-ca.crt
37+
38+
# create CSR
39+
openssl req -new -newkey rsa:4096 -nodes -keyout /tmp/newauth-access.key -out /tmp/newauth-access.csr -subj "/CN=system:admin"
40+
41+
cat << EOF >> /tmp/newauth-access-csr.yaml
42+
apiVersion: certificates.k8s.io/v1
43+
kind: CertificateSigningRequest
44+
metadata:
45+
name: newauth-access
46+
spec:
47+
signerName: kubernetes.io/kube-apiserver-client
48+
groups:
49+
- system:authenticated
50+
request: $(cat /tmp/newauth-access.csr | base64 -w0)
51+
usages:
52+
- client auth
53+
EOF
54+
55+
oc create -f /tmp/newauth-access-csr.yaml
56+
57+
until `oc adm certificate approve newauth-access > /dev/null 2>&1`
58+
do
59+
echo "Unable to approve the csr newauth-access"
60+
sleep 5
61+
done
62+
63+
cluster_name=$(oc config view -o jsonpath='{.clusters[0].name}')
64+
apiserver_url=$(oc config view -o jsonpath='{.clusters[0].cluster.server}')
65+
66+
if [ -f "${external_ip_path}" ]; then
67+
apiserver_url=api.$(cat "${external_ip_path}").nip.io
68+
fi
69+
70+
updated_kubeconfig_path=/opt/crc/kubeconfig
71+
72+
oc get csr newauth-access -o jsonpath='{.status.certificate}' | base64 -d > /tmp/newauth-access.crt
73+
oc config set-credentials system:admin --client-certificate=/tmp/newauth-access.crt --client-key=/tmp/newauth-access.key --embed-certs --kubeconfig="${updated_kubeconfig_path}"
74+
oc config set-context system:admin --cluster="${cluster_name}" --namespace=default --user=system:admin --kubeconfig="${updated_kubeconfig_path}"
75+
oc get secret localhost-recovery-client-token -n openshift-kube-controller-manager -ojsonpath='{.data.ca\.crt}'| base64 -d > /tmp/bundle-ca.crt
76+
oc config set-cluster "${cluster_name}" --server="${apiserver_url}" --certificate-authority=/tmp/bundle-ca.crt \
77+
--kubeconfig="${updated_kubeconfig_path}" --embed-certs
78+
79+
echo "Logging in again to update $KUBECONFIG with kubeadmin token"
80+
COUNTER=0
81+
MAXIMUM_LOGIN_RETRY=500
82+
until `oc login --insecure-skip-tls-verify=true -u kubeadmin -p "$PASS_KUBEADMIN" https://api.crc.testing:6443 --kubeconfig /opt/crc/newkubeconfig > /dev/null 2>&1`
83+
do
84+
if [ $COUNTER == $MAXIMUM_LOGIN_RETRY ]; then
85+
echo "Unable to login to the cluster..., installation failed."
86+
exit 1
87+
fi
88+
echo "Logging into OpenShift with updated credentials try $COUNTER, hang on...."
89+
sleep 5
90+
((COUNTER++))
91+
done

0 commit comments

Comments
 (0)