OSDOCS-12976: updates greenboot scripts MicroShift

Author: ShaunaDiaz

microshift_running_apps/microshift-greenboot-workload-scripts.adoc

@@ -14,12 +14,15 @@ include::modules/microshift-greenboot-how-workload-health-check-scripts-work.ado
 
 include::modules/microshift-greenboot-included-health-checks.adoc[leveloffset=+1]
 
-include::modules/microshift-greenboot-create-health-check-script.adoc[leveloffset=+1]
+include::modules/microshift-greenboot-app-health-check-script.adoc[leveloffset=+1]
+
+include::modules/microshift-greenboot-workload-health-script-ex.adoc[leveloffset=+2]
 
 include::modules/microshift-greenboot-testing-workload-script.adoc[leveloffset=+1]
 
-[id="additional-resources_microshift-greenboot-workload-scripts_{context}"]
 [role="_additional-resources"]
 == Additional resources
+
 * xref:../microshift_install_get_ready/microshift-greenboot.adoc#microshift-greenboot[The greenboot health check]
+
 * xref:../microshift_running_apps/microshift-applications.adoc#microshift-applying-manifests-example_applications-microshift[Auto applying manifests]

modules/microshift-greenboot-app-health-check-script.adoc

@@ -0,0 +1,19 @@
+//Module included in the following assemblies:
+//
+//* microshift_running_apps/microshift-greenboot-workload-scripts.adoc
+
+:_mod-docs-content-type: CONCEPT
+[id="microshift-greenboot-app-health-check-script_{context}"]
+= How to create a health check script for your application
+
+You can create workload or application health check scripts in the text editor of your choice using the example in this documentation. Save the scripts in the `/etc/greenboot/check/required.d` directory. When a script in the `/etc/greenboot/check/required.d` directory exits with an error, greenboot triggers a reboot in an attempt to heal the system.
+
+[NOTE]
+====
+Any script in the `/etc/greenboot/check/required.d` directory triggers a reboot if it exits with an error.
+====
+
+If your health check logic requires any post-check steps, you can also create additional scripts and save them in the relevant greenboot directories. For example:
+
+* You can also place shell scripts you want to run after a boot has been declared successful in `/etc/greenboot/green.d`.
+* You can place shell scripts you want to run after a boot has been declared failed in `/etc/greenboot/red.d`. For example, if you have steps to heal the system before restarting, you can create scripts for your use case and place them in the `/etc/greenboot/red.d` directory.

modules/microshift-greenboot-check-status.adoc

@@ -34,3 +34,5 @@ $ systemctl show --property=ExecMainStatus --value greenboot-healthcheck.service
 ----
 $ cat /run/motd.d/boot-status
 ----
+
+//Q: is all of this still true?

modules/microshift-greenboot-check-update.adoc

@@ -23,4 +23,4 @@ $ sudo grub2-editenv - list | grep ^boot_success
 boot_success=1
 ----
 
-If your command returns `boot_success=0`, either the greenboot health check is still running, or the update is a failure.
+If your command returns `boot_success=0`, either the greenboot health check is still running, or the update is a failure.

modules/microshift-greenboot-create-health-check-script.adoc

@@ -46,4 +46,6 @@ If you customize the values of any environment variable in the `/etc/greenboot/g
 
 * To retain customizations when building system images with {microshift-short}, add the `greenboot.conf` file to a blueprint.
 * To retain customizations when using an RPM installation, apply changes to the `greenboot.conf` file after you install {microshift-short} and greenboot RPMs.
-====
+====
+
+//Q: all still true?

modules/microshift-greenboot-dir-structure.adoc

@@ -34,4 +34,4 @@ Waiting 300s for MicroShift service to be active and not failed
 FAILURE
 ...
 ...
-----
+----

modules/microshift-greenboot-health-check-log.adoc

@@ -6,19 +6,37 @@
 [id="microshift-greenboot-how-workload-health-check-scripts-work_{context}"]
 = How workload health check scripts work
 
-The workload or application health check script described in this tutorial uses the {microshift-short} health check functions that are available in the `/usr/share/microshift/functions/greenboot.sh` file. This enables you to reuse procedures already implemented for the {microshift-short} core services.
+The workload or application health check script described in this tutorial uses the {microshift-short} health check functions already implemented for the {microshift-short} core services. This enables you to reuse those procedures as a convenience for checking your own workloads.
 
 The script starts by running checks that the basic functions of the workload are operating as expected. To run the script successfully:
 
 * Execute the script from a root user account.
 * Enable the {microshift-short} service.
 
-The health check performs the following actions:
+The `microshift healthcheck` command checks whether a workload of the provided type exists and verifies its
+status for the specified timeout duration. The number of ready replicas, that is, pods, must match the expected amount.
 
-* Gets a wait timeout of the current boot cycle for the `wait_for` function.
-* Calls the `namespace_images_downloaded` function to wait until pod images are available.
-* Calls the `namespace_pods_ready` function to wait until pods are ready.
-* Calls the `namespace_pods_not_restarting` function to verify pods are not restarting.
+You can add the following actions to the `microshift healthcheck` command:
+
+* `-v=2` to increase verbosity of the output
+* `--timeout="${WAIT_TIMEOUT_SECS}s"` to override default 300s timeout value
+* `--namespace busybox` to specify the Namespace of the workloads
+* `--deployments busybox-deployment` to specify Deployment to check the readiness of
+
+The `microshift healthcheck` command also accepts the following additional parameters to specify other kinds
+of workloads:
+
+*`--daemonsets`
+* `--statefulsets`
+
+These options take a comma-delimited list of resources, for example, `--daemonsets ovnkube-master,ovnkube-node`.
+
+Alternatively, a `--custom` option can be used with a `JSON` string, for example:
++
+[source,terminal]
+----
+$ microshift healthcheck --custom '{"openshift-storage":{"deployments": ["lvms-operator"], "daemonsets": ["vg-manager"]}, "openshift-ovn-kubernetes":{"daemonsets": ["ovnkube-master", "ovnkube-node"]}}'
+----
 
 [NOTE]
 ====

modules/microshift-greenboot-how-workload-health-check-scripts-work.adoc

@@ -30,29 +30,14 @@ The `40_microshift_running_check.sh` health check script only performs validatio
 |Next
 |`exit 1`
 
-|Wait for Kubernetes API health endpoints to be working and receiving traffic
+|For each core namespace, wait for readiness of the workload
 |Next
 |`exit 1`
-
-|Wait for any pod to start
-|Next
-|`exit 1`
-
-|For each core namespace, wait for images to be pulled
-|Next
-|`exit 1`
-
-|For each core namespace, wait for pods to be ready
-|Next
-|`exit 1`
-
-|For each core namespace, check if pods are not restarting
-|`exit 0`
-|`exit 1`
 |===
 
 [id="validation-wait-period_{context}"]
 == Validation wait period
+
 The wait period in each validation is five minutes by default. After the wait period, if the validation has not succeeded, it is declared a failure. This wait period is incrementally increased by the base wait period after each boot in the verification loop.
 
 * You can override the base-time wait period by setting the `MICROSHIFT_WAIT_TIMEOUT_SEC` environment variable in the `/etc/greenboot/greenboot.conf` configuration file. For example, you can change the wait time to three minutes by resetting the value to 180 seconds, such as `MICROSHIFT_WAIT_TIMEOUT_SEC=180`.

modules/microshift-greenboot-microshift-health-script.adoc

@@ -7,7 +7,7 @@
 [id="microshift-greenboot-access-prerollback-check_{context}"]
 = Accessing prerollback health check output in the system log
 
-You can access the output of health check scripts in the system log. For example, check the results of a prerollback script using the following procedure.
+You can access the output of health check scripts in the system log. For example, check the results of a pre-rollback script using the following procedure.
 
 .Procedure
 
@@ -47,3 +47,8 @@ Script '40_microshift_pre_rollback.sh' SUCCESS
 FINISHED
 redboot-task-runner.service: Deactivated successfully.
 ----
++
+[NOTE]
+====
+In case of a rollback, the pre-rollback script runs the `sudo microshift-cleanup-data --ovn` command to prepare the system for a potential software downgrade.
+====

modules/microshift-greenboot-prerollback-log.adoc

@@ -13,4 +13,4 @@ Health check scripts for updates are installed into the `/etc/greenboot/check/re
 [IMPORTANT]
 ====
 Wait until after an update is declared valid before starting third-party workloads. If a rollback is performed after workloads start, you can lose data. Some third-party workloads create or update data on a device before an update is complete. Upon rollback, the file system reverts to its state before the update.
-====
+====

modules/microshift-greenboot-updates-workloads.adoc

@@ -0,0 +1,55 @@
+/Module included in the following assemblies:
+//
+//* microshift_running_apps/microshift-greenboot-workload-scripts.adoc
+
+:_mod-docs-content-type: REFERENCE
+[id="microshift-greenboot-workload-health-check-script-ex_{context}"]
+= Workload health check script example
+
+The following example uses the {microshift-short} health check script as a template. You can use this example with the provided libraries as a guide for creating basic health check scripts for your applications.
+
+[id="microshift-greenboot-app-health-check-basic-prereqs_{context}"]
+== Basic prerequisites for creating a health check script
+
+* The workload must be installed.
+* You must have root access.
+
+[id="microshift-greenboot-app-health-check-ex_{context}"]
+== Example and functional requirements
+
+You can start with the following example health check script. Modify it for your use case. In your workload health check script, you must complete the following minimum steps:
+
+* Set the environment variables.
+* Define the user workload namespaces.
+* List the expected pod count.
+
+[IMPORTANT]
+====
+Choose a name prefix for your application that ensures it runs after the `40_microshift_running_check.sh` script, which implements the {microshift-short} health check procedure for its core services.
+====
+
+.Example workload health check script
+[source, bash]
+----
+# #!/bin/bash
+set -e
+
+SCRIPT_NAME=$(basename $0)
+
+# Set greenboot to read and execute the workload health check functions library.
+source /usr/share/microshift/functions/greenboot.sh
+#Q: should this still be in the script if we deprecated it? see https://github.com/openshift/microshift/pull/4437/files#diff-01887b3e92e76300a404b954081c1b6c06f7417e7d74b8bdac0d16a071a56c5dR7
+
+# Set the system to automatically stop the script if the user running it is not 'root'.
+if [ $(id -u) -ne 0 ] ; then
+    echo "The '${SCRIPT_NAME}' script must be run with the 'root' user privileges"
+    exit 1
+fi
+
+echo "STARTED"
+
+# Set the wait timeout for the current check based on the boot counter
+WAIT_TIMEOUT_SECS=$(get_wait_timeout)
+
+/usr/bin/microshift healthcheck -v=2 --timeout="${WAIT_TIMEOUT_SECS}s" --namespace busybox --deployments busybox-deployment
+----

modules/microshift-greenboot-workload-health-script-ex.adoc

@@ -21,4 +21,4 @@ $ sudo grub2-editenv - list | grep ^boot_success
 [source,terminal]
 ----
 boot_success=1
-----
+----