OSDOCS-12976: updates greenboot scripts MicroShift

Author: ShaunaDiaz

_topic_maps/_topic_map_ms.yml

@@ -223,8 +223,8 @@ Topics:
   File: microshift-embed-apps-offline-use
 - Name: Embedding applications tutorial
   File: microshift-embedding-apps-tutorial
-- Name: Creating application or workload health check scripts
-  File: microshift-greenboot-workload-scripts
+- Name: Using greenboot for workload health checks
+  File: microshift-greenboot-workload-health-checks
 - Name: Automating application management with GitOps
   File: microshift-gitops
 - Name: Pod security authentication and authorization

microshift_install_get_ready/microshift-greenboot.adoc

@@ -6,19 +6,14 @@ include::_attributes/attributes-microshift.adoc[]
 
 toc::[]
 
-Greenboot is the generic health check framework for the `systemd` service on `rpm-ostree` systems such as {op-system-ostree-first}. This framework is included in {microshift-short} installations with the `microshift-greenboot` and `greenboot-default-health-checks` RPM packages.
+Learn about how greenboot health checks are used with {microshift-short}.
 
-Greenboot health checks run at various times to assess system health and automate a rollback on `rpm-ostree` systems to the last healthy state in cases of software trouble, for example:
-
-* Default health check scripts run each time the system starts.
-* In addition the to the default health checks, you can write, install, and configure application health check scripts to also run every time the system starts.
-* Greenboot can reduce your risk of being locked out of edge devices during updates and prevent a significant interruption of service if an update fails.
-* When a failure is detected, the system boots into the last known working configuration using the `rpm-ostree` rollback capability. This feature is especially useful automation for edge devices where direct serviceability is either limited or non-existent.
-
-A {microshift-short} application health check script is included in the `microshift-greenboot` RPM. The `greenboot-default-health-checks` RPM includes health check scripts verifying that DNS and `ostree` services are accessible. You can create your own health check scripts for the workloads you are running. You can write one that verifies that an application has started, for example.
+include::modules/microshift-greenboot-dir-use-for-scripts.adoc[leveloffset=+1]
 
 include::modules/microshift-greenboot-dir-structure.adoc[leveloffset=+1]
 
+include::modules/microshift-greenboot-included-health-checks.adoc[leveloffset=+1]
+
 include::modules/microshift-greenboot-microshift-health-script.adoc[leveloffset=+1]
 
 include::modules/microshift-greenboot-systemd-journal-data.adoc[leveloffset=+1]
@@ -40,4 +35,4 @@ include::modules/microshift-greenboot-check-update.adoc[leveloffset=+1]
 [id="additional-resources_microshift-greenboot_{context}"]
 [role="_additional-resources_microshift-greenboot"]
 == Additional resources
-* xref:../microshift_running_apps/microshift-greenboot-workload-scripts.adoc#microshift-greenboot-workload-scripts[Greenboot workload health check scripts]
+* xref:../microshift_running_apps/microshift-greenboot-workload-health-checks.adoc#microshift-greenboot-workload-health-checks[Greenboot workload health checks]

microshift_running_apps/microshift-greenboot-workload-health-checks.adoc

@@ -0,0 +1,26 @@
+:_mod-docs-content-type: ASSEMBLY
+[id="microshift-greenboot-workload-health-checks"]
+= Using greenboot for application and workload health checks
+include::_attributes/attributes-microshift.adoc[]
+:context: microshift-greenboot-workload-health-checks
+
+toc::[]
+
+You can use greenboot health checks to assess the health of your workloads and applications.
+
+include::modules/microshift-greenboot-how-workload-health-checks-work.adoc[leveloffset=+1]
+
+include::modules/microshift-greenboot-health-check-command.adoc[leveloffset=+1]
+
+include::modules/microshift-greenboot-app-health-check-script.adoc[leveloffset=+1]
+
+include::modules/microshift-greenboot-workload-health-script-ex.adoc[leveloffset=+2]
+
+include::modules/microshift-greenboot-testing-workload-script.adoc[leveloffset=+2]
+
+[role="_additional-resources"]
+== Additional resources
+
+* xref:../microshift_install_get_ready/microshift-greenboot.adoc#microshift-greenboot[The greenboot health check]
+
+* xref:../microshift_running_apps/microshift-applications.adoc#microshift-applying-manifests-example_applications-microshift[Auto applying manifests]

microshift_running_apps/microshift-greenboot-workload-scripts.adoc

@@ -51,7 +51,7 @@ To begin a {microshift-short} update by embedding in a {op-system-ostree} image,
 To understand more about greenboot, see the following documentation:
 
 * xref:../microshift_install_get_ready/microshift-greenboot.adoc#microshift-greenboot[The greenboot health check]
-* xref:../microshift_running_apps/microshift-greenboot-workload-scripts.adoc#microshift-greenboot-workload-scripts[Greenboot workload health check scripts]
+* xref:../microshift_running_apps/microshift-greenboot-workload-health-checks.adoc#microshift-greenboot-workload-health-checks[Greenboot workload health checks]
 
 [id="microshift-update-options-manual-rpm-updates_{context}"]
 === Manual RPM updates
@@ -91,4 +91,4 @@ You can update {op-system-ostree} or {op-system-base} and update {microshift-sho
 * xref:../microshift_updating/microshift-update-rpms-ostree.adoc#microshift-update-rpms-ostree[Applying updates on an OSTree system]
 * xref:../microshift_updating/microshift-update-rpms-manually.adoc#microshift-update-rpms-manually[Applying updates manually with RPMs]
 * xref:../microshift_install_get_ready/microshift-greenboot.adoc#microshift-greenboot[The greenboot system health check]
-* xref:../microshift_running_apps/microshift-greenboot-workload-scripts.adoc#microshift-greenboot-workload-scripts[Greenboot workload scripts]
+* xref:../microshift_running_apps/microshift-greenboot-workload-health-checks.adoc#microshift-greenboot-workload-health-checks[Greenboot workload health checks]

microshift_updating/microshift-update-options.adoc

@@ -0,0 +1,19 @@
+//Module included in the following assemblies:
+//
+//* microshift_running_apps/microshift-greenboot-workload-health-checks.adoc
+
+:_mod-docs-content-type: CONCEPT
+[id="microshift-greenboot-app-health-check-script_{context}"]
+= How to create a health check script for your application
+
+You can create workload or application health check scripts in the text editor of your choice. Save the scripts in the `/etc/greenboot/check/required.d` directory. When a script in the `/etc/greenboot/check/required.d` directory exits with an error, greenboot triggers a reboot in an attempt to heal the system.
+
+[NOTE]
+====
+Any script in the `/etc/greenboot/check/required.d` directory triggers a reboot if it exits with an error.
+====
+
+If your health check logic requires any post-check steps, you can also create additional scripts and save them in the relevant greenboot directories. For example:
+
+* You can also place shell scripts you want to run after a boot has been declared successful in `/etc/greenboot/green.d`.
+* You can place shell scripts you want to run after a boot has been declared failed in `/etc/greenboot/red.d`. For example, if you have steps to heal the system before restarting, you can create scripts for your use case and place them in the `/etc/greenboot/red.d` directory.

modules/microshift-greenboot-app-health-check-script.adoc

@@ -16,11 +16,10 @@ Access the output of greenboot health check scripts in the system log after an u
 ----
 $ sudo grub2-editenv - list | grep ^boot_success
 ----
-
++
 .Example output for a successful update
 [source,terminal]
 ----
-boot_success=1
+boot_success=1 <1>
 ----
-
-If your command returns `boot_success=0`, either the greenboot health check is still running, or the update is a failure.
+<1> If your command returns `boot_success=0`, either the greenboot health check is still running, or the update is a failure.

modules/microshift-greenboot-check-update.adoc

@@ -46,4 +46,4 @@ If you customize the values of any environment variable in the `/etc/greenboot/g
 
 * To retain customizations when building system images with {microshift-short}, add the `greenboot.conf` file to a blueprint.
 * To retain customizations when using an RPM installation, apply changes to the `greenboot.conf` file after you install {microshift-short} and greenboot RPMs.
-====
+====

modules/microshift-greenboot-create-health-check-script.adoc

@@ -0,0 +1,18 @@
+// Module included in the following assemblies:
+//
+// * microshift_install_get_ready/microshift-greenboot.adoc
+
+:_mod-docs-content-type: CONCEPT
+[id="microshift-greenboot-dir-use-for-scripts_{context}"]
+= How greenboot uses directories to run scripts
+
+Greenboot is the generic health check framework for the `systemd` service on `rpm-ostree` systems such as {op-system-ostree-first}. This framework is included in {microshift-short} installations with the `microshift-greenboot` and `greenboot-default-health-checks` RPM packages.
+
+Greenboot health checks run at various times to assess system health and automate a rollback on `rpm-ostree` systems to the last healthy state in cases of software trouble, for example:
+
+* Default health check scripts run each time the system starts.
+* In addition the to the default health checks, you can write, install, and configure application health check scripts to also run every time the system starts.
+* Greenboot can reduce your risk of being locked out of edge devices during updates and prevent a significant interruption of service if an update fails.
+* When a failure is detected, the system boots into the last known working configuration using the `rpm-ostree` rollback capability. This feature is especially useful automation for edge devices where direct serviceability is either limited or non-existent.
+
+A {microshift-short} application health check script is included in the `microshift-greenboot` RPM. The `greenboot-default-health-checks` RPM includes health check scripts verifying that DNS and `ostree` services are accessible. You can create your own health check scripts for the workloads you are running. You can write one that verifies that an application has started, for example.

modules/microshift-greenboot-dir-structure.adoc

@@ -0,0 +1,40 @@
+//Module included in the following assemblies:
+//
+//* microshift_running_apps/microshift-greenboot-workload-health-checks.adoc
+
+:_mod-docs-content-type: REFERENCE
+[id="microshift-greenboot-microshift-health-check-command_{context}"]
+= How to use the {microshift-short} health check command
+
+The `microshift healthcheck` command checks whether a workload of the provided type exists and verifies its
+status for the specified timeout duration. The number of ready replicas, that is, pods, must match the expected amount.
+
+To run the `microshift healthcheck` command successfully, use the following prerequisites:
+
+* Execute commands from a root user account.
+* Enable the {microshift-short} service.
+
+You can add the following actions to the `microshift healthcheck` command:
+
+* `-v=2` to increase verbosity of the output
+* `--timeout="${WAIT_TIMEOUT_SECS}s"` to override default 300s timeout value
+* `--namespace `_<namespace>_` to specify the namespace of the workloads
+* `--deployments `_<application-deployment>_` to check the readiness of a specific deployment
+
+The `microshift healthcheck` command also accepts the following additional parameters to specify other kinds
+of workloads:
+
+* `--daemonsets`
+* `--statefulsets`
+* These options take a comma-delimited list of resources, for example, `--daemonsets ovnkube-master,ovnkube-node`.
+
+Alternatively, a `--custom` option can be used with a `JSON` string, for example:
+[source,terminal]
+----
+$ microshift healthcheck --custom '{"openshift-storage":{"deployments": ["lvms-operator"], "daemonsets": ["vg-manager"]}, "openshift-ovn-kubernetes":{"daemonsets": ["ovnkube-master", "ovnkube-node"]}}'
+----
+
+[NOTE]
+====
+Restarting pods can indicate a crash loop.
+====

modules/microshift-greenboot-dir-use-for-scripts.adoc

@@ -16,7 +16,7 @@ You can manually access the output of health checks in the system log by using t
 ----
 $ sudo journalctl -o cat -u greenboot-healthcheck.service
 ----
-
++
 .Example output of a failed health check
 [source,terminal]
 ----
@@ -34,4 +34,4 @@ Waiting 300s for MicroShift service to be active and not failed
 FAILURE
 ...
 ...
-----
+----