[enterprise-4.19] OSDOCS-12976: updates greenboot scripts MicroShift

_topic_maps/_topic_map_ms.yml

@@ -215,8 +215,8 @@ Topics:
   File: microshift-embed-apps-offline-use
 - Name: Embedding applications tutorial
   File: microshift-embedding-apps-tutorial
-- Name: Creating application or workload health check scripts
-  File: microshift-greenboot-workload-scripts
+- Name: Using greenboot for workload health checks
+  File: microshift-greenboot-workload-health-checks
 - Name: Automating application management with GitOps
   File: microshift-gitops
 - Name: Pod security authentication and authorization

microshift_install_get_ready/microshift-greenboot.adoc

@@ -6,19 +6,14 @@ include::_attributes/attributes-microshift.adoc[]
 
 toc::[]
 
-Greenboot is the generic health check framework for the `systemd` service on `rpm-ostree` systems such as {op-system-ostree-first}. This framework is included in {microshift-short} installations with the `microshift-greenboot` and `greenboot-default-health-checks` RPM packages.
+Learn about how greenboot health checks are used with {microshift-short}.
 
-Greenboot health checks run at various times to assess system health and automate a rollback on `rpm-ostree` systems to the last healthy state in cases of software trouble, for example:
-
-* Default health check scripts run each time the system starts.
-* In addition the to the default health checks, you can write, install, and configure application health check scripts to also run every time the system starts.
-* Greenboot can reduce your risk of being locked out of edge devices during updates and prevent a significant interruption of service if an update fails.
-* When a failure is detected, the system boots into the last known working configuration using the `rpm-ostree` rollback capability. This feature is especially useful automation for edge devices where direct serviceability is either limited or non-existent.
-
-A {microshift-short} application health check script is included in the `microshift-greenboot` RPM. The `greenboot-default-health-checks` RPM includes health check scripts verifying that DNS and `ostree` services are accessible. You can create your own health check scripts for the workloads you are running. You can write one that verifies that an application has started, for example.
+include::modules/microshift-greenboot-dir-use-for-scripts.adoc[leveloffset=+1]
 
 include::modules/microshift-greenboot-dir-structure.adoc[leveloffset=+1]
 
+include::modules/microshift-greenboot-included-health-checks.adoc[leveloffset=+1]
+
 include::modules/microshift-greenboot-microshift-health-script.adoc[leveloffset=+1]
 
 include::modules/microshift-greenboot-systemd-journal-data.adoc[leveloffset=+1]
@@ -40,4 +35,4 @@ include::modules/microshift-greenboot-check-update.adoc[leveloffset=+1]
 [id="additional-resources_microshift-greenboot_{context}"]
 [role="_additional-resources_microshift-greenboot"]
 == Additional resources
-* xref:../microshift_running_apps/microshift-greenboot-workload-scripts.adoc#microshift-greenboot-workload-scripts[Greenboot workload health check scripts]
+* xref:../microshift_running_apps/microshift-greenboot-workload-health-checks.adoc#microshift-greenboot-workload-health-checks[Greenboot workload health checks]

microshift_running_apps/microshift-greenboot-workload-health-checks.adoc

@@ -0,0 +1,26 @@
+:_mod-docs-content-type: ASSEMBLY
+[id="microshift-greenboot-workload-health-checks"]
+= Using greenboot for application and workload health checks
+include::_attributes/attributes-microshift.adoc[]
+:context: microshift-greenboot-workload-health-checks
+
+toc::[]
+
+You can use greenboot health checks to assess the health of your workloads and applications.
+
+include::modules/microshift-greenboot-how-workload-health-checks-work.adoc[leveloffset=+1]
+
+include::modules/microshift-greenboot-health-check-command.adoc[leveloffset=+1]
+
+include::modules/microshift-greenboot-app-health-check-script.adoc[leveloffset=+1]
+
+include::modules/microshift-greenboot-workload-health-script-ex.adoc[leveloffset=+2]
+
+include::modules/microshift-greenboot-testing-workload-script.adoc[leveloffset=+2]
+
+[role="_additional-resources"]
+== Additional resources
+
+* xref:../microshift_install_get_ready/microshift-greenboot.adoc#microshift-greenboot[The greenboot health check]
+
+* xref:../microshift_running_apps/microshift-applications.adoc#microshift-applying-manifests-example_applications-microshift[Auto applying manifests]

microshift_updating/microshift-update-options.adoc

@@ -51,7 +51,7 @@ To begin a {microshift-short} update by embedding in a {op-system-ostree} image,
 To understand more about greenboot, see the following documentation:
 
 * xref:../microshift_install_get_ready/microshift-greenboot.adoc#microshift-greenboot[The greenboot health check]
-* xref:../microshift_running_apps/microshift-greenboot-workload-scripts.adoc#microshift-greenboot-workload-scripts[Greenboot workload health check scripts]
+* xref:../microshift_running_apps/microshift-greenboot-workload-health-checks.adoc#microshift-greenboot-workload-health-checks[Greenboot workload health checks]
 
 [id="microshift-update-options-manual-rpm-updates_{context}"]
 === Manual RPM updates
@@ -91,4 +91,4 @@ You can update {op-system-ostree} or {op-system-base} and update {microshift-sho
 * xref:../microshift_updating/microshift-update-rpms-ostree.adoc#microshift-update-rpms-ostree[Applying updates on an OSTree system]
 * xref:../microshift_updating/microshift-update-rpms-manually.adoc#microshift-update-rpms-manually[Applying updates manually with RPMs]
 * xref:../microshift_install_get_ready/microshift-greenboot.adoc#microshift-greenboot[The greenboot system health check]
-* xref:../microshift_running_apps/microshift-greenboot-workload-scripts.adoc#microshift-greenboot-workload-scripts[Greenboot workload scripts]
+* xref:../microshift_running_apps/microshift-greenboot-workload-health-checks.adoc#microshift-greenboot-workload-health-checks[Greenboot workload health checks]

modules/microshift-greenboot-app-health-check-script.adoc

@@ -0,0 +1,19 @@
+//Module included in the following assemblies:
+//
+//* microshift_running_apps/microshift-greenboot-workload-health-checks.adoc
+
+:_mod-docs-content-type: CONCEPT
+[id="microshift-greenboot-app-health-check-script_{context}"]
+= How to create a health check script for your application
+
+You can create workload or application health check scripts in the text editor of your choice. Save the scripts in the `/etc/greenboot/check/required.d` directory. When a script in the `/etc/greenboot/check/required.d` directory exits with an error, greenboot triggers a reboot in an attempt to heal the system.
+
+[NOTE]
+====
+Any script in the `/etc/greenboot/check/required.d` directory triggers a reboot if it exits with an error.
+====
+
+If your health check logic requires any post-check steps, you can also create additional scripts and save them in the relevant greenboot directories. For example:
+
+* You can also place shell scripts you want to run after a boot has been declared successful in `/etc/greenboot/green.d`.
+* You can place shell scripts you want to run after a boot has been declared failed in `/etc/greenboot/red.d`. For example, if you have steps to heal the system before restarting, you can create scripts for your use case and place them in the `/etc/greenboot/red.d` directory.

modules/microshift-greenboot-check-update.adoc

@@ -16,11 +16,10 @@ Access the output of greenboot health check scripts in the system log after an u
 ----
 $ sudo grub2-editenv - list | grep ^boot_success
 ----
-
++
 .Example output for a successful update
 [source,terminal]
 ----
-boot_success=1
+boot_success=1 <1>
 ----
-
-If your command returns `boot_success=0`, either the greenboot health check is still running, or the update is a failure.
+<1> If your command returns `boot_success=0`, either the greenboot health check is still running, or the update is a failure.

modules/microshift-greenboot-dir-structure.adoc

@@ -46,4 +46,4 @@ If you customize the values of any environment variable in the `/etc/greenboot/g
 
 * To retain customizations when building system images with {microshift-short}, add the `greenboot.conf` file to a blueprint.
 * To retain customizations when using an RPM installation, apply changes to the `greenboot.conf` file after you install {microshift-short} and greenboot RPMs.
-====
+====

modules/microshift-greenboot-dir-use-for-scripts.adoc

@@ -0,0 +1,18 @@
+// Module included in the following assemblies:
+//
+// * microshift_install_get_ready/microshift-greenboot.adoc
+
+:_mod-docs-content-type: CONCEPT
+[id="microshift-greenboot-dir-use-for-scripts_{context}"]
+= How greenboot uses directories to run scripts
+
+Greenboot is the generic health check framework for the `systemd` service on `rpm-ostree` systems such as {op-system-ostree-first}. This framework is included in {microshift-short} installations with the `microshift-greenboot` and `greenboot-default-health-checks` RPM packages.
+
+Greenboot health checks run at various times to assess system health and automate a rollback on `rpm-ostree` systems to the last healthy state in cases of software trouble, for example:
+
+* Default health check scripts run each time the system starts.
+* In addition the to the default health checks, you can write, install, and configure application health check scripts to also run every time the system starts.
+* Greenboot can reduce your risk of being locked out of edge devices during updates and prevent a significant interruption of service if an update fails.
+* When a failure is detected, the system boots into the last known working configuration using the `rpm-ostree` rollback capability. This feature is especially useful automation for edge devices where direct serviceability is either limited or non-existent.
+
+A {microshift-short} application health check script is included in the `microshift-greenboot` RPM. The `greenboot-default-health-checks` RPM includes health check scripts verifying that DNS and `ostree` services are accessible. You can create your own health check scripts for the workloads you are running. You can write one that verifies that an application has started, for example.

modules/microshift-greenboot-health-check-command.adoc

@@ -0,0 +1,73 @@
+//Module included in the following assemblies:
+//
+//* microshift_running_apps/microshift-greenboot-workload-health-checks.adoc
+
+:_mod-docs-content-type: REFERENCE
+[id="microshift-greenboot-microshift-health-check-command_{context}"]
+= How to use the {microshift-short} health check command
+
+The `microshift healthcheck` command checks whether a workload of the provided type exists and verifies its
+status for the specified timeout duration. The number of ready replicas, that is, pods, must match the expected amount.
+
+To run the `microshift healthcheck` command successfully, use the following prerequisites:
+
+* Execute commands from a root user account.
+* Enable the {microshift-short} service.
+
+You can add the following actions to the `microshift healthcheck` command:
+
+* `-v=2` to increase verbosity of the output
+* `--timeout="${WAIT_TIMEOUT_SECS}s"` to override default 300s timeout value
+* `--namespace `_<namespace>_` to specify the namespace of the workloads
+* `--deployments `_<application-deployment>_` to check the readiness of a specific deployment
++
+.Example command
+[source,terminal]
+----
+$ sudo microshift healthcheck -v=2 --timeout="300s" --namespace busybox --deployments busybox-deployment
+----
++
+.Example output
+[source,text]
+----
+??? I0410 08:54:03.766578    5898 service.go:29] microshift.service is enabled
+??? I0410 08:54:03.766699    5898 service.go:31] Waiting 5m0s for microshift.service to be ready
+??? I0410 08:54:03.768794    5898 service.go:38] microshift.service is ready
+??? I0410 08:54:03.770585    5898 utils.go:34] Waiting for 1 goroutines
+??? I0410 08:54:03.770955    5898 workloads.go:94] Waiting 5m0s for deployment/busybox-deployment in busybox
+??? I0410 08:54:03.777830    5898 workloads.go:132] Deployment/busybox-deployment in busybox is ready
+??? I0410 08:54:03.777858    5898 healthcheck.go:75] Workloads are ready
+----
+
+The `microshift healthcheck` command also accepts the following additional parameters to specify other kinds
+of workloads:
+
+* `--daemonsets`
+* `--statefulsets`
+* These options take a comma-delimited list of resources, for example, `--daemonsets ovnkube-master,ovnkube-node`.
+
+Alternatively, a `--custom` option can be used with a `JSON` string, for example:
+
+[source,terminal]
+----
+$ sudo microshift healthcheck --custom '{"openshift-storage":{"deployments":
+    ["lvms-operator"], "daemonsets": ["vg-manager"]}, "openshift-ovn-kubernetes":
+    {"daemonsets": ["ovnkube-master", "ovnkube-node"]}}'
+----
+
+.Example output
+[source,text]
+----
+??? I0410 08:54:25.291059    5979 service.go:29] microshift.service is enabled
+??? I0410 08:54:25.291167    5979 service.go:31] Waiting 5m0s for microshift.service to be ready
+??? I0410 08:54:25.293188    5979 service.go:38] microshift.service is ready
+??? I0410 08:54:25.294331    5979 workloads.go:58] Waiting 5m0s for daemonset/ovnkube-node in openshift-ovn-kubernetes
+??? I0410 08:54:25.294351    5979 workloads.go:58] Waiting 5m0s for daemonset/ovnkube-master in openshift-ovn-kubernetes
+??? I0410 08:54:25.294331    5979 workloads.go:58] Waiting 5m0s for daemonset/vg-manager in openshift-storage
+??? I0410 08:54:25.294341    5979 workloads.go:94] Waiting 5m0s for deployment/lvms-operator in openshift-storage
+??? I0410 08:54:25.309739    5979 workloads.go:89] Daemonset/ovnkube-node in openshift-ovn-kubernetes is ready
+??? I0410 08:54:25.310213    5979 workloads.go:89] Daemonset/vg-manager in openshift-storage is ready
+??? I0410 08:54:25.310731    5979 workloads.go:132] Deployment/lvms-operator in openshift-storage is ready
+??? I0410 08:54:25.311017    5979 workloads.go:89] Daemonset/ovnkube-master in openshift-ovn-kubernetes is ready
+??? I0410 08:54:25.311189    5979 healthcheck.go:52] Workloads are ready
+----