kubernetes-must-gather

kubernetes-must-gather is a custom must-gather image and collection script for Kubernetes and OpenShift. It should work on AMD/x64, ARM/AARCH, PPC64, and z/Linux. The image is published to Red Hat's Quay.io at quay.io/ibm/kubernetes-must-gather. The image is based on the oc adm must-gather image.

Usage

Basic usage

oc adm must-gather --image=quay.io/ibm/kubernetes-must-gather:latest

Customized usage

Default Behavior

kubernetes-must-gather gathers significantly less than the default oc adm must-gather image because kubernetes-must-gather is designed for a more lightweight and iterative workflow using command line flags. By default, kubernetes-must-gather gathers:

1. oc describe YAMLs for all namespaces for the following resources: nodes pods events securitycontextconstraints
2. Pod logs of pods in CrashLoopBackOff state for all namespaces. Disable with --no-logs-crashloopbackoff

Usage help

oc adm must-gather --image=quay.io/ibm/kubernetes-must-gather:latest -- gather -h

Common flags

• Gather etcd pod logs: --logs-etcd
• Disable gathering CrashLoopBackOff pod logs: --no-logs-crashloopbackoff
• Enable verbose logging: --log=2

oc adm must-gather collection-scripts

kubernetes-must-gather can run any of the other collection scripts available in the upstream oc adm must-gather collection-scripts. Just pass -- and the name of the script (multiple allowed and they're executed in sequence).

For example, to run gather_apirequestcounts:

oc adm must-gather --image=quay.io/ibm/kubernetes-must-gather:latest -- gather --gather_apirequestcounts

Extra output is in the download at must-gather.local.*/quay-io-ibm-kubernetes-must-gather*/requests/

Notes

We generally recommend using a specific tag rather than latest because oc adm must-gather uses an ImagePullPolicy of PullIfNotPresent so if you were to use --image=quay.io/ibm/kubernetes-must-gather:latest once, then you could not get a newer version of the latest image in the same cluster unless you manually deleted that image from the internal image registry. Consult the tag in the release name of the latest release (e.g. 0.1.20250715013 in the URL) to use a specific version.

---

Development

Steps to build locally and publish to an OpenShift registry

1. Update the VERSION= line in gather
2. Set a variable to this version in your shell:

   export VERSION="..."
   

3. Set the namespace/project to push the image to (make sure this namespace exists):

   export NAMESPACE="customimages"
   

4. Build the image for your cluster platform; for example:

   podman build --platform linux/amd64 -t kubernetes-must-gather .
   

5. Get your cluster registry URL:

   REGISTRY=$(oc get route default-route -n openshift-image-registry --template='{{ .spec.host }}')
   

6. Login to your remote image registry with podman:

   podman login --tls-verify=false -u $(oc whoami | sed 's/://g') -p $(oc whoami -t) ${REGISTRY}
   

7. Push the image to the remote image registry (may require exposing the registry):

   podman push --tls-verify=false localhost/kubernetes-must-gather:latest ${REGISTRY}/${NAMESPACE}/kubernetes-must-gather:${VERSION}
   

8. Use the image. For example:

   oc adm must-gather --image=image-registry.openshift-image-registry.svc:5000/${NAMESPACE}/kubernetes-must-gather:${VERSION}
   

Steps to publish a new image to Quay

1. Update the VERSION= line in gather
2. Set a variable to this version in your shell:

   export VERSION="..."
   

3. Create the manifest (error if it exists is okay):

   podman manifest create quay.io/ibm/kubernetes-must-gather:latest
   

4. Remove any existing manifest images:

   for i in $(podman manifest inspect quay.io/ibm/kubernetes-must-gather:latest | jq '.manifests[].digest' | tr '\n' ' ' | sed 's/"//g'); do podman manifest remove quay.io/ibm/kubernetes-must-gather:latest $i; done
   

5. Build the images:

   podman build --platform linux/amd64,linux/ppc64le,linux/s390x,linux/arm64 --jobs=1 --manifest quay.io/ibm/kubernetes-must-gather:latest .
   

6. Log into Quay:

   podman login quay.io
   

7. Push with the version in step 1:

   podman manifest push --all quay.io/ibm/kubernetes-must-gather:latest docker://quay.io/ibm/kubernetes-must-gather:$VERSION
   

8. Test the tag:
   1. Make sure it basically works with usage:

      oc adm must-gather --image=quay.io/ibm/kubernetes-must-gather:$VERSION -- gather -h
      

   2. Run the default must gather:

      oc adm must-gather --image=quay.io/ibm/kubernetes-must-gather:$VERSION
      

9. If testing looks good, push to the latest tag:

   podman manifest push --all quay.io/ibm/kubernetes-must-gather:latest docker://quay.io/ibm/kubernetes-must-gather:latest
   

10. Test the latest tag:

    oc adm must-gather --image=quay.io/ibm/kubernetes-must-gather:latest
    

11. If everything looks good
    1. Add and push a tag:

       git tag $VERSION
       

    2. Then find this new tag in the tag list, click the ellipses, click Create release, and then create a new release.

Files

• LICENSE
• CONTRIBUTING.md
• MAINTAINERS.md
• CHANGELOG.md

Issues and Pull Requests

If you have any questions or issues you can create a new issue here.

Pull requests are very welcome! Make sure your patches are well tested. Ideally create a topic branch for every separate change you make. For example:

1. Fork the repo
2. Create your feature branch (git checkout -b my-new-feature)
3. Commit your changes (git commit -am 'Added some feature')
4. Push to the branch (git push origin my-new-feature)
5. Create new Pull Request

License

All source files must include a Copyright and License header. The SPDX license header is preferred because it can be easily scanned.

If you would like to see the detailed LICENSE click here.

#
# Copyright IBM Corporation. All Rights Reserved.
# SPDX-License-Identifier: Apache-2.0
#

Authors

• Author: Kevin Grigorenko - mailto:[email protected]
• Author: Leon Foret - mailto:[email protected]
• Author: Amar Kalsi - mailto:[email protected]
• Author: Kaifu Wu - mailto:[email protected]