OCPBUGS-62790: Use separate tmpfs for ostree checkout on live ISO

[zaneb]

Installations using ABI/assisted with 16GiB of RAM on the bootstrap node
were failing with "no space left on device" during bootstrapping. The
live ISO environment uses a tmpfs mounted at /var that is sized at 50%
of available RAM. On systems with 16GiB of RAM, this provides only 8GiB
of tmpfs space.

At the beginning of the bootstrap process, node-image-pull.sh creates an
ostree checkout underneath /var/ostree-container. When this is added to
the regular disk space usage of the later parts of the bootstrap, the
peak tmpfs usage hits around 9.4GiB.

This fix creates a separate 5GiB tmpfs to house /var/ostree-container,
so that it is not subject to the limits on the size of /var.

[openshift-ci-robot]

@zaneb: This pull request references Jira Issue OCPBUGS-62790, which is valid.

3 validation(s) were run on this bug

• bug is open, matching expected state (open)
• bug target version (4.21.0) matches configured target version for branch (4.21.0)
• bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, POST)

Requesting review from QA contact:
/cc @mhanss

The bug has been updated to refer to the pull request using the external bug tracker.

In response to this:

Installations using ABI/assisted with 16GiB of RAM on the bootstrap node
were failing with "no space left on device" during bootstrapping. The
live ISO environment uses a tmpfs mounted at /var that is sized at 50%
of available RAM. On systems with 16GiB of RAM, this provides only 8GiB
of tmpfs space.

At the beginning of the bootstrap process, node-image-pull.sh creates an
ostree checkout underneath /var/ostree-container. When this is added to
the regular disk space usage of the later parts of the bootstrap, the
peak tmpfs usage hits around 9.4GiB.

This fix creates a separate 5GiB tmpfs to house /var/ostree-container,
so that it is not subject to the limits on the size of /var.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[zaneb]

This works and is independent of coreos internals, unlike #10055. Plus no resizing of a mounted filesystem. I think this is the way to go.

/verified by @zaneb
/cc @jlebon @andfasano

[openshift-ci-robot]

@zaneb: This PR has been marked as verified by @zaneb.

In response to this:

This works and is independent of coreos internals, unlike #10055. Plus no resizing of a mounted filesystem. I think this is the way to go.

/verified by @zaneb
/cc @jlebon @andfasano

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[zaneb]

/cherry-pick release-4.20

[openshift-cherrypick-robot]

@zaneb: once the present PR merges, I will cherry-pick it on top of release-4.20 in a new PR and assign it to you.

In response to this:

/cherry-pick release-4.20

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[openshift-ci]

@zaneb: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/e2e-agent-two-node-fencing-ipv4	027899c	link	false	/test e2e-agent-two-node-fencing-ipv4
ci/prow/e2e-agent-compact-ipv4-iso-no-registry	027899c	link	false	/test e2e-agent-compact-ipv4-iso-no-registry
ci/prow/okd-scos-e2e-vsphere-ovn	027899c	link	false	/test okd-scos-e2e-vsphere-ovn

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

[zaneb]

coreos/fedora-coreos-config#499 describes the original reason for using an xfs filesystem on top of tmpfs, which was related to an SELinux issue at startup. Since that isn't a concern here, we can greatly simplify by dropping the xfs part.

/verified by @zaneb

[openshift-ci-robot]

@zaneb: This PR has been marked as verified by @zaneb.

In response to this:

coreos/fedora-coreos-config#499 describes the original reason for using an xfs filesystem on top of tmpfs, which was related to an SELinux issue at startup. Since that isn't a concern here, we can greatly simplify by dropping the xfs part.

/verified by @zaneb

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[mhanss]

/verified by @mhanss
https://qe-private-deck-ci.apps.ci.l2s4.p1.openshiftapps.com/view/gs/test-platform-results/pr-logs/pull/openshift_release/68119/rehearse-68119-periodic-ci-openshift-eng-agent-qe-infra-release-4.21-amd64-nightly-vsphere-agent-compact-fips-f7/1994493157932273664

[openshift-ci-robot]

@mhanss: This PR has been marked as verified by @mhanss.

In response to this:

/verified by @mhanss
https://qe-private-deck-ci.apps.ci.l2s4.p1.openshiftapps.com/view/gs/test-platform-results/pr-logs/pull/openshift_release/68119/rehearse-68119-periodic-ci-openshift-eng-agent-qe-infra-release-4.21-amd64-nightly-vsphere-agent-compact-fips-f7/1994493157932273664

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[pawanpinjarkar]

/approve

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: pawanpinjarkar

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [pawanpinjarkar]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[bfournie]

/lgtm

[openshift-ci-robot]

@zaneb: Jira Issue Verification Checks: Jira Issue OCPBUGS-62790
✔️ This pull request was pre-merge verified.
✔️ All associated pull requests have merged.
✔️ All associated, merged pull requests were pre-merge verified.

Jira Issue OCPBUGS-62790 has been moved to the MODIFIED state and will move to the VERIFIED state when the change is available in an accepted nightly payload. 🕓

In response to this:

Installations using ABI/assisted with 16GiB of RAM on the bootstrap node
were failing with "no space left on device" during bootstrapping. The
live ISO environment uses a tmpfs mounted at /var that is sized at 50%
of available RAM. On systems with 16GiB of RAM, this provides only 8GiB
of tmpfs space.

At the beginning of the bootstrap process, node-image-pull.sh creates an
ostree checkout underneath /var/ostree-container. When this is added to
the regular disk space usage of the later parts of the bootstrap, the
peak tmpfs usage hits around 9.4GiB.

This fix creates a separate 5GiB tmpfs to house /var/ostree-container,
so that it is not subject to the limits on the size of /var.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[openshift-cherrypick-robot]

@zaneb: new pull request created: #10140

In response to this:

/cherry-pick release-4.20

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.