DNM Checking CI job result after enabling disk IOPS/RW limitation #2961
Conversation
|
Skipping CI for Draft Pull Request. |
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: The full list of commands accepted by this bot can be found here.
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
|
recheck |
|
Build failed (check pipeline). Post https://softwarefactory-project.io/zuul/t/rdoproject.org/buildset/4dd7e96f32d9444684ac5ec2fd9b7778 ✔️ openstack-k8s-operators-content-provider SUCCESS in 1h 56m 17s |
|
recheck |
|
Build failed (check pipeline). Post https://softwarefactory-project.io/zuul/t/rdoproject.org/buildset/4b2b7e9d11314a3ea6242b61367a51df ❌ openstack-k8s-operators-content-provider FAILURE in 7m 21s |
|
recheck |
|
Build failed (check pipeline). Post https://softwarefactory-project.io/zuul/t/rdoproject.org/buildset/eb552d8ff1bd47cba9b68e96e530c51c ✔️ openstack-k8s-operators-content-provider SUCCESS in 1h 53m 43s |
|
recheck |
|
Build failed (check pipeline). Post https://softwarefactory-project.io/zuul/t/rdoproject.org/buildset/dfedc4772b52442babf1e25b2d542ba6 ✔️ openstack-k8s-operators-content-provider SUCCESS in 1h 54m 41s |
danpawlik
force-pushed
the
check-iops-limitation
branch
from
aeded39 to
3ac70f4
Compare
|
recheck |
1 similar comment
|
recheck |
|
Build failed (check pipeline). Post https://softwarefactory-project.io/zuul/t/rdoproject.org/buildset/72e20fc6686e48509120326f8b730559 ✔️ openstack-k8s-operators-content-provider SUCCESS in 1h 50m 10s |
|
recheck |
|
Increased IOPS and RW - https://review.rdoproject.org/r/c/config/+/57595 - it is just few minutes to finish. Eh. |
|
Build failed (check pipeline). Post https://softwarefactory-project.io/zuul/t/rdoproject.org/buildset/f509e8861f054906b93c36547388136d ✔️ openstack-k8s-operators-content-provider SUCCESS in 1h 53m 23s |
danpawlik
force-pushed
the
check-iops-limitation
branch
2 times, most recently
from
2e96cee to
a2fb65e
Compare
|
Merge Failed. This change or one of its cross-repo dependencies was unable to be automatically merged with the current state of its repository. Please rebase the change and upload a new patchset. |
The CI still have from time to time issue related to "noisy neighbor". We still can ask infra team to apply limitation inside the flavor, but until we don't know what quota can be set, let's do it via systemd. For many services and for CRC (kubelet has set cgroup systemd in /etc/kubernetes/kubelet.conf), so should respect that. More info what value were set are in commit message and PR [1]. [1] https://review.rdoproject.org/r/c/config/+/57582/ Depends-On: openstack-k8s-operators/openstack-operator#1434
danpawlik
force-pushed
the
check-iops-limitation
branch
from
a2fb65e to
b0d0e5e
Compare
|
Build failed (check pipeline). Post https://softwarefactory-project.io/zuul/t/rdoproject.org/buildset/ca4fb80ccbcf419581e1feef033ebe80 ✔️ openstack-k8s-operators-content-provider SUCCESS in 1h 51m 11s |
|
15k IOPS is not enough. |
The kubelet service logs provides many interesting information, that might be helpful to see what is the root cause of failing job. Signed-off-by: Daniel Pawlik <[email protected]>
danpawlik
force-pushed
the
check-iops-limitation
branch
from
b0d0e5e to
fc44237
Compare
|
recheck |
|
recheck |
|
recheck |
After doing few tests [1], the CI job pass without increasing timeout when RW (read write) limit is set to 250MB - let's set that value as default. Also enable disk limitation by default. [1] openstack-k8s-operators/ci-framework#2961 Change-Id: I3b2e81711145d398430b3830ed541040123f4535 Signed-off-by: Daniel Pawlik <[email protected]>
The CI still have from time to time issue related to "noisy neighbor". We still can ask infra team to apply limitation inside the flavor, but until we don't know what quota can be set, let's do it via systemd. For many services and for CRC (kubelet has set cgroup systemd in /etc/kubernetes/kubelet.conf), so should respect that.
More info what value were set are in commit message and PR [1].
[1] https://review.rdoproject.org/r/c/config/+/57582/
Depends-On: openstack-k8s-operators/openstack-operator#1434