OCPBUGS-55755: Create intervals when etcd disk metrics are over upstream recommended thresholds #30413

dgoodwin · 2025-10-22T21:11:39Z

Assisted-by: Cursor AI

This will allow us to correlate problems with time periods when etcd was struggling, and search for job runs experiencing those problems.

It is unclear how valid the upstream thresholds are, we suspect our CI proves they are too low and we operate normally well above this.

Related: openshift/release#70577 will start ingesting metrics into bigquery so we can see how jobs do.

openshift-ci-robot · 2025-10-22T21:12:05Z

@dgoodwin: This pull request references Jira Issue OCPBUGS-55755, which is invalid:

expected the bug to target the "4.21.0" version, but no target version was set

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

The bug has been updated to refer to the pull request using the external bug tracker.

In response to this:

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

openshift-trt · 2025-10-23T01:13:39Z

Risk analysis has seen new tests most likely introduced by this PR.
Please ensure that new tests meet guidelines for naming and stability.

New tests seen in this PR at sha: 0091d4f

"Import the release payload "nightly-arm64" from an external source" [Total: 2, Pass: 2, Fail: 0, Flake: 0]
"[Monitor:etcd-disk-metrics-intervals][Jira:"etcd"] monitor test etcd-disk-metrics-intervals cleanup" [Total: 12, Pass: 12, Fail: 0, Flake: 0]
"[Monitor:etcd-disk-metrics-intervals][Jira:"etcd"] monitor test etcd-disk-metrics-intervals collection" [Total: 12, Pass: 12, Fail: 0, Flake: 0]
"[Monitor:etcd-disk-metrics-intervals][Jira:"etcd"] monitor test etcd-disk-metrics-intervals interval construction" [Total: 12, Pass: 12, Fail: 0, Flake: 0]
"[Monitor:etcd-disk-metrics-intervals][Jira:"etcd"] monitor test etcd-disk-metrics-intervals preparation" [Total: 12, Pass: 12, Fail: 0, Flake: 0]
"[Monitor:etcd-disk-metrics-intervals][Jira:"etcd"] monitor test etcd-disk-metrics-intervals setup" [Total: 12, Pass: 12, Fail: 0, Flake: 0]
"[Monitor:etcd-disk-metrics-intervals][Jira:"etcd"] monitor test etcd-disk-metrics-intervals test evaluation" [Total: 12, Pass: 12, Fail: 0, Flake: 0]
"[Monitor:etcd-disk-metrics-intervals][Jira:"etcd"] monitor test etcd-disk-metrics-intervals writing to storage" [Total: 12, Pass: 12, Fail: 0, Flake: 0]

dgoodwin · 2025-10-24T12:16:24Z

In action: https://sippy.dptools.openshift.org/sippy-ng/job_runs/1981106219183509504/pull-ci-openshift-origin-main-e2e-vsphere-ovn/openshift_origin/30413/intervals?end=2025-10-22T23%3A31%3A19Z&filterText=&intervalFile=e2e-timelines_spyglass_20251022-222305.json&overrideDisplayFlag=0&selectedSources=OperatorAvailable&selectedSources=OperatorProgressing&selectedSources=OperatorDegraded&selectedSources=KubeletLog&selectedSources=EtcdLog&selectedSources=EtcdLeadership&selectedSources=Alert&selectedSources=Disruption&selectedSources=E2EFailed&selectedSources=APIServerGracefulShutdown&selectedSources=KubeEvent&selectedSources=NodeState&selectedSources=CPUMonitor&selectedSources=EtcdDiskCommitDuration&selectedSources=EtcdDiskWalFsyncDuration&start=2025-10-22T21%3A41%3A19Z

neisw · 2025-10-27T13:48:17Z

pkg/monitortests/testframework/etcddiskmetricsintervals/monitortest.go

+		return nil, err
+	}
+
+	if intervals, err := prometheus.EnsureThanosQueriersConnectedToPromSidecars(ctx, prometheusClient); err != nil {


I would change intervals to _ here and explicitly return nil for the error case. Just for clarity (AI Slop I'm sure)

neisw · 2025-10-27T13:50:13Z

pkg/monitortests/testframework/etcddiskmetricsintervals/monitortest.go

+	walFsyncQuery := `histogram_quantile(0.99, rate(etcd_disk_wal_fsync_duration_seconds_bucket{job=~".*etcd.*"}[5m]))`
+	walFsyncMetrics, warningsForWal, err := prometheusClient.QueryRange(ctx, walFsyncQuery, timeRange)
+	if err != nil {
+		return nil, err


Do you want to return nil in this case or allIntervals. Fine if you want to drop all intervals in case of any failure.

neisw · 2025-10-27T13:51:04Z

pkg/monitortests/testframework/etcddiskmetricsintervals/monitortest.go

+
+	walFsyncIntervals, err := w.createIntervalsFromMetrics(logger, walFsyncMetrics, monitorapi.SourceEtcdDiskWalFsyncDuration, w.walFsyncThreshold, "disk WAL fsync duration")
+	if err != nil {
+		return nil, err


same, just confirming you drop previously good intervals

Yeah I'm ok with this, if we error here all bets are off I think.

… thresholds

neisw · 2025-10-27T16:08:30Z

/lgtm

openshift-ci · 2025-10-27T16:11:34Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: dgoodwin, neisw

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [dgoodwin,neisw]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

dgoodwin · 2025-10-28T11:43:23Z

/verify bypass

dgoodwin · 2025-10-28T12:39:00Z

/verified bypass

openshift-ci-robot · 2025-10-28T12:39:12Z

@dgoodwin: The verified label has been added.

In response to this:

/verified bypass

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

dgoodwin · 2025-10-29T11:11:59Z

/jira refresh

openshift-ci-robot · 2025-10-29T11:12:08Z

@dgoodwin: This pull request references Jira Issue OCPBUGS-55755, which is valid. The bug has been moved to the POST state.

3 validation(s) were run on this bug

bug is open, matching expected state (open)
bug target version (4.21.0) matches configured target version for branch (4.21.0)
bug is in the state ASSIGNED, which is one of the valid states (NEW, ASSIGNED, POST)

Requesting review from QA contact:
/cc @geliu2016

In response to this:

/jira refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

openshift-ci-robot · 2025-10-29T11:57:28Z

/retest-required

Remaining retests: 0 against base HEAD f886e4e and 2 for PR HEAD ddfb6d8 in total

openshift-ci-robot · 2025-10-29T16:57:30Z

/retest-required

Remaining retests: 0 against base HEAD a91d3d0 and 1 for PR HEAD ddfb6d8 in total

openshift-ci-robot · 2025-10-30T06:56:57Z

/retest-required

Remaining retests: 0 against base HEAD 1f66b70 and 0 for PR HEAD ddfb6d8 in total

openshift-ci-robot · 2025-10-30T07:56:57Z

/hold

Revision ddfb6d8 was retested 3 times: holding

dgoodwin · 2025-10-30T14:20:42Z

/hold cancel

openshift-ci-robot · 2025-10-30T15:57:55Z

/retest-required

Remaining retests: 0 against base HEAD 8b17b60 and 2 for PR HEAD ddfb6d8 in total

openshift-ci · 2025-10-30T17:41:50Z

@dgoodwin: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/okd-scos-e2e-aws-ovn	`0091d4f`	link	false	`/test okd-scos-e2e-aws-ovn`

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

openshift-ci-robot · 2025-11-02T16:08:17Z

/retest-required

Remaining retests: 0 against base HEAD 387947a and 1 for PR HEAD ddfb6d8 in total

openshift-trt · 2025-11-02T19:53:42Z

Risk analysis has seen new tests most likely introduced by this PR.
Please ensure that new tests meet guidelines for naming and stability.

New Test Risks for sha: ddfb6d8

Job Name	New Test Risk
pull-ci-openshift-origin-main-e2e-aws-ovn-fips	High - "[Monitor:etcd-disk-metrics-intervals][Jira:"etcd"] monitor test etcd-disk-metrics-intervals cleanup" is a new test that was not present in all runs against the current commit.
pull-ci-openshift-origin-main-e2e-aws-ovn-fips	High - "[Monitor:etcd-disk-metrics-intervals][Jira:"etcd"] monitor test etcd-disk-metrics-intervals collection" is a new test that was not present in all runs against the current commit.
pull-ci-openshift-origin-main-e2e-aws-ovn-fips	High - "[Monitor:etcd-disk-metrics-intervals][Jira:"etcd"] monitor test etcd-disk-metrics-intervals interval construction" is a new test that was not present in all runs against the current commit.
pull-ci-openshift-origin-main-e2e-aws-ovn-fips	High - "[Monitor:etcd-disk-metrics-intervals][Jira:"etcd"] monitor test etcd-disk-metrics-intervals preparation" is a new test that was not present in all runs against the current commit.
pull-ci-openshift-origin-main-e2e-aws-ovn-fips	High - "[Monitor:etcd-disk-metrics-intervals][Jira:"etcd"] monitor test etcd-disk-metrics-intervals setup" is a new test that was not present in all runs against the current commit.
pull-ci-openshift-origin-main-e2e-aws-ovn-fips	High - "[Monitor:etcd-disk-metrics-intervals][Jira:"etcd"] monitor test etcd-disk-metrics-intervals test evaluation" is a new test that was not present in all runs against the current commit.
pull-ci-openshift-origin-main-e2e-aws-ovn-fips	High - "[Monitor:etcd-disk-metrics-intervals][Jira:"etcd"] monitor test etcd-disk-metrics-intervals writing to storage" is a new test that was not present in all runs against the current commit.
pull-ci-openshift-origin-main-e2e-gcp-ovn-upgrade	High - "[Monitor:etcd-disk-metrics-intervals][Jira:"etcd"] monitor test etcd-disk-metrics-intervals cleanup" is a new test that was not present in all runs against the current commit.
pull-ci-openshift-origin-main-e2e-gcp-ovn-upgrade	High - "[Monitor:etcd-disk-metrics-intervals][Jira:"etcd"] monitor test etcd-disk-metrics-intervals collection" is a new test that was not present in all runs against the current commit.
pull-ci-openshift-origin-main-e2e-gcp-ovn-upgrade	High - "[Monitor:etcd-disk-metrics-intervals][Jira:"etcd"] monitor test etcd-disk-metrics-intervals interval construction" is a new test that was not present in all runs against the current commit.
pull-ci-openshift-origin-main-e2e-gcp-ovn-upgrade	High - "[Monitor:etcd-disk-metrics-intervals][Jira:"etcd"] monitor test etcd-disk-metrics-intervals preparation" is a new test that was not present in all runs against the current commit.
pull-ci-openshift-origin-main-e2e-gcp-ovn-upgrade	High - "[Monitor:etcd-disk-metrics-intervals][Jira:"etcd"] monitor test etcd-disk-metrics-intervals setup" is a new test that was not present in all runs against the current commit.
pull-ci-openshift-origin-main-e2e-gcp-ovn-upgrade	High - "[Monitor:etcd-disk-metrics-intervals][Jira:"etcd"] monitor test etcd-disk-metrics-intervals test evaluation" is a new test that was not present in all runs against the current commit.
pull-ci-openshift-origin-main-e2e-gcp-ovn-upgrade	High - "[Monitor:etcd-disk-metrics-intervals][Jira:"etcd"] monitor test etcd-disk-metrics-intervals writing to storage" is a new test that was not present in all runs against the current commit.
pull-ci-openshift-origin-main-e2e-metal-ipi-ovn-ipv6	High - "[Monitor:etcd-disk-metrics-intervals][Jira:"etcd"] monitor test etcd-disk-metrics-intervals cleanup" is a new test that was not present in all runs against the current commit.
pull-ci-openshift-origin-main-e2e-metal-ipi-ovn-ipv6	High - "[Monitor:etcd-disk-metrics-intervals][Jira:"etcd"] monitor test etcd-disk-metrics-intervals collection" is a new test that was not present in all runs against the current commit.
pull-ci-openshift-origin-main-e2e-metal-ipi-ovn-ipv6	High - "[Monitor:etcd-disk-metrics-intervals][Jira:"etcd"] monitor test etcd-disk-metrics-intervals interval construction" is a new test that was not present in all runs against the current commit.
pull-ci-openshift-origin-main-e2e-metal-ipi-ovn-ipv6	High - "[Monitor:etcd-disk-metrics-intervals][Jira:"etcd"] monitor test etcd-disk-metrics-intervals preparation" is a new test that was not present in all runs against the current commit.
pull-ci-openshift-origin-main-e2e-metal-ipi-ovn-ipv6	High - "[Monitor:etcd-disk-metrics-intervals][Jira:"etcd"] monitor test etcd-disk-metrics-intervals setup" is a new test that was not present in all runs against the current commit.
pull-ci-openshift-origin-main-e2e-metal-ipi-ovn-ipv6	High - "[Monitor:etcd-disk-metrics-intervals][Jira:"etcd"] monitor test etcd-disk-metrics-intervals test evaluation" is a new test that was not present in all runs against the current commit.
	(...showing 20 of 35 rows)

New tests seen in this PR at sha: ddfb6d8

"[Monitor:etcd-disk-metrics-intervals][Jira:"etcd"] monitor test etcd-disk-metrics-intervals cleanup" [Total: 46, Pass: 46, Fail: 0, Flake: 0]
"[Monitor:etcd-disk-metrics-intervals][Jira:"etcd"] monitor test etcd-disk-metrics-intervals collection" [Total: 46, Pass: 46, Fail: 0, Flake: 0]
"[Monitor:etcd-disk-metrics-intervals][Jira:"etcd"] monitor test etcd-disk-metrics-intervals interval construction" [Total: 46, Pass: 46, Fail: 0, Flake: 0]
"[Monitor:etcd-disk-metrics-intervals][Jira:"etcd"] monitor test etcd-disk-metrics-intervals preparation" [Total: 46, Pass: 46, Fail: 0, Flake: 0]
"[Monitor:etcd-disk-metrics-intervals][Jira:"etcd"] monitor test etcd-disk-metrics-intervals setup" [Total: 46, Pass: 46, Fail: 0, Flake: 0]
"[Monitor:etcd-disk-metrics-intervals][Jira:"etcd"] monitor test etcd-disk-metrics-intervals test evaluation" [Total: 46, Pass: 46, Fail: 0, Flake: 0]
"[Monitor:etcd-disk-metrics-intervals][Jira:"etcd"] monitor test etcd-disk-metrics-intervals writing to storage" [Total: 46, Pass: 46, Fail: 0, Flake: 0]

openshift-ci-robot · 2025-11-03T00:23:41Z

@dgoodwin: Jira Issue OCPBUGS-55755: All pull requests linked via external trackers have merged:

Jira Issue OCPBUGS-55755 has been moved to the MODIFIED state.

In response to this:

Assisted-by: Cursor AI

This will allow us to correlate problems with time periods when etcd was struggling, and search for job runs experiencing those problems.

It is unclear how valid the upstream thresholds are, we suspect our CI proves they are too low and we operate normally well above this.

Related: openshift/release#70577 will start ingesting metrics into bigquery so we can see how jobs do.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

openshift-merge-robot · 2025-11-04T03:57:08Z

Fix included in accepted release 4.21.0-0.nightly-2025-11-03-191704

dgoodwin changed the title ~~Create intervals when etcd disk metrics are over upstream recommended thresholds~~ OCPBUGS-55755: Create intervals when etcd disk metrics are over upstream recommended thresholds Oct 22, 2025

openshift-ci bot requested review from deads2k and p0lyn0mial October 22, 2025 21:12

openshift-ci bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Oct 22, 2025

neisw reviewed Oct 27, 2025

View reviewed changes

Create intervals when etcd disk metrics are over upstream recommended…

ddfb6d8

… thresholds

dgoodwin force-pushed the etcd-disk-intervals branch from 0091d4f to ddfb6d8 Compare October 27, 2025 14:44

openshift-ci bot assigned neisw Oct 27, 2025

openshift-ci bot added the lgtm Indicates that a PR is ready to be merged. label Oct 27, 2025

openshift-ci-robot added the verified Signifies that the PR passed pre-merge verification criteria label Oct 28, 2025

openshift-ci-robot added jira/valid-bug Indicates that a referenced Jira bug is valid for the branch this PR is targeting. and removed jira/invalid-bug Indicates that a referenced Jira bug is invalid for the branch this PR is targeting. labels Oct 29, 2025

openshift-ci bot requested a review from geliu2016 October 29, 2025 11:12

openshift-ci-robot mentioned this pull request Oct 29, 2025

OCPBUGS-55755: gather P50, P95 and P99 for etcd disk metrics from all CI runs openshift/release#70577

Merged

openshift-ci bot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Oct 30, 2025

openshift-ci bot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Oct 30, 2025

openshift-merge-bot bot merged commit 87710bd into openshift:main Nov 3, 2025
20 checks passed

OCPBUGS-55755: Create intervals when etcd disk metrics are over upstream recommended thresholds #30413

OCPBUGS-55755: Create intervals when etcd disk metrics are over upstream recommended thresholds #30413

Conversation

dgoodwin commented Oct 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

openshift-ci-robot commented Oct 22, 2025

Uh oh!

openshift-trt bot commented Oct 23, 2025

Uh oh!

dgoodwin commented Oct 24, 2025

Uh oh!

neisw Oct 27, 2025

Choose a reason for hiding this comment

Uh oh!

dgoodwin Oct 27, 2025

Choose a reason for hiding this comment

Uh oh!

neisw Oct 27, 2025

Choose a reason for hiding this comment

Uh oh!

neisw Oct 27, 2025

Choose a reason for hiding this comment

Uh oh!

dgoodwin Oct 27, 2025

Choose a reason for hiding this comment

Uh oh!

neisw commented Oct 27, 2025

Uh oh!

openshift-ci bot commented Oct 27, 2025

Uh oh!

dgoodwin commented Oct 28, 2025

Uh oh!

dgoodwin commented Oct 28, 2025

Uh oh!

openshift-ci-robot commented Oct 28, 2025

Uh oh!

dgoodwin commented Oct 29, 2025

Uh oh!

openshift-ci-robot commented Oct 29, 2025

Uh oh!

openshift-ci-robot commented Oct 29, 2025

Uh oh!

openshift-ci-robot commented Oct 29, 2025

Uh oh!

openshift-ci-robot commented Oct 30, 2025

Uh oh!

openshift-ci-robot commented Oct 30, 2025

Uh oh!

dgoodwin commented Oct 30, 2025

Uh oh!

openshift-ci-robot commented Oct 30, 2025

Uh oh!

openshift-ci bot commented Oct 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

openshift-ci-robot commented Nov 2, 2025

Uh oh!

openshift-trt bot commented Nov 2, 2025

Uh oh!

Uh oh!

openshift-ci-robot commented Nov 3, 2025

Uh oh!

openshift-merge-robot commented Nov 4, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

dgoodwin commented Oct 22, 2025 •

edited

Loading

openshift-ci bot commented Oct 30, 2025 •

edited

Loading