Skip to content

NO-JIRA: Add docs about junits of a monitortest #30428

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 1 commit into
base: main
Choose a base branch
from
Open

NO-JIRA: Add docs about junits of a monitortest #30428

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 19 additions & 1 deletion pkg/monitortestframework/types.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -52,7 +52,16 @@ type MonitorTest interface {

// CollectData will only be called once near the end of execution, before all Intervals are inspected.
// Errors reported will be indicated as junit test failure and will cause job runs to fail.
// storageDir is for gathering data only, not for writing in this stage. To store data, use WriteContentToStorage
// storageDir is for gathering data only, not for writing in this stage. To store data, use WriteContentToStorage.
// The returned JUnitTestCases should be stable in different runs as the test pass/fail rate
// is calculated in an aggregated to improve CI signal. I.e., if a JUnitTestCase shows up in
// some run, then it should stay in other runs as well.
// See https://docs.ci.openshift.org/docs/release-oversight/improving-ci-signal/#passfail-rates-for-running-jobs-10-times
// for details.
// In addition, we should avoid renaming a JUnitTestCase, e.g., by not using
// any specific numbers that could be changed over time.
// See https://github.com/openshift-eng/ci-test-mapping?tab=readme-ov-file#renaming-tests
// for details.
CollectData(ctx context.Context, storageDir string, beginning, end time.Time) (monitorapi.Intervals, []*junitapi.JUnitTestCase, error)

// ConstructComputedIntervals is called after all InvariantTests have produced raw Intervals.
Expand All @@ -64,6 +73,15 @@ type MonitorTest interface {
// EvaluateTestsFromConstructedIntervals is called after all Intervals are known and can produce
// junit tests for reporting purposes.
// Errors reported will be indicated as junit test failure and will cause job runs to fail.
// The returned JUnitTestCases should be stable in different runs as the test pass/fail rate
// is calculated in an aggregated to improve CI signal. I.e., if a JUnitTestCase shows up in
// some run, then it should stay in other runs as well.
// See https://docs.ci.openshift.org/docs/release-oversight/improving-ci-signal/#passfail-rates-for-running-jobs-10-times
// for details.
// In addition, we should avoid renaming a JUnitTestCase, e.g., by not using
// any specific numbers that could be changed over time.
// See https://github.com/openshift-eng/ci-test-mapping?tab=readme-ov-file#renaming-tests
// for details.
EvaluateTestsFromConstructedIntervals(ctx context.Context, finalIntervals monitorapi.Intervals) ([]*junitapi.JUnitTestCase, error)

// WriteContentToStorage writes content to the storage directory that is collected by openshift CI.
Expand Down