The Operation will be canceled. The next steps may not contain expected logs.

[tarekgh]

Build

https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_build/results?buildId=696281

Build leg reported

Build / browser-wasm linux Release LibraryTests / Send to Helix

Pull Request

dotnet/runtime#103009

Known issue core information

Fill out the known issue JSON section by following the step by step documentation on how to create a known issue

 {
    "ErrorMessage" : "The Operation will be canceled. The next steps may not contain expected logs.",
    "BuildRetry": false,
    "ErrorPattern": "",
    "ExcludeConsoleLog": false
 }

@dotnet/dnceng

Release Note Category

• Feature changes/additions
• Bug fixes
• Internal Infrastructure Improvements

Release Note Description

Additional information about the issue reported

No response

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng-public/public/_build/results?buildId=696281
Error message validated: [The Operation will be canceled. The next steps may not contain expected logs.]
Result validation: ✅ Known issue matched with the provided build.
Validation performed at: 6/4/2024 8:08:48 PM UTC

Report

Build	Definition	Step Name	Console log	Pull Request
2677593	dotnet-winforms	🛡 Guardian: TSAUpload	Log
2677513	dotnet-winforms	🛡 Guardian: CredScan (Binary)	Log
2677422	dotnet-runtime	Send Utf8JsonWriterFuzzer to OneFuzz	Log
2677448	dotnet-runtime	Checkout dotnet-runtime@main to s	Log
2677442	dotnet-runtime	Initialize containers	Log
2677441	dotnet-runtime	Pre-job: Download secrets: HelixProdKV	Log
2677440	dotnet-runtime	Component Detection (auto-injected by policy)	Log
2677430	dotnet-runtime	Agent Pool Validation	Log
1001409	dotnet/runtime	Send to Helix	Log	dotnet/runtime#113313
1002176	dotnet/runtime	AOT compile CoreCLR tests	Log
2677326	dotnet-dotnet	Build (centOSStreamContainer)	Log
2677228	dotnet-runtime	Send job to Helix (Unix)	Log
1001857	dotnet/aspire	Build	Log
2677030	dotnet-dotnet	Build	Log
1001715	dotnet/aspire	Run helix tests	Log
2677011	dotnet-runtime	Run performance job script	Log
1001653	dotnet/aspire	Run helix tests	Log
2676998	dotnet-dotnet	Secure Supply Chain Analysis (auto-injected by policy)	Log
1001556	dotnet/aspire	Run non-helix tests	Log
1001479	dotnet/runtime	AOT compile CoreCLR tests	Log
2676882	dotnet-dotnet	Build (centOSStreamContainer)	Log
2674217	dotnet-runtime	Build product	Log
2676873	dotnet-dotnet	Validate Hosted Pool Information (1ES PT)	Log
2676655	dotnet-dotnet	Build	Log
1001158	dotnet/runtime	Publish	Log	dotnet/runtime#114110
2676791	dotnet-runtime	Publish AOT Mono Artifacts	Log
2676756	dotnet-runtime	Download Build Artifact LinuxMonoAOTx64	Log
1001264	dotnet/roslyn	Run Integration Tests	Log	dotnet/roslyn#76906
1001274	dotnet/aspire	Build	Log
1001268	dotnet/aspire	Build	Log
1001204	dotnet/runtime	Publish Build Assets	Log	dotnet/runtime#113916
2676700	dotnet-runtime	Send job to Helix (Unix)	Log
1001126	dotnet/runtime	Publish Build Assets	Log	dotnet/runtime#114105
2676724	dotnet-dotnet-docker	Validate Hosted Pool Information (1ES PT)	Log
1001050	dotnet/runtime	AOT compile CoreCLR tests	Log	dotnet/runtime#114080
2676654	dotnet-dotnet	Checkout dotnet-dotnet@sign-and-harden-pkgs to s	Log
1000655	dotnet/runtime	Send to Helix	Log	dotnet/runtime#113313
1000894	dotnet/roslyn	Test	Log	dotnet/roslyn#77942
2676570	dotnet-performance	Send job to Helix (Windows)	Log
2676502	dotnet-runtime	Send Utf8JsonWriterFuzzer to OneFuzz	Log
2676525	dotnet-performance	AzureCLI	Log
2676496	dotnet-runtime	Send job to Helix (Unix)	Log
2676492	dotnet-roslyn	Build	Log
1000700	dotnet/aspire	Build	Log
1000617	dotnet/runtime	AOT compile CoreCLR tests	Log
1000608	dotnet/runtime	AOT compile CoreCLR tests	Log	dotnet/runtime#114080
1000525	dotnet/aspire	Run helix tests	Log
2676184	dotnet-dotnet	Build (centOSStreamContainer)	Log
2676145	dotnet-cli-lab	Build and Publish	Log
2676135	dotnet-cli-lab	🛡 Guardian: ARMory	Log
2676133	dotnet-cli-lab	Build	Log
2676119	dotnet-cli-lab	Build and Publish	Log
2676107	dotnet-cli-lab	🛡 Guardian: Check CodeSign Validation Results (1ES PT)	Log
2676079	dotnet-dotnet	Validate Signing	Log
983276	dotnet/runtime	Send to Helix	Log	dotnet/runtime#113313
2676058	dotnet-dotnet	Download Blob Artifacts	Log
999931	dotnet/runtime	Publish Build Assets	Log	dotnet/runtime#114075
999955	dotnet/runtime	AOT compile CoreCLR tests	Log
999795	dotnet/runtime	Publish	Log	dotnet/runtime#114076
999918	dotnet/runtime	AOT compile CoreCLR tests	Log	dotnet/runtime#114080
999940	dotnet/runtime	Build	Log	dotnet/runtime#114016
999884	dotnet/runtime	AOT compile CoreCLR tests	Log	dotnet/runtime#110818
999770	dotnet/runtime	Publish	Log	dotnet/runtime#113988
999831	dotnet/runtime	Publish Build Assets	Log	dotnet/runtime#114077
999739	dotnet/aspire	Run helix tests	Log
999371	dotnet/roslyn	Test	Log
999343	dotnet/sdk	🟣 Run TestBuild Tests	Log	dotnet/sdk#48023
999127	dotnet/runtime	Send to Helix	Log
994633	dotnet/runtime	Send to Helix	Log	dotnet/runtime#113740
999322	dotnet/sdk	🟣 Run TestBuild Tests	Log	dotnet/sdk#48019
999254	dotnet/performance	Send job to Helix (Unix)	Log	dotnet/performance#4805
999142	dotnet/runtime	AOT compile CoreCLR tests	Log	dotnet/runtime#110818
999186	dotnet/aspire	Run helix tests	Log
999079	dotnet/runtime	AOT compile CoreCLR tests	Log
999066	dotnet/runtime	AOT compile CoreCLR tests	Log	dotnet/runtime#113880
999067	dotnet/runtime	AOT compile CoreCLR tests	Log	dotnet/runtime#113880
999012	dotnet/aspire	Run helix tests	Log
998914	dotnet/performance	Send job to Helix (Windows)	Log	dotnet/performance#4802
998671	dotnet/runtime	AOT compile CoreCLR tests	Log	dotnet/runtime#114052
998638	dotnet/runtime	AOT compile CoreCLR tests	Log
2675166	dotnet-runtime	Send to Helix	Log
998501	dotnet/runtime	AOT compile CoreCLR tests	Log
2674745	dotnet-dotnet	🛡 Guardian: AntiMalware Scanner (Binary)	Log
998378	dotnet/runtime	Send tests to Helix (Windows)	Log	dotnet/runtime#113976
998361	dotnet/runtime	Send tests to Helix (Windows)	Log	dotnet/runtime#114040
998357	dotnet/runtime	Send tests to Helix (Windows)	Log	dotnet/runtime#113407
998352	dotnet/runtime	Send tests to Helix (Windows)	Log
998350	dotnet/runtime	AOT compile CoreCLR tests	Log
2674761	dotnet-maui	🛡 Guardian: CredScan (Source)	Log
2674685	dotnet-dotnet	Build	Log
998143	dotnet/runtime	AOT compile CoreCLR tests	Log	dotnet/runtime#110818
998124	dotnet/runtime	AOT compile CoreCLR tests	Log
2674490	dotnet-runtime	Run performance job script	Log
2674432	dotnet-runtime	Send tests to Helix (Unix)	Log	#48810
2674239	dotnet-dotnet	🛡 Guardian: ESLint (TypeScript)	Log
2674242	dotnet-cli-lab	Build and Publish	Log
2674235	dotnet-cli-lab	🛡 Component Governance	Log
2674223	dotnet-dotnet	Build (azurelinuxArm64MuslCrossContainer)	Log
2674141	dotnet-dotnet	Publish Test Results	Log
2674142	dotnet-dotnet	🛡 Guardian: AntiMalware Scanner (Binary)	Log
Displaying 100 of 985 results

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
34	180	985

[dkurepa]

These appear to be timeout cancellations, and most of them are happening on the same runtime leg that uses OSX.1200.ARM64.Open, looking at grafana, they do indeed appear to be quite high. Will check with the team on what we can do about this

[dkurepa]

For pipelines experiencing timeouts on osx.1200.arm64.open we recommend using osx.14.arm64.open, as this queue has a lot more machines

[dkurepa]

There is also a number of Publish Using Darc timeouts, these are in 99% of cases AzDo Api issues. During the Publish Using Darc job, we queue up a Maestro Promotion pipeline, that downloads the artifacts from the build calling it, and publishes them where they need to go. While publishing is happening, the build pipeline keeps busy waiting for the promotion pipeline to be done, checking the status of it every 60 seconds. In cases where this times out, the build most likely did get promoted, the AzDo API just kept telling us the wrong thing

[ivanpovazan]

@dkurepa for context, the same timeouts are also caught with this tracking #1883
where I asked about potentially bumping the queues to osx.13.amd/arm64, would that suffice?
(bumping to osx.14 is not an option right now as we still want testing on osx.13 which is the minimum supported OS version)