BDN should not delete temporary build directories for failed builds

[kg]

Right now the documentation for troubleshooting build failures has this as a first, required step:

Configure BenchmarkDotNet to keep auto-generated benchmark files (they are being removed after benchmark is executed by default).

And all the methods of doing this appear to require me to locate the benchmark in question and then modify its source code intrusively, which will get blown away if I revert my git checkout etc. Anyone else troubleshooting will also need to do this. Maybe I don't understand BDN well enough and there's some obvious way to feed --keepfiles through without modifying the code that calls BenchmarkSwitcher, like using a magic environment variable?

Anyway, I think the default should be to keep the directory around, or at a minimum there should be an environment var you can set to produce this. In the first place, it says 'removed after benchmark is executed', but that isn't an accurate characterization of this behavior, since the benchmark never executed (due to the build failing). It took me a while to even figure out that my attempts at getting a binlog out of this were working, and that BDN was just deleting the logs. I can imagine some disk space usage issues if someone runs thousands of failed benchmark builds (and accumulates temporary output directories), but that seems like a corner case with multiple easy solutions.

[adamsitnik]

@kg good point! I've sent #1567 to address the problem.

[kg]

It looks like this may've regressed, or BDN isn't sensing a failed build in my current scenario. I'm following the instructions from the dotnet/performance wiki to run benchmarks using a custom corerun, like this:

E:\Documents\Projects\performance\src\benchmarks\micro>E:\Documents\Projects\runtime\dotnet.cmd run -c Release -f net10.0 --artifacts "E:\Documents\Projects\benchmark-results\runtime" --coreRun "E:\Documents\Projects\runtime\artifacts\bin\testhost\net10.0-windows-Release-x64\shared\Microsoft.NETCore.App\10.0.0\CoreRun.exe" --filter "*CtorGivenSize<String>*"

But the restore and publish fail with nonzero exit codes, and BDN doesn't seem to notice. As a result it doesn't keep the temporary build directories around so I can't manually reproduce and investigate the build failure since the folder(s) in question are gone:

E:\Documents\Projects\performance\src\benchmarks\micro>E:\Documents\Projects\runtime\dotnet.cmd run -c Release -f net10.0 --artifacts "E:\Documents\Projects\benchmark-results\runtime" --coreRun "E:\Documents\Projects\runtime\artifacts\bin\testhost\net10.0-windows-Release-x64\shared\Microsoft.NETCore.App\10.0.0\CoreRun.exe" --filter "*CtorGivenSize<String>*"
E:\Documents\Projects\runtime\.dotnet
E:\Documents\Projects\runtime\.dotnet\sdk\10.0.100-alpha.1.24610.7\Sdks\Microsoft.NET.Sdk\targets\Microsoft.NET.EolTargetFrameworks.targets(32,5): warning NETSDK1138: The target framework 'net7.0' is out of support and will not receive security updates in the future. Please refer to https://aka.ms/dotnet-core-support for more information about the support policy.
// Validating benchmarks:
// ***** BenchmarkRunner: Start   *****
// ***** Found 8 benchmark(s) in total *****
// ***** Building 1 exe(s) in Parallel: Start   *****
// start dotnet  restore /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 /p:Deterministic=true /p:Optimize=true /p:IntermediateOutputPath="E:\Documents\Projects\performance\artifacts\bin\MicroBenchmarks\Release\net10.0\d84672c5-5a3d-4073-8e51-757c5b0f4da4\obj\Release\net10.0/" /p:OutDir="E:\Documents\Projects\performance\artifacts\bin\MicroBenchmarks\Release\net10.0\d84672c5-5a3d-4073-8e51-757c5b0f4da4\bin\Release\net10.0\publish/" /p:OutputPath="E:\Documents\Projects\performance\artifacts\bin\MicroBenchmarks\Release\net10.0\d84672c5-5a3d-4073-8e51-757c5b0f4da4\bin\Release\net10.0\publish/" in E:\Documents\Projects\performance\artifacts\bin\MicroBenchmarks\Release\net10.0\d84672c5-5a3d-4073-8e51-757c5b0f4da4
// command took 0.02 sec and exited with -2147450735
// start dotnet  publish -c Release --no-restore /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 /p:Deterministic=true /p:Optimize=true /p:IntermediateOutputPath="E:\Documents\Projects\performance\artifacts\bin\MicroBenchmarks\Release\net10.0\d84672c5-5a3d-4073-8e51-757c5b0f4da4\obj\Release\net10.0/" /p:OutDir="E:\Documents\Projects\performance\artifacts\bin\MicroBenchmarks\Release\net10.0\d84672c5-5a3d-4073-8e51-757c5b0f4da4\bin\Release\net10.0\publish/" /p:OutputPath="E:\Documents\Projects\performance\artifacts\bin\MicroBenchmarks\Release\net10.0\d84672c5-5a3d-4073-8e51-757c5b0f4da4\bin\Release\net10.0\publish/" --output "E:\Documents\Projects\performance\artifacts\bin\MicroBenchmarks\Release\net10.0\d84672c5-5a3d-4073-8e51-757c5b0f4da4\bin\Release\net10.0\publish/" in E:\Documents\Projects\performance\artifacts\bin\MicroBenchmarks\Release\net10.0\d84672c5-5a3d-4073-8e51-757c5b0f4da4
// command took 0.01 sec and exited with -2147450735
// ***** Done, took 00:00:00 (0.29 sec)   *****
// Found 8 benchmarks:
//   CtorGivenSize<String>.Array: Job-WWTTVM(PowerPlanMode=00000000-0000-0000-0000-000000000000, Toolchain=CoreRun, IterationTime=250ms, MaxIterationCount=20, MinIterationCount=15, WarmupCount=1) [Size=512]
//   CtorGivenSize<String>.List: Job-WWTTVM(PowerPlanMode=00000000-0000-0000-0000-000000000000, Toolchain=CoreRun, IterationTime=250ms, MaxIterationCount=20, MinIterationCount=15, WarmupCount=1) [Size=512]
//   CtorGivenSize<String>.HashSet: Job-WWTTVM(PowerPlanMode=00000000-0000-0000-0000-000000000000, Toolchain=CoreRun, IterationTime=250ms, MaxIterationCount=20, MinIterationCount=15, WarmupCount=1) [Size=512]
//   CtorGivenSize<String>.Dictionary: Job-WWTTVM(PowerPlanMode=00000000-0000-0000-0000-000000000000, Toolchain=CoreRun, IterationTime=250ms, MaxIterationCount=20, MinIterationCount=15, WarmupCount=1) [Size=512]
//   CtorGivenSize<String>.Queue: Job-WWTTVM(PowerPlanMode=00000000-0000-0000-0000-000000000000, Toolchain=CoreRun, IterationTime=250ms, MaxIterationCount=20, MinIterationCount=15, WarmupCount=1) [Size=512]
//   CtorGivenSize<String>.Stack: Job-WWTTVM(PowerPlanMode=00000000-0000-0000-0000-000000000000, Toolchain=CoreRun, IterationTime=250ms, MaxIterationCount=20, MinIterationCount=15, WarmupCount=1) [Size=512]
//   CtorGivenSize<String>.SortedList: Job-WWTTVM(PowerPlanMode=00000000-0000-0000-0000-000000000000, Toolchain=CoreRun, IterationTime=250ms, MaxIterationCount=20, MinIterationCount=15, WarmupCount=1) [Size=512]
//   CtorGivenSize<String>.ConcurrentDictionary: Job-WWTTVM(PowerPlanMode=00000000-0000-0000-0000-000000000000, Toolchain=CoreRun, IterationTime=250ms, MaxIterationCount=20, MinIterationCount=15, WarmupCount=1) [Size=512]

// **************************
// Benchmark: CtorGivenSize<String>.Array: Job-WWTTVM(PowerPlanMode=00000000-0000-0000-0000-000000000000, Toolchain=CoreRun, IterationTime=250ms, MaxIterationCount=20, MinIterationCount=15, WarmupCount=1) [Size=512]
// *** Execute ***
// Launch: 1 / 1
Did not find E:\Documents\Projects\performance\artifacts\bin\MicroBenchmarks\Release\net10.0\d84672c5-5a3d-4073-8e51-757c5b0f4da4\bin\Release\net10.0\publish\d84672c5-5a3d-4073-8e51-757c5b0f4da4.dll, but the folder contained:
d84672c5-5a3d-4073-8e51-757c5b0f4da4.dll.config
Executable E:\Documents\Projects\performance\artifacts\bin\MicroBenchmarks\Release\net10.0\d84672c5-5a3d-4073-8e51-757c5b0f4da4\bin\Release\net10.0\publish\d84672c5-5a3d-4073-8e51-757c5b0f4da4.dll not found
ExitCode != 0 and no results reported
No Workload Results were obtained from the run.

After the run the folder in question doesn't exist. Is it expected that restores or publishes can fail with a nonzero exit code but benchmark runs will continue and succeed despite that? It seems to me that a failed restore/publish should probably halt the benchmark run (I had to set this tiny filter to even see that the restore/publish were failing, since they were getting scrolled out of the buffer by all the failed benchmark runs.)

[timcassell]

@kg You can try adding --keepFiles argument. That bug should be fixed by #2535.

[kg]

Thanks, --keepfiles got me far enough to repro the actual problem:

The command could not be loaded, possibly because:
  * You intended to execute a .NET application:
      The application 'restore' does not exist.
  * You intended to execute a .NET SDK command:
      A compatible .NET SDK was not found.

Requested SDK version: 10.0.100-alpha.1.24573.1
global.json file: E:\Documents\Projects\performance\global.json

Installed SDKs:
9.0.101 [C:\Program Files\dotnet\sdk]

Install the [10.0.100-alpha.1.24573.1] .NET SDK or update [E:\Documents\Projects\performance\global.json] to match an installed SDK.

Learn about SDK resolution:
https://aka.ms/dotnet/sdk-not-found

I've tried using DOTNET_ROOT to point it to a 10 SDK but it seems like that environment var doesn't work anymore? I haven't been able to find documentation on what to do here. I'm running BDN itself using the dotnet from my checkout, which is a 10 SDK, but BDN is using the dotnet.exe from my ProgramFiles even if I override PATH to put reporoot\dotnet.cmd first in the search order.

Anyway, thanks, it looks like that PR will fix it :)

[adamsitnik]

@kg could you please try specifying the path to dotnet (executable) via --cli $exactPath command line argument? thanks!