Add go.benchFlags to extract benchmark flags from testFlags

[cristaloleg]

Is your feature request related to a problem? Please describe.
Go 1.17 provides a new flag -shuffle which changes order of the tests to a random one.
But this flags is bothersome for benchmarks. In other words "I don't care about order of benchmarks, I want the results".
Also using this flag complicates benchmark comparison (due to order, hah!).
Same problem with a -race flag, not so useful in benchmarks for most of the cases.

Describe the solution you'd like
Have go.benchFlags parameter where user can specify benchmark only flags.

Describe alternatives you've considered
Disabling -shuffle or -race flag for tests or just use terminal for benchmarks.
1st solution disable good Go test features.
2nd is just less simpler than click "run benchmark" (or so).

Additional context
Nothing to add.

[hyangah]

This proposes a new go.benchFlags:

• go.buildFlags -> apply to build. apply to test if go.testFlags is not set. apply to benchmark if go.benchFlags is not set.
• go.testFlags -> apply to test. apply to bench if go.benchFlags is not set.
• go.benchFlags -> apply to bench.

Note: when a Benchmark function runs as part of test runs, only go.buildFlags or go.testFlags are used.

Agree that disabling -shuffle/-race unconditionally for tests or benchmarks are not ideal. There are cases where these flags make sense in tests and benchmarks (e.g. compare performance with/without -race flag).

I think it's reasonable. Does anyone have an objection or a different approach we should consider?

[cristaloleg]

No objections, it's even better, after your edits users will be able to configure build flags too. Great! Probably title can be also updated now, 'cause feature request covers other cases.

Edit: didn't know that buildFlags already exists 👀

[cristaloleg]

Hey, can I send a patch for it?

[firelizzard18]

It's increasingly looking like the current set of options (buildFlags and friends) are a bad fit for the current state of tooling, i.e. gopls. golang/go#65757 discusses an alternative.