Make --strict imply --strict-config / xfail_strict?

[The-Compiler]

With pytest 4.5.0 (May 2019), we introduced --strict-markers as a replacement for --strict, and said:

The existing --strict option has the same behavior currently, but can be augmented in the future for additional checks.

With #7286, now we also have --strict-config. Yet, --strict is still an alias for --strict-markers.

Given that pytest 6 is allowed to do breaking changes, wouldn't it make sense for --strict to imply both --strict-markers and --strict-config now?

(Though, on a second thought: Do we actually need --strict-markers in pytest 6 still, given the warning system? Probably still useful as a convenient way to turn those warnings into errors?)

This would also make pytest more in line with other tools with a --strict flag - mypy and tsc (the typescript compiler) come to mind, where --strict implies various different strict flags.

cc @gnikonorov (who added --strict-config), @webknjaz and @DahlitzFlorian (some existing discussion in #7233), and @nicoddemus (#5023) - what do you think?

[webknjaz]

I think it's a good idea to have some --strict-all-the-things that would imply enabling all strictness-related settings. What I'm not sure about is how it should be called. I'm okay with just --strict but Bruno mentioned earlier that it used to mean just "strict-markers" for a while so such semantics change may be confusing. OTOH personally I can live with that and I'm not really concerned about this name. I think it should be decided by the folks who feel differently.

[webknjaz]

The only thing I want to add is that I'd probably expect it to also be a config value rather than just a CLI option.

[Zac-HD]

I like the idea of --strict or strict = True implying all the strictness options.

Whether we still need it with the warnings system or not, I think we should keep it until pytest 7 rather than scrambling to make such a large change now.

[DahlitzFlorian]

I really like the idea of using --strict to activate all --strict* options. However, I would suggest adding a note to the documentation that --strict will have that behavior in version x.x and people should be aware of it. Of course, we already added a note that it might happen, but an explicit note that it is going to happen in version x.x is highly preferred.

IMHO we can discuss whether we need to wait until version 7 or can include it in some minor version of pytest 6.

[symonk]

Should we consider giving the user the option?

--strict=all | --strict=marker | --strict=config

Im just wondering if we add future 'strictness' but have then coupled the user into all or none situations.

[The-Compiler]

@symonk We already have --strict-markers and --strict-config, I don't think we should have a second way to do the same thing.

[DahlitzFlorian]

Totally agree with @The-Compiler. In fact, --strict should be a shortcut for using all --strict* options.

[gnikonorov]

No objections from me at all. --strict should imply all strict flags IMO, as @DahlitzFlorian mentions above.

[The-Compiler]

I opened a PR: #7529

[graingert]

Given that pytest 6 is allowed to do breaking changes

Can we flip the default here so markers default to strict in v6?

What about xfail_strict

[The-Compiler]

Can we flip the default here so markers default to strict in v6?

We already have a warning for that since 4.5.0, so it should be quite clear what's going on even without --strict-markers. -1 on making that the default now, that seems a bit late.

What about xfail_strict

That one is much more controversial and I'm not sure if starting the discussion again will produce a different outcome. See #1299 and #814.