consider linting test/run-pass/*.rs to ensure all have assert/assert_eq

[pnkfelix]

spawned off of the story in #23112

We should consider making the compiletest system reject run-pass tests that contain no calls to assert! nor assert_eq!

(Better still would be to also reject if no such call is reached during an execution of the test, but that would probably require we add testing-specific assertion macros.)

We could add a comment flag to allow tests to opt out of this requirement (e.g. if they are regression tests that are indeed just checking that we do not segfault and do not have a clear assertion to include).

[pnkfelix]

(of course, one cannot stop people from writing weak tests via such requirements, since they may just put in silly assertions. but then again, I think a silly assertion will stick out more during the review process than the absence of any assertion will.)

[steveklabnik]

Triage: no changes.

[tommyip]

I want to take a stab at doing this. Should I use regex to test the presence of assertions or is there a better way?

[Mark-Simulacrum]

I would either add this to compiletest (src/tools/compiletest) or make it a tidy check (src/tools/tidy). If in compile test, I would probably look for the code that loads in tests. I'm somewhat preferring using tidy for this, but would be fine with either. I'm also not entirely sure we even want this -- tests without assert_eq/assert seem fine to me, if for example they're testing for a lack of segfault or similar. We also don't have a compile-pass test suite so some run-pass tests are maybe just compile-pass tests (again, without assert_eq/assert). So I'm not sure that this is entirely the best issue to work on; if you're interested in this area I would look at one of these: #40713 or #33581. I can probably provide some details on thoughts on either issue, just leave a comment if interested.

If you want to implement this, feel free though. I think that we'll run into a lot of false positives, but I could be wrong. Let me know if there's anything we can help you with!

[steveklabnik]

Triage: I don't think anyone took mark up on his offer, as far as I know.

[Mark-Simulacrum]

I'm going to close this as I'm still not sure that this is entirely a good idea, in particular since we have ~1597 such files today.

Generated the list (https://gist.github.com/Mark-Simulacrum/cb65bb73dcc43914f2aae416370ec83b) with:

rg --multiline --multiline-dotall 'run-pass.*assert' src/test/ui -l | sort > /tmp/with-assert
rg --multiline --multiline-dotall 'run-pass' src/test/ui -l | sort > /tmp/all
diff -U0 /tmp/all /tmp/with-assert | rg -v '@@'