Skip to content

consider linting test/run-pass/*.rs to ensure all have assert/assert_eq #23113

Closed
Closed
consider linting test/run-pass/*.rs to ensure all have assert/assert_eq#23113
Labels
A-testsuiteArea: The testsuite used to check the correctness of rustcC-enhancementCategory: An issue proposing an enhancement or a PR with one.

Description

@pnkfelix
pnkfelix
opened on Mar 6, 2015

spawned off of the story in #23112

We should consider making the compiletest system reject run-pass tests that contain no calls to assert! nor assert_eq!

(Better still would be to also reject if no such call is reached during an execution of the test, but that would probably require we add testing-specific assertion macros.)

We could add a comment flag to allow tests to opt out of this requirement (e.g. if they are regression tests that are indeed just checking that we do not segfault and do not have a clear assertion to include).

Activity

  1. pnkfelix commented on Mar 6, 2015

    @pnkfelix
    pnkfelix
    on Mar 6, 2015
    ContributorAuthor

    (of course, one cannot stop people from writing weak tests via such requirements, since they may just put in silly assertions. but then again, I think a silly assertion will stick out more during the review process than the absence of any assertion will.)

  2. pnkfelix
    added
    A-testsuiteArea: The testsuite used to check the correctness of rustc
    on Mar 6, 2015

  3. steveklabnik commented on May 24, 2016

    @steveklabnik
    steveklabnik
    on May 24, 2016
    Contributor

    Triage: no changes.

  4. tommyip commented on May 16, 2017

    @tommyip
    tommyip
    on May 16, 2017
    Contributor

    I want to take a stab at doing this. Should I use regex to test the presence of assertions or is there a better way?

  5. Mark-Simulacrum commented on May 16, 2017

    @Mark-Simulacrum
    Mark-Simulacrum
    on May 16, 2017
    Member

    I would either add this to compiletest (src/tools/compiletest) or make it a tidy check (src/tools/tidy). If in compile test, I would probably look for the code that loads in tests. I'm somewhat preferring using tidy for this, but would be fine with either. I'm also not entirely sure we even want this -- tests without assert_eq/assert seem fine to me, if for example they're testing for a lack of segfault or similar. We also don't have a compile-pass test suite so some run-pass tests are maybe just compile-pass tests (again, without assert_eq/assert). So I'm not sure that this is entirely the best issue to work on; if you're interested in this area I would look at one of these: #40713 or #33581. I can probably provide some details on thoughts on either issue, just leave a comment if interested.

    If you want to implement this, feel free though. I think that we'll run into a lot of false positives, but I could be wrong. Let me know if there's anything we can help you with!

  6. Mark-Simulacrum
    added
    C-enhancementCategory: An issue proposing an enhancement or a PR with one.
    on Jul 22, 2017

  7. steveklabnik commented on Mar 11, 2019

    @steveklabnik
    steveklabnik
    on Mar 11, 2019
    Contributor

    Triage: I don't think anyone took mark up on his offer, as far as I know.

  8. Mark-Simulacrum commented on Aug 31, 2019

    @Mark-Simulacrum
    Mark-Simulacrum
    on Aug 31, 2019
    Member

    I'm going to close this as I'm still not sure that this is entirely a good idea, in particular since we have ~1597 such files today.

    Generated the list (https://gist.github.com/Mark-Simulacrum/cb65bb73dcc43914f2aae416370ec83b) with:

    rg --multiline --multiline-dotall 'run-pass.*assert' src/test/ui -l | sort > /tmp/with-assert
rg --multiline --multiline-dotall 'run-pass' src/test/ui -l | sort > /tmp/all
diff -U0 /tmp/all /tmp/with-assert | rg -v '@@'
  9. Mark-Simulacrum
    closed this as completedon Aug 31, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    A-testsuiteArea: The testsuite used to check the correctness of rustcC-enhancementCategory: An issue proposing an enhancement or a PR with one.

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions