deprecate runtest in favor of testopts (was: Add pretestopts and testops)

[wpoely86]

For #447 (EB 4.0)

• We should add a pretestopts just like prebuildopts
• Rename runtests to testopts

So we have %(pretestopts) make %(testopts) in the configuremake block

[boegel]

@wpoely86 Adding support for pretestopts and testopts should be done earlier, and then runtest should be deprecated.

If this is done early enough, we can drop support for runtest in EB 4.0 in favor of (pre)testopts.

The right place for this issue would have been in framework btw (although it does involve changing all uses of runtest to testopts in easyblocks/easyconfigs.

[boegel]

Support for (pre)testopts was added in #2793

@wpoely86 Do we still want to go ahead and deprecate runtest in favor of testopts for EasyBuild 4.0?

cc @akesandgren

[wpoely86]

It would make sense?

[akesandgren]

Sounds like a plan.

[boegel]

I guess it would, it's just a bit more work on top of other things for EasyBuild v4.0, which involves deprecating runtest in framework (in a way that doesn't break stuff that is still using it), and updating all easyblocks/easyconfigs that use it...

[boegel]

I had a quick look at this, and deprecating runtest is less trivial than it may seem at first. The problem is that it means different things in different contexts...

• 1. In the default EasyBlock.test_step implementation, it can be used to specify the full test command to run (for example runtest = "./run_tests.sh"). I'm not sure we should just use testopts in this case, since the name doesn't fit. Do we want to introduce a testcmd to replace this runtest interpretation?
• 2. In ConfigureMake (and derivatives), you can specify the make target that should be used to run the tests (for example runtest = 'check' is equivalent with running make check); in this case we can just switch to testopts.
• 3. runtest is expected to have a boolean value in various custom easyblocks, and is often set to True as default , indicating that the tests for that software to be run while leaving the door open to skip the tests using runtest = False. Also here testopts is not a good replacement. I guess here we can get away with recommending to use skipsteps = ['test'] as a replacement for runtest = False, or do we introduce a dedicated parameter for this interpretation too?

@akesandgren, @wpoely86: feedback welcome!

[wpoely86]

I agree on only using testopts for point ii.

[boegel]

@wpoely86 OK, but what do we do with use cases i) and iii) for which runtest is used now as well?

[wpoely86]

I would keep it. It's a different thing and and not the thing I had in mind with testopts. This should go with buildopts etc. Point (i) is completely general and point (iii) is also independent (although a other name wouldn't have been a bad idea).

[boegel]

Getting rid of runtest won't be happening for EasyBuild v4.0.

We can limit the use cases of it over 4.x release, and perhaps make a more drastic change in EasyBuild v5.0...