kola: don't run reprovisioning tests if only a subset is requested

[jlebon]

Some upstreams just want to run a subset of tests. #118 split out the
reprovisioning tests, but it then changed the semantics for users that
only want to run a specific pattern. Now, those CIs are always running
reprovisioning tests even though they're not interested.

[dustymabe]

I'm a bit concerned about the number of knobs we have in here already. I feel like we're re-implementing a lot of things that are just baked into kola directly.

Is this causing a failure or just undesirable? Before #118 were those upstreams running the reprovision tests?

[dustymabe]

another way to say this: I think the direction we should go is to have a mode of kola() that is just a single kola run (bareKola: True) without the nuance of what's going on inside this function. The caller will have to provide things like extraArgs: '--tag '!reprovision' then. but they don't have to fight with the function either.

I did something similar over in 4f1e7f0?diff=split&w=1

so my strawman here is that if a user doesn't want basic scenarios or they want reprovision tests separated out they pass bareKola or something and they have more control rather than adding a new knob each time we do something nuanced.

[jlebon]

Yeah, there's a lot here that should be lowered into kola proper. I think this all would be simpler if kola testiso and kola run-upgrade were merged into kola run as regular test cases.

Anyway, I've reworked this now so it switches on extraArgs instead of adding a new parameter. This retains to a much closer degree the semantics we had before #118.

[jlebon]

another way to say this: I think the direction we should go is to have a mode of kola() that is just a single kola run (bareKola: True) without the nuance of what's going on inside this function.

Not sure. I think that'd be making this function even more branch-y. :) I think having a default set of tests to run is good, and most upstreams do rely on that. We just need to give the flexibility when they need to slightly tweak that set. We don't want any tweaking to require having to re-encode everything else this function does they do still want.

[dustymabe]

Since I've been in this code for the past day I'm trying to think about how this meshes with the re-work I've been doing in #120. I just added a bit on top.. What do you think about this logic:

https://github.com/dustymabe/coreos-ci-lib/blob/dusty-example/vars/kola.groovy#L78-L91

[jlebon]

Since I've been in this code for the past day I'm trying to think about how this meshes with the re-work I've been doing in #120. I just added a bit on top.. What do you think about this logic:

dustymabe/coreos-ci-lib@dusty-example/vars/kola.groovy#L78-L91

Yeah, I think that sounds good to me!

[dustymabe]

Since I've been in this code for the past day I'm trying to think about how this meshes with the re-work I've been doing in #120. I just added a bit on top.. What do you think about this logic:
dustymabe/coreos-ci-lib@dusty-example/vars/kola.groovy#L78-L91

Yeah, I think that sounds good to me!

this is now part of one of the commits in https://github.com/coreos/coreos-ci-lib/pull/120/files

[jlebon]

Closing in favour of #120.