runtests requires phantomjs running and app being served

[ngm]

Is it aplet's responsibility to make sure they're up and running?

[ngm]

In some ways it's a set of preconditions that need running before we run the tests.

It's a bit like e.g. before_script in travis.

For running these tests, I don't want to end up recreating some kind of CI task runner. Plenty of things already exist for that.

At the moment it feels I'm heading that way a bit. I started with a bash script and then migrated it to python when the product line/feature model management of it was becoming too complex for bash. But I don't want to reinvent the wheel.

Perhaps things need to be inverted, so that a script (e.g. bash, or a travis script) can call into aplet? I'm currently finding being able to run aplet commands for the various steps much simpler than it was with the various bash scripts, however, I think it could be at the expense of some generality.

I think runtests should run under the assumption that each product that it is testing has already been set up and correctly configured and that all test dependencies are running. It should just get given a command to run and an application to run that command against. The test phase could mostly happen without aplet, except what it brings to the table is figuring out which subset of tests to run based on the feature model and the product configuration. So potentially that function should be exposed from aplet for the framework running the tests to use, which would of course be dependent on what flags the framework takes.

Then, you could also argue, if something above aplet is aware of the fact that this is a product line, then that's going to have to do product line work, and we're back where we started.

Maybe it's something like travis' matrices, that expands out for all the products, and aplet is called per product.

Dunno. Think for my own purposes right now I'll just stick with coding it in aplet, while trying to keep it reasonably configurable. And in future think about how to make it more general purpose.

[ngm]

Added in to the code for now, needs more work though as discussed above.