Scrap approval of test results, introduce third state (yellow)

[mbuechse]

For fear of false positives, we invented a system where test results would have to be approved by an auditor.

This system is too labor-intensive, and has also been confusing many people and therefore hurting transparency.

Therefore it shall be scrapped. In its stead, we shall work with three states:

• fail
• indeterminate (or inconclusive, or partial, or ...)
• pass

We can still have two views: one that shows 'indeterminate' as green (because the cloud hasn't failed any required test; maybe show an additional asterisk), and one that shows 'indeterminate' as yellow (because the cloud hasn't passed all required tests).

The difference compared to today is as follows:

• our current views show different test results (one being restricted to approved results)
• the two future views would show exactly the same test results (but with slightly different colors).

To be fully precise, there shall actually be four states:

• fail (some required test has failed)
• miss (some required test is missing)
• indeterminate (some required test did not produce a result)
• pass (all required tests have passed)

For the end-user, the former two would be RED, the latter two would be GREEN (where indeterminate could get an additional asterisk or something).

To the engineer, the first one would be RED, the middle two would be YELLOW, and the final one would be GREEN.

[mbuechse]

Request for comment @fkr @depressiveRobot @garloff @jklare :-)