Testing

This page describes the custom test framework used to test Visual2 instructions against data generated by another simulator (thus far only Visual has been hooked to do this).

For information on how to use Expecto and FsCheck to test Visual2 add-ons go to the using tests page.

Files with test data in: ./app/test-data.txt.

Test file format is currently not well documented: however randomised sets of tests in the correct format are generated by the documented DSL in VisualRandomTestGen. The default test files are sufficient to give good acceptance testing for all emulator instructions and can be found under app/test-data.

In addition to the test files there are some files of allowed tests. An allowed test is one where the results are allowed to be different from VisUAL, and therefore an error, because the correct spec is different from what VisUAL currently implements. These have name ALLOWEDxxxx.txt for some xxxx. The test names and numbers inside any such file identify which of the other tests (each separate test has a name and number) are allowed.

If there are no errors for a test file except on tests in the ALLOWED files, the test file will display as Passed.

Note that this is not perfect. Ideally the allowed test files should specify the CORRECT results and be used instead of the VisUAL-generated results. This is not implemented. The framework is however easy to use. For example, if all generated ERROR output files are copied as ALLOWED files and put into test-data, all the tests will run and pass.

For debug purposes, if a file app/focus.txt is added to this directory only its tests will be run, otherwise all test files will be run.

Help->Run emulator tests from GUI will run the test data files.

Output files are put in .app/test-results.

• ERRORSxxx.txt - assembler that errors or delivers different results from code that runs under VisUAL. NB - these errors may represent Visual spec errors.
• BETTERsxxx.txt - assembler that runs Ok under VisUAL2 but errors under VisUAL. This looser spec may be desired, or problematic.
• OKsxxx.txt - assembler that runs identically under VisUAL and VisUAL2.

Caveats

This framework will not test the values of PC at the end of the code fragment and therefore branches must be tested by embedding them in code that changes register values according to branch result.

This framework will not test memory writes, however reads and writes can be tested indirectly by leaving trace data values in registers.

Generating Test Data

Large random testsets can be generated using VisualRandomTestGen. Code from this could be extracted to generate programmatic test data (though this would require the program used to be correct).