From 2cb571a29883c7fa1e1140f56aa1fd6f89fe67f5 Mon Sep 17 00:00:00 2001 From: Leigh McCulloch <351529+leighmcculloch@users.noreply.github.com> Date: Thu, 16 Jan 2025 15:58:41 +1000 Subject: [PATCH 1/5] Update testing guide structure and add coverage testing --- docs/build/guides/testing/code-coverage.mdx | 53 +++++++++++++++++++ .../build/guides/testing/coverage-testing.mdx | 9 ---- ...differential-tests-with-test-snapshots.mdx | 2 +- .../guides/testing/differential-tests.mdx | 2 +- docs/build/guides/testing/fuzzing.mdx | 2 +- .../integration-tests-with-mainnet-data.mdx | 3 +- .../guides/testing/integration-tests.mdx | 2 +- .../build/guides/testing/mutation-testing.mdx | 4 +- .../guides/testing/test-contract-auth.mdx | 1 + .../guides/testing/test-contract-events.mdx | 1 + 10 files changed, 63 insertions(+), 16 deletions(-) create mode 100644 docs/build/guides/testing/code-coverage.mdx delete mode 100644 docs/build/guides/testing/coverage-testing.mdx diff --git a/docs/build/guides/testing/code-coverage.mdx b/docs/build/guides/testing/code-coverage.mdx new file mode 100644 index 000000000..8a5def0c1 --- /dev/null +++ b/docs/build/guides/testing/code-coverage.mdx @@ -0,0 +1,53 @@ +--- +title: Code Coverage +hide_table_of_contents: true +description: Code coverage tools find code not tested. +sidebar_position: 10 +--- + +Measuring code coverage uses tools to identify lines of code that are and aren't executed by tests. Code coverage stats can give us an idea of how much of a contract is actually tested by its tests. + +:::tip + +Mutation testing is another form of coverage testing. See [Mutation Testing]. + +::: + +In rust projects the `cargo-llvm-cov` tool can be used to generate coverage stats, HTML reports, and lcov files that IDEs will load to display the coverage in the code editor. + +Install `cargo-llvm-cov` before proceeding with the other commands. + +``` +cargo install cargo-llvm-cov +``` + +## How to Get Coverage Stats + +Run the test subcommand that will run the tests and output the stats per file. + +``` +cargo llvm-cov test +``` + +## How to Generate a Coverage Report with Code + +Run the test subcommand that will run the tests and output a set of HTML files showing which lines of code are covered. + +``` +cargo llvm-cov test --html --open +``` + +The output of the command will indicate where the HTML file has been written. Open the file in a browser. + +## How to Generate an LCOV File for IDEs + +Run the test subcommand that will run the tests and output a single `lcov.info` file. + +``` +cargo llvm-cov test --lcov --output-path=lcov.info +``` + +Load the `lcov.info` file into your IDE using it's coverage feature. In VSCode this can be done by installing the [Coverage Gutters] extension and executing the `Coverage Gutters: Watch` command. + +[Coverage Gutters]: https://marketplace.visualstudio.com/items?itemName=ryanluker.vscode-coverage-gutters +[Mutation Testing]: mutation-testing.mdx diff --git a/docs/build/guides/testing/coverage-testing.mdx b/docs/build/guides/testing/coverage-testing.mdx deleted file mode 100644 index 0979d2849..000000000 --- a/docs/build/guides/testing/coverage-testing.mdx +++ /dev/null @@ -1,9 +0,0 @@ ---- -title: Coverage Testing -hide_table_of_contents: true -description: Coverage testing finds code not tested. -sidebar_position: 10 -draft: true ---- - -**TODO: Fill in example.** diff --git a/docs/build/guides/testing/differential-tests-with-test-snapshots.mdx b/docs/build/guides/testing/differential-tests-with-test-snapshots.mdx index 30b649ceb..61aad39bb 100644 --- a/docs/build/guides/testing/differential-tests-with-test-snapshots.mdx +++ b/docs/build/guides/testing/differential-tests-with-test-snapshots.mdx @@ -2,7 +2,7 @@ title: Differential Tests with Test Snapshots hide_table_of_contents: true description: Differential testing using automatic test snapshots. -sidebar_position: 8 +sidebar_position: 9 --- Tests are written to ensure that contracts behave today as expected, and in the future as well. Over time a contract may change and in all software development there remains the possibility of changes causing side-effects that are unexpected. Testing is one of the ways that we identify unexpected changes. diff --git a/docs/build/guides/testing/differential-tests.mdx b/docs/build/guides/testing/differential-tests.mdx index e9036b2f3..9a5674f8f 100644 --- a/docs/build/guides/testing/differential-tests.mdx +++ b/docs/build/guides/testing/differential-tests.mdx @@ -2,7 +2,7 @@ title: Differential Tests hide_table_of_contents: true description: Differential testing detects unintended changes. -sidebar_position: 7 +sidebar_position: 8 --- Differential testing is the testing of two things to discover differences in their behavior. diff --git a/docs/build/guides/testing/fuzzing.mdx b/docs/build/guides/testing/fuzzing.mdx index 6b588c996..396422b82 100644 --- a/docs/build/guides/testing/fuzzing.mdx +++ b/docs/build/guides/testing/fuzzing.mdx @@ -2,7 +2,7 @@ title: Fuzzing hide_table_of_contents: true description: Fuzzing and property testing to find unexpected behavior. -sidebar_position: 5 +sidebar_position: 7 --- Fuzzing is the process of providing random data to programs to identify unexpected behavior, such as crashes and panics. diff --git a/docs/build/guides/testing/integration-tests-with-mainnet-data.mdx b/docs/build/guides/testing/integration-tests-with-mainnet-data.mdx index a8fa9ecd3..04fb6f4e6 100644 --- a/docs/build/guides/testing/integration-tests-with-mainnet-data.mdx +++ b/docs/build/guides/testing/integration-tests-with-mainnet-data.mdx @@ -2,8 +2,7 @@ title: Integration Tests with Mainnet Data hide_table_of_contents: true description: Integration testing uses dependency contracts instead of mocks. -sidebar_position: 4 -draft: true +sidebar_position: 6 --- Testing with mainnet data is another form of [integration test], where not only mainnet contracts can be used, but also their data on mainnet. diff --git a/docs/build/guides/testing/integration-tests.mdx b/docs/build/guides/testing/integration-tests.mdx index 06003d7eb..b11d5defa 100644 --- a/docs/build/guides/testing/integration-tests.mdx +++ b/docs/build/guides/testing/integration-tests.mdx @@ -2,7 +2,7 @@ title: Integration Tests hide_table_of_contents: true description: Integration testing uses dependency contracts instead of mocks. -sidebar_position: 3 +sidebar_position: 5 --- Integration tests are tests that include the integration between components, and so test a larger scope such as other contracts. diff --git a/docs/build/guides/testing/mutation-testing.mdx b/docs/build/guides/testing/mutation-testing.mdx index 8bccad902..f731e1c0f 100644 --- a/docs/build/guides/testing/mutation-testing.mdx +++ b/docs/build/guides/testing/mutation-testing.mdx @@ -2,7 +2,7 @@ title: Mutation Testing hide_table_of_contents: true description: Mutation testing finds code not tested. -sidebar_position: 9 +sidebar_position: 11 --- Mutation testing is making changes to a program, either manually or automatically, to identify changes that can be made that don't get caught by tests. @@ -33,3 +33,5 @@ The `cargo-mutants` tool can be used to automatically and iteratively modify the Code that is identified as not covered by a test will be outputted as a `MISSED` line in the output. Diffs of each change that was attempted can be found in the `mutants.out/diff` directory. + +[code coverage]: code-coverage.mdx diff --git a/docs/build/guides/testing/test-contract-auth.mdx b/docs/build/guides/testing/test-contract-auth.mdx index 3cac5898c..054deb1b6 100644 --- a/docs/build/guides/testing/test-contract-auth.mdx +++ b/docs/build/guides/testing/test-contract-auth.mdx @@ -2,6 +2,7 @@ title: Test Authorization hide_table_of_contents: true description: Write tests that test contract authorization. +sidebar_position: 3 --- Tests can assert on the auths that are expected to occur. diff --git a/docs/build/guides/testing/test-contract-events.mdx b/docs/build/guides/testing/test-contract-events.mdx index 07f28562d..9360d1592 100644 --- a/docs/build/guides/testing/test-contract-events.mdx +++ b/docs/build/guides/testing/test-contract-events.mdx @@ -2,6 +2,7 @@ title: Test Events hide_table_of_contents: true description: Write tests that test contract events. +sidebar_position: 4 --- Tests can assert on events that are expected to be published. From eb10f9e243286e0f773f2a409c292bbc91750f30 Mon Sep 17 00:00:00 2001 From: Leigh McCulloch <351529+leighmcculloch@users.noreply.github.com> Date: Thu, 16 Jan 2025 17:04:51 +1000 Subject: [PATCH 2/5] add code coverage for fuzzing --- docs/build/guides/testing/code-coverage.mdx | 7 +++++ docs/build/guides/testing/fuzzing.mdx | 32 +++++++++++++++++++++ 2 files changed, 39 insertions(+) diff --git a/docs/build/guides/testing/code-coverage.mdx b/docs/build/guides/testing/code-coverage.mdx index 8a5def0c1..28468c61e 100644 --- a/docs/build/guides/testing/code-coverage.mdx +++ b/docs/build/guides/testing/code-coverage.mdx @@ -49,5 +49,12 @@ cargo llvm-cov test --lcov --output-path=lcov.info Load the `lcov.info` file into your IDE using it's coverage feature. In VSCode this can be done by installing the [Coverage Gutters] extension and executing the `Coverage Gutters: Watch` command. +:::info + +Measuring code coverage in fuzz tests requires different tooling. See [Fuzzing]. + +::: + [Coverage Gutters]: https://marketplace.visualstudio.com/items?itemName=ryanluker.vscode-coverage-gutters [Mutation Testing]: mutation-testing.mdx +[Fuzzing]: fuzzing.mdx diff --git a/docs/build/guides/testing/fuzzing.mdx b/docs/build/guides/testing/fuzzing.mdx index 396422b82..02a923769 100644 --- a/docs/build/guides/testing/fuzzing.mdx +++ b/docs/build/guides/testing/fuzzing.mdx @@ -116,6 +116,37 @@ There is another tool for fuzzing Rust code, `cargo-afl`. See the [Rust Fuzz boo ::: +### How to Get Code Coverage of Fuzz Tests + +Getting code coverage data for fuzz tests requires some different tooling than when doing the same for regular Rust tests. + +1. Run the fuzz tests until it has produced a corpus, just as in step 7 above. + + ``` + cargo +nightly fuzz coverage --sanitizer thread fuzz_target_1 + ``` + +2. Install the llvm-tools for the nightly compiler. + + ``` + rustup component add --toolchain nightly llvm-tools-preview + ``` + +3. Run the fuzz coverage command that'll execute the corpus and write coverage data to the coverage directory. + + ``` + cargo +nightly fuzz coverage --sanitizer thread fuzz_target_1 + ``` + +4. + +:::tip + +To get code coverage of regular Rust tests, see [Code Coverage]. + +::: + + [increment example]: https://github.com/stellar/soroban-examples/blob/main/increment/src/lib.rs [Differential Testing with Test Snapshots]: ./differential-tests-with-test-snapshots.mdx [stellar contract fetch]: ../../../tools/developer-tools/cli/stellar-cli.mdx#stellar-contract-fetch @@ -124,3 +155,4 @@ There is another tool for fuzzing Rust code, `cargo-afl`. See the [Rust Fuzz boo [stellar/rs-soroban-sdk#1360]: https://github.com/stellar/rs-soroban-sdk/issues/1360 [fuzzing example]: ../../smart-contracts/example-contracts/fuzzing.mdx [Rust Fuzz Book]: https://rust-fuzz.github.io/book +[Code Coverage]: code-coverage.mdx From 1ce0a5627af81610ea0e783aa1d82ef0f1c1139c Mon Sep 17 00:00:00 2001 From: Leigh McCulloch <351529+leighmcculloch@users.noreply.github.com> Date: Thu, 16 Jan 2025 17:07:33 +1000 Subject: [PATCH 3/5] more --- docs/build/guides/testing/fuzzing.mdx | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-) diff --git a/docs/build/guides/testing/fuzzing.mdx b/docs/build/guides/testing/fuzzing.mdx index 02a923769..5c91f5323 100644 --- a/docs/build/guides/testing/fuzzing.mdx +++ b/docs/build/guides/testing/fuzzing.mdx @@ -132,13 +132,19 @@ Getting code coverage data for fuzz tests requires some different tooling than w rustup component add --toolchain nightly llvm-tools-preview ``` -3. Run the fuzz coverage command that'll execute the corpus and write coverage data to the coverage directory. +3. Run the fuzz coverage command that'll execute the corpus and write coverage data to the coverage directory in the `profdata` format. ``` cargo +nightly fuzz coverage --sanitizer thread fuzz_target_1 ``` -4. +4. Run the llvm-cov command to convert the profdata file to an lcov file. + + ``` + TODO + ``` + + Load the `lcov.info` file into your IDE using it's coverage feature. In VSCode this can be done by installing the [Coverage Gutters] extension and executing the `Coverage Gutters: Watch` command. :::tip @@ -156,3 +162,4 @@ To get code coverage of regular Rust tests, see [Code Coverage]. [fuzzing example]: ../../smart-contracts/example-contracts/fuzzing.mdx [Rust Fuzz Book]: https://rust-fuzz.github.io/book [Code Coverage]: code-coverage.mdx +[Coverage Gutters]: https://marketplace.visualstudio.com/items?itemName=ryanluker.vscode-coverage-gutters From 17f94a72aedc526ac3ed36d413919de459dd1734 Mon Sep 17 00:00:00 2001 From: Leigh McCulloch <351529+leighmcculloch@users.noreply.github.com> Date: Fri, 17 Jan 2025 11:54:16 +1000 Subject: [PATCH 4/5] update wording in fuzzing guide tip --- docs/build/guides/testing/fuzzing.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/build/guides/testing/fuzzing.mdx b/docs/build/guides/testing/fuzzing.mdx index 5c91f5323..006c47fae 100644 --- a/docs/build/guides/testing/fuzzing.mdx +++ b/docs/build/guides/testing/fuzzing.mdx @@ -148,7 +148,7 @@ Getting code coverage data for fuzz tests requires some different tooling than w :::tip -To get code coverage of regular Rust tests, see [Code Coverage]. +To measure code coverage of regular Rust tests, see [Code Coverage]. ::: From ef9d87c584e844cd025b4a65b172a76ee7c9e836 Mon Sep 17 00:00:00 2001 From: Leigh McCulloch <351529+leighmcculloch@users.noreply.github.com> Date: Fri, 17 Jan 2025 17:32:13 +1000 Subject: [PATCH 5/5] add llvm-cov commands --- docs/build/guides/testing/fuzzing.mdx | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/docs/build/guides/testing/fuzzing.mdx b/docs/build/guides/testing/fuzzing.mdx index 006c47fae..d874c5fd0 100644 --- a/docs/build/guides/testing/fuzzing.mdx +++ b/docs/build/guides/testing/fuzzing.mdx @@ -123,7 +123,7 @@ Getting code coverage data for fuzz tests requires some different tooling than w 1. Run the fuzz tests until it has produced a corpus, just as in step 7 above. ``` - cargo +nightly fuzz coverage --sanitizer thread fuzz_target_1 + cargo +nightly fuzz run --sanitizer thread fuzz_target_1 ``` 2. Install the llvm-tools for the nightly compiler. @@ -141,7 +141,11 @@ Getting code coverage data for fuzz tests requires some different tooling than w 4. Run the llvm-cov command to convert the profdata file to an lcov file. ``` - TODO + LLVM_TOOLS_PATH=$(dirname $(find $(rustc +nightly --print sysroot) -name 'llvm-cov')) + $LLVM_TOOLS_PATH/llvm-cov export -instr-profile=fuzz/coverage/fuzz_target_1/coverage.profdata \ + -format=lcov \ + -object target/aarch64-apple-darwin/coverage/aarch64-apple-darwin/release/fuzz_target_1 \ + --ignore-filename-regex "rustc" > lcov.info ``` Load the `lcov.info` file into your IDE using it's coverage feature. In VSCode this can be done by installing the [Coverage Gutters] extension and executing the `Coverage Gutters: Watch` command.