CI: run unit/integration tests for each crate of the workspace

[dannywillems]

Workspace crates to test:

(actual crate-name in brackets)

• core (mina-core)
• macros (mina-macros)
• ledger (mina-tree)
• snark
• p2p
• p2p/testing (p2p-testing)
• p2p/libp2p-rpc-behaviour (libp2p-rpc-behaviour)
• node
• node/account (mina-node-account)
• node/common (mina-node-common)
• node/native (mina-node-native)
• node/web (mina-node-web)
• node/invariants (mina-node-invariants)
• node/testing (mina-node-testing)
• cli
• cli/replay_dynamic_effects (replay_dynamic_effects)
• vrf
• mina-p2p-messages
• poseidon
• tools/transport (mina-transport)
• tools/bootstrap-sandbox (mina-bootstrap-sandbox)
• tools/gossipsub-sandbox (mina-gossipsub-sandbox)
• tools/hash-tool (hash-tool)
• tools/ledger-tool (ledger-tool)
• tools/salsa-simple (salsa-simple)
• tools/fuzzing (transaction_fuzzer)
• tools/archive-breadcrumb-compare (mina-archive-breadcrumb-compare)
• tools/heartbeats-processor (heartbeats-processor)
• tools/webrtc-sniffer (webrtc-sniffer)
• producer-dashboard (mina-producer-dashboard)
• fuzzer (mina-fuzzer)

Plan

Phase 1: Consolidate Unit Tests and Introduce nextest

• In the new workflow, create a single job unit-tests that runs all unit tests for all crates using cargo nextest run --workspace --lib --all-features..

Phase 2: Dynamic Integration Test Jobs

We don't want to have to manually look at all the jobs to identify which crates are or aren't tested.

• Create a "discover" job that programmatically finds all crates with integration tests.
  • This job should output a JSON array of crate names.
• Create a "matrix" job that uses the output of the "discover" job to dynamically generate a matrix of integration test jobs.
  • Each job in the matrix will run the integration tests for a single crate.
• Create a "final" job that depends on all the jobs in the matrix and provides a single "pass/fail" signal for the entire integration test suite.

[richardpringle]

@dannywillems, moving your original description to a comment:

When we made the patch 1549, we realized that we were not running all the unit and integrations tests of every crate in the CI. We only run the following:

• ledger
• mina-node-native
• p2p-messages
• vrf
• p2p

The main reason is that when o1Labs took over the development of the Rust node, only these crates where tested in the CI, and we started from there. Over the weeks, we added more CI workflows, and realized that different tests were ignored, failing or flaky, see this meta-issue.

We would like to ensure that all existing tests are executed in the CI at each push/PR, and when we add a new test, it is actually executed in the CI. It has not been always the case, with, for instance, the fix for the bug 1464, implemented here. We noticed back then that the tests for the crate mina-node-native were not executed in the CI, see the patch 1549.

A patch to run all unit and integration tests has been started in 1550, but it hasn't been completed and it seems there are some failing tests.

To close this issue, we would like to finish the patch 1550 by:

• ensuring that the CI workflow defined in tests.yaml is actually running the unit/integration tests for each crate, and each individual step is correctly configured. For instance, sqlite and sqlx must be properly configured for heartbeat-processors and some linking issues must be fixed, as for webrtc-sniffer.
• fix the build issues if required - it is the case for mina-tree or p2p.
• report in the meta issue [Meta issue] List of failing/empty/flaky tests #524 the failing tests. As part of the task, if the fix is easy and quick (a few minutes), the assignee can take the liberty of doing so.

Some libraries might be old, and unused for now, and we might want to get rid of it soon if we realize they are not useful. However, it would be handled in a follow-up work.

As part of this task, the assignee can suggest a new format to track the failing/ignoring/flaky tests in the meta issue.

[richardpringle]

@dannywillems

Some libraries might be old, and unused for now, and we might want to get rid of it soon if we realize they are not useful. However, it would be handled in a follow-up work.

Do we know which libraries aren't used?

[dannywillems]

@dannywillems

Some libraries might be old, and unused for now, and we might want to get rid of it soon if we realize they are not useful. However, it would be handled in a follow-up work.

Do we know which libraries aren't used?

I think they are all used but by different softwares that we are not currently running, like the archive node. The crates in the subdirectory tools might be related to the memory profiler or the network debugger. We've got open issues to revamp these projects soon.

The crate poseidon will have to be moved to proof-systems too.