Organization of Test Code
We separate tests into
- unit tests
- narrower integration tests
- wider integration tests
- end-to-end-tests
1. Unit Tests
What?
Unit tests are tests that only test a small internal function.
If it does not need a component to spin up, it can probably go here.
NATS and Database are acceptable here, but if you are using NATS to communicate with other components, those components should be mocked.
Where?
It should go inside the same file as the function that is being tested.
How?
From the project root (this directory), execute:
cargo install cargo-nextest
cargo nextest run
2. Narrower Integration Tests
What?
Integration tests are testing the interaction between two or more components.
It is narrow if the components are all inside the same crate.
Where?
It should go inside the /tests directory of the crate.
How?
The arminius-lib-testing crate provides a test runner and utility functions
to spin up and execute integration tests.
3. Wider Integration Tests
What?
Integration tests are testing the interaction between two or more components.
It is Wider if the components are in different crates.
Where?
It should go inside the /tests directory of the monolith crate.
How?
The arminius-lib-testing crate provides a test runner and utility functions
to spin up and execute integration tests.
4. End-to-End Tests
What?
If the test is testing the code from the perspective of the users it should go here. Further documentation about the end-to-end tests can be found on the dedicated page.
Where?
crates/tests/arminius-tests-endtoend is your friend.
5. Complement
What?
Warning
Running Complement tests on Darwin is not supported. This is tracked in #2268
What?
The Matrix foundation has Complement, which is their own e2e testing suite for homeservers. We also have support for testing Arminius against Complement. Further documentation about the Complement tests can be found on the dedicated page.
6. Local Federation
What?
Included in Arminius is the local federation utility, to run Arminius in a local federation (optionally also with other homeservers supported by the end-to-end tests).
To run it, ensure you have the Arminius docker images available and:
export E2E_TESTS_BASE_CONFIGURATION=arminius
export CARGO_TARGET_DIR="$(pwd)/target"
cargo run -p arminius-tests-localfederation -- --element-desktop
Two element-desktop sessions will spawn (This requires that you have element-desktop installed, which is not included in the devshell for space reasons. You can leave out the --element-desktop argument, and connect with your own client if you wish.).
The one marked as hs1 has an account user1 and the one marked as hs2 has an account user2.
Both use Password1! as their password.
To run Synapse alongside of Arminius, you can set E2E_TESTS_HS2_CONFIGURATION=synapse before starting the local federation.
Configuring the homeservers used to launch local federation follows the same convention as the E2E test framework.
Coverage
Every job that collects coverage uploads its lcov tracefiles as a GitLab
artifact, under a per-job reports/<job>/ directory: the three test slices
(arminius-tests, nuss-tests, nuss-e2e-tests), and the end-to-end tests that
run against the server containers, whose raw profiles
scripts-dev/export-e2e-coverage.sh turns into one tracefile per component.
Each of those jobs also converts its tracefiles into a Cobertura report, which is what annotates the merge request diff - GitLab combines every job’s into one overlay.
The percentage can’t work that way, since GitLab averages what it scrapes per
job, weighting a tiny slice like the whole workspace. So merge_coverage merges
all the tracefiles with coverage-merge as a line-level union
(nix/packages/merge-lcov.py) - a line counts once, and as covered if any job
covered it - and that union’s percentage is the headline number.
Locally, one command builds the three slices (or fetches them from the cache) and merges them:
nix build .#coverage-report
# The headline percentage, as CI computes it:
cat result/coverage-summary
# result/ also contains:
# cobertura.xml - the merged report, as consumed by GitLab
# merged.info - the merged lcov, for the HTML report below
This is an approximation of the CI number: the container end-to-end coverage needs a full dockerized run, so it is not part of this derivation. To include it, or to merge any other set of tracefiles, call the same tool the pipeline uses - it takes an output directory and any number of tracefiles:
coverage-merge ./coverage result/lcov.info target/coverage/*.info
For a browsable, line-by-line HTML report, run genhtml over the merged lcov.
It ships with lcov, which is not in the devshell (it pulls in a large
graphing/font closure), so fetch it on demand:
nix shell nixpkgs#lcov --command genhtml result/merged.info --output-directory coverage-html
# then open coverage-html/index.html
To iterate on a single slice’s coverage without merging, build that slice
directly and point genhtml at its lcov.info, e.g.:
nix build .#nuss-tests-coverage
nix shell nixpkgs#lcov --command genhtml result/lcov.info --output-directory coverage-html