Complement

Warning

Running Complement tests on Darwin is not supported. This is tracked in #2268

The Matrix foundation has Complement, which is their own e2e testing suite for homeservers. We also have support for testing Arminius against Complement.

In nix/packages/complement_utils/failing_csapi_tests.txt and nix/packages/complement_utils/failing_federation_tests.txt, we store the complete list of Complement test cases which currently fail in Arminius.

Usage

In the nix shell, three utility scripts are available in your path:

• complement-csapi-runner and complement-federation-runner build Arminius as a Complement base image and run all client-server or server-server tests not in the denylists, respectively. It has two modes (auto-detected from DOCKER_HOST, overridable with COMPLEMENT_IN_DIND=true/false): locally it runs the Complement binaries directly against your docker daemon; in CI on the GitLab Docker executor it runs Complement inside a container on the DinD daemon (--network host + mounted socket), because the DinD service there is network-isolated. In the dev shell just run complement-csapi-runner or complement-federation-runner; otherwise nix run .#complement-csapi-runner or .#complement-federation-runner. See nix/packages/complement_utils/default.nix and nix/packages/complement_runner_image.nix.
• complement-csapi-hacker builds and mounts arminius-server-monolith into a “bare” Arminius Complement image, and runs the Complement client-server tests with your given arguments. Example: complement-csapi-hacker -test.run TestVersionStructure
• complement-federation-hacker mounts your monolith build from $CARGO_TARGET_DIR into a “bare” Arminius Complement image, and runs the Complement server-server tests with your given arguments. Example: complement-federation-hacker -test.run TestInboundFederationKeys

The -hacker scripts require a reachable local Docker daemon and that the CARGO_TARGET_DIR environment variable is set explicitly. For example, this invocation of complement-federation-runner runs only the federation test TestACLs:

export CARGO_TARGET_DIR="$(pwd)/target"
complement-federation-hacker -test.run '^TestACLs$'

-test.run accepts a regular expression, so the anchors prevent this command from also running TestACLsForEDUs.

If the environment variable “COMPLEMENT_DIR” is set, the Complement checkout in the given directory will be used, instead of the Complement version defined by the Nix shell.

Firewall

Some federation tests in Complement make use of a mocked Matrix server running on the host machine, instead of a docker container. To make the mock server reachable from Arminius when Arminius is being run under Complement, you may need to modify your firewall rules. On NixOS, this can be achieved by:

networking.firewall.extraCommands = "iptables -A INPUT -i br-+ -j ACCEPT";

See also the upstream Complement documentation.

Troubleshooting

/override/arminius-server-monolith: cannot execute: required file not found

This error results from the container not containing the dynamic libraries needed by the arminius-server-monolith binary. Typically, this results from an old version of the arminius-empty-complement image, which has dynamic libraries not matching those of the devshell. You just need to delete the old arminius-empty-complement images:

docker image ls --filter 'reference=arminius-empty-complement*' --format='{{.ID}}' | xargs docker image rm