End-to-End Testing for Matrix Homeservers

This crate provides a comprehensive (WIP) end-to-end testing solution for Matrix Homeserver implementations. Leveraging the matrix_sdk crate for client-server communication, it enables validation of server behavior against both client application expectations and established Homeserver specifications.

Architecture

The test suite operates through the following workflow:

1. Spawns Docker containers running configured homeserver instances, initializing test environments using blueprint configurations and runtime parameters (set via environment variables).
2. Executes test cases using Matrix client operations via matrix_sdk.

Running the Tests

By default end-to-end tests are disabled inside nextest configuration. To run only the end-to-end-test you can use this command:

cargo nextest run -p arminius-tests-endtoend

Or if you prefer to include it inside the whole test suit of this workspace:

cargo nextest run -E 'all()' --ignore-default-filter

Also, here is a way to compile and run test only for a specific test target

cargo nextest run -p arminius-tests-endtoend --test TEST_TARGET [TEST_NAME]

Please note that for running the tests for Arminius in the current repository, images must be built beforehand.

Using nix you can run the following:

Warning

The below nix build will not work on aarch64-darwin. This is tracked in #2268

# This will build images and generates the script for
# importing them to the docker local registry.
nix build .#containers-arminius-all-test-login

# And then run the script that was generated

./result/bin/import_script

Supported Homeservers

This framework can test multiple homeservers. You can adjust which homeserver you are running using environment variables

arminius

Run Arminius as a series of Docker containers, one for each component. This is the one we run in CI. The binary in the container can be overwritten with one on your local filesystem, in order to hack on Arminius. To do so, set one of the following environment variables to a path on your local system:

• E2E_TESTS_HS{n}_ARMINIUS_ADMIN_BINARY_OVERRIDE
• E2E_TESTS_HS{n}_ARMINIUS_CLIENT_BINARY_OVERRIDE
• E2E_TESTS_HS{n}_ARMINIUS_FEDERATION_INPUT_BINARY_OVERRIDE
• E2E_TESTS_HS{n}_ARMINIUS_FEDERATION_OUTPUT_BINARY_OVERRIDE
• E2E_TESTS_HS{n}_ARMINIUS_ROOMS_BINARY_OVERRIDE

When using binary overrides, it might make sense to have a “blank” docker image, into which you mount your binary override. To build a blank docker image using Nix, run nix build .#arminius-blank-image && ./result | docker load.

native (Default)

Run Arminius, but instead of multiple Docker containers, run the Monolith mode in the same process. This is useful for development, due to generally being faster to start than building and running a Docker container.

continuwuity

Runs the official Continuwuity Docker image.

Currently this is not tested in the CI, but we will probably change this at some point.

synapse

The Synapse homeserver, in a Docker container.

Configuration Management

Environment variables control test configuration:

Global Configuration

• E2E_TESTS_DISALLOW_DEFAULT_IMAGES: If set, require that the configuration and image environment variables are set explicitly
• E2E_TESTS_BASE_CONFIGURATION: Homeserver implementation to test (Options: native, arminius, synapse | Default: native)
• E2E_TESTS_BASE_SYNAPSE_IMAGE: Container image name:tag (Synapse default: matrixdotorg/synapse:latest)
• E2E_TESTS_BASE_ARMINIUS_ADMIN_IMAGE: Container image name:tag (Default: arminius-admin:latest)
• E2E_TESTS_BASE_ARMINIUS_CLIENT_IMAGE: Container image name:tag (Default: arminius-client:latest)
• E2E_TESTS_BASE_ARMINIUS_ROOMS_IMAGE: Container image name:tag (Default: arminius-rooms:latest)
• E2E_TESTS_BASE_ARMINIUS_FEDERATION_INPUT_IMAGE: Container image name:tag (Default: arminius-federation-input:latest)
• E2E_TESTS_BASE_ARMINIUS_FEDERATION_OUTPUT_IMAGE: Container image name:tag (Default: arminius-federation-output:latest)
• E2E_TESTS_BASE_ARMINIUS_NATS_IMAGE: Container image name:tag (Default: nats:latest, per testcontainers)
• E2E_TESTS_BASE_NGINX_IMAGE: Container image name:tag (Default: nginx:latest)
• E2E_TESTS_BASE_ARMINIUS_POSTGRES_IMAGE: Container image name:tag (Default: postgres:latest, per testcontainers)
• E2E_TESTS_BASE_CONTINUWUITY_IMAGE: Container image name:tag (Default: ghcr.io/continuwuity/continuwuity:latest, per testcontainers)

Per-Role Overrides

Imagine these scenarios:

1. You want your test to send a message from a Homeserver using Synapse implementation to one that is using Arminius implementation.
2. You want your test to send message from an earlier version of Arminius to a newer version of it.

In these scenarios, the tests that spin up two different Homeservers, assign a number to each homeserver, calling it a role id. And for each role id you can overwrite the global and default homeserver implementation and image for each. This way, you can configure individual Homeservers using:

• E2E_TESTS_HS{role_id}_CONFIGURATION: Configuration override (Applies to homeserver with specified u8 role identifier)
• E2E_TESTS_HS{role_id}_SYNAPSE_IMAGE: Image override (Applies to homeserver with specified u8 role identifier)
• E2E_TESTS_HS{role_id}_ARMINIUS_{component}_IMAGE: Image override (Applies to homeserver with specified u8 role identifier). The components are:
  • ADMIN
  • CLIENT
  • FEDERATION_INPUT
  • FEDERATION_OUTPUT
  • ROOMS
  • NATS
  • NGINX
  • POSTGRES

The test suite uses fallback behavior:

1. Attempt role-specific configuration
2. Fall back to global configuration if:

• Role-specific variable undefined
• Invalid configuration detected

Debugging Tips and Tricks

Filtering Logs

For example, to filter logs from one of the two homeservers:

RUST_LOG="arminius_tests_endtoend::container[homeserver{homeserver_role=2}]=DEBUG"

Accessing Synapse’s Database

If you want to inspect a end-to-end test’s resulting database state for Synapse:

1. Throw a sleep in the test where you want to inspect the database state at.
2. Identify the running docker container using docker ps
3. Copy the database file from the container using docker cp <CONTAINER_ID>:/data/homeserver.db .
4. Run a nix shell that has sqlite3 using nix-shell -p sqlite
5. Open the database using sqlite3 ./homeserver.db

Requirements

• Docker or podman daemon running locally
• User must have a docker access without a need for root privileges
• Available TCP ports in range 8000-9000

Implementation Notes

The test infrastructure utilizes testcontainers-rs for container management.

A Word of Caution Regarding Running Tests with Arminius

Running the tests for arminius means that it may run tens of containers at the same moment, which can be quite RAM intensive.

Cleanup

If tests are stopped unexpectedly, for instance with CTRL+C, the test suite will not clean up running containers, and this will have to be done manually.

# Remove containers
docker system prune -af --filter 'label=project.arminius.env=e2e'
# Remove volumes
docker volume ls -q | grep '^e2e-' | xargs -r docker volume rm

Images

The end to end tests pull several images to run the matrix homeservers. For the synapse target, it pulls matrixdotorg matrixdotorg/synapse, unless overwritten by an environment variable, while for the arminius target, it pulls nats and postgres.