Arminius

Welcome to the documentation for Arminius.

Whether you’re looking to deploy the project, explore its architectural design, or join the development effort, this documentation has you covered.

Warning

Arminius is not yet ready to be used in production. At its current state, deploying it and federating with other rooms could lead to Arminius messing up the state of these rooms.

Furthermore, parts of the code currently assume a development environment or are unfinished in ways that make it insecure for a public-facing deployment. Therefore, we currently advise against deploying Arminius publicly.

GitLab

The GitLab repository is:

• https://gitlab.com/bundesmessenger/backend/neo/arminius/arminius
• Apache-2.0 License

Rustdoc

The rustdoc is also hosted alongside this documentation. It can be found here.

Admin OpenAPI

The OpenAPI documentation for the Admin service can be found here

Setup

This chapter covers setting up Arminius for either development or for deploying it.

Important

Currently, the deployment setup still relies on tools and workflows also used in development. Therefore you should read the chapter “development setup” first.

All later chapters assume you have set up your development environment as described in that chapter. Expect this to change in the future.

📚 Pages

• Development Setup
• Deployment

Development Setup

These docs detail getting an environment set up for developing Arminius.

Nix Setup

Arminius is developed using Nix.

If you do not yet have Nix installed, we recommend you install the Lix implementation of Nix.

Important

The Lix installer will ask whether you would like to

Enable flakes? ([Y]es/[n]o):

Arminius depends on flakes. We therefore recommend you enable them.

The documentation always assumes that you are inside the nix shell provided with the project

You can enter the nix shell using

nix develop

Tip

To automatically enter the nix shell when entering the project directory, you can install direnv and add an .envrc to the project directory like so:

cp .envrc.example .envrc && direnv allow

Unlike nix develop which always uses bash, direnv transparently loads the nix shell environment into your current shell.

Entering the development shell should get you the following message:

Welcome to the Arminius development shell :)

[[[lints and checks]]]

  ast-grep scan         - Run the repository lint scan.
  ast-grep scan --error - Run the CI-equivalent lint scan.
  ast-grep test         - Run the ast-grep rule fixture tests.
  nix fmt               - Auto-format all files in the project.
  prek                  - Run pre-commit hooks on currently staged changes
  prek --stage pre-push - *Also* slow pre-commit hooks
  prek -s main -o HEAD  - Run pre-commit hooks on all commits in the current branch

[[general commands]]

  menu                  - prints this menu

Running Background Services

To reset and start NATS, PostgreSQL, and run migrations:

export $(dev-services up | xargs -L 1)

This helper is also available in source form in scripts-dev/dev-services.sh.

Running Arminius

Arminius is designed as a series of microservices. But for development, we support running all services as one monolith:

export ARMINIUS_HOST=127.0.0.1:8080
export ARMINIUS_NAME=127.0.0.1:8080
cargo run --package arminius-server-monolith --features test_login

You can now connect with your Matrix client to http://localhost:8080, and create an account!

With Nuss

To run Arminius with Nuss you should in one shell run Nuss:

export $(dev-services up nuss| xargs -L 1)
export BASE_URL=http://localhost:9000
cargo run --package nuss -- --dev-mode

and in another shell run Arminius:

export $(dev-services up with-nuss| xargs -L 1)
export ARMINIUS_HOST=127.0.0.1:8080
export ARMINIUS_NAME=127.0.0.1:8080
cargo run --package arminius-server-monolith

Deployment

This chapter describes the supported ways of deploying Arminius.

Long-term deployments will be supported by means of the helm chart.

Warning

Arminius is not ready for production use. If you deploy it and federate with other servers, Arminius can corrupt the state of the rooms they share.

Some parts of the code assume a development environment. Others are unfinished in ways that make a public-facing deployment insecure. Do not deploy Arminius where it is reachable from the public internet.

📚 Pages

• Helm Chart
• Reverse Proxy Setup

The Arminius Helm Chart

The eventual goal is to have Arminius deployable on Kubernetes. Therefore, we are currently developing a helm chart. This helm chart is not production ready, but just the least amount of helm chart we could write to get something that runs, such that we can start work on all the tooling around kubernetes that will be necessary.

The application chart lives in charts/arminius. Cluster dependencies for local development (Postgres, NATS, Gateway API, optional Zitadel) live in charts/dev-services.

Running the Helm Chart with Kind and Tilt

These instructions assume that you have a Docker daemon running on your system. Additionally, you’ll need kind, tilt, kubernetes, and an array of other packages which are included in the Nix devshell.

First, you’ll want to add the following line to your /etc/hosts :

127.0.0.1 matrix.local zitadel.local openobserve.local

This is needed because Zitadel needs its own hostname, distinct from your Matrix server. If you’re running NixOS, you can make this change permanent with

networking.extraHosts = "127.0.0.1 matrix.local zitadel.local";

Afterwards, you need to use tilt to start the homeserver:

kind create cluster --config kind_config.yaml
helm repo add nats https://nats-io.github.io/k8s/helm/charts/
helm dependency build charts/dev-services
tilt up

To log in, we recommend first opening https://matrix.local/_matrix/federation/v1/version in a Chrome-based browser to create an exception for the self-signed TLS certificate. Afterwards, you will be able to select the homeserver matrix.local on app.element.io. You can log in with the Zitadel admin account, with the username zitadel-admin@zitadel.zitadel.local and password Password1!.

If you want to look at the traces OpenObserve is started at openobserve.local with the username root@example.com and password Complexpass#123.

To delete your cluster again, use the command kind delete cluster.

On macOS

There is no native aarch64-darwin container runtime, so the solution on macOS is generally to run an aarch64-linux VM which you then install kind into.

However, in order to be able to build these images on darwin using the nix flake, you need to use the nix-darwin linux builder. If using nix-darwin, see below:

nix.linux-builder = {
  enable = true;
  config.virtualisation = {
    cores = 6;
    darwin-builder.memorySize = 24 * 1024;
    # The default 20G is not enough: the workspace's target directory alone
    # goes past that.
    darwin-builder.diskSize = 100 * 1024;
  };
};

Check it works with nix build nixpkgs#legacyPackages.aarch64-linux.hello. The first tilt up then builds the whole Rust workspace inside that VM, which takes a while; afterwards only changed crates are rebuilt.

Hacking

Famedly’s Helm Chart Guidelines can be found here

Chart values are validated against charts/arminius/values.schema.json (JSON Schema Draft-07), generated with helm schema --draft 7. After changing values.yaml annotations, regenerate the schema and commit the result.

Troubleshooting, Inspecting Your Cluster

When you have a cluster up and running, you can look at it using kubectl. This is not a complete list of commands available in kubectl. kubernetes is very big and complicated. But if something fails in the previous setup, these are probably the commands you’d run to figure out what.

kubectl get pods

The command kubectl get pods prints all pods which are running on the cluster in the selected namespace. Resource names are derived from the helm release name, which the Tiltfile pins to arminius for the application chart. The Arminius pods should look like this, alongside the charts/dev-services pods (Postgres, NATS, the Gateway, and Zitadel):

NAME                                                       READY   STATUS    RESTARTS        AGE
deploy-arminius-admin-7df7dc99c7-2zkvx                     1/1     Running   1 (5m37s ago)   6m9s
deploy-arminius-client-64bd9f4dd-blzv6                     1/1     Running   1 (5m37s ago)   6m9s
deploy-arminius-federation-input-69484689f4-kdwx5          1/1     Running   1 (5m37s ago)   6m9s
deploy-arminius-federation-output-5c9f8b7d64-w4np2         1/1     Running   1 (5m37s ago)   6m9s
deploy-arminius-nuss-6b8c47d5f9-x2mrt                      1/1     Running   0               6m9s
deploy-arminius-scim-7c4d9f8b56-qk8vn                      1/1     Running   1 (5m37s ago)   6m9s
deploy-arminius-rooms-7d75d8f654-dn2gx                     1/1     Running   2 (5m7s ago)    6m9s
deploy-arminius-rooms-7d75d8f654-pqhwc                     1/1     Running   2 (5m7s ago)    6m9s
deploy-arminius-rooms-7d75d8f654-r7tpz                     1/1     Running   1 (5m37s ago)   6m9s
deploy-arminius-rooms-7d75d8f654-z5skk                     1/1     Running   2 (5m7s ago)    6m9s

If the STATUS of some pod is not Running as expected, you can use the kubectl describe pod <NAME> to inspect the state of your pod, and kubectl logs <NAME> to get logs from it.

Rootless Container with Podman

If you run kind with rootless Podman, the cluster cannot bind the privileged ports 80 and 443 that the Gateway needs, so lower the unprivileged port start:

echo 'net.ipv4.ip_unprivileged_port_start=80' | sudo tee /etc/sysctl.d/99-rootless-ports.conf
sudo sysctl --system

The dev-services Helmchart

For dev-services an additional Helm-Chart is provided. It provides the same purpose as the other options for development setups and covers the following services:

• NATS (for dev purposes)
• A CNPG cluster for Arminius (for dev purposes)
• Single-Node Postgres for Nuss (for dev purposes)
• Zitadel (for dev purposes)
• Gateway API configuration resources
• OpenObserve (for dev purposes)

Scope of the Dev-Services Helmchart

The Dev-Services shall not be used for production services. It serves the following purposes:

• run in local Nix/Tilt driven environments
• run in Development Deployments

For flexible integration into local/remote pre-provisioned clusters which come often with some services by default (like Gateway resources), it’s necessary to make all components in the Dev-Services toggleable.

Reverse Proxy Setup

The only supported way to run Arminius is with a reverse proxy, Arminius does not support TLS natively.

The Helm chart provides a reverse proxy setup through Gateway API HTTPRoutes. Each HTTP component’s httpRoute block in charts/arminius/values.yaml sets its hostnames and, optionally, its own Gateway. Routes without their own Gateway attach to the one in httpRoute.parentRefs. The chart README describes these values.

For any other reverse proxy, forward requests:

• To /arminius/admin/ to the admin service
• To /_matrix/client/ to the client service
• To /.well-known/matrix/client to the client service
• To /_matrix/federation/ to the federation-input service
• To /_matrix/key/ to the federation-input service
• To /.well-known/openid-configuration, /oauth/, /login/ and /scim/ to the Nuss service

Forwarded IPs

Arminius gets client IP addresses from the X-Forwarded-For header. Since Arminius only supports running in a reverse proxy, it trusts any such header it receives, no matter its origin. That means, if the reverse proxy allows external clients to pass X-Forwarded-For headers through the reverse proxy, it may be possible for clients to spoof their IP Address.

Administration and Configuration

This chapter describes configuring and operating a running Arminius deployment and its direct dependencies.

Startup Behavior

For cloud-native environments, it is generally preferable that Arminius microservices start even if PostgreSQL or NATS are unavailable.

Set the environment variable ARMINIUS_LAZY to enable this behavior. PostgreSQL recognizes 1 or true and defers the initial connection until the first query. NATS recognizes any nonempty value and retries the initial connection.

Tip

Example:

export ARMINIUS_LAZY=true
tilt up [...]

📚 Pages

• PostgreSQL
• NATS
• Identity Provider
• Outbound Networking
• Telemetry Export
• Admin API

PostgreSQL

Arminius connects to a PostgreSQL database using environment variables. If DATABASE_URL is set, connection options are derived from it. Otherwise, PGHOST, PGPORT, PGUSER, PGPASSWORD, and PGDATABASE configure the connection.

See Startup Behavior to defer the initial database connection.

NATS

Arminius requires a connection to NATS, which is configured through environment variables:

• NATS_URLS configures the URLs used to connect to a NATS server.
• ARMINIUS_NATS_CERT and ARMINIUS_NATS_KEY configure client TLS.

See Startup Behavior to configure retrying the initial NATS connection.

Identity Provider

Upstream Identity Provider

By default, Arminius comes with the Nuss IdP in dev mode, providing no real authentication. Instead of using dev mode, you can configure an upstream IdP in Nuss by setting the environment variables of Nuss.

These are the instructions to configure Zitadel as an upstream IdP:

arminius-server-scim

1. In the organisation you want to sync, create the Service User “arminius” (Access Token Type: JWT).
2. Click on “Actions” > “Generate Client Secret”. Set SCIM_CLIENT_ID and SCIM_CLIENT_SECRET environment variables of arminius-server-scim to the shown credentials.
3. Set SCIM_TOKEN_ENDPOINT of arminius-server-scim to http://$ZITADEL/oauth/v2/token.

Nuss

1. In the same organisation, create the Service User “nuss” in the Zitadel Console (Access Token Type: Bearer).
2. Click on “Actions” > “Generate Client Secret”. Set CLIENT_ID and CLIENT_SECRET environment variables of nuss to the shown credentials.
3. Add the Service User as a manager to the organisation, with the permission “Org Owner Viewer”.
4. Set SCIM_BASE_URL to http://$ZITADEL/scim/v2/$ORG_ID/ (replace the variables) and SCOPES to openid urn:zitadel:iam:org:project:id:zitadel:aud.
5. Add an application in Zitadel (in any project in any organisation). Select “Web” > “PKCE”. The Redirect URI should be configured to http://$NUSS_SERVER/login (by default, it’s localhost:9000).
6. Set OIDC_CLIENT_ID to the shown client id. Set OIDC_BASE_URL to http://$ZITADEL/.
7. Set BASE_URL to the user-facing URL for Nuss.
8. Under “Token Settings” set the “Auth Token Type” to “JWT”.

Custom Identity Provider

If your IdP directly provides the required custom extensions, you can omit Nuss and configure the OIDC-related environment variables directly in Arminius.

Set OIDC_BASE_URL to the OIDC provider.

Outbound Networking

Outgoing Proxy

Arminius supports using a proxy for all outgoing requests. To use it, set the environment variable ARMINIUS_PROXY to a URL. We support everything supported by Reqwest.

Custom CA

Arminius supports setting a single custom Certificate Authority, for federation. Set the environment variable ARMINIUS_EXTRA_CA_CERT to a filesystem path which has a PEM-encoded CA certificate.

Telemetry Export

Arminius can report traces, logs and metrics to an OpenTelemetry compatible service.

You can set the endpoint to report to using the OTEL_EXPORTER_OTLP_ENDPOINT variable. If this variable is unset, no metrics or traces are collected and logs are only provided on stdout.

You may also want to configure authorization headers using OTEL_EXPORTER_OTLP_HEADERS.

More information about those variables can be found here: https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/protocol/exporter.md

Admin API

This document describes the Arminius Admin API and how to authenticate against it.

Overview

The Arminius Admin API provides additional functionality for server administrators. It is a complement to the Matrix APIs, providing ergonomic management of rooms, users, and other server properties.

Authentication

The Admin API authenticates requests with the Matrix access token from the user’s OpenID Connect (OIDC) session.

Warning

Arminius does not yet check authorization. Any user with an active session is a server admin. Do not expose the Admin API publicly. See FIXME(#2215).

API Documentation

The API documentation is available in the form of an OpenAPI specification.

Code Style & Development Practices

This section explains the different Code Style and Development Practices that ought to be respected when working on the implementation of the homeserver.

📚 Pages

• Coding Style
• Formatting and Linting
  • ast-grep
• Code Reviews
• Pre-Merge Duties
• Documentation
• Organization of Test Code
• Database Queries
• Error Handling
• Telemetry
• Continuous Integration

Coding Style

Centralized Crates Versions

All dependency crate versions are declared in the root Cargo.toml. Member crates should declare any required features in their own Cargo.toml files.

In practice, it means that:

• The root Cargo.toml must contain entries in the form <crate-name> = { version = "X.Y", default-features = false}.
• The members Cargo.toml must contain entries in the form <crate-name> = { workspace = true, features = [<features>] }.

Note

Some crates will not compile without a minimum set of features declared. In that case, only enable these features in the root Cargo.toml while still disabling the default ones (e.g <crate-name> = { version = "x.y.z", default-features = false, features = [<mandatory-features>] }).

Additionally, patch versions are omitted from version declarations (e.g. 1.0 instead of 1.0.5).

API Endpoint Function Signatures

We generally follow the same template for our API endpoints, to make the code easier to navigate.

Here is the template to apply for any handler:

/// Handler for `<METHOD> <ENDPOINT>` (See <URL>).
///
/// Optional: Additional description.
///
/// Optional: Notes,warnings, precisions if needed.
async fn ENDPOINT_handler(
	RumaRequest(create_tag_request): RumaRequest<ENDPOINT::Request>,
) -> Result<ArminiusResponse<ENDPOINT::Response>, InternalServerError> {

An example:

/// Handler for `PUT /_matrix/client/v3/user/{userId}/rooms/{roomId}/tags/{tag}`
/// (See <https://spec.matrix.org/latest/client-server-api/#put_matrixclientv3useruseridroomsroomidtagstag>).
///
/// This endpoint is used to add user tags to rooms.
///
/// Warning: The name of a tag MUST NOT exceed 255 bytes.
async fn create_tag_handler(
	database: State<DefaultDb>,
	AuthenticatedClaims(claims): AuthenticatedClaims,
	RumaRequest(create_tag_request): RumaRequest<tag::create_tag::v3::Request>,
) -> Result<ArminiusResponse<tag::create_tag::v3::Response>, AddTagError> {

Working with Time

If you can, use std::time::Instant. For instance, if you’re just measuring time passing inside of a process, this is the way to go, because this API is robust against clock jumps, and generally faster than SystemTime or calendar time.

If you need to store time somewhere else, you should use chrono. time is not allowed in this project, since we already have chrono. This is also checked by forbidden_deps.sh. See the Database Architecture docs for conventions on storing timestamps in PostgreSQL.

FIXMEs and TODOs

The three following syntaxes are supported.

• TODO: <reason>: These indicate areas where the current code works correctly but could be optimized, refactored or improved. Every non-trivial TODO should have their associated issue indicated in parenthesis (e.g TODO(#123): Add cross-signing keys to the response.).
• FIXME(#<issue>): <reason>: These indicate something that is wrong, incorrect, incomplete and must be fixed before reaching a production ready codebase. All of these must mention a corresponding existing issue.
• FIXME(PR): <reason>: These help clarify topics that needs to be addressed before being able to merge the Pull Request. They can be used to ask questions to reviewers, or simply be used as reminders for the developer.

Tip

This section is about comments, not the Rust built-in todo!() macro. If you use such macro, it should therefore be preceded by a FIXME like so:

fn x_squared(x: usize) -> usize {
  // FIXME(#123): Implement the rest of this function.
  todo!()
}

Module Documentation

Rust supports two kinds of documentation for modules:

/// Below is a module which produces foos
mod foo_producer {
  fn produce_foo() {

  }
}

and

mod foo_producer {
  //! This module produces foos
  fn produce_foo() {

  }
}

This project prefers the second module-internal documentation style, and for consistency this is enforced by a rule with ast-grep.

Formatting and Linting

This section covers repository formatting, pre-commit validations, custom linter rules, and CI quality gates in Arminius.

Getting Started

The supported development environment is the Nix development shell. Entering nix develop or using direnv provides pinned versions of every formatter and linter.

For shell setup and caching details, see the Development Setup.

Common Commands

The following table summarizes the primary tools and their effects:

Optional Git Hook

prek and prek --all-files work without installing a Git hook. To also run those checks automatically on every commit, install the hook from inside the Nix development shell:

prek install

Pre-Merge Quality Gate

Before submitting a merge request or requesting review:

1. Run nix fmt to ensure all files adhere to project formatting.
2. Run prek --all-files to verify repository hygiene, licenses, and spellings.
3. Run ast-grep scan --error to ensure custom architectural rules pass.
4. Run nix flake check to verify static checks locally.

Reference

Configuration File Ownership

Nix modules from the pinned engineering-standards flake generate several repository-root configuration files. Each generated file includes a managed-by: engineering-standards header that forbids hand edits and lists the regenerate command.

Do not edit those files by hand. Instead, change the sources:

• Famedly Engineering Standards
• Local Nix formatter module: nix/formatter-module.nix
• Local pre-commit module: nix/prek-pre-commit.nix

To regenerate configuration files after modifying Nix modules, run:

nix run .#filegen-activate

Formatting

nix fmt invokes treefmt with the complete formatter set required by this repository.

To debug a single formatter, inspect the flags in nix/formatter-module.nix and in the pinned Famedly Engineering Standards treefmt module. Those sources are authoritative for the dev-shell and CI.

Pre-Commit Checks

The repository uses prek for pre-commit checks. Commands are listed under “Common Commands”.

prek also runs ast-grep scan --error for staged Rust files. prek --all-files runs it for all Rust files, but changes limited to .ast-grep/ or sgconfig.yml still rely on the separate flake check.

Linting and Policy Checks

Getting Started

Run all flake checks with:

nix flake check

Note, that this command performs CPU-/memory-intensive operations and takes at least a couple of minutes to complete, even on modern machines.

For ast-grep setup, commands, and custom rule documentation, see our ast-grep documentation.

CI Execution

GitLab CI splits the flake checks into parallel source-checks and rust-checks jobs. Each job builds a named check derivation, for example:

nix build -L .#checks.x86_64-linux.ast-grep-lints

The CI pipeline also runs the pre-commit job separately. nix flake check does not replace that job or run nix fmt.

Rust Lints

Clippy and the compiler lints catch many problems before review. CI turns every warning into an error, so a lint that fires blocks the merge request.

Clippy

The clippy flake check runs Clippy over the whole workspace in the rust-checks CI job. It passes --all-features --all-targets -- --deny warnings. Run the same check in the development shell with:

cargo clippy --all-features --all-targets -- --deny warnings

You can also build the flake check with nix build -L .#checks.x86_64-linux.clippy.

Lint Configuration

Lints are defined in the [workspace.lints.rust] and [workspace.lints.clippy] tables of the root Cargo.toml.

Workspace crates must opt in with [lints] workspace = true in their own Cargo.toml.

.clippy.toml holds options for individual lints.

Documentation Lints

Every item needs a doc comment, whether it is public or private. The lints missing_docs and clippy::missing_docs_in_private_items enforce this.

Module documentation sits inside the module, as //! comments. The ast-grep rule external-module-comments enforces this, and Coding Style shows both forms.

Lint Exceptions

When a lint does not fit one item, silence it for that item with #[expect] and a reason:

#[expect(
	clippy::too_many_lines,
	reason = "Being a big match statement, the complexity doesn't really increase with length that much"
)]
pub fn construct_outgoing_request(&self, /* ... */) { /* ... */ }

Two lints enforce this form. clippy::allow_attributes flags #[allow(...)] and asks for #[expect(...)] instead. clippy::allow_attributes_without_reason flags any exception without a reason.

ast-grep

In this project, we use ast-grep (Website) to implement our own linter rules. The language server command is ast-grep lsp.

Rule Index

sgconfig.yml loads every rule file in .ast-grep/rules. The is-test-function.yml file is a helper used by other rules and is not an independent lint.

Why These Rules Exist

The rules encode repository conventions that compiler and formatter checks do not cover. They keep error reporting, test coverage, tracing, query layout, module structure, and compiler reliability consistent.

Run the Checks

For instructions on setting up the Nix development environment, entering the development shell, and using direnv, see the development setup guide. The Nix development environment already includes ast-grep.

If you do not use Nix, follow the ast-grep installation guide to install ast-grep.

Run the repository lint scan with:

ast-grep scan

This reports lint findings without changing files.

To run the CI-equivalent lint scan, treat all findings as errors:

ast-grep scan --error

If the optional prek Git hook is installed, this scan also runs automatically via prek on staged Rust files.

Run the ast-grep rule fixture tests separately:

ast-grep test

Rules

comment-follows-attribute

What It Does

Check that we consistently order doc-comments before attributes, not after.

Why Restrict This

To ensure our code is consistent.

Example

#[derive(SomeAttribute)]
/// A struct which is cool
///
/// This is a struct which is cool. This implies a temperature below 5°C,
/// except in July, where anything below 35°C is considered cooler than cool.
/// Ice-cold.
pub struct MyCoolStruct;

should instead be expressed as

/// A struct which is cool
///
/// This is a struct which is cool. This implies a temperature below 5°C,
/// except in July, where anything below 35°C is considered cooler than cool.
/// Ice-cold.
#[derive(SomeAttribute)]
pub struct MyCoolStruct;

external-module-comments

What It Does

Check that module documentation is inside the module it documents.

Why Restrict This

Keeping module documentation with the module implementation makes it easier to find and maintain.

Example

// lib.rs
/// Documentation for the foo module.
mod foo;

should instead be expressed as

// lib.rs
mod foo;

// foo.rs
//! Documentation for the foo module.

inconsistent-eyre-return

What It Does

Checks that we consistently use the full path syntax color_eyre::eyre::Result<()> for return values.

Why Restrict This

To ensure our code is consistent.

Example

fn my_doubling_function() -> Result<u8, color_eyre::eyre::Error> {
  u8 * 2
}

should instead be expressed as

fn my_doubling_function() -> color_eyre::eyre::Result<u8> {
  u8 * 2
}

inline-tests

What It Does

Check that every unit test is stored in its own file in the source tree, opposed to being inline.

Why Restrict This

llvm-cov has problems distinguishing test code from production code when counting coverage, resulting in warnings about our tests themselves having insufficient coverage. By having tests in a distinct file, we can make sure that we only check whether production code is covered.

Example

// file.rs
#[cfg(test)]
mod tests {
  #[test]
  fn my_test() {
    assert_eq!(2, 2);
  }
}

should instead be expressed as

// file.rs
mod tests;

// file/tests.rs

#![cfg(test)]

#[test]
fn my_test() {
  assert_eq!(2, 2);
}

or

// file.rs
#[cfg(test)]
mod tests;

// file/tests.rs
#[test]
fn my_test() {
  assert_eq!(2, 2);
}

long-queries

What It Does

Check that we have no inline sqlx::query! invocations which exceed 50 lines.

Why Restrict This

At this length, the queries are nicer to read in their own file. And we get to use pg_format.

Example

sqlx::query!(r#"
  SELECT
    row1,
    row2,
    ...,
    row100
  FROM
      foo_fighters;
"#).await?

should instead be expressed as

sqlx::query_file!("src/fetch_foo_fighters.sql")

missing-context

What It Does

Check that try expressions in non-test functions returning an eyre result have .context().

Why Restrict This

Context gives errors enough information to identify the operation that failed.

Example

fn read_config() -> color_eyre::eyre::Result<String> {
  Ok(std::fs::read_to_string("config.toml")?)
}

should instead be expressed as

fn read_config() -> color_eyre::eyre::Result<String> {
  Ok(std::fs::read_to_string("config.toml").context("reading config.toml")?)
}

missing-install

What It Does

Check that test functions returning color_eyre::eyre::Result call color_eyre::install()?;.

Why Restrict This

Installing color_eyre gives failed tests the same readable error reports as other application code.

Example

#[test]
fn my_test() -> color_eyre::eyre::Result<()> {
  assert_eq!(2, 2);
  Ok(())
}

should instead be expressed as

#[test]
fn my_test() -> color_eyre::eyre::Result<()> {
  color_eyre::install()?;
  assert_eq!(2, 2);
  Ok(())
}

missing-instrument

What It Does

Check that every database query has the #[tracing::instrument] annotation

Why Restrict This

We want every database query to create its own span, such that it’s easier to get the overview of the performance of individual queries.

Example

impl FooQueries : Database {
  async fn fetch_foo(&mut self) -> Result<i32, DatabaseError> {
    /* some code */
    Ok(0)
  }
}

should instead be expressed as

impl FooQueries : Database {
  // Arguments to the query will be logged if not skipped, be mindful of data sensitivity
  #[tracing::instrument(skip(self))]
  async fn fetch_foo(&mut self) -> Result<i32, DatabaseError> {
    /* some code */
    Ok(0)
  }
}

pub-use-parent

What It Does

Check that every instance of pub use only exports child elements. It does this by looking for instances of pub use crate::*. Sometimes, this can actually be legitimate, specifically in lib.rs, since it has no parent modules. But in this case, pub use crate::module can simply be replaced in pub use module, which is easier to read.

Why Restrict This

Having a clear hierarchical structure, where modules are only responsible for exporting their child modules makes visibility rules easier to follow.

Example

// src/lib.rs

mod mod1
mod mod2;

pub use mod2::my_cool_function;

// src/mod1.rs

pub fn my_cool_function() {

}

// src/mod2.rs

pub use crate::mod2::my_cool_function;

should instead be expressed as

// src/lib.rs

mod mod1
mod mod2;
pub use mod1::my_cool_function;

// src/mod1.rs

pub fn my_cool_function() {

}

test-attr-last

What It Does

Check that the attribute marking a function as a test is the last attribute.

Why Restrict This

Keeping the test marker last gives test declarations a consistent form. It also makes it much easier to recognize tests in ast-grep lints.

Example

#[test]
#[tracing::instrument]
fn my_test() {}

should instead be expressed as

#[tracing::instrument]
#[test]
fn my_test() {}

test-without-returntype

What It Does

Check that recognized test functions return color_eyre::eyre::Result<()>.

Why Restrict This

Returning a result gives tests better error reporting than panicking.

Example

#[test]
fn my_test() {
  assert_eq!(2, 2);
}

should instead be expressed as

#[test]
fn my_test() -> color_eyre::eyre::Result<()> {
  color_eyre::install()?;
  assert_eq!(2, 2);
  Ok(())
}

Code Reviews

Every pull request to Arminius requires a review to ensure quality criteria in the pull request are met and to prevent obvious mistakes from slipping in.

Code reviews can be done by anyone, but to merge a PR a review from at least one code owner is necessary. To help with spreading the load a “review roulette” will randomly assign a reviewer. This however does not mean other team members should not review the PR if they have time. It only exists to force one person to deprioritize other tasks to ensure a timely review. Be aware that the review roulette might assign team members, who are currently on vacation or otherwise can’t take up the review duty. In those cases manual intervention is necessary.

Expectations for a Code Review

We expect code reviews to be:

• timely. A PR author is blocked until a review is done as such we prefer to never have anybody wait more than 2 days for a review.
• empathic. The author has spent a lot of effort usually on the code they are presenting for review. At the same time the reviewer only has so much time to spend on reviews. And both sides might have gaps in understanding of design tradeoffs or parts of the code base. As such the author needs to ensure documentation is clear and helpful as well be open to suggestions and the reviewer needs to be open to their suggestions having been tried already and be aware their feedback might be understood different from what they intended.
• positive. Reviews mostly point out faults. This can bias a review to be something negative. However reviews should be considered something positive, where both participants can learn something and issues can be caught before they cause breakage. As such reviewers and PR authors should be open to learning and praise good solutions or suggestions. The tone should generally be collaborative instead of demanding or degrading.
• helpful. Pointing out an issue is helpful. Providing guidance on how to remediate an issue is a lot more helpful. Only giving feedback about an issue might leave the author with an unclear path on how to “please” the reviewer. If you have concrete suggestions or ideas, state them. It reduces frustration and guess work and speeds up back and forth between parties.

To help with this we expect:

• PRs to be only marked as ready for review, when necessary documentation and changes are finished. Reviewing drafts or WIP can lead to a lot of wasted work. Before undrafting your PR also always do a self review of your own work. There is no need to involve others in a review when you can notice obvious issues yourself already.
• Feedback to be complete and actionable. Unclear feedback can never be resolved and holds up reviews.
• Mutual respect between all review participants. There is no benefit to bringing your ego into a review. The benefit of the team and the code base should always be the highest priority in a review over individual preferences.

Pre-Merge Duties

Before any merge request can be merged, every merge request author has to make sure that they have completed all of their pre-merge duties.

• It is the duty of the merge request author to fulfill the list of duties and to mark them as completed
• It is the duty of all merge request reviewers to verify the correct completion of the duties
• If the duties are not fulfilled (the boxes are not all ticked), the merge request in question cannot be merged.

The pre-merge duties checklist is embedded in the merge request template and automatically included in every new merge request.

Documentation

This document documents how we document things. Funny, right?

Documentation Guidelines

Documentation should assume general familiarity with the basic concepts of the Matrix specification. Wherever possible, especially at functions implementing an endpoint of the Matrix specification, a link should be made to the relevant section. The primary purpose of the documentation in Arminius is to document our implementation and choices made when implementing the specification, not documenting the Matrix protocol.

Running Locally

Using Nix

If you have Nix installed, you can render the docs using:

  nix build .#docs
  firefox ./result/index.html

Without Nix

First, install the required crates: cargo install mdbook mdbook-mermaid

Initialize mdbook-mermaid using: mdbook-mermaid install docs/

Build the database documentation (requires a running psql database with applied migrations): ./docs/document_tables/document_tables.sh

And open them in your browser (automatically reloading on change): mdbook serve --open docs/

Formatting

Markdown files are formatted automatically via nix fmt using mdformat and the mdformat-footnote plugin. For repository formatting configuration and exclusions, see Formatting and Linting.

Visual Documentation: Diagrams and Admonitions

This section covers how to enhance your documentation with visual diagrams using Mermaid and highlighted callouts using admonitions.

Mermaid Diagrams

Mermaid is a powerful diagramming and charting tool that renders Markdown-inspired text definitions to create diagrams dynamically. It’s perfect for technical documentation because diagrams are defined as code, making them version-controllable and easy to maintain.

Use Mermaid when you need to:

• Visualize system architecture or data flow
• Document processes, workflows, or decision trees
• Show relationships between components
• Illustrate sequence interactions between services
• Create organizational charts or hierarchies
• Document database schemas or entity relationships

Don’t use Mermaid when:

• You need highly detailed technical drawings
• Custom styling is critical to the diagram’s purpose

Supported Diagram Types

Flowcharts

Perfect for processes, decision flows, and system overviews.

```mermaid
flowchart TD
    A[Start] --> B{Is user logged in?}
    B -->|Yes| C[Show dashboard]
    B -->|No| D[Show login form]
    D --> E[Authenticate]
    E --> F{Valid credentials?}
    F -->|Yes| C
    F -->|No| G[Show error message]
    G --> D
```

flowchart TD
    A[Start] --> B{Is user logged in?}
    B -->|Yes| C[Show dashboard]
    B -->|No| D[Show login form]
    D --> E[Authenticate]
    E --> F{Valid credentials?}
    F -->|Yes| C
    F -->|No| G[Show error message]
    G --> D

When to use: Process documentation, user flows, system logic, troubleshooting guides.

Sequence Diagrams

Ideal for showing interactions between different actors or systems over time.

```mermaid
sequenceDiagram
    participant User
    participant Frontend
    participant API
    participant Database

    User->>Frontend: Submit form
    Frontend->>API: POST /api/users
    API->>Database: INSERT user data
    Database-->>API: Success response
    API-->>Frontend: 201 Created
    Frontend-->>User: Show success message
```

sequenceDiagram
    participant User
    participant Frontend
    participant API
    participant Database

    User->>Frontend: Submit form
    Frontend->>API: POST /api/users
    API->>Database: INSERT user data
    Database-->>API: Success response
    API-->>Frontend: 201 Created
    Frontend-->>User: Show success message

When to use: API documentation, microservice interactions, authentication flows, integration guides.

Entity Relationship Diagrams

Perfect for database design and data modeling.

```mermaid
erDiagram
    CUSTOMER ||--o{ ORDER : places
    ORDER ||--|{ LINE-ITEM : contains
    PRODUCT ||--o{ LINE-ITEM : "ordered in"

    CUSTOMER {
        int id PK
        string name
        string email
        date created_at
    }

    ORDER {
        int id PK
        int customer_id FK
        date order_date
        decimal total
    }
```

erDiagram
    CUSTOMER ||--o{ ORDER : places
    ORDER ||--|{ LINE-ITEM : contains
    PRODUCT ||--o{ LINE-ITEM : "ordered in"

    CUSTOMER {
        int id PK
        string name
        string email
        date created_at
    }

    ORDER {
        int id PK
        int customer_id FK
        date order_date
        decimal total
    }

When to use: Database documentation, data architecture guides, schema migration documentation.

Best Practices for Mermaid

1. Keep diagrams focused: Each diagram should illustrate one concept or process
2. Use descriptive labels: Make node and edge labels clear and meaningful
3. Maintain consistent styling: Use similar formatting across diagrams in your documentation
4. Test rendering: Always verify your diagrams render correctly in your documentation system
5. Version control: Since Mermaid diagrams are text, they can be easily tracked in version control
6. Add context: Always accompany diagrams with explanatory text

Syntax Tips

• Use meaningful IDs for nodes: user_login[User Login] instead of A[User Login]
• Leverage subgraphs for grouping related components
• Keep line lengths reasonable for better readability in source code

Admonitions

Admonitions are highlighted callout boxes that draw attention to important information. They help break up dense text and guide readers to critical details.

Documentation can be found in the mdbook documentation.

Organization of Test Code

We separate tests into

1. unit tests
2. narrower integration tests
3. wider integration tests
4. end-to-end-tests

1. Unit Tests

What?

Unit tests are tests that only test a small internal function.

If it does not need a component to spin up, it can probably go here.

NATS and Database are acceptable here, but if you are using NATS to communicate with other components, those components should be mocked.

Where?

It should go inside the same file as the function that is being tested.

How?

From the project root (this directory), execute:

cargo install cargo-nextest
cargo nextest run

2. Narrower Integration Tests

What?

Integration tests are testing the interaction between two or more components.

It is narrow if the components are all inside the same crate.

Where?

It should go inside the /tests directory of the crate.

How?

The arminius-lib-testing crate provides a test runner and utility functions to spin up and execute integration tests.

3. Wider Integration Tests

What?

Integration tests are testing the interaction between two or more components.

It is Wider if the components are in different crates.

Where?

It should go inside the /tests directory of the monolith crate.

How?

The arminius-lib-testing crate provides a test runner and utility functions to spin up and execute integration tests.

4. End-to-End Tests

What?

If the test is testing the code from the perspective of the users it should go here. Further documentation about the end-to-end tests can be found on the dedicated page.

Where?

crates/tests/arminius-tests-endtoend is your friend.

5. Complement

What?

Warning

Running Complement tests on Darwin is not supported. This is tracked in #2268

What?

The Matrix foundation has Complement, which is their own e2e testing suite for homeservers. We also have support for testing Arminius against Complement. Further documentation about the Complement tests can be found on the dedicated page.

6. Local Federation

What?

Included in Arminius is the local federation utility, to run Arminius in a local federation (optionally also with other homeservers supported by the end-to-end tests).

To run it, ensure you have the Arminius docker images available and:

export E2E_TESTS_BASE_CONFIGURATION=arminius
export CARGO_TARGET_DIR="$(pwd)/target"
cargo run -p arminius-tests-localfederation -- --element-desktop

Two element-desktop sessions will spawn (This requires that you have element-desktop installed, which is not included in the devshell for space reasons. You can leave out the --element-desktop argument, and connect with your own client if you wish.). The one marked as hs1 has an account user1 and the one marked as hs2 has an account user2. Both use Password1! as their password.

To run Synapse alongside of Arminius, you can set E2E_TESTS_HS2_CONFIGURATION=synapse before starting the local federation. Configuring the homeservers used to launch local federation follows the same convention as the E2E test framework.

Coverage

Every job that collects coverage uploads its lcov tracefiles as a GitLab artifact, under a per-job reports/<job>/ directory: the three test slices (arminius-tests, nuss-tests, nuss-e2e-tests), and the end-to-end tests that run against the server containers, whose raw profiles scripts-dev/export-e2e-coverage.sh turns into one tracefile per component.

Each of those jobs also converts its tracefiles into a Cobertura report, which is what annotates the merge request diff - GitLab combines every job’s into one overlay.

The percentage can’t work that way, since GitLab averages what it scrapes per job, weighting a tiny slice like the whole workspace. So merge_coverage merges all the tracefiles with coverage-merge as a line-level union (nix/packages/merge-lcov.py) - a line counts once, and as covered if any job covered it - and that union’s percentage is the headline number.

Locally, one command builds the three slices (or fetches them from the cache) and merges them:

nix build .#coverage-report

# The headline percentage, as CI computes it:
cat result/coverage-summary

# result/ also contains:
#   cobertura.xml - the merged report, as consumed by GitLab
#   merged.info   - the merged lcov, for the HTML report below

This is an approximation of the CI number: the container end-to-end coverage needs a full dockerized run, so it is not part of this derivation. To include it, or to merge any other set of tracefiles, call the same tool the pipeline uses - it takes an output directory and any number of tracefiles:

coverage-merge ./coverage result/lcov.info target/coverage/*.info

For a browsable, line-by-line HTML report, run genhtml over the merged lcov. It ships with lcov, which is not in the devshell (it pulls in a large graphing/font closure), so fetch it on demand:

nix shell nixpkgs#lcov --command genhtml result/merged.info --output-directory coverage-html
# then open coverage-html/index.html

To iterate on a single slice’s coverage without merging, build that slice directly and point genhtml at its lcov.info, e.g.:

nix build .#nuss-tests-coverage
nix shell nixpkgs#lcov --command genhtml result/lcov.info --output-directory coverage-html

End-to-End Testing for Matrix Homeservers

This crate provides a comprehensive (WIP) end-to-end testing solution for Matrix Homeserver implementations. Leveraging the matrix_sdk crate for client-server communication, it enables validation of server behavior against both client application expectations and established Homeserver specifications.

Architecture

The test suite operates through the following workflow:

1. Spawns Docker containers running configured homeserver instances, initializing test environments using blueprint configurations and runtime parameters (set via environment variables).
2. Executes test cases using Matrix client operations via matrix_sdk.

Running the Tests

By default end-to-end tests are disabled inside nextest configuration. To run only the end-to-end-test you can use this command:

cargo nextest run -p arminius-tests-endtoend

Or if you prefer to include it inside the whole test suit of this workspace:

cargo nextest run -E 'all()' --ignore-default-filter

Also, here is a way to compile and run test only for a specific test target

cargo nextest run -p arminius-tests-endtoend --test TEST_TARGET [TEST_NAME]

Please note that for running the tests for Arminius in the current repository, images must be built beforehand.

Using nix you can run the following:

Warning

The below nix build will not work on aarch64-darwin. This is tracked in #2268

# This will build images and generates the script for
# importing them to the docker local registry.
nix build .#containers-arminius-all-test-login

# And then run the script that was generated

./result/bin/import_script

Supported Homeservers

This framework can test multiple homeservers. You can adjust which homeserver you are running using environment variables

arminius

Run Arminius as a series of Docker containers, one for each component. This is the one we run in CI. The binary in the container can be overwritten with one on your local filesystem, in order to hack on Arminius. To do so, set one of the following environment variables to a path on your local system:

• E2E_TESTS_HS{n}_ARMINIUS_ADMIN_BINARY_OVERRIDE
• E2E_TESTS_HS{n}_ARMINIUS_CLIENT_BINARY_OVERRIDE
• E2E_TESTS_HS{n}_ARMINIUS_FEDERATION_INPUT_BINARY_OVERRIDE
• E2E_TESTS_HS{n}_ARMINIUS_FEDERATION_OUTPUT_BINARY_OVERRIDE
• E2E_TESTS_HS{n}_ARMINIUS_ROOMS_BINARY_OVERRIDE

When using binary overrides, it might make sense to have a “blank” docker image, into which you mount your binary override. To build a blank docker image using Nix, run nix build .#arminius-blank-image && ./result | docker load.

native (Default)

Run Arminius, but instead of multiple Docker containers, run the Monolith mode in the same process. This is useful for development, due to generally being faster to start than building and running a Docker container.

continuwuity

Runs the official Continuwuity Docker image.

Currently this is not tested in the CI, but we will probably change this at some point.

synapse

The Synapse homeserver, in a Docker container.

Configuration Management

Environment variables control test configuration:

Global Configuration

• E2E_TESTS_DISALLOW_DEFAULT_IMAGES: If set, require that the configuration and image environment variables are set explicitly
• E2E_TESTS_BASE_CONFIGURATION: Homeserver implementation to test (Options: native, arminius, synapse | Default: native)
• E2E_TESTS_BASE_SYNAPSE_IMAGE: Container image name:tag (Synapse default: matrixdotorg/synapse:latest)
• E2E_TESTS_BASE_ARMINIUS_ADMIN_IMAGE: Container image name:tag (Default: arminius-admin:latest)
• E2E_TESTS_BASE_ARMINIUS_CLIENT_IMAGE: Container image name:tag (Default: arminius-client:latest)
• E2E_TESTS_BASE_ARMINIUS_ROOMS_IMAGE: Container image name:tag (Default: arminius-rooms:latest)
• E2E_TESTS_BASE_ARMINIUS_FEDERATION_INPUT_IMAGE: Container image name:tag (Default: arminius-federation-input:latest)
• E2E_TESTS_BASE_ARMINIUS_FEDERATION_OUTPUT_IMAGE: Container image name:tag (Default: arminius-federation-output:latest)
• E2E_TESTS_BASE_ARMINIUS_NATS_IMAGE: Container image name:tag (Default: nats:latest, per testcontainers)
• E2E_TESTS_BASE_NGINX_IMAGE: Container image name:tag (Default: nginx:latest)
• E2E_TESTS_BASE_ARMINIUS_POSTGRES_IMAGE: Container image name:tag (Default: postgres:latest, per testcontainers)
• E2E_TESTS_BASE_CONTINUWUITY_IMAGE: Container image name:tag (Default: ghcr.io/continuwuity/continuwuity:latest, per testcontainers)

Per-Role Overrides

Imagine these scenarios:

1. You want your test to send a message from a Homeserver using Synapse implementation to one that is using Arminius implementation.
2. You want your test to send message from an earlier version of Arminius to a newer version of it.

In these scenarios, the tests that spin up two different Homeservers, assign a number to each homeserver, calling it a role id. And for each role id you can overwrite the global and default homeserver implementation and image for each. This way, you can configure individual Homeservers using:

• E2E_TESTS_HS{role_id}_CONFIGURATION: Configuration override (Applies to homeserver with specified u8 role identifier)
• E2E_TESTS_HS{role_id}_SYNAPSE_IMAGE: Image override (Applies to homeserver with specified u8 role identifier)
• E2E_TESTS_HS{role_id}_ARMINIUS_{component}_IMAGE: Image override (Applies to homeserver with specified u8 role identifier). The components are:
  • ADMIN
  • CLIENT
  • FEDERATION_INPUT
  • FEDERATION_OUTPUT
  • ROOMS
  • NATS
  • NGINX
  • POSTGRES

The test suite uses fallback behavior:

1. Attempt role-specific configuration
2. Fall back to global configuration if:

• Role-specific variable undefined
• Invalid configuration detected

Debugging Tips and Tricks

Filtering Logs

For example, to filter logs from one of the two homeservers:

RUST_LOG="arminius_tests_endtoend::container[homeserver{homeserver_role=2}]=DEBUG"

Accessing Synapse’s Database

If you want to inspect a end-to-end test’s resulting database state for Synapse:

1. Throw a sleep in the test where you want to inspect the database state at.
2. Identify the running docker container using docker ps
3. Copy the database file from the container using docker cp <CONTAINER_ID>:/data/homeserver.db .
4. Run a nix shell that has sqlite3 using nix-shell -p sqlite
5. Open the database using sqlite3 ./homeserver.db

Requirements

• Docker or podman daemon running locally
• User must have a docker access without a need for root privileges
• Available TCP ports in range 8000-9000

Implementation Notes

The test infrastructure utilizes testcontainers-rs for container management.

A Word of Caution Regarding Running Tests with Arminius

Running the tests for arminius means that it may run tens of containers at the same moment, which can be quite RAM intensive.

Cleanup

If tests are stopped unexpectedly, for instance with CTRL+C, the test suite will not clean up running containers, and this will have to be done manually.

# Remove containers
docker system prune -af --filter 'label=project.arminius.env=e2e'
# Remove volumes
docker volume ls -q | grep '^e2e-' | xargs -r docker volume rm

Images

The end to end tests pull several images to run the matrix homeservers. For the synapse target, it pulls matrixdotorg matrixdotorg/synapse, unless overwritten by an environment variable, while for the arminius target, it pulls nats and postgres.

Complement

Warning

Running Complement tests on Darwin is not supported. This is tracked in #2268

The Matrix foundation has Complement, which is their own e2e testing suite for homeservers. We also have support for testing Arminius against Complement.

In nix/packages/complement_utils/failing_csapi_tests.txt and nix/packages/complement_utils/failing_federation_tests.txt, we store the complete list of Complement test cases which currently fail in Arminius.

Usage

In the nix shell, three utility scripts are available in your path:

• complement-csapi-runner and complement-federation-runner build Arminius as a Complement base image and run all client-server or server-server tests not in the denylists, respectively. It has two modes (auto-detected from DOCKER_HOST, overridable with COMPLEMENT_IN_DIND=true/false): locally it runs the Complement binaries directly against your docker daemon; in CI on the GitLab Docker executor it runs Complement inside a container on the DinD daemon (--network host + mounted socket), because the DinD service there is network-isolated. In the dev shell just run complement-csapi-runner or complement-federation-runner; otherwise nix run .#complement-csapi-runner or .#complement-federation-runner. See nix/packages/complement_utils/default.nix and nix/packages/complement_runner_image.nix.
• complement-csapi-hacker builds and mounts arminius-server-monolith into a “bare” Arminius Complement image, and runs the Complement client-server tests with your given arguments. Example: complement-csapi-hacker -test.run TestVersionStructure
• complement-federation-hacker mounts your monolith build from $CARGO_TARGET_DIR into a “bare” Arminius Complement image, and runs the Complement server-server tests with your given arguments. Example: complement-federation-hacker -test.run TestInboundFederationKeys

The -hacker scripts require a reachable local Docker daemon and that the CARGO_TARGET_DIR environment variable is set explicitly. For example, this invocation of complement-federation-runner runs only the federation test TestACLs:

export CARGO_TARGET_DIR="$(pwd)/target"
complement-federation-hacker -test.run '^TestACLs$'

-test.run accepts a regular expression, so the anchors prevent this command from also running TestACLsForEDUs.

If the environment variable “COMPLEMENT_DIR” is set, the Complement checkout in the given directory will be used, instead of the Complement version defined by the Nix shell.

Firewall

Some federation tests in Complement make use of a mocked Matrix server running on the host machine, instead of a docker container. To make the mock server reachable from Arminius when Arminius is being run under Complement, you may need to modify your firewall rules. On NixOS, this can be achieved by:

networking.firewall.extraCommands = "iptables -A INPUT -i br-+ -j ACCEPT";

See also the upstream Complement documentation.

Troubleshooting

/override/arminius-server-monolith: cannot execute: required file not found

This error results from the container not containing the dynamic libraries needed by the arminius-server-monolith binary. Typically, this results from an old version of the arminius-empty-complement image, which has dynamic libraries not matching those of the devshell. You just need to delete the old arminius-empty-complement images:

docker image ls --filter 'reference=arminius-empty-complement*' --format='{{.ID}}' | xargs docker image rm

Database

The crate arminius-lib-db contains our basic interface to execute queries and manage transactions

Queries are meant to be defined in their own crates. The name of such a crate should adhere to the pattern arminius-lib-db-<table-or-partition-name>. As a rule of thumb, one such crate should be created per table. However, there may be situations where it makes sense to have one crate being responsible for multiple tables, if those are so closely related to each other that it makes no sense to use them in isolation.

Application code may only depend on arminius-lib-db / arminius-lib-db-* to interface with the database, but must not use sqlx directly.

See the module-level documentation in arminius-lib-db/src/lib.rs for more details and examples.

Query Functions

• Functions should use the statically checked query functions from sqlx - sqlx::query!, sqlx::query_as!, sqlx::query_file!, and family

• Function names must be declarative and indicate the type of operation. For example:

  • Select statements must start with fetch_
  • Insert statements with insert_
  • Delete statements with delete_
  • Update statements with update_
  • Conditional insert or update statements with upsert_

• Be explicit about which columns you select. SELECT * might be tempting, but will possibly select more columns than expected and will silently change the columns returned when the schema changes.

• Query functions must return SQL-related errors, not API errors. This enforces a clear separation between database logic and API layers.

• Functions must not accept or return any complex Ruma-api-specific structs (e.g. Ruma Responses). Only primitive types or simple domain types that don’t implement complex functionality should be used for input.

• Functions should only implement database operations, not entire endpoints.

• Functions should use strong types for their parameters where possible. Wrappers around primitive types that just validate the format of the input are explicitly encouraged.

  • Example: ruma::OwnedUserId is a wrapper around String that verifies the string is a correct MXID

• Function must not return tuples. Tuples force the caller to remember (or look up) the meaning of each position, which hurts readability and maintainability. Structs also provide better forward compatibility. If the function needs to return an additional field, this is easier to maintain with structs than with tuples.

• The sqlx method fetch_optional is generally preferred to fetch_one.

  • Whereas fetch_one returns a Result<_, sqlx::Error> where the error may be sqlx::Error::RowNotFound, fetch_optional returns a Result<Option<_>, sqlx::Error>, where sqlx::Error::RowNotFound is ruled out. This allows us to explicitly handle the case of a missing row, and consider all other database errors unrecoverable. With fetch_one it is far too easy to write code that returns a 500 when it should return a 404

• sqlx is not perfect at telling when a column may or may not be null. If you know a query can’t be null, prefer forcing non-null by writing SELECT column as "column!" FROM function() as your query, instead of handling it in Rust.

Tip

The sqlx::query! macro returns anonymous structs, so their type cannot be described. If you find yourself in this situation:

async fn fetch_all_users(conn: &mut PgConnection) -> Result</* WTF DO I PUT HERE */, sqlx::Error> {
 sqlx::query!("SELECT column FROM  users").fetch_all(conn).await
}

you should use the query_as! macro in cases where it is easily possible to map records into their respective representation without doing any sophisticated kind of transformation. Alternatively your function might use query! internally as long as it immediately converts the return type to a named type, but query_as! is preferred where possible.

• query functions should be traced using the tracing::instrument macro to give us insights into the database usage. Long term we will refine the rules around that or move to automated instrumentation for those queries to limit what data is collected in those traces, but until then all query functions should generally be instrumented.

Disclaimer: This project is still under active development. If better conventions are identified in the future, this documentation will be updated.

Related Pages

• Database Architecture – schema conventions, table rules, crate structure
• Storing JSON in PostgreSQL – json vs jsonb decisions

Error Handling

How an application handles errors ultimately also decides how the application fails. As we strive to produce very reliable software, it is important we get our error handling right. So this document includes some best practices for error handling that this project should follow.

In general this follows a priority list, although exceptions might exist:

1. Errors that can be avoided should be avoided. E.g. use strongly typed function parameters instead of parsing inputs repeatedly.
2. If errors can be handled gracefully they should be handled. Errors should be handled as closely to the source as possible.
3. Errors that can’t be handled should be clear, descriptive, but also not leak sensitive information across external API boundaries.
4. If nobody can handle an error, the application should fail early and predictably.

Panicking

In the ideal version of our code, a panic never happens, no matter how badly a user may mess up. Any code that the compiler thinks may panic, should have an annotation, describing why this can never happen:

  let a = vec![0];
  // SAFETY: We just created a vector with one element, so it will have a first
  // element
  #[allow(clippy::unwrap_used)]
  let first = a.get(0).unwrap();

Instead, we generally use the thiserror crate to make it easier to define errors.

In services, we try to use thiserror for custom error types when it makes sense to do so. color_eyre may also be used, but this should be done sparingly and only when a custom error type is too much boilerplate than it’s worth. When handling an error with color_eyre, the actual error must be passed, any text representation of the error should be attached with .context().

For invalid configuration, we also don’t use panics, for aesthetic reasons. The code

panic!("Missing environment variable");

returns

thread 'main' panicked at /app/example.rs:2:5:
Missing environment variable
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

which is just down-right ugly. Instead, with an error type, we can use the typing system for our errors, define them centrally, and completely control how they’re displayed.

Parse Don’t Validate

To avoid errors as much as possible, we should use strongly typed parameters. This means that the function:

pub fn add_sender_to_event(event: &mut Event, sender: String) -> Result<(), SenderUserIdIsInvalid>

is bad. Instead, we should prefer

pub fn parse_user_id(user_id: String) -> Result<UserId, UserIdIsInvalid>

pub fn add_sender_to_event(event: &mut Event, sender: UserId)

This has the advantage that all error handling, in regards to the format of the user ID that we are working with, is done only once, instead of every function having to handle the error.

Low-Level Library Functions.

For low-level library functions, that may be used in multiple places by different components in Arminius, it’s often a good idea to define one error type per function. In some cases similar functions might share an error type, if they fail in the same ways. Such as in crates/libs/arminius-lib-roomservice-client/src/subject_builder.rs::

/// Errors that occur when unescaping a NATS token
#[allow(missing_docs)]
#[derive(Debug, thiserror::Error, Serialize, Deserialize, PartialEq)]
pub enum TokenEscapeError {
	#[error("Unknown escape code in token {0}")]
	UnknownEscapeCode(char),
	#[error("Last character of escaped token is backslash")]
	LastCharIsBackslash,
}

/// Unescape a string that has been replaced with [escape_token]
fn unescape_token(escaped_token: &str) -> Result<String, TokenEscapeError>

This code is nice, because it’s modular. Everything you need to know about the unescape_token function, and how it can go wrong is contained in the same file as the function, which makes it easy to find, and easy to modify without it impacting the rest of our code.

Returning Errors to Users

Note

For more information on #[derive(MatrixApiError)], see the relevant rustdocs

Error types that return to users should implement the MatrixApiError trait (when returned over an API specified by the Matrix specification). A good example is our implementation of the Matrix endpoint GET /_matrix/client/*/user/{userId}/account_data/{type}:

/// Error for when unset account data is queried.
#[derive(thiserror::Error, Debug, MatrixApiError)]
#[error("Account data not found")]
#[matrix_api_error(status_code = StatusCode::NOT_FOUND, error_kind = ErrorKind::NotFound)]
struct AccountDataNotFound;

#[allow(missing_docs, clippy::missing_docs_in_private_items)]
#[derive(thiserror::Error, Debug, MatrixApiError, FromDbError)]
enum GetAccountDataError {
	/// Unrecoverable error.
	/// See <https://bundesmessenger.gitlab.io/backend/neo/arminius/arminius/03-development/errors.html#unrecoverable-errors>
	#[error(transparent)]
	InternalServerError(#[from] InternalServerError),
	#[error(transparent)]
	AccountActionNotAuthorized(#[from] AccountActionNotAuthorized),
	#[error(transparent)]
	AccountDataNotFound(#[from] AccountDataNotFound),
}

/// Endpoint for getting global account data.
async fn global_account_data_get(
	database: State<Database>,
	AuthenticatedClaims(claims): AuthenticatedClaims,
	RumaRequest(request): RumaRequest<get_global_account_data::v3::Request>,
) -> Result<ArminiusResponse<get_global_account_data::v3::Response>, GetAccountDataError> {
	auth::check_user_matches(claims.user_id.as_ref(), request.user_id.as_ref())?;

  let time_tomorrow = chrono::Utc::now()
    .checked_add_days(1)
    .unrecoverable_error("Timestamp incorrect, near overflow")?;

	Ok(ArminiusResponse(get_global_account_data::v3::Response::new(Raw::from_json(
		db::account_data::fetch_global_account_data_by_user_and_type(
			database.get_pool(),
			request.user_id.as_str(),
			&request.event_type.to_string(),
		)
		.await?
		.ok_or(AccountDataNotFound)?,
	))))
}

To define an individual error type, we define the struct AccountDataNotFound. We use the thiserror::Error derive macro to make a rust error type, which uses the error attribute to set an error message. We then use the MatrixApiError trait to extend this error type to a MatrixAPIError, which using the attribute matrix_api_error will have the correct HTTP status code, and matrix error kind when turned into a response. The error message is taken from the Rust Error.

For the function global_account_data_get, we then define an enum of the errors this function can return (these enums will often end up shared between functions that return similar errors). Using the MatrixApiError macro, we implement MatrixApiError for the enum. When GetAccountDataError is turned into a response, it will use the MatrixApiError trait of the corresponding value, to create the HTTP response.

In the function, we return errors at three places:

• On the first line, we call check_user_matches, which may return AccountActionNotAuthorized, handling it simply with ?
• Further down, we get the time tomorrow. If adding a single day to the current time overflow, we cannot recover from this, and use .unrecoverable_error (See the next section)
• Further down, we use the From<DatabaseError> implementation provided by the #[derive(FromDbError)] attribute on GetAccountDataError, as a substitute for .unrecoverable_error
• Finally, if the database returns a None, we return AccountDataNotFound

Tip

You don’t have to implement one error type for each handler. In client-component/src/account_data.rs, the setters and getters for global and room-local account data ended up with the same error type.

Unrecoverable Errors

Note

For more information about InternalServerError, see the relevant rustdoc

In some cases the type system does not allow us to express, that certain errors should never happen. An example of this might be that a string in the database does not follow the usual schema (e.g. a wrongly formatted user ID, such as ":jim@homeserver.gov"). If an administrator manually modified those strings to not follow our schema, we have no way to recover from this.

For these kind of errors, we create an InternalServerError. For any Result type, we have an extension trait which provides the function unrecoverable_error(self, context: str), which maps any error into an InternalServerError. If there is an error, the context string and callsite will be logged, and the user will get

  {
		"errcode": "M_UNKNOWN",
		"error": "Internal server error"
	}

Errors in Tests

To make tests easier to develop, we have all our tests return color_eyre::eyre::Result<()>. Instead of having to write .expect("some message") after every theoretically fallible function which should never actually fail, we can just use the try-operator ?. And we get prettier output, that’s easier to parse at a glance. There is an ast-grep rule that checks whether all test functions actually return color_eyre::eyre::Result<()>.

Telemetry

We aim to have the server be fully observable to developers and administrators without compromising data protection. The exact approach is still getting refined, but we have some basic rules in the mean time.

Having different versions of the otel tracing crates in the dependency tree can cause problems with the tracing infrastructure like configuration not applying properly.

You can test for this using `cargo tree -i tracing-opentelemetry -v`.

Traces

• We collect traces for every incoming http request. This is handled automatically using an Axum layer. This also accepts external tracing headers allowing for cross-service traces.
• We generally try to trace every database call to give us basic performance insights. Long term we might lower that for performance reasons or to reduce data ending up in traces.
• We trace requests across NATS by inserting and extracting tracing headers. Necessary utilities can be found in arminius-lib-nats in the opentelemetry module.

Logs

Logs are automatically forwarded to OTel when configured.

Currently logs are not automatically attached to tracing spans, but they will be once we depend on tracing-opentelemetry 0.32 or higher: https://github.com/tokio-rs/tracing-opentelemetry/releases/tag/v0.32.0

Metrics

TBD. We currently don’t have a structured approach to metrics yet.

Continuous Integration

As this project is developed on GitHub, we use GitHub Actions as our continuous integration (CI).

Nix Efforts

Currently, an experiment is underway to convert parts of our CI to Nix. The reasons are:

• Nix workflows are often faster, due to clever caching
• They are easier to adapt for workflows that require air-gapping
• Less vendor lock-in to GitHub - On any CI provider, you will only have to put nix flake check in your yaml
• If you have Nix installed on your computer, you can run it locally with ease, compared to GitHub Actions-defined Workflows which needs a 100GiB container
• Nix allows for deterministic builds, which makes our CI less likely to flake based on updates of our dependencies we are not pinning properly

For people who cannot or will not install Nix, this should have little difference compared to GH Actions. Instead of modifying the GitHub Actions YAML, and pushing to a pull request to see whether it works, they can simply edit whatever code the Nix code is calling to see whether it works.

nehwgeneration

nehwgeneration–ngen for short–is a code generation utility, converting OpenAPI 3.1 spec to Rust Axum route handler function signatures, mapping to existing types, instead of generating its own. It uses flexible configuration files to control how OpenAPI schemas, properties, and parameters are translated into idiomatic Rust types.

On the following pages, you can learn more about configuring and using nehwgeneration.

Writing Configuration Files

nehwgeneration is a type-mapping based code generator. This means, that while it generates server code for you, it is your task to tell it which struct, enums and co. you’d like to use. This is done by writing a config file, specifying matching-rules, and finally, writing conversion rules.

This chapter will teach you how to write both simple and intricate configuration files for ngen.

General Structure

Love it or hate it, ngen uses TOML as its configuration language. ngen requires a reference to a singular, well-formed configuration file to run.

This chapter goes in-depth about what this config file looks like, explaining what makes a config file “well-formed” and what to avoid.

ngen does not assume a particular ordering of configuration file sections.

Header

version = 1

The “header” section of an ngen file is currently very simple, requiring a field version, which holds an integer value from 0 to 255 (inclusive).

This field currently does not serve any purpose other than future proofing. Its intended purpose is to be able to facilitate programmatic distinction between configuration file versions, should this be needed in the future.

general

Every ngen configuration file must contain a section [general], formatted as such:

[general]
# May be empty; array of conversion rules
convert = []
# Boolean value; whether the rules in the `convert` array are
# being considered when generating type mappings
enabled = true
# String, valid path; Where the auto-generated code should be placed.
output = "./example/src/auto_gen/matrix_v113.rs"

Fallback Behavior

Read the chapter “Priority Resolution” in “Route Matching” to learn more.

output

The path in the output directory must be readable and writable by the operating system user executing ngen. Both absolute and relative paths are allowed. The file that is being pointed to in the path will be replaced entirely with the auto-generated code generated by ngen.

For more information on what happens to this file and parent mod.rs files, see the “Generating Code” chapter.

Custom Rulesets

In addition to the general ruleset, an unlimited amount of custom rulesets may be present in the configuration file. Custom rulesets allow you to apply different conversion rules to specific groups of API routes based on regex pattern matching.

Custom rulesets are formatted as such:

[admin_api]         # String, unique per config file; Name of custom ruleset
include_routes = [] # Array of String; Regex patterns of route paths to include in this ruleset
exclude_routes = [] # Array of String; Regex patterns of route paths to exclude from this ruleset
convert = []        # Array of Object; Conversion rules to apply to all matched routes

Route Matching

Custom rulesets use regex patterns to determine which OpenAPI routes they should apply to:

• include_routes: Array of regex patterns. If any pattern matches a route path, that route is included in this ruleset (unless explicitly excluded).
• exclude_routes: Array of regex patterns. If any pattern matches a route path, that route is excluded from this ruleset, even if it matches an include pattern.

Exclude patterns take precedence over include patterns.

Example

[matrix_client_v1]
include_routes = ['^/_matrix/client/v1/.*$']  # Include all Matrix client v1 routes
exclude_routes = ['^/_matrix/client/v1/admin/.*$']  # But exclude admin routes
convert = [
    { from = "roomId", to = "RoomId" },
    { from = "userId", to = "UserId" }
]

[admin_api]
include_routes = [
    '^/admin/.*$',                              # All admin routes
    'specific/route/\{id\}/actions'             # Plus this specific route
]
exclude_routes = ['^/admin/health$']            # Except health check
convert = [
    { from = "adminId", to = "AdminUserId" }
]

Priority and Fallback Behavior

Rulesets have priority based on their specificity and definition order:

1. Custom rulesets are processed first, in the order they appear in the configuration file
2. General ruleset acts as a fallback for routes not matched by any custom ruleset
3. Rule inheritance: If the general ruleset is enabled, its rules are merged into custom rulesets, but custom ruleset rules take precedence

Priority Example

[general]
convert = [
    { from = "userId", to = "String" },      # Fallback mapping
    { from = "roomId", to = "String" }       # Fallback mapping
]
enabled = true

[matrix_client_v1]
include_routes = ['^/_matrix/client/v1/.*$']
convert = [
    { from = "roomId", to = "RoomId" }       # Overrides general ruleset for roomId
    # userId inherits from general ruleset (maps to "String")
]

Best Practices

1. Order matters: Define more specific rulesets before general ones
2. Use descriptive names: Ruleset names should clearly indicate their purpose
3. Test regex patterns: Ensure your include/exclude patterns match the intended routes. You can use https://regex101.com/ to help with this. Don’t forget to set the regex engine flavor to “Rust”!

Route Matching

Route matching is the process by which ngen determines which conversion rules to apply to specific OpenAPI endpoints. This system uses regex pattern matching to provide flexible and powerful control over code generation.

Overview

Route matching works through a two-stage process:

1. Pattern Matching: Each custom ruleset defines include_routes and exclude_routes patterns
2. Priority Resolution: ngen determines which ruleset has the highest priority for each route

Pattern Syntax

Route patterns use standard regex syntax to match OpenAPI path strings. The matching is performed against the full path as it appears in the OpenAPI specification.

Basic Patterns

[api_routes]
include_routes = [
    "^/api/.*$",           # All routes starting with /api/
    "^/users/[0-9]+$",     # User routes with numeric IDs
    "/specific/route$"      # Exact match for specific route
]

Path Parameter Handling

OpenAPI path parameters (like {userId}) appear as literal text in the patterns:

[user_routes]
include_routes = [
    "^/users/\\{userId\\}/.*$",      # Routes under /users/{userId}/
    "^/rooms/\\{roomId\\}/events$"   # Specific parameterized route
]

Note: Curly braces {} are regex metacharacters, so they must be escaped with \\{ and \\} when you want to match them literally.

Common Pattern Examples

API Versioning

# Match all v1 routes
[api_v1]
include_routes = ["^/api/v1/.*$"]

# Match all v2 routes
[api_v2]
include_routes = ["^/api/v2/.*$"]

# Match multiple versions
[legacy_api]
include_routes = ["^/api/v[12]/.*$"]

Feature-Based Routing

# User management endpoints
[user_management]
include_routes = [
    "^/users/.*$",
    "^/auth/.*$",
    "^/profile/.*$"
]

# Admin endpoints
[admin_routes]
include_routes = ["^/admin/.*$"]
exclude_routes = ["^/admin/health$"]  # Exclude health check

Matrix Protocol Example

# Matrix Client-Server API v1
[matrix_client_v1]
include_routes = ["^/_matrix/client/v1/.*$"]
exclude_routes = [
    "^/_matrix/client/v1/admin/.*$",     # Admin routes handled separately
    "^/_matrix/client/v1/health$"        # Health check
]

# Matrix Server-Server API
[matrix_federation]
include_routes = ["^/_matrix/federation/.*$"]

Include vs Exclude Logic

The matching logic follows these rules:

1. Include First: Route must match at least one include_routes pattern
2. Exclude Override: If route matches any exclude_routes pattern, it’s excluded
3. No Include = Match All: Empty include_routes array matches all routes

Truth Table

Example Behavior

[example_ruleset]
include_routes = ["^/api/.*$", "^/users/.*$"]
exclude_routes = ["^/api/internal/.*$", "^/users/admin/.*$"]

Given these patterns:

• /api/posts → ✅ Included (matches include, no exclude match)
• /api/internal/logs → ❌ Excluded (matches both, exclude wins)
• /users/123 → ✅ Included (matches include, no exclude match)
• /users/admin/settings → ❌ Excluded (matches both, exclude wins)
• /health → ❌ Excluded (no include match)

Priority Resolution

When multiple rulesets could apply to the same route, ngen uses a priority system:

Priority Order (Highest to Lowest)

1. Custom rulesets in configuration file order (first wins)
2. General ruleset as fallback

Priority Example

[general]
convert = [{ from = "id", to = "String" }]
enabled = true

[specific_api]      # Higher priority (defined first)
include_routes = ["^/api/v2/.*$"]
convert = [{ from = "id", to = "ApiV2Id" }]

[generic_api]       # Lower priority (defined second)
include_routes = ["^/api/.*$"]
convert = [{ from = "id", to = "GenericId" }]

For route /api/v2/users:

• Both specific_api and generic_api match
• specific_api wins due to higher priority
• id fields map to ApiV2Id

Advanced Matching Techniques

Negative Lookahead

Use regex negative lookahead to create complex exclusions:

[non_admin_routes]
include_routes = ["^/api/(?!admin).*$"]  # All /api/ routes except /api/admin/*

Multiple Route Groups

Combine multiple patterns for complex routing:

[content_api]
include_routes = [
    "^/posts/.*$",
    "^/comments/.*$",
    "^/media/.*$",
    "^/tags/.*$"
]
exclude_routes = [
    ".*/draft/.*",      # Exclude any draft routes
    ".*/internal/.*"    # Exclude any internal routes
]

Case Sensitivity

Regex patterns are case-sensitive by default. Use case-insensitive flags if needed:

[case_insensitive]
include_routes = ["(?i)^/API/.*$"]  # Matches /api/, /API/, /Api/, etc.

Debugging Route Matching

Testing Patterns

To test if your patterns work correctly:

1. Use regex tools: Test patterns with online regex testers
2. Check logs: ngen provides debug logging about route matching
3. Start simple: Begin with broad patterns and refine

Common Mistakes

1. Forgetting anchors: Use ^ and $ to match full paths

   # Wrong: matches any route containing "api"
   include_routes = ["api"]
   
   # Right: matches routes starting with /api/
   include_routes = ["^/api/.*$"]
   

2. Unescaped special characters:

   # Wrong: braces are regex metacharacters
   include_routes = ["/users/{id}"]
   
   # Right: escape literal braces
   include_routes = ["/users/\\{id\\}"]
   

3. Overly broad exclusions:

   # Dangerous: might exclude too much
   exclude_routes = [".*admin.*"]
   
   # Better: more specific pattern
   exclude_routes = ["^/admin/.*$", ".*/admin/.*$"]
   

Best Practices

1. Order rulesets by specificity: Most specific first, most general last
2. Use descriptive names: Make ruleset purposes clear
3. Test thoroughly: Verify patterns match intended routes
4. Document complex patterns: Add comments explaining regex logic
5. Keep it simple: Prefer multiple simple patterns over one complex pattern
6. Use anchors: Always use ^ and $ for precise matching

Conversion Rules

Conversion rules are the core of ngen’s type mapping system. They define how OpenAPI schema types, properties, and parameters are converted into Rust types during code generation.

Overview

A conversion rule maps a source type or property (from OpenAPI) to a target Rust type. Rules can match based on:

1. Property names - Direct mapping by field name
2. Schema titles - Mapping entire schemas by their title
3. JSON types - Mapping by primitive JSON types
4. Property-based matching - Advanced matching based on schema structure
5. Payload filtering - Applying rules only to specific HTTP methods or status codes

Basic Rule Structure

{ from = "source", to = "target" }

• from: What to match (property name, schema title, or type pattern)
• to: Target Rust type to generate

Warning

When specifying the to mapping, you should use fully qualified Rust type names, except for core primitive types like String, u64, bool, etc.

For example:

• Use chrono::DateTime<Utc> instead of DateTime<Utc>
• Use std::collections::HashMap<String, String> instead of HashMap<String, String>
• Use uuid::Uuid instead of Uuid

If you do not do this, the resulting, generated code will not compile.

Simple Examples

[general]
convert = [
    { from = "userId", to = "UserId" },           # Property name mapping
    { from = "email", to = "String" },            # Property to primitive type
    { from = "UserRequest", to = "CreateUser" },  # Schema title mapping
]

Property Name Matching

The most simple type of rule matches properties by their name:

convert = [
    { from = "user_id", to = "UserId" },
    { from = "room_id", to = "RoomId" },
    { from = "timestamp", to = "chrono::DateTime<Utc>" },
    { from = "is_admin", to = "bool" },
]

This maps any OpenAPI property with these names to the corresponding Rust types.

Example

The following JSON object would match the above rules:

{
  "user_id": "@alice:example.com",
  "room_id": "!abc123:example.com",
  "timestamp": "2024-01-15T10:30:00Z",
  "is_admin": true
}

The properties would be converted to:

• user_id → UserId type
• room_id → RoomId type
• timestamp → chrono::DateTime<Utc> type
• is_admin → bool type

Schema Title Matching

Schema title matching applies to entire OpenAPI schemas:

convert = [
    { from = "CreateUserRequest", to = "UserCreateRequest" },
    { from = "UserResponse", to = "User" },
    { from = "ErrorResponse", to = "ApiError" },
]

When ngen encounters a schema with title: "CreateUserRequest", it generates a Rust type named UserCreateRequest.

Type-Based Matching

Type-based rules match OpenAPI properties by their JSON schema type using the t# prefix:

convert = [
    { from = "t#string", to = "String" },
    { from = "t#int", to = "i64" },
    { from = "t#bool", to = "bool" },
    { from = "t#array", to = "Vec<T>" },
    { from = "t#object", to = "HashMap<String, Value>" },
]

These act as fallback rules when no specific property name matches.

Available JSON Types

• t#string - String values
• t#int - Integer values
• t#number - Numeric values (integers and floats)
• t#bool - Boolean values
• t#array - Array/list values
• t#object - Object/map values
• t#null - Null values

Property-Based Matching

Property-based matching allows rules to match schemas based on their internal structure using where clauses:

convert = [
    # Match schemas that have ALL specified properties
    { from = "UserRequest", to = "CreateUserRequest", where = {
        all_properties = ["name", "email", "password"]
    }},

    # Match schemas that have EXACTLY these properties (no more, no less)
    { from = "LoginRequest", to = "AuthRequest", where = {
        exact_properties = ["username", "password"]
    }},

    # Match schemas that have ANY of these properties
    { from = "PaymentRequest", to = "PaymentMethod", where = {
        any_properties = ["credit_card", "paypal_email", "bank_account"]
    }},
]

Where Clause Options

all_properties

Schema must contain all listed properties (but may have additional ones):

{ from = "UserSchema", to = "User", where = {
    all_properties = ["id", "name"]
}}

Matches schemas with id, name, and optionally other properties.

exact_properties

Schema must contain exactly these properties (no more, no less):

{ from = "LoginSchema", to = "LoginRequest", where = {
    exact_properties = ["username", "password"]
}}

Only matches schemas with exactly username and password properties.

any_properties

Schema must contain at least one of the listed properties:

{ from = "ContactSchema", to = "ContactInfo", where = {
    any_properties = ["email", "phone", "address"]
}}

Matches schemas with email OR phone OR address (or any combination).

Combined Conditions

You can combine multiple where conditions:

{ from = "OrderSchema", to = "OrderRequest", where = {
    all_properties = ["customer_id", "items"],    # Must have these
    any_properties = ["credit_card", "paypal"]    # Plus at least one payment method
}}

Type-Based Property Matching

Property-based matching also works with type patterns:

{ from = "StringOnlyRequest", to = "TextRequest", where = {
    all_properties = ["t#string"]  # All properties must be strings
}}

Payload Filtering

Payload filtering restricts when rules apply based on HTTP methods or response status codes:

Request Method Filtering

convert = [
    { from = "UserData", to = "CreateUserRequest", payload = {
        type = "request",
        method = ["POST", "PUT"]
    }},
    { from = "UserData", to = "UserQuery", payload = {
        type = "request",
        method = ["GET"]
    }},
]

Response Status Filtering

convert = [
    { from = "ErrorResponse", to = "ClientError", payload = {
        type = "response",
        status = [400, 401, 403, 404]
    }},
    { from = "ErrorResponse", to = "ServerError", payload = {
        type = "response",
        status = [500, 502, 503]
    }},
]

Payload Types

• type = "request": Apply only to request bodies
• type = "response": Apply only to response bodies
• method: Array of HTTP methods (GET, POST, PUT, DELETE, etc.)
• status: Array of HTTP status codes (200, 404, 500, etc.)

Rule Priority and Matching Order

Rules are evaluated in the following priority order (highest to lowest):

1. Property-based matching with where clauses
2. Property name matching (exact field names)
3. Schema title matching (schema-level titles)
4. Type-based matching (t# patterns)

Priority Example

convert = [
    # Highest priority: property-based matching
    { from = "UserData", to = "DetailedUser", where = {
        all_properties = ["id", "name", "email", "profile"]
    }},

    # Medium priority: property name matching
    { from = "name", to = "UserName" },

    # Lower priority: schema title matching
    { from = "UserData", to = "BasicUser" },

    # Lowest priority: type-based matching
    { from = "t#string", to = "String" },
]

For a schema titled “UserData” with properties id, name, email, profile:

• The property-based rule matches first → generates DetailedUser
• Other rules are ignored for this schema
• Individual properties still use their specific rules

Advanced Features

Complex Type Mappings

convert = [
    # Generic types
    { from = "items", to = "Vec<Item>" },
    { from = "metadata", to = "HashMap<String, Value>" },

    # Optional types
    { from = "optional_field", to = "Option<String>" },

    # Custom wrapper types
    { from = "timestamp", to = "DateTime<Utc>" },
    { from = "uuid", to = "Uuid" },

    # Domain-specific types
    { from = "room_id", to = "matrix_types::RoomId" },
    { from = "event_id", to = "matrix_types::EventId" },
]

Composition Schema Support

Property-based matching works with OpenAPI composition schemas (allOf, anyOf, oneOf):

# This rule matches schemas composed of multiple sub-schemas
{ from = "ComposedRequest", to = "ComplexRequest", where = {
    all_properties = ["base_field", "extension_field", "composed_field"]
}}

The rule will match if the combined schema (including all composed parts) contains the specified properties.

Generating Code

This section of the ngen documentation covers interacting with the nehwgeneration CLI to generate code.

Command-Line Options

In order to not provide redundant information, please execute

cargo run -p nehwgeneration -- --help

from the workspace root directory to list all available command line options, or

nehwgeneration --help

in case you have installed nehwgeneration on your system.

Configuring the Output Directory

To generate code, you will need a valid ngen.toml configuration file, complete with a configured general configuration.

Consider the following output entry in the general configuration:

[general]
output = "./server-crate/src/ngen/mod.rs"

nehwgeneration will attempt the following, when trying to save the generated code to disk:

1. Determine, whether ngen is already a valid, included module in the Rust source tree.
   • If not: Will add a pub mod ngen statement to the parent directories’ main.rs, lib.rs or mod.rs to attempt and include the module.
   • If so: Continue
2. If ngen/mod.rs already exists, all contents of that file will be completely overwritten with the auto-generated ngen code. If the file does not already exist, ngen will create it, and add the generated code to that file.

Tip

It is advised to give ngen-generated code its own module folder.

Interfacing with Generated Code from Rust

nehwgeneration generates a custom trait featuring all generated methods in form of axum route handler stubs. To use the auto-generated code, you will need to implement this trait and add all of the methods to an axum router.

Warning

Automatic generation of an axum router is a planned feature.

Warning

Do not implement the trait inside files generated by nehwgeneration, as the tool will overwrite all of your changes to that file the next time it is commanded to work on this file.

Architecture

This document outlines the high-level architecture of Arminius and provides some context on how that architecture was reached. In the future some of these decisions may be revisited.

Context

The architecture in Arminius follows a few core principles:

• To provide potentially infinite scalability, there should not be any central process. Instead everything should be potentially redundant and be able to fail over.
• There are external interfaces: Client traffic, federation traffic, administration and application services/extensions. The architecture has to route data between those interfaces with various constraints. Ideally reverse proxy configurations just need to route prefixes, not specific endpoints, to those components.

To simplify architectural decisions there are some assumptions:

• A single database can scale far enough: This has mostly worked out on the largest homeserver currently in existence and it avoids synchronising separate sources of truth like in a more decentralised microservice system. Currently this database is PostgreSQL.
• All internal traffic can be done over an abstract messaging fabric with potential database lookups. Currently this fabric is NATS.

This leads us to the following basic set of components:

• A client service to deal with requests from clients, (potentially) application services, and (potentially) a more powerful extension system based on the latter. The application service API is largely based on the client API with only a few additions. Usually the client service only receives requests from clients. Potentially the best way to shard the client service is based on the user ID or access token.
• A federation output service to send queued up transactions as well as queries to other homeservers. There is a shared retry and backoff logic for external services, so this may potentially shard based on the destination. This is, apart from appservice transactions, the only real outbound connection.
• A federation input service to deal with incoming requests from other homeservers. Rate limits are usually applied based on the incoming service, so it may be beneficial to shard based on the incoming server name or IP.
• To bind those services together and because rooms are the central decentralisation mechanism in Matrix, there is a room service. Room data needs to be processed in order in some cases and the room service should generally be the single source of truth about a room. As a result sharding based on the rooms may be the best solution.
• Then there is another API for administration. To ensure availability and to potentially restrict access to only the internal network, the admin service is a separate component. It also potentially deals with cross-cutting concerns.
• Another component is reserved for authentication. On the one hand we need to sync users to the homeserver database, which is done by the SCIM service. On the other hand we also need to deal with OIDC authentication. It is good practice to separate this out into a separate domain/url. As such we have 2 components to deal with identity and authentication related topics. A SCIM service for syncing and the intermediate IdP-like service NUSS to connect to an upstream IdP and add Matrix-specific OIDC features.

External to the homeserver we have clients, other homeservers, a potential admin frontend as well as application services/extensions. Traffic between our homeserver and such components may route through the public internet and may include forward or reverse proxies in between.

High-Level Architecture

The following diagram shows the high-level architecture of Arminius. Some flows have been simplified to make the diagram more readable.

---
config:
  flowchart:
    curve: step
  theme: "redux-color"
  look: neo
  layout: elk
---

flowchart LR
  %% Define the various components of the system
  HS[External Homeservers]@{shape: processes}
  Clients[Clients]
  %% FIXME: this shape is only supported after mermaid 1.17 @{shape: browser}
  FedIn[Federation Input Component]@{shape: processes}
  FedOut[Federation Output Component]@{shape: processes}
  ROOM[Roomserver]@{shape: processes}
  ClientServ[Client Component]@{shape: processes}
  SCIM[SCIM Component]@{shape: processes}
  NUSS[NUSS/IDP]@{shape: processes}
  AppServ[Extensions/Appservices]@{shape: processes}
  Admin[Admin Component]@{shape: processes}
  AdminFE[Admin Frontend]
  %% FIXME: this shape is only supported after mermaid 1.17 @{shape: browser}
  DB[Database]@{shape: database}

  %% Dotted border class
  classDef dotted stroke-dasharray: 5 5
  class AdminFE dotted
  class AppServ dotted

  %% Declare links between components
  subgraph Internal components
    direction TB
    
    
    ClientServ <== NATS ==> ROOM
    ClientServ <==> DB

    ROOM <==> DB

    FedOut <===> DB

    FedIn <===> DB
    FedIn <== NATS ==> ROOM

    FedOut <== NATS ==> ROOM

    Admin <-...->|administration to all components| DB
    %% invisible link for putting admin and scim next to each other.
    Admin ~~~~ SCIM

    SCIM ====> DB
  end

  subgraph EE[External Entities]
    direction LR
    
    Clients
    AppServ
    AdminFE
    NUSS
    HS
  end

  %% Invisible link for alignment
  HS ~~~ FedOut 
  FedOut e1@---> HS
  HS e2@---> FedIn
  Clients --> ClientServ

  AppServ <-.-> ClientServ
  AdminFE -.-> Admin
  NUSS --> SCIM

  %% Animate certain links
  e1@{ animate: true }
  e2@{ animate: true }

Note

Dotted components (Admin Frontend and Extensions) are not yet implemented.

Arminius is structured as a distributed monolith. The services are split up to allow us to scale them independently, but we’re not interested in having services developed or deployed independently.

Therefore, all services also share a single database. Based on other homeservers in the ecosystem, we think that a single database will be able to scale sufficiently for our needs.

Communication between components is done via NATS.

Components

Detailed information on the individual components can be found in their dedicated pages.

• Admin Component
• Client Component
• Federation Input Component
• Federation Output Component
• Roomserver
• SCIM Client Component
• Nuss

Folder Structure

All rust code is contained in the crates/ directory. The crates/ directory is split up into the directories libs/, servers/, tests/ and tools/.

• servers/ - contains our binaries, which run parts of the Arminius homeserver
• libs/ - libraries used in our other crates
• tests/ - crates that contain tests
• tools/ - developer tooling (e.g. nehwgeneration)

We decided on this hierarchical folder structure, instead of a flat crates/ directory due to the high amount of crates in our project. All crates are prefixed with arminius in order to not conflict with external crates. The servers, libs and tests path is also included in crate names, in order to make the module path match the file path.

📚 Other Pages

• Rate Limiting
• Storing JSON payloads in PostgreSQL
• Sync

Admin Component

This component implements an Administration API. Since administrators accomplishing their tasks by having direct database access is quite error-prone, we prefer providing them with an API. The admin API is implemented as a separate component in order to facilitate independent scaling, and putting it behind a firewall. The Admin Component is also responsible for setup tasks, like performing database migrations and generating/rotating server keys.

Client Component

This component implements all the logic of Matrix’s Client-Server API.

Code Organization

arminius-server-client/src
├── auth.rs
├── handlers
│   ├── account_data.rs
│   ├── account.rs
│   └── user_keys.rs
├── lib.rs
└── main.rs

• handlers/: Contains one Component for each logical group of endpoints in the matrix API

Federation Input Component

The Federation Input Component implements the incoming parts of Matrix’s federation algorithm defined in the Server-Server API.

Federation Output Component

The federation output component implements the outbound parts of Matrix’s federation algorithm defined in the Server-Server API. It is responsible for delivering events or messages to remote homeservers and handling their responses. In the context of this project, the component serves as the outbound gateway for cross-server communication.

Scope

The federation output component is responsible for federating room events (PDUs and EDUs) to other homeservers. Additionally, other components, such as the federation input component, can request the output component to fetch data from remote homeservers.

Note

In summary, the responsibilities of the federation output component are:

• pushing events from federated rooms to other homeservers, and
• requesting data from other homeservers when required by other components.

Message Delivery

Warning

Message delivery from other components to the federation output component MUST be handled via a message broker.

In the current implementation, NATS is used. However, the underlying structs and traits are generic enough to support any other message broker.

By using JetStream instead of core NATS, we ensure durable message persistence, retry support, and message ordering within a stream.

A single JetStream stream named federation-output-stream is configured to capture all subjects using a wildcard: federation.output.*.

Prioritization is achieved using two worker pools: a transaction worker pool and a miscellaneous (misc) worker pool. More details are provided in the Message Processing section.

Assumptions about the Message Broker

The only assumptions made about the message broker are:

• messages not acknowledged within a specific period of time are redelivered, and
• messages explicitly not acknowledged are also redelivered.

These assumptions are not strict requirements—other brokers can be used—but this may weaken the guarantees provided to the server.

Message Processing

Messages are processed in an asynchronous event loop. After receiving a message, the bundler groups it into a task.

• Transaction tasks: multiple messages are bundled into one task.
• Misc tasks: a single message forms the task content.

This abstraction allows the misc and transaction task managers to remain generic over the message broker’s implementation details.

While the bundler fetches events from the broker, HTTP workers continuously request new tasks from the bundler.

The HTTP workers are responsible for performing the actual HTTP requests. They report task results back to the bundler, which acknowledges all messages associated with the task.

To prevent starvation of misc requests, two worker pools are maintained:

• The misc worker pool prioritizes misc tasks.
• The transaction worker pool prioritizes transaction tasks.

The number of workers in each pool is configurable via environment variables, allowing adaptation to different use cases. If a pool has no available tasks, its workers will assist the other pool to avoid wasted resources.

The current design of the message processing pipeline is illustrated in the following diagram:

Communication between the Bundler and HTTP Workers

In the current implementation, communication between the bundler and workers is coordinated through shared mutable state. Access is synchronized via a lock, ensuring requests to the pool are queued in FIFO order.

Further implementation details can be found in the Federation output crate.

Task and Message Lifecycle

The lifecycle of messages can be described by the following diagram:

1. Another component sends a request to the federation output component via the message broker.
2. The bundler receives the message and either creates a new task or associates it with an existing task.
3. An HTTP worker fetches a new task from the task manager.
4. The HTTP worker sends the request to the remote homeserver.
5. The HTTP worker reports the outcome to the manager.
6. The manager acknowledges or does not acknowledge the messages associated with the task.
   • Acknowledged → message is considered delivered.
   • Not acknowledged → message broker redelivers the message to retry the task.
7. (Optional) The outcome of the HTTP request is reported back to the requesting component.

Guarantees

The guarantees of this component depend on the underlying message broker infrastructure. Each guarantee is justified with its dependencies:

• Transaction prioritization: Transactions are prioritized by size.
• Restart safety: The component maintains internal transaction pools.
  • On restart, these pools are lost.
  • If the broker redelivers unacknowledged messages, tasks are reconstructed.
  • If the broker lacks redelivery, this guarantee is lost.
• Exclusivity: For Matrix transactions, one active transaction per homeserver marks the server as occupied. Two transactions cannot be sent simultaneously to the same server.
• Infallibility: Task processing is fail-safe. Errors are logged, messages are not acknowledged, and unaffected tasks continue execution.
• No starvation of misc tasks: Dedicated pools ensure misc tasks receive processing power. Prioritization can be tuned via pool size configuration.

What the Component Does Not Guarantee

• In-order processing: New messages may be processed later if existing tasks have higher priority.

Room Service

This component is an internal service that implements all the various Room-related actions defined in the Matrix Room Version specifications.

Code Organization

crates/libs/arminius-lib-rooms/
├── src
│   ├── create_room.rs
│   ├── event_ordering.rs
│   ├── events.rs
│   ├── lib.rs
│   └── state.rs
crates/servers/arminius-server-rooms/
├── build.rs
├── Cargo.toml
├── src
│   ├── handlers
│   │   ├── create_room.rs
│   │   ├── get_member_events.rs
│   │   └── send_state_event_handler.rs
│   ├── lib.rs
│   └── main.rs
└── tests
    └── room_creation.rs

The logic for the roomserver is split up into these two crates:

• arminius-lib-rooms contains basic functions for modifying the room DAG. It is used by arminius-server-rooms to implement the API that it exposes to other services, and by the tests of arminius-lib-db-sync to set up its test cases. This can only be depended on by arminius-server-rooms, which is enforced by forbidden_deps.sh

• arminius-server-rooms is the actual room service, which implements an API that lets other services modify the room DAG.

Consumer Group

Solving the scalability challenge is very important for Arminius, and a core part of how we tackled it is by implementing Consumer Groups over NATS. This document outlines the insides of this central piece of code.

The following example describes the process triggered by the Client Component when one of its connected clients wants to send a message to a room.

graph LR;
    ID1["(1) Client Component"] -- "rooms.{room_id}.{command}" --> ID2["(2) NATS"];
    ID2 -- "rooms.{partition}.{room_id}.{command}" --> ID3["(3) Consumer Client 1"];
    ID2 -- "rooms.{partition}.{room_id}.{command}" --> ID4["(3) Consumer Client 2"];

1. The client component makes a request over NATS with the subject rooms.{room_id}.{command}.
2. Upon reception, the NATS server uses subject mapping (and especially here Deterministic subject partitioning) in order to transform the rooms.{room_id}.{command} subject into rooms.{PARTITION}.{room_id}.{command}, where PARTITION is a random number between 0 and the number of partition parts configured in our server.
3. The consumer clients are each subscribed to different partitions of the NATS subject and they get to consume the message sent.

With this approach, one can easily add new consumer clients by simply spawning them and let them share amongst themselves the range of partitions of the stream.

SCIM Client Component

SCIM (System for Cross-domain Identity Management) is an HTTP-based protocol. Arminius uses SCIM to exchange user information with an upstream identity provider (IdP). RFC 7643 specifies that the service provider (the server) “provides identity information via the SCIM protocol”, so the SCIM server is the IdP. This is further clarified by RFC 7642 where the service provider is the “authoritative identity data store”. Therefore, arminius acts as a SCIM client that connects to the IdP.

The SCIM Client Component continuously polls an upstream SCIM endpoint to synchronize users and sessions to the local database. Note that the upstream SCIM server is responsible to provide consistent data. Therefore, sessions have to reference existing users at any point in time. Consequently, an incremental response that deletes users must also explicitly delete any sessions that belong to them and still exist.

SCIM Extensions

The SCIM Client Component requires an upstream IdP that supports the custom extensions specified in this chapter.

Incremental Sync with Long Polling

On the SCIM root endpoint, these query parameters are added:

• sync: This query parameter is set to the nextSync value from a previous response. When this query parameter is specified and recognized by the server, the server SHOULD send an incremental response.
• timeout: When an incremental response would be empty, the server MUST wait up to timeout seconds until responding.

The Resources array of an incremental ListResponse only contains resources that were added or modified. These attributes are added to ListResponse:

• nextSync: Opaque String value for the next sync request. A server supporting this extension MUST set this attribute for full and incremental responses.
• Deletions: Array of String IDs of each deleted resource. The presence of this attribute indicates an incremental response (even if it is empty).

OIDC Sessions

Arminius expects the custom resource type OidcSession. The IdP must create such resources on OIDC login. The id common attribute of this resource must match OIDC’s sid (Session ID) claim. The resource uses this schema definition:

{
  "name": "OidcSession",
  "description": "A SCIM object representing an OIDC session created on the IdP",
  "attributes": [
    {
      "name": "userId",
      "type": "reference",
      "multiValued": false,
      "description": "the user associated with the OIDC session",
      "required": true,
      "mutability": "readOnly",
      "referenceTypes": ["User"],
    },
    {
      "name": "scope",
      "type": "string",
      "multiValued": false,
      "description": "the space-delimited scopes of the OIDC session",
      "required": true,
      "mutability": "readOnly",
    }
  ]
}

🥜Nuss

Arminius adopts the modern OIDC-native approach of Matrix, sourcing out authentication to a separate component, the IdP (Identity Provider). In contrast to simpler applications, Matrix imposes some extended requirements on the IdP: First, Matrix allows users to choose arbitrary clients by design, so Dynamic Client Registration is an integral part of Matrix OIDC. Also, sessions are long-lived, so they should be manageable. For example, users must be able to log out lost devices remotely. Additionally, sessions are managed both by the IdP (e.g., the refresh token) and the homeserver (e.g., E2E device keys). To provide a good user experience and reduce the attack surface, the sessions on the IdP and on the homeserver should always be in-sync. These features are not commonly provided by third-party IdPs.

To bridge this gap, we provide Nuss (New User and Session Service) along with Arminius, although it is a separate component. For example, it uses a separate database and communicates with Arminius over SCIM and OIDC. It provides the SCIM extensions needed by the scim-client-component. Nuss aims not to implement any Matrix-specific behavior. All of its features could be reasonably implemented by other IdPs, although our SCIM extensions are not standardized.

Nuss contains an OIDC server providing long-lived sessions based on short-lived OIDC sessions from an upstream IdP (such as Zitadel). Nuss contains a SCIM client to sync users periodically from upstream. Nuss provides a webhook endpoint to trigger immediate SCIM synchronization from upstream. Nuss provides users and sessions as a SCIM server. Currently, only a small subset of SCIM is implemented in the server. The SCIM server implements a custom extension for incremental sync with long polling, so clients (such as Arminius) can watch for user or session changes.

Nuss itself connects to an upstream IdP with SCIM, to continuously synchronize the list of users in Nuss with the upstream IdP. This way, any user which is deactivated on the upstream IdP will quickly be deactivated in Nuss, and therefore in Arminius. Additionally, by regularly synchronizing the list of users via SCIM, they will show up in search on the Matrix homeserver immediately. Without SCIM, using only OIDC, users would only show up in search when Nuss became aware of them on their first login.

Implementation Goals

• No unauthenticated storage: Unauthenticated clients are unable to cause Nuss to store data, except logs. Motivation: Attackers aren’t able to fill up server storage. Attackers aren’t able to store illegal data on the server. The unauthenticated endpoints are stateless, so they can be scaled to infinity in case of DoS attacks.
• No unauthenticated server traffic: Unauthenticated clients are unable to cause Nuss to start any network requests or connections to other servers. Motivation: Attackers aren’t able to proxy DoS attacks through Nuss. They have to target other servers or Nuss directly.

Database

Nuss mostly follows the same conventions as Arminius, and it relies on arminius-lib-db for creating a database connection. There are some points where it currently deviates from conventions, including that queries are defined in a submodule rather than in a separate crate, and the nid naming convention for surrogate keys is not followed. Nuss autogenerates its own separate table descriptions.

Database Tables

audit

Old versions of the rows in users and sessions

                                                                                                               Table "public.audit"
  Column   |           Type           | Collation | Nullable | Default | Storage  | Compression | Stats target |                                                            Description                                                            
-----------+--------------------------+-----------+----------+---------+----------+-------------+--------------+-----------------------------------------------------------------------------------------------------------------------------------
 id        | text                     |           | not null |         | extended |             |              | ID as in the source table (users or sessions)
 timestamp | timestamp with time zone |           | not null | now()   | plain    |             |              | When the row was changed
 kind      | audit_kind               |           | not null |         | plain    |             |              | The kind of table modification
 type      | resource_type            |           | not null |         | plain    |             |              | Which table the modification refers to, either "User" or "OidcSession"
 data      | jsonb                    |           |          |         | extended |             |              | The data from the new contents after the change. Saved as a JSON blob, the schema depends on the table the modification refers to
Indexes:
    "audit_pkey" PRIMARY KEY, btree (id, "timestamp")
Access method: heap

jwks

Keys for JWT encoding

                                                                                  Table "public.jwks"
   Column    |           Type           | Collation | Nullable |           Default            | Storage  | Compression | Stats target |                   Description                   
-------------+--------------------------+-----------+----------+------------------------------+----------+-------------+--------------+-------------------------------------------------
 id          | integer                  |           | not null | generated always as identity | plain    |             |              | Key ID, generated as consecutive integers
 created     | timestamp with time zone |           | not null | now()                        | plain    |             |              | When the key was created
 stop_issue  | timestamp with time zone |           | not null | now() + '1 day'::interval    | plain    |             |              | When we should stop issuing JWTs using this key
 expiry      | timestamp with time zone |           | not null | now() + '2 days'::interval   | plain    |             |              | When this key expires
 private_key | bytea                    |           | not null |                              | extended |             |              | The key itself
Access method: heap

sessions

User login sessions

                                                                                  Table "public.sessions"
      Column       |           Type           | Collation | Nullable |           Default            | Storage  | Compression | Stats target |                  Description                  
-------------------+--------------------------+-----------+----------+------------------------------+----------+-------------+--------------+-----------------------------------------------
 id                | integer                  |           | not null | generated always as identity | plain    |             |              | Session ID, generated as consecutive integers
 created           | timestamp with time zone |           | not null | now()                        | plain    |             |              | When the session was created
 user_id           | text                     |           | not null |                              | extended |             |              | ID of the user this session belongs to
 scope             | text                     |           | not null |                              | extended |             |              | OAuth scope for the session
 refresh_token     | text                     |           |          |                              | extended |             |              | Token for refreshing the session
 nonce             | text                     |           |          |                              | extended |             |              | OpenID nonce
 code              | text                     |           |          |                              | extended |             |              | OpenID authorization code
 code_challenge    | text                     |           |          |                              | extended |             |              | OpenID PKCE code challenge
 upstream_id_token | text                     |           |          |                              | extended |             |              | The ID token from the upstream IdP
Foreign-key constraints:
    "sessions_user_id_fkey" FOREIGN KEY (user_id) REFERENCES users(id) ON DELETE CASCADE
Triggers:
    add_audit_session_delete AFTER DELETE ON sessions FOR EACH ROW EXECUTE FUNCTION audit_session_delete()
    add_audit_session_insert AFTER INSERT ON sessions FOR EACH ROW EXECUTE FUNCTION audit_session_insert()
    sessions_sync_trigger AFTER INSERT OR DELETE OR UPDATE ON sessions FOR EACH STATEMENT EXECUTE FUNCTION send_resource_notification()
Access method: heap

users

Most recent info about users

                                                                   Table "public.users"
  Column  |           Type           | Collation | Nullable | Default | Storage  | Compression | Stats target |                Description                 
----------+--------------------------+-----------+----------+---------+----------+-------------+--------------+--------------------------------------------
 id       | text                     |           | not null |         | extended |             |              | String uniquely identifying the user
 created  | timestamp with time zone |           | not null | now()   | plain    |             |              | When the user was created
 modified | timestamp with time zone |           | not null |         | plain    |             |              | When the user was last modified
 data     | jsonb                    |           | not null |         | extended |             |              | Arbitrary JSON data from the SCIM upstream
Indexes:
    "users_pkey" PRIMARY KEY, btree (id)
Referenced by:
    TABLE "sessions" CONSTRAINT "sessions_user_id_fkey" FOREIGN KEY (user_id) REFERENCES users(id) ON DELETE CASCADE
Triggers:
    add_audit_user_delete AFTER DELETE ON users FOR EACH ROW EXECUTE FUNCTION audit_user_delete()
    add_audit_user_insert AFTER INSERT ON users FOR EACH ROW EXECUTE FUNCTION audit_user_insert()
    add_audit_user_update AFTER UPDATE ON users FOR EACH ROW EXECUTE FUNCTION audit_user_update()
    add_update_user_modified BEFORE INSERT OR UPDATE ON users FOR EACH ROW EXECUTE FUNCTION update_user_modified()
    check_audit_timestamps_trigger AFTER INSERT OR UPDATE ON users DEFERRABLE INITIALLY DEFERRED FOR EACH ROW EXECUTE FUNCTION check_audit_timestamps()
    users_sync_trigger AFTER INSERT OR DELETE OR UPDATE ON users FOR EACH STATEMENT EXECUTE FUNCTION send_resource_notification()
Access method: heap

Database

This section of the documentation is all about the Arminius PostgreSQL database.

A list of tables, their columns and attributes is provided. This is auto-generated, to avoid users and developers having to use the psql command to look at how the database is set up. Alternatives considered to auto-generated schema documentation were:

• Manually written schema descriptions
  • Manually writing a copy of the schema in markdown format is very expensive to maintain, and prone to documentation rot, which would confuse more than inform, and make any changes to the database needlessly annoying
• Just comments in migration files
  • Depending solely on migration files makes viewing the state of the database difficult in the future, when we may have multiple migrations.

Tradeoffs between Query Complexity and Performance

CRUD operations are simpler to read and debug for most people, but consistency and performance requirements can make fewer, more complicated queries more beneficial than multiple simpler queries.

Multiple queries have additional network latency and splitting up queries makes it harder to reason about ids and relations used over multiple tables.

Rules for Database Tables

All surrogate keys should have the name nid.

Foreign keys should use the format <TABLE>_nid, where <TABLE> is the table we are referencing, unless the relationship between the tables is not an obvious “belongs to” relation. That is, events have a column room_nid, to specify which room the event belongs to, while to-device messages have a sender_nid, which references the users table.

All tables and columns should have a postgres comment, this is also checked in the CI.

For storing time, always use timestamp with time zone. This type is stored internally as UTC, which sqlx and chrono are aware of. Therefore, we can pass chrono::DateTime<chrono::Utc> directly to the database, and get it back later, which is always what we want. The timestamp without time zone in postgres is only really suitable for timestamps the application is not meant to understand. For more information, see “Don’t do this” on the PostgreSQL Wiki.

Communication with the Database

The crates/libs/db/arminius-lib-db crate is primarily responsible for providing a connection to the database. Individual queries are then in database crates, such as crates/libs/db/arminius-lib-db-keys or crates/libs/db/arminius-lib-db-user, which provide extension traits for the database type from crates/libs/db/arminius-lib-db.

The crates in crates/libs/db are the only crates which should have a direct dependency on sqlx.

Do Not Use NIDs Outside of the Database Layer

We decided on this rule so that NIDs are internal to the database layer. Therefore, types that will be returned by SQL queries should not mention nor contain any such NID column. Instead, call these queries using the Matrix ID of things (for example, query a user using its user_id, not its nid).

Crate Documentation

For the crate documentation, see arminius-lib-db/src/lib.rs

Related Pages

• Tables
• Database Query Conventions, writing query functions
• Storing JSON in PostgreSQL, how we store JSON payloads
• Nuss database, the tables for Nuss, which don’t quite follow the same standards yet.

Database Tables

auth_events

Table registering which events authorize which

                                                            Table "public.auth_events"
     Column     |  Type  | Collation | Nullable | Default | Storage | Compression | Stats target |                  Description                   
----------------+--------+-----------+----------+---------+---------+-------------+--------------+------------------------------------------------
 auth_event_nid | bigint |           | not null |         | plain   |             |              | The event which authenticates some other event
 event_nid      | bigint |           | not null |         | plain   |             |              | The event which is being authenticated
Foreign-key constraints:
    "auth_events_auth_event_nid_fkey" FOREIGN KEY (auth_event_nid) REFERENCES events(nid) ON DELETE CASCADE
    "auth_events_event_nid_fkey" FOREIGN KEY (event_nid) REFERENCES events(nid) ON DELETE CASCADE
Access method: heap

database_import_progress

Tracks the progress of a homeserver migration import per table, so an interrupted import can be resumed.

                                                                                                                                          Table "public.database_import_progress"
     Column     |           Type           | Collation | Nullable | Default | Storage  | Compression | Stats target |                                                                                              Description                                                                                              
----------------+--------------------------+-----------+----------+---------+----------+-------------+--------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 table_name     | text                     |           | not null |         | extended |             |              | The record type / table being imported (e.g. 'users').
 cursor         | text                     |           | not null |         | extended |             |              | The sort key of the last successfully imported row. Passed back to the exporter (under the query parameter it advertises in the export header's 'cursor_param') to resume the export from this point.
 imported_count | bigint                   |           | not null | 0       | plain    |             |              | The number of rows imported so far for this table.
 completed      | boolean                  |           | not null | false   | plain    |             |              | Whether the import for this table has finished (the exporter's trailer record was received).
 updated_at     | timestamp with time zone |           | not null | now()   | plain    |             |              | When this row was last updated.
Indexes:
    "database_import_progress_pkey" PRIMARY KEY, btree (table_name)
Access method: heap

devices

Local Devices

                                                                                                         Table "public.devices"
      Column       |  Type   | Collation | Nullable |                   Default                    | Storage  | Compression | Stats target |                                        Description                                         
-------------------+---------+-----------+----------+----------------------------------------------+----------+-------------+--------------+--------------------------------------------------------------------------------------------
 nid               | bigint  |           | not null | generated by default as identity             | plain    |             |              | Surrogate key for the device
 user_nid          | integer |           | not null |                                              | plain    |             |              | User to which this device belongs
 device_id         | text    |           | not null |                                              | extended |             |              | ID for the device, according to the matrix spec
 idp_id            | text    |           |          |                                              | extended |             |              | ID for the device when talking to the identity provider
 revoked_token_tag | bigint  |           | not null | 0                                            | plain    |             |              | All token tags less than this count as revoked
 stream_id         | bigint  |           | not null | locking_nextval('user_keys_streamids'::text) | plain    |             |              | The stream ID of the device creation. Used to compute the `device_lists` of sync requests.
Indexes:
    "devices_pkey" PRIMARY KEY, btree (nid)
    "devices_stream_id_key" UNIQUE CONSTRAINT, btree (stream_id)
    "devices_user_nid_device_id_key" UNIQUE CONSTRAINT, btree (user_nid, device_id)
    "devices_user_nid_idp_id_key" UNIQUE CONSTRAINT, btree (user_nid, idp_id)
Foreign-key constraints:
    "devices_user_nid_fkey" FOREIGN KEY (user_nid) REFERENCES users(nid) DEFERRABLE
Referenced by:
    TABLE "event_transactions" CONSTRAINT "event_transactions_sender_device_nid_fkey" FOREIGN KEY (sender_device_nid) REFERENCES devices(nid) ON DELETE CASCADE
    TABLE "fallback_keys" CONSTRAINT "fallback_keys_device_nid_fkey" FOREIGN KEY (device_nid) REFERENCES devices(nid) ON DELETE CASCADE
    TABLE "filters" CONSTRAINT "filters_device_nid_fkey" FOREIGN KEY (device_nid) REFERENCES devices(nid) ON DELETE CASCADE
    TABLE "one_time_keys" CONSTRAINT "one_time_keys_device_nid_fkey" FOREIGN KEY (device_nid) REFERENCES devices(nid) ON DELETE CASCADE
    TABLE "syncv4_connections" CONSTRAINT "syncv4_connections_device_nid_fkey" FOREIGN KEY (device_nid) REFERENCES devices(nid) ON DELETE CASCADE
    TABLE "to_device_messages" CONSTRAINT "to_device_messages_device_nid_fkey" FOREIGN KEY (device_nid) REFERENCES devices(nid) ON DELETE CASCADE
    TABLE "user_keys" CONSTRAINT "user_keys_device_nid_fkey" FOREIGN KEY (device_nid) REFERENCES devices(nid) ON DELETE CASCADE
Access method: heap

event_ordering

Table which stores a linear ordering of events. See also https://bundesmessenger.gitlab.io/backend/neo/arminius/arminius/rustdoc/arminius_lib_db_room/ordering/index.html

                                                                               Table "public.event_ordering"
  Column   |   Type   | Collation | Nullable | Default | Storage  | Compression | Stats target |                                        Description                                         
-----------+----------+-----------+----------+---------+----------+-------------+--------------+--------------------------------------------------------------------------------------------
 event_nid | bigint   |           | not null |         | plain    |             |              | The event which occupies a position in our ordering
 room_nid  | integer  |           | not null |         | plain    |             |              | The room for which our event is ordered. Should always match the event
 ordering  | bigint[] |           | not null |         | extended |             |              | The actual ordering. Ordering of events is achieved by sorting this list lexicographically
Indexes:
    "event_ordering_event_nid_key" UNIQUE CONSTRAINT, btree (event_nid)
    "event_ordering_room_nid_ordering_key" UNIQUE CONSTRAINT, btree (room_nid, ordering)
    "linear_ordering" btree (ordering)
Check constraints:
    "event_ordering_ordering_check" CHECK (array_position(ordering, NULL::bigint) IS NULL)
Foreign-key constraints:
    "event_ordering_event_nid_fkey" FOREIGN KEY (event_nid) REFERENCES events(nid) ON DELETE CASCADE
    "event_ordering_room_nid_fkey" FOREIGN KEY (room_nid) REFERENCES rooms(nid) ON DELETE CASCADE
Access method: heap

event_transactions

Table that stores references to events which were sent as part of a call to PUT /_matrix/client/v3/rooms/{roomId}/send/{eventType}/{txnId}

                                                                                                            Table "public.event_transactions"
      Column       |          Type          | Collation | Nullable | Default | Storage  | Compression | Stats target |                                                            Description                                                            
-------------------+------------------------+-----------+----------+---------+----------+-------------+--------------+-----------------------------------------------------------------------------------------------------------------------------------
 event_nid         | bigint                 |           | not null |         | plain    |             |              | The event which was inserted as part of the transaction
 sender_device_nid | bigint                 |           | not null |         | plain    |             |              | The device which sent the message. Used for de-duplication
 txn_id            | character varying(255) |           | not null |         | extended |             |              | User-defined transaction ID used for de-duplication. See https://spec.matrix.org/v1.14/client-server-api/#transaction-identifiers
 room_nid          | integer                |           | not null |         | plain    |             |              | The room the event was sent to. Part of the de-duplication key
 event_type        | text                   |           | not null |         | extended |             |              | The type of the event, as in the `/send/{eventType}` path. Part of the de-duplication key
Indexes:
    "event_transactions_pkey" PRIMARY KEY, btree (event_nid)
    "event_transactions_dedup_key" UNIQUE, btree (sender_device_nid, room_nid, event_type, txn_id)
Foreign-key constraints:
    "event_transactions_event_nid_fkey" FOREIGN KEY (event_nid) REFERENCES events(nid) ON DELETE CASCADE
    "event_transactions_room_nid_fkey" FOREIGN KEY (room_nid) REFERENCES rooms(nid) ON DELETE CASCADE
    "event_transactions_sender_device_nid_fkey" FOREIGN KEY (sender_device_nid) REFERENCES devices(nid) ON DELETE CASCADE
Access method: heap

events

Events table for storing all event data

                                                                                                                Table "public.events"
   Column    |  Type   | Collation | Nullable |                       Default                       | Storage  | Compression | Stats target |                                              Description                                               
-------------+---------+-----------+----------+-----------------------------------------------------+----------+-------------+--------------+--------------------------------------------------------------------------------------------------------
 nid         | bigint  |           | not null | generated by default as identity                    | plain    |             |              | Surrogate key for events
 event_id    | text    |           | not null |                                                     | extended |             |              | The event id per the matrix spec
 room_nid    | integer |           | not null |                                                     | plain    |             |              | Room to which the event belongs
 event_json  | json    |           | not null |                                                     | extended |             |              | The event contents, in federation format
 depth       | bigint  |           | not null |                                                     | plain    |             |              | For the first event in a room, should be 1. For all following events, the max depth of an ancestor + 1
 received_ts | bigint  |           | not null | (EXTRACT(epoch FROM now()) * 1000::numeric)::bigint | plain    |             |              | Server receipt time in milliseconds
 stream_id   | bigint  |           | not null | locking_nextval('eventids'::text)                   | plain    |             |              | Position in the /sync stream
Indexes:
    "events_pkey" PRIMARY KEY, btree (nid)
    "events_room_member_state_key" btree (room_nid, (event_json ->> 'state_key'::text)) WHERE (event_json ->> 'type'::text) = 'm.room.member'::text
    "events_room_nid_event_id_key" UNIQUE CONSTRAINT, btree (room_nid, event_id)
    "events_room_stream" btree (room_nid, stream_id)
    "events_stream_id_key" UNIQUE CONSTRAINT, btree (stream_id)
Foreign-key constraints:
    "events_room_nid_fkey" FOREIGN KEY (room_nid) REFERENCES rooms(nid) ON DELETE CASCADE
Referenced by:
    TABLE "auth_events" CONSTRAINT "auth_events_auth_event_nid_fkey" FOREIGN KEY (auth_event_nid) REFERENCES events(nid) ON DELETE CASCADE
    TABLE "auth_events" CONSTRAINT "auth_events_event_nid_fkey" FOREIGN KEY (event_nid) REFERENCES events(nid) ON DELETE CASCADE
    TABLE "event_ordering" CONSTRAINT "event_ordering_event_nid_fkey" FOREIGN KEY (event_nid) REFERENCES events(nid) ON DELETE CASCADE
    TABLE "event_transactions" CONSTRAINT "event_transactions_event_nid_fkey" FOREIGN KEY (event_nid) REFERENCES events(nid) ON DELETE CASCADE
    TABLE "events_state_snapshots" CONSTRAINT "events_state_snapshots_event_nid_fkey" FOREIGN KEY (event_nid) REFERENCES events(nid) ON DELETE CASCADE
    TABLE "leaf_events" CONSTRAINT "leaf_events_event_nid_fkey" FOREIGN KEY (event_nid) REFERENCES events(nid) ON DELETE CASCADE
    TABLE "previous_events" CONSTRAINT "previous_events_event_nid_fkey" FOREIGN KEY (event_nid) REFERENCES events(nid) ON DELETE CASCADE
    TABLE "previous_events" CONSTRAINT "previous_events_previous_event_nid_fkey" FOREIGN KEY (previous_event_nid) REFERENCES events(nid) ON DELETE CASCADE
    TABLE "read_receipts" CONSTRAINT "read_receipts_event_nid_fkey" FOREIGN KEY (event_nid) REFERENCES events(nid) ON DELETE CASCADE
    TABLE "state_snapshot_state_events" CONSTRAINT "state_snapshot_state_events_state_event_nid_fkey" FOREIGN KEY (state_event_nid) REFERENCES events(nid) ON DELETE CASCADE
    TABLE "state_snapshots" CONSTRAINT "state_snapshots_history_visibility_event_nid_fkey" FOREIGN KEY (history_visibility_event_nid) REFERENCES events(nid) ON DELETE SET NULL
    TABLE "state_snapshots" CONSTRAINT "state_snapshots_server_acl_event_nid_fkey" FOREIGN KEY (server_acl_event_nid) REFERENCES events(nid) ON DELETE SET NULL
    TABLE "verification_metadata" CONSTRAINT "verification_metadata_event_nid_fkey" FOREIGN KEY (event_nid) REFERENCES events(nid) ON DELETE CASCADE
Triggers:
    require_room_lock BEFORE INSERT ON events FOR EACH ROW EXECUTE FUNCTION assert_room_locked()
Access method: heap

events_state_snapshots

Table storing relations between events, and the state snapshots which store state events that apply to them

                                                            Table "public.events_state_snapshots"
       Column       |  Type  | Collation | Nullable | Default | Storage | Compression | Stats target |                      Description                      
--------------------+--------+-----------+----------+---------+---------+-------------+--------------+-------------------------------------------------------
 event_nid          | bigint |           | not null |         | plain   |             |              | The state event, to which some state snapshot applies
 state_snapshot_nid | bigint |           | not null |         | plain   |             |              | Some state snapshot, which applies to a state event
Indexes:
    "events_state_snapshots_pkey" PRIMARY KEY, btree (event_nid)
Foreign-key constraints:
    "events_state_snapshots_event_nid_fkey" FOREIGN KEY (event_nid) REFERENCES events(nid) ON DELETE CASCADE
    "events_state_snapshots_state_snapshot_nid_fkey" FOREIGN KEY (state_snapshot_nid) REFERENCES state_snapshots(nid) ON DELETE CASCADE
Access method: heap

fallback_keys

Fallback keys for a user

                                                                           Table "public.fallback_keys"
    Column     |  Type   | Collation | Nullable |             Default              | Storage  | Compression | Stats target |                     Description                      
---------------+---------+-----------+----------+----------------------------------+----------+-------------+--------------+------------------------------------------------------
 nid           | integer |           | not null | generated by default as identity | plain    |             |              | Surrogate key for the key
 key_algorithm | text    |           | not null |                                  | extended |             |              | The algorithm used in the key
 key_name      | text    |           | not null |                                  | extended |             |              | A name for the key
 device_nid    | integer |           | not null |                                  | plain    |             |              | The device which owns the private key
 key_object    | jsonb   |           | not null |                                  | extended |             |              | The public key part of the key
 used          | boolean |           | not null | false                            | plain    |             |              | Whether or not the fallback has already been claimed
Indexes:
    "fallback_keys_pkey" PRIMARY KEY, btree (nid)
    "fallback_key_per_device" UNIQUE CONSTRAINT, btree (device_nid, key_algorithm)
Foreign-key constraints:
    "fallback_keys_device_nid_fkey" FOREIGN KEY (device_nid) REFERENCES devices(nid) ON DELETE CASCADE
Access method: heap

federated_invites

Invites received over federation, for rooms we are not active in

                                                                                                  Table "public.federated_invites"
    Column    |   Type   | Collation | Nullable |                Default                 | Storage  | Compression | Stats target |                                            Description                                            
--------------+----------+-----------+----------+----------------------------------------+----------+-------------+--------------+---------------------------------------------------------------------------------------------------
 nid          | bigint   |           | not null | generated by default as identity       | plain    |             |              | Surrogate key for the invite
 stream_id    | bigint   |           | not null | locking_nextval('invited_rooms'::text) | plain    |             |              | Stream ID for the federated_invite. See the sync docs in the external documentation for more info
 user_nid     | integer  |           | not null |                                        | plain    |             |              | The user that has been invited to a room
 room_id      | text     |           | not null |                                        | extended |             |              | The matrix room ID of the room being invited to
 state_events | bigint[] |           |          |                                        | extended |             |              | List of state events in the stripped state, giving a preview of the room
Indexes:
    "federated_invites_pkey" PRIMARY KEY, btree (nid)
    "federated_invites_stream_id_key" UNIQUE CONSTRAINT, btree (stream_id)
Foreign-key constraints:
    "federated_invites_user_nid_fkey" FOREIGN KEY (user_nid) REFERENCES users(nid)
Access method: heap

federated_knocks

Rooms we have knocked on over federation

                                                                                                   Table "public.federated_knocks"
    Column    |   Type   | Collation | Nullable |                Default                 | Storage  | Compression | Stats target |                                            Description                                            
--------------+----------+-----------+----------+----------------------------------------+----------+-------------+--------------+---------------------------------------------------------------------------------------------------
 nid          | bigint   |           | not null | generated by default as identity       | plain    |             |              | Surrogate key for the knock
 stream_id    | bigint   |           | not null | locking_nextval('invited_rooms'::text) | plain    |             |              | Stream ID for the federated_invite. See the sync docs in the external documentation for more info
 user_nid     | integer  |           | not null |                                        | plain    |             |              | The user that has been invited to a room
 room_id      | text     |           | not null |                                        | extended |             |              | The matrix room ID of the room being invited to
 state_events | bigint[] |           |          |                                        | extended |             |              | List of state events in the stripped state, giving a preview of the room
Indexes:
    "federated_knocks_pkey" PRIMARY KEY, btree (nid)
    "federated_knocks_stream_id_key" UNIQUE CONSTRAINT, btree (stream_id)
Foreign-key constraints:
    "federated_knocks_user_nid_fkey" FOREIGN KEY (user_nid) REFERENCES users(nid)
Access method: heap

federated_to_device_transactions

Transaction tracker for incoming to-device EDUs

                                           Table "public.federated_to_device_transactions"
 Column | Type | Collation | Nullable | Default | Storage  | Compression | Stats target |                 Description                 
--------+------+-----------+----------+---------+----------+-------------+--------------+---------------------------------------------
 origin | text |           | not null |         | extended |             |              | The homeserver that sent us the transaction
 txnid  | text |           | not null |         | extended |             |              | The actual transaction ID.
Indexes:
    "federated_to_device_transactions_origin_txnid_key" UNIQUE CONSTRAINT, btree (origin, txnid)
Access method: heap

filters

Filters for endpoints that user have uploaded.

                                                                                                                                        Table "public.filters"
    Column    |         Type         | Collation | Nullable |           Default           | Storage  | Compression | Stats target |                                                                            Description                                                                             
--------------+----------------------+-----------+----------+-----------------------------+----------+-------------+--------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------
 id           | uuid                 |           | not null |                             | plain    |             |              | The primary key of the filter used to retrieve the filter by the user that created it. This is stored as UUID, but it is generated and used as ULID value.
 user_nid     | integer              |           | not null |                             | plain    |             |              | The user that has created the filter. Only this user should have access to the filter.
 device_nid   | bigint               |           | not null |                             | plain    |             |              | The filter for the device that created it. The life time of the filter is bound to the existence of this device.
 account_data | event_filter         |           |          |                             | extended |             |              | Filters for events related to account data that are not part of the rooms to include
 presence     | event_filter         |           |          |                             | extended |             |              | Filters for the updates to presence to include
 room         | room_filter          |           |          |                             | extended |             |              | Filters to be applied to room data
 event_format | event_format_enum    |           | not null | 'client'::event_format_enum | plain    |             |              | The format for the events. "federation" should send the events in a raw event as received, while the "client" should send it in a format suitable for the clients.
 event_fields | dot_separated_path[] |           |          |                             | extended |             |              | Address the fields of the events to include.
Indexes:
    "filters_pkey" PRIMARY KEY, btree (id)
    "idx_filters_dedup" UNIQUE, btree (user_nid, device_nid, generate_filter_hash(account_data, presence, room, event_format, event_fields))
Foreign-key constraints:
    "filters_device_nid_fkey" FOREIGN KEY (device_nid) REFERENCES devices(nid) ON DELETE CASCADE
    "filters_user_nid_fkey" FOREIGN KEY (user_nid) REFERENCES users(nid) ON DELETE CASCADE
Access method: heap

forgotten_rooms

List of rooms that users have marked as “forgotten”. If all our local users have forgotten a room, we should delete it.

                                                       Table "public.forgotten_rooms"
  Column  |  Type   | Collation | Nullable | Default | Storage | Compression | Stats target |                  Description                  
----------+---------+-----------+----------+---------+---------+-------------+--------------+-----------------------------------------------
 room_nid | integer |           | not null |         | plain   |             |              | The room that has been marked as forgotten
 user_nid | integer |           | not null |         | plain   |             |              | The user which has marked a room as forgotten
Indexes:
    "forgotten_rooms_room_nid_user_nid_key" UNIQUE CONSTRAINT, btree (room_nid, user_nid)
Foreign-key constraints:
    "forgotten_rooms_room_nid_fkey" FOREIGN KEY (room_nid) REFERENCES rooms(nid) ON DELETE CASCADE
    "forgotten_rooms_user_nid_fkey" FOREIGN KEY (user_nid) REFERENCES users(nid) ON DELETE CASCADE
Access method: heap

global_account_data

Global account data for each user

                                                                                                                   Table "public.global_account_data"
  Column   |  Type   | Collation | Nullable |                     Default                     | Storage  | Compression | Stats target |                                                           Description                                                            
-----------+---------+-----------+----------+-------------------------------------------------+----------+-------------+--------------+----------------------------------------------------------------------------------------------------------------------------------
 user_nid  | integer |           | not null |                                                 | plain    |             |              | The user to which this account data belongs
 data_type | text    |           | not null |                                                 | extended |             |              | Type of the tag. See https://spec.matrix.org/v1.14/client-server-api/#client-behaviour-17
 data      | jsonb   |           | not null |                                                 | extended |             |              | A JSON blob the user stores
 stream_id | bigint  |           | not null | locking_nextval('account_data_streamids'::text) | plain    |             |              | Stream ID of the user's account data. See https://bundesmessenger.gitlab.io/backend/neo/arminius/arminius/architecture/sync.html
Indexes:
    "global_account_data_pkey" PRIMARY KEY, btree (user_nid, data_type)
    "global_account_data_stream_id_key" UNIQUE CONSTRAINT, btree (stream_id)
Foreign-key constraints:
    "global_account_data_user_nid_fkey" FOREIGN KEY (user_nid) REFERENCES users(nid) ON DELETE CASCADE
Access method: heap

leaf_events

Leaves of the room event DAG

                                                     Table "public.leaf_events"
  Column   |  Type   | Collation | Nullable | Default | Storage | Compression | Stats target |             Description              
-----------+---------+-----------+----------+---------+---------+-------------+--------------+--------------------------------------
 room_nid  | integer |           | not null |         | plain   |             |              | The room to which an event is a leaf
 event_nid | bigint  |           | not null |         | plain   |             |              | The event which is a leaf
Indexes:
    "leaf_events_room_nid_event_nid_key" UNIQUE CONSTRAINT, btree (room_nid, event_nid)
Foreign-key constraints:
    "leaf_events_event_nid_fkey" FOREIGN KEY (event_nid) REFERENCES events(nid) ON DELETE CASCADE
    "leaf_events_room_nid_fkey" FOREIGN KEY (room_nid) REFERENCES rooms(nid) ON DELETE CASCADE
Triggers:
    require_room_lock BEFORE INSERT ON leaf_events FOR EACH ROW EXECUTE FUNCTION assert_room_locked()
Access method: heap

local_signatures

Cross-signing signatures for local keys

                                                                                                     Table "public.local_signatures"
      Column      |   Type   | Collation | Nullable |                   Default                    | Storage  | Compression | Stats target |                                         Description                                          
------------------+----------+-----------+----------+----------------------------------------------+----------+-------------+--------------+----------------------------------------------------------------------------------------------
 signing_user_nid | integer  |           | not null |                                              | plain    |             |              | Reference to the user who issued the signature
 signing_key_type | key_type |           | not null |                                              | plain    |             |              | Type of key that issued the signature. See the `KEY_TYPE` enum in the `user_keys` migration.
 signing_key_id   | text     |           | not null |                                              | extended |             |              | Key ID of the key that issued the signature
 signed_key_nid   | bigint   |           | not null |                                              | plain    |             |              | Link to the signed key. Will delete the signature on deletion.
 signature        | bytea    |           | not null |                                              | extended |             |              | The actual signature
 stream_id        | bigint   |           | not null | locking_nextval('user_keys_streamids'::text) | plain    |             |              | The stream ID to tell when the key was inserted. This reuses the keys streamids.
Indexes:
    "local_signatures_stream_id_key" UNIQUE CONSTRAINT, btree (stream_id)
    "unique_signature" UNIQUE CONSTRAINT, btree (signing_user_nid, signing_key_id, signed_key_nid)
Foreign-key constraints:
    "local_signatures_signed_key_nid_fkey" FOREIGN KEY (signed_key_nid) REFERENCES user_keys(nid) ON DELETE CASCADE
    "local_signatures_signing_user_nid_fkey" FOREIGN KEY (signing_user_nid) REFERENCES users(nid) ON DELETE CASCADE
Access method: heap

one_time_keys

To-be-issued one time keys for a user

                                                                                             Table "public.one_time_keys"
     Column      |  Type   | Collation | Nullable |             Default              | Storage  | Compression | Stats target |                                      Description                                       
-----------------+---------+-----------+----------+----------------------------------+----------+-------------+--------------+----------------------------------------------------------------------------------------
 nid             | integer |           | not null | generated by default as identity | plain    |             |              | Surrogate key for the OTK
 key_algorithm   | text    |           | not null |                                  | extended |             |              | The algorithm used in the OTK
 key_name        | text    |           | not null |                                  | extended |             |              | A name for the OTK
 device_nid      | integer |           | not null |                                  | plain    |             |              | The device which owns the private key
 key_object      | jsonb   |           | not null |                                  | extended |             |              | The public key part of the OTK
 upload_revision | integer |           | not null | generated always as identity     | plain    |             |              | An integer used to order the claimed key by sending back the oldest uploaded one first
Indexes:
    "one_time_keys_pkey" PRIMARY KEY, btree (nid)
    "user_device_key_unique" UNIQUE CONSTRAINT, btree (device_nid, key_object)
Foreign-key constraints:
    "one_time_keys_device_nid_fkey" FOREIGN KEY (device_nid) REFERENCES devices(nid) ON DELETE CASCADE
Access method: heap

presence

User presence information

                                                                                             Table "public.presence"
     Column      |           Type           | Collation | Nullable | Default | Storage  | Compression | Stats target |                                        Description                                         
-----------------+--------------------------+-----------+----------+---------+----------+-------------+--------------+--------------------------------------------------------------------------------------------
 user_nid        | integer                  |           | not null |         | plain    |             |              | The user for which we are recording presence
 last_active     | timestamp with time zone |           | not null | now()   | plain    |             |              | The time at which the user was last active
 status          | character varying(255)   |           |          |         | extended |             |              | A custom status the user can set
 manual_presence | character varying(100)   |           |          |         | extended |             |              | Users can set their presence manually with PUT /_matrix/client/v3/presence/{userId}/status
Indexes:
    "presence_pkey" PRIMARY KEY, btree (user_nid)
Foreign-key constraints:
    "presence_user_nid_fkey" FOREIGN KEY (user_nid) REFERENCES users(nid)
Access method: heap

previous_events

Table registering which events precede which

                                                                          Table "public.previous_events"
       Column       |  Type  | Collation | Nullable | Default | Storage | Compression | Stats target |                                 Description                                 
--------------------+--------+-----------+----------+---------+---------+-------------+--------------+-----------------------------------------------------------------------------
 previous_event_nid | bigint |           | not null |         | plain   |             |              | The event which precedes the event referenced by `event_nid`
 event_nid          | bigint |           | not null |         | plain   |             |              | The event which predecessor is the event referenced by `previous_event_nid`
Foreign-key constraints:
    "previous_events_event_nid_fkey" FOREIGN KEY (event_nid) REFERENCES events(nid) ON DELETE CASCADE
    "previous_events_previous_event_nid_fkey" FOREIGN KEY (previous_event_nid) REFERENCES events(nid) ON DELETE CASCADE
Access method: heap

read_receipts

Users room read receipts

                                                                                                                                 Table "public.read_receipts"
    Column    |           Type           | Collation | Nullable |                     Default                      | Storage  | Compression | Stats target |                                                           Description                                                            
--------------+--------------------------+-----------+----------+--------------------------------------------------+----------+-------------+--------------+----------------------------------------------------------------------------------------------------------------------------------
 nid          | integer                  |           | not null | generated by default as identity                 | plain    |             |              | Surrogate key for the read receipt
 room_nid     | integer                  |           | not null |                                                  | plain    |             |              | The room of the read receipt
 user_id      | text                     |           | not null |                                                  | extended |             |              | The user owning the read receipt
 event_nid    | integer                  |           | not null |                                                  | plain    |             |              | The event ID the receipt acknowledges up to
 receipt_type | receipt_type             |           | not null |                                                  | plain    |             |              | The type of read receipt
 thread       | text                     |           |          |                                                  | extended |             |              | Optional. The thread the receipt applies to
 ts           | timestamp with time zone |           | not null |                                                  | plain    |             |              | The timestamp the receipt was sent at
 stream_id    | bigint                   |           | not null | locking_nextval('read_receipts_streamids'::text) | plain    |             |              | Stream ID of the users read receipts. See https://bundesmessenger.gitlab.io/backend/neo/arminius/arminius/architecture/sync.html
Indexes:
    "read_receipts_pkey" PRIMARY KEY, btree (nid)
    "read_receipts_room_nid_user_id_receipt_type_thread_key" UNIQUE CONSTRAINT, btree (room_nid, user_id, receipt_type, thread) NULLS NOT DISTINCT
    "read_receipts_stream_id_key" UNIQUE CONSTRAINT, btree (stream_id)
Foreign-key constraints:
    "read_receipts_event_nid_fkey" FOREIGN KEY (event_nid) REFERENCES events(nid) ON DELETE CASCADE
    "read_receipts_room_nid_fkey" FOREIGN KEY (room_nid) REFERENCES rooms(nid) ON DELETE CASCADE
Access method: heap

room_account_data

Room specific account data for each user

                                                                                                                    Table "public.room_account_data"
  Column   |  Type   | Collation | Nullable |                     Default                     | Storage  | Compression | Stats target |                                                           Description                                                            
-----------+---------+-----------+----------+-------------------------------------------------+----------+-------------+--------------+----------------------------------------------------------------------------------------------------------------------------------
 user_nid  | integer |           | not null |                                                 | plain    |             |              | The user to which this room-specific account data belongs
 room_nid  | integer |           | not null |                                                 | plain    |             |              | The room to which this room-specific account data belongs
 data_type | text    |           | not null |                                                 | extended |             |              | See documentation for the table `global_account_data`
 data      | jsonb   |           | not null |                                                 | extended |             |              | An arbitrary JSON blob, the actual uploaded data
 stream_id | bigint  |           | not null | locking_nextval('account_data_streamids'::text) | plain    |             |              | Stream ID of the user's account data. See https://bundesmessenger.gitlab.io/backend/neo/arminius/arminius/architecture/sync.html
Indexes:
    "room_account_data_pkey" PRIMARY KEY, btree (user_nid, room_nid, data_type)
    "room_account_data_stream_id_key" UNIQUE CONSTRAINT, btree (stream_id)
Foreign-key constraints:
    "room_account_data_room_nid_fkey" FOREIGN KEY (room_nid) REFERENCES rooms(nid) ON DELETE CASCADE
    "room_account_data_user_nid_fkey" FOREIGN KEY (user_nid) REFERENCES users(nid) ON DELETE CASCADE
Access method: heap

room_keys

Room keys stored in backups

                                                             Table "public.room_keys"
        Column         |  Type   | Collation | Nullable | Default | Storage  | Compression | Stats target |              Description               
-----------------------+---------+-----------+----------+---------+----------+-------------+--------------+----------------------------------------
 room_keys_version_nid | integer |           | not null |         | plain    |             |              | Link to the backup keys are part of
 room_nid              | integer |           | not null |         | plain    |             |              | Link to the room the keys belong to
 session_id            | text    |           | not null |         | extended |             |              | Name of the session the keys belong to
 backup_data           | jsonb   |           | not null |         | extended |             |              | Actual room keys backup content
Indexes:
    "room_keys_room_keys_version_nid_room_nid_session_id_key" UNIQUE CONSTRAINT, btree (room_keys_version_nid, room_nid, session_id)
Foreign-key constraints:
    "room_keys_room_keys_version_nid_fkey" FOREIGN KEY (room_keys_version_nid) REFERENCES room_keys_versions(nid) ON DELETE CASCADE
    "room_keys_room_nid_fkey" FOREIGN KEY (room_nid) REFERENCES rooms(nid) ON DELETE CASCADE
Access method: heap

room_keys_versions

Room keys backups

                                                                    Table "public.room_keys_versions"
  Column   |  Type   | Collation | Nullable |             Default              | Storage  | Compression | Stats target |                   Description                   
-----------+---------+-----------+----------+----------------------------------+----------+-------------+--------------+-------------------------------------------------
 nid       | integer |           | not null | generated by default as identity | plain    |             |              | Surrogate key for room_keys_versions
 user_nid  | integer |           |          |                                  | plain    |             |              | User owning the room keys backup
 version   | bigint  |           | not null |                                  | plain    |             |              | Version of the room keys backup
 auth_data | jsonb   |           | not null |                                  | extended |             |              | Algorithm dependent data of the backup
 etag      | bigint  |           | not null | 1                                | plain    |             |              | Unique integer identifying the backup's version
Indexes:
    "room_keys_versions_pkey" PRIMARY KEY, btree (nid)
    "room_keys_versions_user_nid_version_key" UNIQUE CONSTRAINT, btree (user_nid, version)
Foreign-key constraints:
    "room_keys_versions_user_nid_fkey" FOREIGN KEY (user_nid) REFERENCES users(nid) ON DELETE CASCADE
Referenced by:
    TABLE "room_keys" CONSTRAINT "room_keys_room_keys_version_nid_fkey" FOREIGN KEY (room_keys_version_nid) REFERENCES room_keys_versions(nid) ON DELETE CASCADE
Access method: heap

room_state_snapshots

The state of a room, when all the latest_events have been resolved

                                                         Table "public.room_state_snapshots"
       Column       |  Type   | Collation | Nullable | Default | Storage | Compression | Stats target |                 Description                  
--------------------+---------+-----------+----------+---------+---------+-------------+--------------+----------------------------------------------
 room_nid           | integer |           | not null |         | plain   |             |              | The room for which we are storing the state
 state_snapshot_nid | bigint  |           | not null |         | plain   |             |              | The state snapshot which applies to the room
Indexes:
    "room_state_snapshots_pkey" PRIMARY KEY, btree (room_nid)
Foreign-key constraints:
    "room_state_snapshots_room_nid_fkey" FOREIGN KEY (room_nid) REFERENCES rooms(nid) ON DELETE CASCADE
    "room_state_snapshots_state_snapshot_nid_fkey" FOREIGN KEY (state_snapshot_nid) REFERENCES state_snapshots(nid) ON DELETE CASCADE
Access method: heap

rooms

Rooms known to server

                                                                              Table "public.rooms"
    Column    |  Type   | Collation | Nullable |             Default              | Storage  | Compression | Stats target |                     Description                     
--------------+---------+-----------+----------+----------------------------------+----------+-------------+--------------+-----------------------------------------------------
 nid          | integer |           | not null | generated by default as identity | plain    |             |              | Surrogate key for the room
 room_id      | text    | C         | not null |                                  | extended |             |              | Room ID, as used in the Matrix protocol
 room_version | text    |           | not null |                                  | extended |             |              | Room version, as defined in the Matrix protocol
 public       | boolean |           | not null | false                            | plain    |             |              | Whether to show the room in the published room list
Indexes:
    "rooms_pkey" PRIMARY KEY, btree (nid)
    "rooms_room_id_key" UNIQUE CONSTRAINT, btree (room_id)
Referenced by:
    TABLE "event_ordering" CONSTRAINT "event_ordering_room_nid_fkey" FOREIGN KEY (room_nid) REFERENCES rooms(nid) ON DELETE CASCADE
    TABLE "event_transactions" CONSTRAINT "event_transactions_room_nid_fkey" FOREIGN KEY (room_nid) REFERENCES rooms(nid) ON DELETE CASCADE
    TABLE "events" CONSTRAINT "events_room_nid_fkey" FOREIGN KEY (room_nid) REFERENCES rooms(nid) ON DELETE CASCADE
    TABLE "forgotten_rooms" CONSTRAINT "forgotten_rooms_room_nid_fkey" FOREIGN KEY (room_nid) REFERENCES rooms(nid) ON DELETE CASCADE
    TABLE "leaf_events" CONSTRAINT "leaf_events_room_nid_fkey" FOREIGN KEY (room_nid) REFERENCES rooms(nid) ON DELETE CASCADE
    TABLE "read_receipts" CONSTRAINT "read_receipts_room_nid_fkey" FOREIGN KEY (room_nid) REFERENCES rooms(nid) ON DELETE CASCADE
    TABLE "room_account_data" CONSTRAINT "room_account_data_room_nid_fkey" FOREIGN KEY (room_nid) REFERENCES rooms(nid) ON DELETE CASCADE
    TABLE "room_keys" CONSTRAINT "room_keys_room_nid_fkey" FOREIGN KEY (room_nid) REFERENCES rooms(nid) ON DELETE CASCADE
    TABLE "room_state_snapshots" CONSTRAINT "room_state_snapshots_room_nid_fkey" FOREIGN KEY (room_nid) REFERENCES rooms(nid) ON DELETE CASCADE
    TABLE "syncv4_room_streams" CONSTRAINT "syncv4_room_streams_room_nid_fkey" FOREIGN KEY (room_nid) REFERENCES rooms(nid)
Access method: heap

scim_sync_token

Table used to store the single sync token

                                                             Table "public.scim_sync_token"
   Column   |  Type  | Collation | Nullable | Default | Storage  | Compression | Stats target |                       Description                        
------------+--------+-----------+----------+---------+----------+-------------+--------------+----------------------------------------------------------
 nid        | bigint |           | not null | 0       | plain    |             |              | ID (always 0, because there is only a single sync token)
 sync_token | text   |           | not null |         | extended |             |              | the latest sync token returned by the SCIM server
Indexes:
    "scim_sync_token_pkey" PRIMARY KEY, btree (nid)
Check constraints:
    "scim_sync_token_nid_check" CHECK (nid = 0)
Access method: heap

server_key_responses

Most recent responses to key queries on other servers. Used for notary key queries

                                                         Table "public.server_key_responses"
   Column    | Type | Collation | Nullable | Default | Storage  | Compression | Stats target |                      Description                      
-------------+------+-----------+----------+---------+----------+-------------+--------------+-------------------------------------------------------
 server_name | text |           | not null |         | extended |             |              | The name of the server from which we got the response
 response    | text |           | not null |         | extended |             |              | The actual signed canonical JSON response
Indexes:
    "server_key_responses_pkey" PRIMARY KEY, btree (server_name)
Access method: heap

server_private_keys

Keys used by this homeserver

                                                                               Table "public.server_private_keys"
   Column    |           Type           | Collation | Nullable | Default | Storage  | Compression | Stats target |                                 Description                                  
-------------+--------------------------+-----------+----------+---------+----------+-------------+--------------+------------------------------------------------------------------------------
 key_version | text                     |           | not null |         | extended |             |              | Alphanumeric string uniquely identifying this key for the homeserver
 key_pair    | bytea                    |           |          |         | extended |             |              | The key pair as a DER-encoded PKCS#8 v2 document w. public key, used by ruma
 verify_key  | bytea                    |           | not null |         | extended |             |              | The public key
 created_at  | timestamp with time zone |           | not null | now()   | plain    |             |              | The time at which the key was created
 expired_at  | timestamp with time zone |           |          |         | plain    |             |              | The time at which the key expired
Indexes:
    "server_private_keys_pkey" PRIMARY KEY, btree (key_version)
Check constraints:
    "nonexpired_keys_available" CHECK (expired_at IS NOT NULL OR key_pair IS NOT NULL)
Access method: heap

server_public_keys

Public keys for other homeservers

                                                            Table "public.server_public_keys"
    Column     |           Type           | Collation | Nullable | Default | Storage  | Compression | Stats target |             Description              
---------------+--------------------------+-----------+----------+---------+----------+-------------+--------------+--------------------------------------
 server_name   | text                     |           | not null |         | extended |             |              | The server which the keys belongs to
 key_algorithm | text                     |           | not null |         | extended |             |              | The algorithm used for the key
 key_name      | text                     |           | not null |         | extended |             |              | The name used for the key
 public_key    | bytea                    |           | not null |         | extended |             |              | The actual key
 expires_at    | timestamp with time zone |           | not null |         | plain    |             |              | The time at which the key expires
Indexes:
    "server_public_keys_pkey" PRIMARY KEY, btree (server_name, key_name, key_algorithm)
Access method: heap

state_snapshot_state_events

Table storing relations between state snapshots, and the state events that make them up

                                                        Table "public.state_snapshot_state_events"
       Column       |  Type  | Collation | Nullable | Default | Storage | Compression | Stats target |                    Description                     
--------------------+--------+-----------+----------+---------+---------+-------------+--------------+----------------------------------------------------
 state_snapshot_nid | bigint |           | not null |         | plain   |             |              | The state snapshot which contains some state event
 state_event_nid    | bigint |           | not null |         | plain   |             |              | A state event which is part of some snapshot
Indexes:
    "state_snapshot_state_events_state_snapshot_nid_state_event__key" UNIQUE CONSTRAINT, btree (state_snapshot_nid, state_event_nid)
Foreign-key constraints:
    "state_snapshot_state_events_state_event_nid_fkey" FOREIGN KEY (state_event_nid) REFERENCES events(nid) ON DELETE CASCADE
    "state_snapshot_state_events_state_snapshot_nid_fkey" FOREIGN KEY (state_snapshot_nid) REFERENCES state_snapshots(nid) ON DELETE CASCADE
Triggers:
    maintain_state_snapshot_pointers AFTER INSERT OR DELETE ON state_snapshot_state_events FOR EACH ROW EXECUTE FUNCTION maintain_state_snapshot_pointers()
Access method: heap

state_snapshots

Surrogate table to store collections of state events. Used to store state for events and rooms

                                                                                                                                       Table "public.state_snapshots"
            Column            |  Type  | Collation | Nullable |             Default              | Storage | Compression | Stats target |                                                                            Description                                                                             
------------------------------+--------+-----------+----------+----------------------------------+---------+-------------+--------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------
 nid                          | bigint |           | not null | generated by default as identity | plain   |             |              | The numerical ID of a event state
 server_acl_event_nid         | bigint |           |          |                                  | plain   |             |              | The effective `m.room.server_acl` event in this snapshot, if any. `m.room.server_acl` always has an empty state key, as per the spec (v1.19/client-server/10.32.2)
 history_visibility_event_nid | bigint |           |          |                                  | plain   |             |              | The effective `m.room.history_visibility` event in this snapshot, if any
Indexes:
    "state_snapshots_pkey" PRIMARY KEY, btree (nid)
    "state_snapshots_history_visibility_event_nid" btree (history_visibility_event_nid) WHERE history_visibility_event_nid IS NOT NULL
    "state_snapshots_server_acl_event_nid" btree (server_acl_event_nid) WHERE server_acl_event_nid IS NOT NULL
Foreign-key constraints:
    "state_snapshots_history_visibility_event_nid_fkey" FOREIGN KEY (history_visibility_event_nid) REFERENCES events(nid) ON DELETE SET NULL
    "state_snapshots_server_acl_event_nid_fkey" FOREIGN KEY (server_acl_event_nid) REFERENCES events(nid) ON DELETE SET NULL
Referenced by:
    TABLE "events_state_snapshots" CONSTRAINT "events_state_snapshots_state_snapshot_nid_fkey" FOREIGN KEY (state_snapshot_nid) REFERENCES state_snapshots(nid) ON DELETE CASCADE
    TABLE "room_state_snapshots" CONSTRAINT "room_state_snapshots_state_snapshot_nid_fkey" FOREIGN KEY (state_snapshot_nid) REFERENCES state_snapshots(nid) ON DELETE CASCADE
    TABLE "state_snapshot_state_events" CONSTRAINT "state_snapshot_state_events_state_snapshot_nid_fkey" FOREIGN KEY (state_snapshot_nid) REFERENCES state_snapshots(nid) ON DELETE CASCADE
Access method: heap

stripped_state_events

Stripped state events, which make up federated invites

                                                          Table "public.stripped_state_events"
  Column   |  Type  | Collation | Nullable |             Default              | Storage  | Compression | Stats target |           Description           
-----------+--------+-----------+----------+----------------------------------+----------+-------------+--------------+---------------------------------
 nid       | bigint |           | not null | generated by default as identity | plain    |             |              | Surrogate key for the event
 content   | json   |           | not null |                                  | extended |             |              | The actual content of the event
 sender    | text   |           | not null |                                  | extended |             |              | The sender of the event
 state_key | text   |           | not null |                                  | extended |             |              | The state_key of the event
 type      | text   |           | not null |                                  | extended |             |              | The event type
Indexes:
    "stripped_state_events_pkey" PRIMARY KEY, btree (nid)
Access method: heap

syncv4_connections

Users sync v4 connections. Linked to (user_id, device_id, conn_id) triplets.

                                                                                 Table "public.syncv4_connections"
   Column   |  Type   | Collation | Nullable |             Default              | Storage  | Compression | Stats target |                               Description                                
------------+---------+-----------+----------+----------------------------------+----------+-------------+--------------+--------------------------------------------------------------------------
 nid        | bigint  |           | not null | generated by default as identity | plain    |             |              | Surrogate key of the connection.
 device_nid | integer |           | not null |                                  | plain    |             |              | Link to the device owning this connection
 conn_id    | text    |           |          |                                  | extended |             |              | User-set connection ID. Can be null. Must be unique per-user per-device.
Indexes:
    "syncv4_connections_pkey" PRIMARY KEY, btree (nid)
    "syncv4_connections_device_nid_conn_id_key" UNIQUE CONSTRAINT, btree (device_nid, conn_id) NULLS NOT DISTINCT
Foreign-key constraints:
    "syncv4_connections_device_nid_fkey" FOREIGN KEY (device_nid) REFERENCES devices(nid) ON DELETE CASCADE
Referenced by:
    TABLE "syncv4_positions" CONSTRAINT "syncv4_positions_connection_nid_fkey" FOREIGN KEY (connection_nid) REFERENCES syncv4_connections(nid) ON DELETE CASCADE
Access method: heap

syncv4_positions

Sync v4 positions within their connections. We store one per response.

                                                                                                                                Table "public.syncv4_positions"
     Column     |  Type  | Collation | Nullable |                    Default                    | Storage | Compression | Stats target |                                                                      Description                                                                       
----------------+--------+-----------+----------+-----------------------------------------------+---------+-------------+--------------+--------------------------------------------------------------------------------------------------------------------------------------------------------
 position       | bigint |           | not null | locking_nextval('syncv4_pos_streamids'::text) | plain   |             |              | Surrogate key of the position. This is a global stream ID (See https://bundesmessenger.gitlab.io/backend/neo/arminius/arminius/architecture/sync.html)
 connection_nid | bigint |           | not null |                                               | plain   |             |              | Link to the connection this position belongs to
 prev_position  | bigint |           |          |                                               | plain   |             |              | Link to the parent position within the same connection. Will be null if it's the first ever position in the connection, or if the parent got deleted.
Indexes:
    "syncv4_positions_position_key" UNIQUE CONSTRAINT, btree ("position")
Foreign-key constraints:
    "syncv4_positions_connection_nid_fkey" FOREIGN KEY (connection_nid) REFERENCES syncv4_connections(nid) ON DELETE CASCADE
    "syncv4_positions_prev_position_fkey" FOREIGN KEY (prev_position) REFERENCES syncv4_positions("position") ON DELETE SET NULL
Referenced by:
    TABLE "syncv4_positions" CONSTRAINT "syncv4_positions_prev_position_fkey" FOREIGN KEY (prev_position) REFERENCES syncv4_positions("position") ON DELETE SET NULL
    TABLE "syncv4_room_streams" CONSTRAINT "syncv4_room_streams_position_fkey" FOREIGN KEY ("position") REFERENCES syncv4_positions("position") ON DELETE CASCADE
Access method: heap

syncv4_room_streams

Tracks, for each stream, for each room, what was already sent down the connection

                                                                                   Table "public.syncv4_room_streams"
    Column     |     Type      | Collation | Nullable | Default | Storage | Compression | Stats target |                                          Description                                           
---------------+---------------+-----------+----------+---------+---------+-------------+--------------+------------------------------------------------------------------------------------------------
 position      | bigint        |           | not null |         | plain   |             |              | Link to the position that sent this stream
 stream_name   | stream_name   |           | not null |         | plain   |             |              | Name of the stream this entry applies to
 room_nid      | integer       |           | not null |         | plain   |             |              | Link to the room this entry applies to
 stream_status | stream_status |           | not null |         | plain   |             |              | LIVE means the room was fully caught up. PREVIOUSLY means we sent up to the `up_to` stream ID.
 up_to         | bigint        |           |          |         | plain   |             |              | Set only if the `stream_status` is PREVIOUSLY. Indicates up to which `stream_id` we sent
Indexes:
    "syncv4_room_streams_position_stream_name_room_nid_key" UNIQUE CONSTRAINT, btree ("position", stream_name, room_nid)
Foreign-key constraints:
    "syncv4_room_streams_position_fkey" FOREIGN KEY ("position") REFERENCES syncv4_positions("position") ON DELETE CASCADE
    "syncv4_room_streams_room_nid_fkey" FOREIGN KEY (room_nid) REFERENCES rooms(nid)
Access method: heap

to_device_messages

Messages sent to local devices. See https://spec.matrix.org/v1.15/client-server-api/#send-to-device-messaging

                                                                                                             Table "public.to_device_messages"
        Column         |  Type  | Collation | Nullable |                      Default                      | Storage  | Compression | Stats target |                                              Description                                              
-----------------------+--------+-----------+----------+---------------------------------------------------+----------+-------------+--------------+-------------------------------------------------------------------------------------------------------
 to_device_message_nid | bigint |           | not null | generated by default as identity                  | plain    |             |              | Surrogate key for the message
 device_nid            | bigint |           | not null |                                                   | plain    |             |              | The intended recipient of the message
 event_type            | text   |           | not null |                                                   | extended |             |              | A string, representing the event type, for instance `m.new_device`
 event_content         | json   |           | not null |                                                   | extended |             |              | The content, an arbitrary JSON blob
 sender                | text   |           | not null |                                                   | extended |             |              | The user who sent the event
 stream_id             | bigint |           | not null | locking_nextval('device_message_streamids'::text) | plain    |             |              | Stream id. See https://bundesmessenger.gitlab.io/backend/neo/arminius/arminius/architecture/sync.html
Indexes:
    "to_device_messages_pkey" PRIMARY KEY, btree (to_device_message_nid)
    "to_device_messages_stream_id_key" UNIQUE CONSTRAINT, btree (stream_id)
Foreign-key constraints:
    "to_device_messages_device_nid_fkey" FOREIGN KEY (device_nid) REFERENCES devices(nid) ON DELETE CASCADE
Access method: heap

user_keys

User device keys and cross-signing keys

                                                                                                                   Table "public.user_keys"
   Column   |   Type   | Collation | Nullable |                   Default                    | Storage  | Compression | Stats target |                                                       Description                                                       
------------+----------+-----------+----------+----------------------------------------------+----------+-------------+--------------+-------------------------------------------------------------------------------------------------------------------------
 nid        | bigint   |           | not null | generated by default as identity             | plain    |             |              | Surrogate key for user_keys
 key_type   | key_type |           | not null |                                              | plain    |             |              | Type of the key, see the `KEY_TYPE` enum
 user_nid   | integer  |           | not null |                                              | plain    |             |              | ID of the user owning the key
 device_nid | integer  |           |          |                                              | plain    |             |              | ID of the device owning the key iff key_type=device
 key_id     | text     |           | not null |                                              | extended |             |              | Matrix key ID
 key_object | jsonb    |           | not null |                                              | extended |             |              | The actual key object
 stream_id  | bigint   |           | not null | locking_nextval('user_keys_streamids'::text) | plain    |             |              | Stream ID of the users keys. See https://bundesmessenger.gitlab.io/backend/neo/arminius/arminius/architecture/sync.html
Indexes:
    "user_keys_pkey" PRIMARY KEY, btree (nid)
    "user_keys_key_type_user_nid_device_nid_key" UNIQUE CONSTRAINT, btree (key_type, user_nid, device_nid) NULLS NOT DISTINCT
    "user_keys_stream_id_key" UNIQUE CONSTRAINT, btree (stream_id)
Check constraints:
    "user_keys_check" CHECK (key_type <> 'device'::key_type AND device_nid = NULL::integer OR key_type = 'device'::key_type AND device_nid <> NULL::integer)
Foreign-key constraints:
    "user_keys_device_nid_fkey" FOREIGN KEY (device_nid) REFERENCES devices(nid) ON DELETE CASCADE
    "user_keys_user_nid_fkey" FOREIGN KEY (user_nid) REFERENCES users(nid) ON DELETE CASCADE
Referenced by:
    TABLE "local_signatures" CONSTRAINT "local_signatures_signed_key_nid_fkey" FOREIGN KEY (signed_key_nid) REFERENCES user_keys(nid) ON DELETE CASCADE
Access method: heap

users

Local Users

                                                                                Table "public.users"
     Column      |  Type   | Collation | Nullable |             Default              | Storage  | Compression | Stats target |                     Description                      
-----------------+---------+-----------+----------+----------------------------------+----------+-------------+--------------+------------------------------------------------------
 nid             | integer |           | not null | generated by default as identity | plain    |             |              | Surrogate key for the user
 user_id         | text    |           | not null |                                  | extended |             |              | Mxid for the user
 idp_id          | text    |           |          |                                  | extended |             |              | ID for the user with the identity provider
 display_name    | text    |           |          |                                  | extended |             |              | The users human-readable display name
 avatar_id       | text    |           |          |                                  | extended |             |              | Link to the users avatar
 keys_backup_seq | bigint  |           | not null | 0                                | plain    |             |              | Version number of the user's latest room keys backup
Indexes:
    "users_pkey" PRIMARY KEY, btree (nid)
    "users_idp_id_key" UNIQUE CONSTRAINT, btree (idp_id)
    "users_user_id_key" UNIQUE CONSTRAINT, btree (user_id)
Referenced by:
    TABLE "devices" CONSTRAINT "devices_user_nid_fkey" FOREIGN KEY (user_nid) REFERENCES users(nid) DEFERRABLE
    TABLE "federated_invites" CONSTRAINT "federated_invites_user_nid_fkey" FOREIGN KEY (user_nid) REFERENCES users(nid)
    TABLE "federated_knocks" CONSTRAINT "federated_knocks_user_nid_fkey" FOREIGN KEY (user_nid) REFERENCES users(nid)
    TABLE "filters" CONSTRAINT "filters_user_nid_fkey" FOREIGN KEY (user_nid) REFERENCES users(nid) ON DELETE CASCADE
    TABLE "forgotten_rooms" CONSTRAINT "forgotten_rooms_user_nid_fkey" FOREIGN KEY (user_nid) REFERENCES users(nid) ON DELETE CASCADE
    TABLE "global_account_data" CONSTRAINT "global_account_data_user_nid_fkey" FOREIGN KEY (user_nid) REFERENCES users(nid) ON DELETE CASCADE
    TABLE "local_signatures" CONSTRAINT "local_signatures_signing_user_nid_fkey" FOREIGN KEY (signing_user_nid) REFERENCES users(nid) ON DELETE CASCADE
    TABLE "presence" CONSTRAINT "presence_user_nid_fkey" FOREIGN KEY (user_nid) REFERENCES users(nid)
    TABLE "room_account_data" CONSTRAINT "room_account_data_user_nid_fkey" FOREIGN KEY (user_nid) REFERENCES users(nid) ON DELETE CASCADE
    TABLE "room_keys_versions" CONSTRAINT "room_keys_versions_user_nid_fkey" FOREIGN KEY (user_nid) REFERENCES users(nid) ON DELETE CASCADE
    TABLE "user_keys" CONSTRAINT "user_keys_user_nid_fkey" FOREIGN KEY (user_nid) REFERENCES users(nid) ON DELETE CASCADE
Access method: heap

verification_metadata

Metadata recorded when attempting to verify the signature of an event

                                                                                      Table "public.verification_metadata"
         Column         |           Type           | Collation | Nullable | Default | Storage | Compression | Stats target |                                     Description                                     
------------------------+--------------------------+-----------+----------+---------+---------+-------------+--------------+-------------------------------------------------------------------------------------
 event_nid              | bigint                   |           | not null |         | plain   |             |              | The ID of the event that verification was attempted for
 attempted_at           | timestamp with time zone |           | not null |         | plain   |             |              | Timestamp when verification was attempted
 signature_verification | parts_verified           |           | not null |         | plain   |             |              | Signature verification outcome for the event
 soft_failed            | boolean                  |           | not null |         | plain   |             |              | Whether the event was soft-failed, i.e. authorization against the room state failed
Indexes:
    "verification_metadata_pkey" PRIMARY KEY, btree (event_nid)
Foreign-key constraints:
    "verification_metadata_event_nid_fkey" FOREIGN KEY (event_nid) REFERENCES events(nid) ON DELETE CASCADE
Access method: heap

Rate Limiting

Overview

The goal of rate limiting is to control the amount of incoming and outgoing traffic to protect system stability, prevent resource exhaustion, and defend against certain types of attacks.

Key Concepts

Requests per Second (RPS)

The number of API calls allowed within a one-second timeframe.

Rate Limit Window

The time period during which requests are counted.

Throttling

Temporarily slowing down or delaying requests when limits are approached.

Quota

The maximum number of requests allowed in a longer period (e.g., daily/monthly).

Common Implementation Methods

Token Bucket

A token bucket algorithm allows for bursts of requests while maintaining an average rate limit. Tokens are added to a bucket at a constant rate, and each request consumes a token. When the bucket is empty, further requests are throttled.

Leaky Bucket

Similar to token bucket but processes requests at a constant rate, removing bursts. Requests enter a queue and are processed at a fixed rate.

Fixed Window

Divides time into fixed windows (e.g., 1-minute intervals) and allows a certain number of requests per window. Simple but can allow request spikes at window boundaries.

Sliding Window

Considers a rolling time period rather than fixed intervals, providing smoother rate limiting. More complex to implement but avoids boundary problems.

HTTP Response Headers

{
  "X-RateLimit-Limit": "100",       // Total requests allowed in period
  "X-RateLimit-Remaining": "75",    // Requests remaining in current period
  "X-RateLimit-Reset": "1627834096" // Timestamp when the limit resets
}

Rate Limiting Strategies

IP-Based

Limits requests based on client IP address.

User-Based

Limits requests based on authenticated user.

Room-Based

Room based. What did you expect?

File-Based

Limit too much access on a certain file or resource.

Service-Based

Limits for specific services or microservices → Rate limit if a stream is falling behind.

Federation Transactions

Deny transactions, if federating servers act maliciously (because of malicious users, for example).

Handling Rate Limit Exceeding

When a client exceeds rate limits, the API should respond with a 429 (Too Many Requests) status code and provide information about when the client can retry.

HTTP/1.1 429 Too Many Requests
Content-Type: application/json
Retry-After: 30

{
  "errcode": "M_LIMIT_EXCEEDED",
  "error": "Too many requests",
  "retry_after_ms": 30000
}

Best Practices

• Implement exponential backoff in clients for graceful handling of rate limits
• Provide clear documentation of rate limits for API consumers
• Consider different tiers of rate limits for different customer segments
• Monitor and analyze rate limit hits to adjust limits as needed
• Implement soft limits with warnings before hard limits are reached

Tools and Libraries

• NATS: High-performance messaging system for distributed rate limiting
• Rust libraries: axum, tower, tower-http
• Custom implementation: Building rate limiting directly in Rust services using NATS for coordination
• Alternative options: Rate limit (Lyft), Envoy for API gateway rate limiting

How Do We Make Rate Limits Configurable? What Are Our Requirements?

• Per power level
• scalable, reliable: emergency announcements should be retrievable
• Some delay is acceptable, but we don’t want an interruption of the service.
  • General limits (x requests per second) may have a delay before kicking in
  • Blocking a foreign server should be instant

Storing JSON Payloads in PostgreSQL

Arminius ingests and stores JSON payloads, some of which are cryptographically signed. For these signed payloads, it is critical that the exact byte-for-byte representation is preserved.

Decisions

Signed JSON Payloads Will Be Stored Using PostgreSQL’s json Data Type

This type stores an exact copy of the input text, preserving all whitespace and key order, which is essential for validating cryptographic signatures. The downside is that such fields need to be validated and parsed whenever they are queried or selected. Also, they cannot be indexed, leading to full table scans in most situations.

Unsigned JSON Payloads Will Be Stored Using PostgreSQL’s jsonb Data Type

This type stores json data in an optimized binary format, which is more efficient with regards to parsing and storage. Whitespaces and key order are not preserved for this type, which should not be a problem for non-signed payloads.

All JSON Fields That Are Frequently Queried or Joined Will Also Be Stored as Normalized, Single-Value Columns

This enables the creation of standard indexes on these columns, ensuring efficient query and join performance, which is not always optimal when querying inside JSON or JSONB blobs.

Alternatives Considered

Storing All JSON as jsonb

Rejected due to the need for exact textual preservation for signed payloads, which jsonb does not guarantee.

Storing Only as Text

Rejected because json and jsonb types enforce JSON validity.

Sync and Message Storage

Matrix has an endpoint /sync, and other related endpoints, which allows users to retrieve a stream of events. This needs to be considered when implementing event storage if we want to support it at a scale.

Requirements for Message Storage and Retrieval

• We want to let clients paginate events and fetch them in real time, using the following endpoints:
  • The V3 sync endpoint
  • The /messages endpoint
  • The sliding sync endpoint
• We want to do relational-style queries on events, to find the current state for instance, or find rooms the user has been invited to
• We want to be able to fetch old events quickly
• For unencrypted rooms, we want to support server-side full-text search
• We want to be able to insert events at arbitrary points in the timeline, if the federation has been partitioned for a while
• We want to be able to support multiple events together
  • When creating a new room, we want to ensure that all the initial events are considered as a bundle. We don’t want a partial room creation.

Sync’s Batch Token

Batch token is a string token that includes the information about the state event streams (Both PDUs and EDUs) in the server. This token is given to the user for different purposes.

In the legacy /sync it is used to fill the since query parameter. This string is not created by the client, rather it is created by the server from previous /sync calls as the next_batch field in the body of the response.

For us this Token looks like this s_123571293 where the first part s_ specifies that it is a sync token, and the other parts are specifying positions in the different streams of events so that the server know until which event the client has received events from that stream.

So for now, the current structure is s_<stream_id>, where the stream_id is specifying the position of the stream for room events (or PDUs).

Roomserver Solution

The roomserver solution would involve, for each sync request, the client service making a request to each room service, such that all rooms were queried for updates. This, however, would make syncing very expensive. Since roomservers are in charge of reading and writing to rooms, sync requests would block writing to a room. The alternative, where the client API makes direct SQL queries to the database, would block far less. In larger deployments, there is also the option of having a secondary, read-only database for the sync API.

Naive Solution

The naive solution for this would be to use the primary key for the events table, which we know is given by a postgresql sequence.

But this has the issue that SQL sequences can end up committed out of order.

Essentially, as transaction A inserts an event, it reserves event id 1. A second writer will start transaction B, insert an event, and reserve event 2. But it is entirely possible that transaction B will complete before transaction A, resulting in a database where there is an event 2, but no event 1. If we then implement sync by paginating through event ids, we will return event id 2, and the next page should logically start at event id 2. But doing this, the event with id 1 will never end up in the stream.

Current Solution

TL;DR

We use an advisory lock to restrict our reads to sequence numbers that have been committed. In crates/libs/db/arminius-lib-db/migrations/00_locking_sequence.sql we define two functions, locking_nextval() and max_safe_seq(), which can be used for this purpose.

When fetching events from the database for sync responses, we simply include a WHERE id <= max_safe_seq('eventids') in order to make sure we don’t advance our index past the point where all sequence numbers belong to committed rows.

The relevant part of the since token is then just the highest stream_id returned.

By only restricting the reads, instead of restricting the writes we avoid throughput being throttled. But a long-running transaction will also hinder all syncs in advancing. The main goal of implementing sync using advisory locks was getting a solution rapidly, which is guaranteed to work, and enabled us to begin implementing the rest of the homeserver. However, this solution has not been tested at all with large server loads, and we therefore expect it to fail. We assume that this decision will have to be revisited at a later point, when we are able to test our homeserver at high load.

On Insertion of Events:

We have a table called events.

On each row, there is a column called “stream_id,” which is a unique 64-bit signed integer.

Filling of this field happens using a locked incrementing sequence (called eventids) by calling the locking_nextval function. This function is called by default inside PostgreSQL.

On the Reading of Events:

This solution is (almost) irrelevant when there is no since token and the full state is required. So we just ignore that part.

So there are two tokens that are important to us

• The since token that the clients send in the request.
• The next_batch token that we should give the user in the response.

Therefore, simply put, the server asks PostgreSQL to give us every event with a stream_id greater than the event stream_id (that is parsed from the since token) and smaller than or equivalent to the max safe sequence number.

How the max safe sequence is decided is based on an SQL function we have already defined called max_safe_seq.

max_safe_seq looks into every lock for this sequence, finds the minimum number locked in the sequence, and returns that, or if there is no lock, it returns the maximum 64-bit signed integer (9_223_372_036_854_775_807).

The next_batch token is the maximum stream_id of the events returned.

The Expected Behavior

Here is the entire timeline I will use to explain the characteristics of the system.

I will continue with a detailed explanation of each part.

sequenceDiagram
    autonumber
    actor c1 as Client 1
    actor c2 as Client 2
    participant sv as Server
    participant pq as Postgresql
    Note over pq: max_safe_seq=9_223_372_036_854_775_807

    %% initial message
    rect AliceBlue
        c1->>+sv: send_message
        sv-->>+pq: insert event1
        pq-->>-sv: (stream_id: 9)
        sv->>-c1: 200
    end

    %% sync attempt
    rect DarkKhaki
        c2-->>+sv: /sync
        Note over pq,sv: because there is no since=?  we request everything.
        sv-->>+pq: Select every event that is from 0 to the last id that is not in a transaction but is less than everything else that is in it.
        Note over pq: max_safe_seq=9_223_372_036_854_775_807
        pq-->>-sv: [event1]
        sv -->>-c2: [event1], next_batch: s_9
    end

    %% let's do a long transaction...
    rect DarkSeaGreen
        Note over c1,pq: now lets do a long running transaction.
        c1->>+sv: create_room

        sv-->>+pq: BEGIN
        sv-->>+pq: insert event2
        pq-->>-sv: (stream_id: 10)

        %% sync attempt
        rect DarkKhaki
            c2-->>+sv: /sync?=s_9
            sv-->>+pq: Select every event that is from 9 to the last id that is not in a transaction but is less than everything else that is in it.
            Note over pq: max_safe_seq=9
            pq-->>-sv: [<empty>]
            sv -->>-c2: [<empty>], next_batch: s_9
        end

        sv-->>+pq: insert event3
        pq-->>-sv: (stream_id: 11)

        %%  this is to show that this message is not going to be shown
        rect AliceBlue
            Note over c2,sv: client 2 sends a message
            c2->>+sv: send_message
            sv-->>+pq: insert event4
            pq-->>-sv: (stream_id: 12)
            sv->>-c2: 200
        end

        %% sync attempt
        rect DarkKhaki
            c2-->>+sv: /sync?=s_9
            sv-->>+pq: Select every event that is from 9 to the last id that is not in a transaction but is less than everything else that is in it.
            Note over pq: max_safe_seq=9
            Note over c2,sv: but it will not receive anything yet.
            Note over c2,sv: because the stream_id is more than 9 which is the current max_safe_seq.
            pq-->>-sv: [<empty>]
            sv -->>-c2: [<empty>], next_batch: s_9
        end

        sv-->>+pq: insert event5
        pq-->>-sv: (stream_id: 13)

        sv-->>+pq: COMMIT
        pq->>-sv: ok
        sv->>-c1: 200
    end

    %% sync attempt
    rect DarkKhaki
        c2-->>+sv: /sync?=s_9
        sv-->>+pq: Select every event that is from 9 to the last id that is not in a transaction but is less than everything else that is in it.
        Note over pq: max_safe_seq=9_223_372_036_854_775_807
        pq-->>-sv: [event2, event3, event4, event5]
        sv -->>-c2: [event2, event3, event4, event5], next_batch: s_13
    end

1. Normal Insertion of an Event

Let’s imagine that we evaluate the max_safe_seq() function when there is no transaction running; the result would be the maximum 64-bit signed integer.

1. Initially, the highest stream_id in the database is 8, and no stream_ids are locked.
2. To insert a message. Client 1 sends a request to the server.
3. The server implicitly starts a transaction by sending an insert query to the database. While this is going on, the max_safe_seq() will evaluate to 8.
4. PostgreSQL returns success.
5. and the server returns success to the client.

sequenceDiagram
    autonumber
    actor c1 as Client 1
    actor c2 as Client 2
    participant sv as Server
    participant pq as Postgresql
    Note over pq: max_safe_seq=9_223_372_036_854_775_807

    %% initial message
    rect AliceBlue
        c1->>+sv: `send_message`
        sv-->>+pq: insert event1
        pq-->>-sv: (stream_id: 9)
        sv->>-c1: 200
    end

Normal business as usual.

2. First Sync

Client 2 now wants to sync.

1. It sends a sync request without any since tokens. For the sake of a story, imagine this is their first sync with the new device! Yaaaay!
2. The server requests every event that has a stream_id less than max_safe_seq. Here, because there is no transaction running, the max_safe_seq evaluates to i64::MAX, which basically covers every event.
3. The only event in our imaginary awesome introverted server is event1, which was the message sent by client 1 in the previous section.
4. server returns that event to the client, along with a token with the stream_id of the event1, which is 9. (The token in this example is s_9)

sequenceDiagram
    actor c1 as Client 1
    actor c2 as Client 2
    participant sv as Server
    participant pq as Postgresql
    Note over pq: max_safe_seq=9_223_372_036_854_775_807
    %% sync attempt
    rect DarkKhaki
        c2-->>+sv: `/sync`
        Note over pq,sv: because there is no `since=?`  we request everything.
        sv-->>+pq: Select every event that is from 0 to the last id that is not in a transaction but is less than everything else that is in it.
        Note over pq: max_safe_seq=9_223_372_036_854_775_807
        pq-->>-sv: [event1]
        sv -->>-c2: [event1], next_batch: s_9
    end

3. The Long Transaction

Now let’s do something more ambitious.

Let’s see what happens when there is a longer transaction going on afterward.

Let’s say that client 1 is creating a room. And client 2, being the socially active person we know and love, is syncing and sending messages to another while client 1 is at it.

sequenceDiagram
    autonumber
    actor c1 as Client 1
    actor c2 as Client 2
    participant sv as Server
    participant pq as Postgresql
    Note over pq: max_safe_seq=9_223_372_036_854_775_807
    %% let's do a long transaction...
    rect DarkSeaGreen
        Note over c1,pq: now lets do a long running transaction.
        c1->>+sv: `create_room`

        sv-->>+pq: BEGIN
        sv-->>+pq: insert event2
        pq-->>-sv: (stream_id: 10)

        %% sync attempt
        rect DarkKhaki
            c2-->>+sv: `/sync?=s_9`
            sv-->>+pq: Select every event that is from 9 to the last id that is not in a transaction but is less than everything else that is in it.
            Note over pq: max_safe_seq=9
            pq-->>-sv: [<empty>]
            sv -->>-c2: [<empty>], next_batch: s_9
        end

        sv-->>+pq: insert event3
        pq-->>-sv: (stream_id: 11)

        %%  this is to show that this message is not going to be shown
        rect AliceBlue
            Note over c2,sv: client 2 sends a message
            c2->>+sv: `send_message`
            sv-->>+pq: insert event4
            pq-->>-sv: (stream_id: 12)
            sv->>-c2: 200
        end

        %% sync attempt
        rect DarkKhaki
            c2-->>+sv: `/sync?=s_9`
            sv-->>+pq: Select every event that is from 9 to the last id that is not in a transaction but is less than everything else that is in it.
            Note over pq: max_safe_seq=9
            Note over c2,sv: but it will not receive anything yet.
            Note over c2,sv: because the stream_id is more than 9 which is the current max_safe_seq.
            pq-->>-sv: [<empty>]
            sv -->>-c2: [<empty>], next_batch: s_9
        end

        sv-->>+pq: insert event5
        pq-->>-sv: (stream_id: 13)

        sv-->>+pq: COMMIT
        pq->>-sv: ok
        sv->>-c1: 200
    end

1. Client 1 sends a create_room request to the server.
2. Server starts a transaction.
3. Server inserts a transaction.
4. The stream_id is at 10 now.
5. Client 2 sends a sync request to the server with the s_9 token they received from the previous section.
6. The server queries PostgreSQL for every event that has a stream_id more than 9 and less than or equal to the max_safe_seq at this point, which is also 9. So, basically nothing.
7. No rows are returned.
8. The next_batch token here will be the same token as the since token.
9. The server, continuing the request of client 1, inserts event3…
10. database sends back stream_id of 11.
11. Client 2, excited, now sends a message.
12. event4 is the client2 send_message event now inserted into the database…
13. …with stream_id of 12
14. 200 to the client 2 for their message request is returned.
15. Client 2 feels insecure, so they sync once more; maybe client 1 has already responded. Poor guy.
16. Again, the max_safe_seq is still at 9 because the transaction has not released the lock yet, so…
17. Nothing is returned.
18. and the next_batch token will remain the same. They don’t even see their messaging event, adding to the feeling of ghosting.
19. server sends a query for inserting event5.
20. with stream_id = 13.
21. and the server sends COMMIT.
22. PostgreSQL sends back success for the transaction commit.
23. Client 1, unaware of the long, gigantic drama that awaits them, receives a 200 for the room creation request.

4. The Final Sync

Now for the final sync.

Client 2 sends another sync request with the since token of s_9.

This time the max_safe_seq evaluates to the i64::MAX (because there is no other transaction going that locks the sequence).

This means that event2, event3, event4 and event5 are all available.

Client 2, facing the reality of what they have done, will receive all these events with an eventually changed next_batch token of s_13.

sequenceDiagram

    actor c1 as Client 1
    actor c2 as Client 2
    participant sv as Server
    participant pq as Postgresql
    Note over pq: max_safe_seq=9_223_372_036_854_775_807

    %% sync attempt
    rect DarkKhaki
        c2-->>+sv: `/sync?=s_9`
        sv-->>+pq: Select every event that is from 9 to the last id that is not in a transaction but is less than everything else that is in it.
        Note over pq: max_safe_seq=9_223_372_036_854_775_807
        pq-->>-sv: [event2, event3, event4, event5]
        sv -->>-c2: [event2, event3, event4, event5], next_batch: s_13
    end

Architecture Decision Records

This section contains architecture decisions which have been made, but not yet implemented. When making design decisions for Arminius, which exceed some threshold, you should first create a PR which describes your decision to this directory, for others to review. Afterwards, you create a follow-up PR that implements the decision.

When implementing something from this directory, the implementation PR should also delete the ADR from this directory, and move it into the regular docs.

Non-Implemented ADRs

• Room Events: Sign, Redact, Verify
• Message Pagination Tokens
• Visibility of events over federation
• Database Structure
• NATS Messaging Abstraction
• Homeserver Migration
• Nuss and IdP at Famedly
• Permissions in Arminius
• User info mapping in Nuss
• Dynamic Configuration
• Simplified Sliding Sync

ADR: ESIG-RDAC

Author: Flori Ava Star Weber

Co-Authors: Rasmus Kjær

Reviewers: Pierre

For room events, we’d like to sign them and verify signatures and authorization, and also be able to handle redactions. This ADR outlines the architecture of arminius-lib-event, a library crate designed to be an easy to use “one-stop-shop” for everything we’d conceivably like to do with room events throughout arminius. Since ruma already implements 85% of “core” behaviors, such as

• Verifying the auth chain for an event
• Canonicalizing events
• Redacting an event

in some form, arminius-lib-event is an abstraction, aimed at bringing all of these behaviours together, creating a type that neatly interfaces with the rest of our code.

Note

Historical Information

As of 2025-12-17, the branch flori/esig-rdac contained a naive sketch of what such a crate might look like. After a first round of reviews pointed out this approaches’ flaws and shortcomings, this ADR was written.

Redesigning RoomEvent

I’d go with my original idea of encoding the state of the event into the type system using generics. States would, in this example, be RoomEvent<Draft>, RoomEvent<Unsigned>, RoomEvent<Signed> and RoomEvent<Verified>. These states would represent the complete lifecycle of an event in all cases:

RoomEvent<Draft>

A Draft RoomEvent contains only content and type and would replace RawEvent. It is essentially just raw user intent (e.g. “Xenia says Hello”).

RoomEvent<Unsigned>

Has auth_events and prev_events added. This would replace the InsertableEvent

RoomEvent<Signed>

Hash event_id added, and signatures generated.

RoomEvent<Verified>

Same as Signed, but with additional metadata on when the verification was attempted. Verification failures get represented through their own error type, such as a enum ValidationError { ... }, containing all the different ways in which a verification can fail.

Tip

A “verification attempt” describes an instance of the verifying function being called on a Signed event. The phrase makes no assumption about whether the verification succeeds. The metadata gets added regardless of whether the verification succeeded.

This invariant ensures that events are guaranteed to be verified at compile-time.

State Transition

stateDiagram-v2
    userEvent: An incoming client request that should create some event
    federatedEvent: Other homeservers
    databaseEvent: Postgres Database

    draftEvent: RoomEvent&ltDraft&gt
    unsignedEvent: RoomEvent&ltUnsigned&gt
    signedEvent: RoomEvent&ltSigned&gt
    verifiedEvent: RoomEvent&ltVerified&gt

    userEvent --> draftEvent: A user can create a draft event
    draftEvent --> unsignedEvent: Our own homeserver attaches auth_events and prev_events
    unsignedEvent --> verifiedEvent: Our own homeserver signs our own events. An event signed by us is inherently verified.
    signedEvent --> verifiedEvent: Our own homeserver can verify foreign events

    federatedEvent --> signedEvent: We can receive events from other homeservers

    databaseEvent --> verifiedEvent: We can read events from the database
    verifiedEvent --> databaseEvent: We can write verified events to the database

    verifiedEvent --> federatedEvent: We can send verified events to other HSs

    classDef eventOrigin fill:white
    class userEvent, federatedEvent, databaseEvent eventOrigin

Draft to Unsigned

To make an Unsigned out of a Draft, the Draft must be contextualized into the DAG of the room.

An extremely naive function (not taking account current database schemas) could take the origin_server_ts, prev_events and auth_events. Auth checks also happen in this step. This invariant also means that Unsigned events are guaranteed to be authorized, which I think is very neat, and has the potential of eliminating a whole class of bugs, because the compiler prevents you from reaching the .sign() method without a successful contextualization function call.

Signing

RoomEvent<Unsigned> exposes a method to sign the event. This yields a RoomEvent<Verified> on success.

Redactions

A redacted event is also just one of Unsigned, Signed, Verified.

Note

Alternative considerations

I previously thought of representing redacted events through their own Redacted state, and using clever trait boundary shenanigans to allow state transitions from both Signed and Redacted into Verified. However, this potentially adds a lot of complexity for something that can be simplified/represented differently.

Verifying

The state transition from Signed to Verified is a method on Signed: async fn verify(key_provider: impl KeyFetcher) -> Result<RoomEvent<Verified>, ValidationError>. KeyFetcher would be a trait, abstracting the fetching logic, implemented by any federation client.

Alternatively…

…don’t abstract with KeyFetcher, just give a FederationNatsClient or something like that directly. Idk. Abstracting feels like we’d do our future selves a favor.

Federation Versus Client Events

RoomEvent<Verified> would expose these methods:

• .to_pdu(), returning a PDU with DAG links and whatnot
• .to_client_event(ctx: ClientContext), where ClientContext contains age and transaction_id to populate the unsigned field. Also strips federation fields.

Intended Use Cases

User Fetching a Single Event from the Database

When fetching an event from the database, we receive it as a RoomEvent<Verified>. There are a wide variety of kinds of unsigned information that the user may want, depending on their use case. We provide some ctx: ClientContext in order to construct this context, and then use .to_client_event(ctx)

Receiving an Event from a Client

When a user sends the event, we take the event content and type they sent us, and construct a RoomEvent<Draft>. We then perform the transition Draft -> Unsigned -> Signed, and write it to the database.

Receiving an Event over Federation

When we receive an event from another homeserver, we instantiate it as RoomEvent<Unsigned>. We then use the transition Unsigned -> Verified

Sending an Event over Federation

If we have an event in the database, that another homeserver needs to see, we can fetch the RoomEvent<Verified> from the database.

Message Pagination Tokens

Issue: #187

Matrix rooms are represented as DAGs of events. But making a good UI for that is quite difficult, so the spec requires that we linearize the events. However, the spec doesn’t require us to do this in a specific manner, so this ADR specifies how we decided to implement a consistent linearization in Arminius.

This ADR starts by presenting the necessary background around how federation and DAGs work in Matrix, and how it complicates message ordering.

We then describe the approaches taken by Synapse and Dendrite, and the Stitched Ordering proposal recently made by Andy Balaam and Rich van der Hoof, the tradeoffs of these different implementations, and suggestions for what we should do in Arminius.

Goal

Our goal is to find some ordering, which we can use for every endpoint visible to the client, such that any pair of messages will always appear in the same order. Matrix clients request ordered lists of messages with the following (non-deprecated) endpoints:

• GET /_matrix/client/v3/sync
• GET /_matrix/client/v3/rooms/{roomId}/messages
• GET /_matrix/client/v1/rooms/{roomId}/relations/{eventId}
• GET /_matrix/client/v1/rooms/{roomId}/relations/{eventId}/{relType}
• GET /_matrix/client/v3/rooms/{roomId}/context/{eventId}

Note

This ADR only concerns the ordering events, as returned to the client. Internal representation of events, and the server-server API, remains unchanged.

Background

Matrix Federation

Matrix is a federated protocol, designed for availability and partition tolerance. This means that Matrix should continue working when other homeservers are unreachable, and that when the other homeservers again become reachable the federation should self-repair. This self-repair should not only consist in allowing future messages to be federated after a longer net-split, but any divergent conversations which happened in the meantime must also be reconciled into a coherent history, visible to all users.

The way that this is achieved is by maintaining room history, not as a linear list of events, but as a graph. When adding a new message to the graph, it’s added as a child of all previous leaf events. By giving up on the requirement that all matrix homeservers agree to one ordering of the messages in a room, we allow homeservers to accept messages from clients, even if one or all of the other homeservers in a room are currently down.

Example

Adrian, Bartholemew, and Cedric are discussing how to spend their evening. But after Bartholemew suggests the theatre, Adrian’s homeserver loses its internet connection.

graph LR
    E1("Adrian: Hey folks, what are we doing tonight?")
     --> E2("Bartholemew: We could go to the theatre")
     --> E3("Cedric: Which theatre?");

    E2 --> E4("Adrian: Theatre is dead.")

This does not preclude either Cedric or Adrian from posting messages in the chat, however. Since the homeservers do not have to agree on the ordering of messages in the chat, each homeserver can continue adding messages to the chat, even without talking to the other homeservers. The new messages are simply both appended as children to the last event the homeservers know about, which for both new messages is Bartholemew’s message. When the connection to Adrian’s homeserver is restored, the homeservers exchange the messages they missed in the meantime. This results in Bartholemew’s message having two children.

Warning

Netsplit can last an arbitrarily long time. In practice this means that any homeserver can append elements as children of any other event in the DAG.

Backfills and Federation Transactions

The two major ways Matrix homeservers learn about events on other homeservers is through federation transactions, and backfills. Whenever a user on a homeserver sends a message, the homeserver is supposed to include this message in federation transactions to all other homeservers in the room.

But homeservers can also request events from other homeservers. If a homeserver was down for a long time, or hasn’t been in the room since it was created and is therefore missing events, homeservers can use endpoints such as /GET /_matrix/federation/v1/backfill/{roomId} to retrieve old messages.

Topological Ordering

As mentioned Matrix rooms are represented as Directed Acyclic Graphs (DAG). Creating a topological ordering of a Matrix room consists of putting all the events into a list, ordered such that if event \(a\) is an ancestor of \(b\) in the DAG, then \(a\) MUST come before \(b\) in the list.

Depending on how complicated the DAG gets, there can be an unbounded amount of different valid topological orderings for a given DAG.

Showing the user a topologically ordered list of events is desirable, as human conversations generally follow causality. An event that is a descendant of some other event in the DAG may be in reply to it, and should therefore be shown afterwards.

Stream Order

The stream order is the order in which the events arrive at our homeserver. The matrix protocol presents absolutely no guarantees in which order events will arrive at our homeserver, and therefore, this is a really bad ordering from a user perspective.

Synapse does actually order responses to /sync solely by the stream ordering, the argument being that any new non-backfilled event to your homeserver should be visible to the user. In order for backfilled events not to show up as new in the timeline, backfilled events get a stream ID starting at -1 and counting down.

Depth

Each event in a DAG will have a depth. The depth is a positive integer. The depth of the root event (m.room.create) will be one. For any other event, the depth will be the maximum height of any parent event plus one.

While the federation spec does specify that other homeservers should send the depth of an events they provide, we generally cannot trust this field to be correct, since this leads to a long series of security issues.

Synapse and Dendrite

I looked a bit at what Synapse and Dendrite are doing. They both order events in the /messages API by the depth. If two events have the same depth, they break ties by stream order. For the /sync API, they order events solely by the stream order, to make sure that new events, excluding the backfilled ones, are always shown to the user as soon as the homeserver receives them.

The best documentation of how this works is found in Synapse

Backfilling events is simply appending events to the events table, computing the depth based on their parents, and setting the stream ID, in reverse order (starting from -1, going down).

Example

Consider the following event graph, which we have created on the blue homeserver:

graph LR
    E1(Event 1
    depth=1,stream_id = 1)
    --> E2(Event 2
    depth=2,stream_id = 2)
    --> E3(Event 3
    depth=3,stream_id = 3)
    --> E4(Event 4
    depth=4,stream_id = 4)
    --> E5(Event 5
    depth=5,stream_id = 5);

We are very happy right now, because our DAG is a timeline without any extra effort whatsoever! But then the orange homeserver, from which we were partitioned from for a while, comes back online. In their first federation request after coming back online, they send us a branch of conversation, which is also a child of event 2:

graph LR
    E1(Event 1
    depth=1,stream_id = 1)
    --> E2(Event 2
    depth=2,stream_id = 2)
    --> E3(Event 3
    depth=3,stream_id = 3)
    --> E4(Event 4
    depth=4,stream_id = 4)
    --> E5(Event 5
    depth=5,stream_id = 5);

    E2 --> E6(Event 6
    depth=3,stream_id = 6):::hs2
   --> E7(Event 7
  depth=4,stream_id = 7):::hs2;

classDef hs2 fill:#f96;

When we then linearize the events, by ordering them by depth and breaking ties by the stream order, and we get the timeline:

graph LR
    E1(Event 1
    depth=1,stream_id = 1)
    --> E2(Event 2
    depth=2,stream_id = 2)
    --> E3(Event 3
    depth=3,stream_id = 3)
    --> E6(Event 6
    depth=3,stream_id = 6):::hs2
    --> E4(Event 4
    depth=4,stream_id = 4)
    --> E7(Event 7
    depth=4,stream_id = 7):::hs2
    --> E5(Event 5
    depth=5,stream_id = 5);


classDef hs2 fill:#f96;

Stitched Ordering

There isn’t really a good notion in the spec of what the ordering of messages is supposed to be. Synapse has two inconsistent orderings, the one from /sync, and the one from /messages. Rich van der Hoff and Andy Balaam have been looking into a solution, and recently came up with Stitched Ordering.

Here, an order on events and gaps is maintained, with gaps being “ghost events” we know exist, but do not have synchronized to our homeserver. New events are always inserted at the end of the order, with the exception of when it matches a gap, at which point the new event is inserted into the gap. When several events are received as part of a federation transaction, we can also put them next to each other. So our example would look like this:

graph LR
    E1(Event 1)
    --> E2(Event 2)
    --> E3(Event 3)
    --> E4(Event 4)
    --> E5(Event 5)
    --> E6(Event 6):::hs2
    --> E7(Event 7):::hs2;


classDef hs2 fill:#f96;

Stitched ordering is designed to be compatible with the design goals of the /sync ordering as described in the Synapse and Dendrite section, specifically in showing newly federated (non-backfilled) events as soon as possible to the client. But in order to achieve this, they sacrifice the topological ordering property. As Matrix provides no guarantees to the order in which events are delivered as part of federation transactions, we can receive an event, which is an ancestor of all events known to our homeserver, as part of a federation transaction. This event, Stitched Ordering would simply place at the end, making stitched ordering non-topological.

Warning

This ADR does not contain a complete description of stitched ordering. Much of the following will not make sense if you read the description of stitched ordering linked above.

Message Pagination Token Format

The message pagination token can simply be the nid of the last event in the response to a pagination API, as the order doesn’t change based on when the user starts paginating. In the next request to the pagination API, when the user sends us an event nid as a token, we return the range starting at, but not including the event referred to by the pagination token. Backpaging into the linear order would then be:

SELECT
    event_json
FROM
    events
    JOIN stitched_ordering ON events.nid = stitched_ordering.event_nid
WHERE
    stitched_ordering.order < (SELECT order FROM stitched_ordering WHERE event_id = $1)
    AND events.visible
LIMIT 10;

Interactions with /sync

According to the stitched ordering MSC, the /sync endpoint should also be using the linear ordering. For our implementation, the stream ID will live on, as a way to tell the client API which subset of events in the database should be looked on when constructing responses to /sync.

Let \(R\) be the set of all messages in a room.

When we get a set of messages \(M \subseteq R\), which all have a stream_id greater than since, we recursively construct the sync response for this room \(S\) as the set of messages where:

1. \(m \in M\) and there is no other message in the room with a higher linear_order
2. \(m \in M\) and there exists a message \(m’ \in S\) such that \(m < m’\) but there is no message \(m’’ \in R\) such that \(m < m’’ < m’\).

This guarantees that the ordering provided by /sync is consistent with the linear ordering, but only shows new events in the timeline, not backfilled ones, which is the same behaviour as Synapse and Dendrite currently provide.

Note

Stitched Ordering does not result in other branches of events being lost from /sync. In the stitched ordering example above, the distinct branch of conversation is put after the existing events in the timeline, ensuring that future calls to /sync will learn of the new events in the timeline.

Only backfilled events, which fill out gaps we have passed by in the conversation, will not be sent to clients.

Possible Problems

Topological Ordering

The ordering used in Synapse and Dendrite is topological. However, to maintain a correct topological ordering you have to, for every new event you have, know a full path from your new event to the ordering you’re maintaining.

The three solutions for this are basically:

• Give up on making events visible to the user as soon as you receive them. When you receive an orphan event, fetch the entire ancestry, until you reach an event in your local copy of the DAG.
• Give up on having a consistent ordering. When you receive a new event, you make a guess to where an event may fit in, but you may move the event later when you find out your guess was wrong.
• Give up on having a topological ordering of history. Instead of making sure that we know where new events fit in the ordering, put new events at the end. (Stitched Ordering)

A previous version of this ADR used the first solution. But since when recovering from a fork the amount of messages between the leaves of the fork, which we receive over federation, and the fork point is unbounded, we would potentially have to fetch a lot of messages that a user does not care about, before fetching the last 50 messages from the other branch. And these 50 messages may very well be the messages the user looks at to figure out that they do not care about the rest.

Note however, that to order a new event linearly, we only need to have one parent of the new event in our ordering, because it being a descendant of an element in our ordering is enough evidence that it is not an unknown ancestor of our ordering. So backfilling events does not stop us from receiving or sending new messages, but can be done in the background.

Slow Backfill

graph LR
E1(Event 1)
-->E2(Event 2)
-->E3(Event 3)
-->E4(Event 4);

E2-->E6@{ shape: braces, label: "Evil predecessors"}
-->E7(Evil Event);

In the above DAG, we have a pleasant conversation in Event 1 through 4. However, an evil homeserver has sent us “Evil Event”, which references an unknown number of events, between it and some point in our DAG.

In the stitched ordering, we would have:

graph LR
E1(Event 1)
-->E2(Event 2)
-->E6@{ shape: braces, label: "Gap: Evil predecessors"}
-->E7(Evil Event)
-->E3(Event 3)
-->E4(Event 4);

Fetching the Evil Predecessors will take a very long time. So what should our homeserver do, when a client asks for the message history before message 3? The two options, without touching the matrix spec are:

1. Ignore the gap/fetch messages in the background, and return Evil Event, Event 2 and the predecessors of event 2 first (This is what is suggested by the Stitched Ordering MSC)
2. Don’t return from the /messages call, until we get all the Evil Predecessors from some other homeserver.

The gaps draft MSC should also be noted. This MSC modifies the client API, letting us return information about known gaps in the timeline to clients. But that is not as relevant right now, since no clients support it.

For now, and for our first implementation of the /messages endpoint, I would simply want to, when asked for three predecessors of Event 3, to return Event 1, Event 2, Evil Event, ignoring the gap. The gap could, possibly, be fetched in the background, such that subsequent calls to /messages would contain the evil predecessors.

Overeager Backfill

No matter which ordering we decide, there is a possible attack in not configuring a rate-limit for outgoing backfill requests. If a malicious homeserver sent an event, with a lot of prev_events, it could make our homeserver DOS third homeservers, by sending too many backfill requests. Therefore, outgoing backfill requests should be rate-limited in some manner.

Conclusion

For a design of message ordering, we need to make three decisions, which have been outlined in the following tables:

Handling Orphan Events

As discussed in Potential Problem: Topological Ordering, when receiving an event that we cannot trace back to an ancestor in our existing DAG, we have three options:

Moving events around in the ordering is, in my opinion, a non-starter. It will inevitably mess up client expectations, and gives a bad impression to the user.

A lot of people in the Matrix community seem to have very strong opinions that a homeserver should show incoming (non-backfilled) messages as soon as possible, and would therefore prefer #3. For most chat applications, however I think #2 is far more reasonable.

I suggest making a homeserver that serves both use cases well. We implement #2, with a configurable amount of “effort” put in before giving up, and reverting to #3. When a new event is received, we try for (a number between 0 and infinite) minutes to get the full lineage of the event before adding it to the ordering. If we fail, we make a guess, and put it at the end, like the Stitched Ordering MSC suggests. If the Stitched Ordering MSC ever got merged, and we stuck with #2, it still wouldn’t look like, from the outside, as if we broke the spec. It would just look like federation transactions were delayed a lot.

Displaying Gaps

When a user requests a part of the timeline, but we know we have a gap in this part of the DAG, we need to pick between the three approaches:

Since the first option is a potential security issue, and the third one isn’t in the Matrix spec yet, I suggest we go with the second one.

An objection may be that it is easy to miss events. But since backfilled events in Synapse and Dendrite are also just put in the middle of the timeline, with no notification of the user done, and I’ve never personally had a problem with it, I think it’s fine.

Ordering of Events in /messages

Suggested Approach

For the three comparison tables above, I suggest we go with “Wait until full lineage of event is available”, “Backfill in background, letting user scroll on”, and “Stitched Ordering”.

To implement the stitched ordering in SQL, I suggest we use a lexicographic ordering key for our events.

Appendix: Storing Lists in Postgres with Sublinear Insertion

Warning

The version of the algorithm described here is outdated. A better design was found while implementing. For an up-to-date description of the ordering, see the rustdoc

The only feasible way to store a list in postgres, where we can insert events at any point in the list without having to renumber all subsequent elements, is by doing lexicographic ordering on a column.

Of course we could also be storing a linked list, requiring us to only update the predecessor and the successor element when inserting a new one. But this would make reading from the database prohibitively expensive.

The solution to this is to use a lexicographic ordering key. Lexicographic ordering is exactly the same way you order stuff in a dictionary. If you have two words “apple” and “angle”, and want to know which word should come first, you compare the first letter. Whatever word has the smallest first letter, is the smallest word. If both words have the same first letter, you continue until you find two different letters. In our example, we find that the second letter of each word is different. Since “n” is smaller than “p”, we now know that “angle” is smaller than “apple”.

If we have two words, where one is a shorter version of the other one, such as “tree” and “treeness”, the shorter word is smallest.

Let’s apply this principle to storing an ordered grocery list in postgresql. We would create the schema:

CREATE TABLE grocery_list (
  item TEXT PRIMARY KEY,
  ordering BIGINT[] NOT NULL
)

where items on our grocery list are just strings, stores as TEXT in postgresql, and our ordering is an array of bigints.

To create our initial shopping list, we execute some SQL statements:

INSERT INTO grocery_list (item, ordering)
    VALUES ('apple','{1}'),
    ('pear','{2}'),
    ('banana', '{3}');

If we want to insert rye bread directly after “pear”, we can do

INSERT INTO grocery_list (item, ordering) VALUES ('rye bread', '{2, 1}');

This would give us the grocery list:

If we then want to insert “potatoes” directly after “pear”, but before “rye bread”, we can do:

INSERT INTO grocery_list (item, ordering) VALUES ('potatoes', '{2, 0}');

This would give us the grocery list:

Algorithm

Given a list \(L \in \mathbb{N}^*\), to find an unoccupied space directly after any item \(\mathbf{a}\), we can use the following algorithm:

1. Let \(\mathbf{a’} = (a_1, a_2, \dots, a_n + 1)\) be a new ordering key, the same as \(\mathbf{a}\) but with the last item incremented by one. If \(\mathbf{a’} \not \in L\), return \(\mathbf{a’}\)
2. Otherwise, there is already an element directly after \(\mathbf{a}\), and we have to expand the list to insert our new element between \(\mathbf{a}\) and \(\mathbf{a’}\). Let \(\mathbf{a’‘} =(a_1, a_2, \dots a_n, 1)\) be \(\mathbf{a}\), with a one appended. If \(\mathbf{a’‘} \not \in L\), return \(\mathbf{a’’}\)
3. If \(\mathbf{a’‘} \in L\), then we let \(\mathbf{a’‘’}\) be \((a_1, a_2, \dots, a_n, x)\), where \(x\) is the largest integer \(<1\) such that \(\mathbf{a’‘’} \not \in L\).

Given a list \(L\) constructed using this algorithm, if you use the algorithm to insert a new element \(\mathbf{d}\) directly after \(\mathbf{b}\), there will be no element \(\mathbf{c} \in L’\) such that \(\mathbf{b} < \mathbf{c} < \mathbf{d}\).

Proof

Let \(\mathbf{d}\) be our new element inserted directly after \(\mathbf{b}\), and assume there exists an element \(\mathbf{c}\) such that \(\mathbf{b} < \mathbf{c} < \mathbf{d}\).

• If \(\mathbf{d}\) was created using rule 1, then in order for \(\mathbf{c}\) to be \(\mathbf{b} < \mathbf{c} < \mathbf{d}\), it must be the case that \(\mathbf{c}\) has the form \((b_1, b_2, \dots, b_n, x)\), and must therefore have been inserted using rule 2 or 3 after \(\mathbf{a}\). But rules 2 and 3 only would apply if \(\mathbf{d} \in L\) was already in the list, leading to a contradiction.

• If \(\mathbf{d}\) was created using rule 2, then \(\mathbf{c}\) must have the form \((a_1, a_2, \dots, a_n, x)\), where \(x\) is less than one. But that rule would only apply if \(\mathbf{D} \in L\), leading to a contradiction.

• If \(\mathbf{d}\) was created using rule 3, then \(\mathbb{d}\) must have the form \((a_1, a_2, \dots, a_n, x_d)\) \(\mathbf{c}\) must have the form \((a_1, a_2, \dots, a_n, x_c)\), where \(x_c < x_d\) is less than one. But since \(x_d\) was the highest possible integer, before \(x_c\) was in the list, this is a contradiction.

Defragmentation

The ordering key does not have to grow indefinitely. If the ordering key we can always renumber all the elements in our list, to only use length-one ordering keys.

An example (non-performant, only meant for demonstration) SQL query could be:

UPDATE
    grocery_list l
SET
    ordering = ARRAY[l2.seqnum]
FROM (
    SELECT
        l2.*,
        row_number() OVER (ORDER BY ordering) AS seqnum
    FROM
        grocery_list l2) l2
WHERE
    l2.item = l.item;

This SQL query would take the list:

and rewrite it to:

Visibility of Events over Federation

When Matrix homeservers share rooms with each other, they should generally share the events in the rooms as well. But for various reasons, there may be events that a homeserver should be able to see, which are not in its local database. This may be because a network failure may have caused a transaction to get dropped, or because a homeserver may have joined a room late and does not have the full backlog.

To remedy this, the Matrix server-server specification contains several endpoints which allow homeservers to query each other about the events in rooms. However, there is not a lot of specification about when these events actually should be sent, and when they should not. We obviously don’t want just anybody to be able to access any event on our homeserver. This document lays out some rules for what events we share with other homeservers, and which we do not.

Since the specification itself does not describe this, we should instead lean on the behaviour of Synapse. This document describes how Synapse handles the visibility of events over federation, and argues why our API should behave the same way.

What Does Synapse Do

The rules for fetching events in Synapse depend on the endpoint used. The Matrix server-server specification has two kinds of endpoints for requesting events: The endpoints which are restricted to returning state events, and those that return both state and message-like events.

For both types of endpoints, Synapse will refuse requests for events in which the requesting homeserver is not currently a member. For this purpose, Synapse has the function assert_host_in_room which checks whether the current state of the room contains a membership event for a user from the foreign homeserver where the membership is join. If no user from the foreign homeserver is currently in the room, it returns a 403 error.

Endpoints Which Return Only State Events

For endpoints which can only request state events, assert_host_in_room is the only check which will be performed before returning the requested events. In effect, any server in a room can view all state events in the history of the room.

An example of an endpoint which returns the current state of the room is GET /_matrix/federation/v1/state/{roomId} (Synapse handler).

Endpoints Which Return State and Message Events

For endpoints which may return message-like events Synapse performs additional filtering in addition checking for current membership. Only events which at least one of the users hosted on the requesting homeserver may be allowed to see are sent to the requesting homeserver in full, while other events are redacted before sending. Synapse does this using the function filter_events_for_server, which applies the redaction algorithm to all events that foreign homeservers ask for that no user on the homeserver is allowed to see according to the rules in the history visibility module of the client-server-specification.

An example of an endpoint that may return both state and message-like events is GET /_matrix/federation/v1/event/{eventId} (Synapse handler)

Reasoning Behind Redacting

Returning all the events in the DAG in redacted format, instead of simply not returning them at all, is necessary for each homeserver to have a complete view of the structure of the DAG, which in turn is necessary for state resolution. Many parts of state resolution, such as reverse topological power ordering and mainline ordering, depend on homeservers being able to sort state events topologically in the DAG. Sorting events topologically is achieved by sorting them on the “depth” field of the PDU. But blindly trusting the value of “depth” set by foreign homeservers can lead to security issues, and should therefore never be done. Since the DAG is a hash graph, where each event contains a reference to the hashes of previous events, the only way to know that each event is telling the truth about its position in the DAG, is by having access to the whole chain from the root to the event itself. Therefore, for every homeserver to be able to participate in the room and perform state resolution, every homeserver should be able to see the complete DAG, possibly with redacted events.

Sending events to a homeserver, that nobody on the homeserver should be able to see of course has security implications. In effect, a homeserver joining a room where it should have no access to history, will be able to see the timestamps of all messages ever sent in the room. However, redacting message-like events before sending them to the requesting homeserver at least removes the content of the messages.

What Arminius Should Do

A spec-issue is filed under #1519 in the Arminius repository.

Outside of that I think the safest path without a spec is duplicating Synapse’s behaviour, in order to maintain compatibility where there is no specification.

There are some areas where I think that the Synapse implementation is less than optimal, but improvements to that should be discussed in the framework of an MSC.

Database Structure #4

We’ve redesigned the database layer a couple of times by now. Currently, any service can access any database, which is a situation we would like to resolve. Therefore, it’s time to refactor the database layer again.

Goals

• We want to separate our database queries from the rest of our code
• We want a solid overview of which services can write to which tables. This way, any guarantees a single service sets up about its database can be upheld.
• We want every service to read from every table
• We don’t want any SQL injections

A Short History of Our Database Layer

First Iteration - Database Everywhere

When we started needing a database, we just threw in sqlx::query! calls wherever we needed, directly in the handlers. The only rule was that the macro had to be used, to ensure that the queries were compiled offline, such that introducing SQL injection vulnerabilities would be very difficult.

Second Iteration - Database Modules

We wanted a clearer separation between the database code, and the rest of the code, and we therefore created a db module in each service crate, which would contain all the queries. This also had the goal of making queries reusable.

However, queries were not reusable across services, something that is a goal, as spelled out in our architecture documentation.

Third Iteration - Database Crates

In order to make the code easier to navigate, we split the database modules into their own crate in the crates/libs/db/ directory. This also introduced the DefaultDb abstraction, where each database crate would expose a trait with all its available queries, and implement it for the DefaultDb struct.

This also ended up letting us compile and test database crates individually, something that turned out to be quite beneficial. For instance, with complicated queries such as message ordering or sync, this has been a big boon.

The goal would be that every table, had roughly one crate for all queries relating to it. But this did not make much sense, considering that JOINs exist. The rule quickly got many exceptions, to the point where it didn’t make much sense any more.

The Next Iteration

Each service in Arminius should have a database user. For each table there exists one user, which has the privileges to update the table. All users are allowed to read from all tables. When the monolith is spawned, it gets a special monolith user, which is allowed to read from and write to all tables.

We split our database crates up into two crates per service. A write crate, which is only usable by the service which is supposed to write to its tables, and a read crate, which is allowed to read from the tables. We use sqlx.toml files to ensure that each db crate and each service uses the correct postgres account to connect to the database.

Example

The arminius-server-client owns the global_account_data table. Therefore, all queries which write to the global_account_data table are found in arminius-lib-db-client-write, and all queries which read from the table are found in arminius-lib-db-client-read. Only arminius-lib-db-client-read may be depended on by other crates than arminius-server-client.

ADR: NATS Messaging Abstraction

Author: Michael Schleicher

This ADR proposes a type-safe, mockable messaging abstraction for inter-service communication in Arminius, and asks reviewers to agree on its interface. This document describes what that design is, why it is shaped that way, and which alternatives were weighed.

What This Is Really About

Arminius is event-driven: services coordinate by emitting events and answering commands. Which events a service emits and which commands it answers is the contract it offers the rest of the system - and that is domain logic, owned by the domain.

The goal of this abstraction is to let a domain declare its messages as plain Rust types (e.g. “emits RoomCreated”, “answers GetDevice with a device id”) and write its logic against those types alone. How a message travels - NATS subjects, JetStream, serialization, acknowledgement - is an architectural concern the domain should not have to care about; changing the transport must not ripple into domain logic.

So the domain side stays clean: a message is just a type, domain code reads as emit this event / handle this command, and the transport (with any performance tuning) lives behind the seam. The rest of this ADR is the technical means to that end.

Context

Today, services share a common NatsConnection type that handles connecting and stream setup. On top of it, each service hand-writes its own messaging layer: a wrapper client, a subject type, and the payload serialisation, all built directly against NatsConnection and async-nats/JetStream. Nothing is shared, so the same plumbing is reinvented per service and every wrapper stays bound to the concrete transport.

This has two recurring problems:

• Transport-coupled. Every wrapper depends on async_nats/JetStream directly. Subject layout, stream names, retention, and consumer configuration are spread across these per-service wrappers rather than centralized.
• Untestable without a broker. Because the transport is hard-wired, exercising a handler in a unit test requires a running NATS instance.

We want a bus where a message is published and received purely through traits - no per-service wrapper and no manual serde at the callsite - where subject and JetStream configuration are centralized inside the abstraction, and where handlers can be unit-tested without a running NATS instance.

Decision

Define messaging as a set of transport-agnostic traits: Event/Command describe a message, Publisher/Subscriber and Requester/Responder move it. Domain code depends only on these traits; the NATS/JetStream implementation lives behind them (see Crate Organisation). The rest of this document specifies that interface.

Message Types

There are two different message types we currently support in Arminius, Events and Commands. First we define the interface that the domain logic uses to publish, subscribe, request, and respond to those messages. A concrete message is then a plain Rust type that implements the matching carrier trait, which the interface traits are generic over.

Messages should support compatibility to older messages and versioning to provide safety for future changes. All messages will use JSON as a transport format.

The following signatures and trait bounds shown here are not set in stone. Migrating real callsites may surface a need to adjust them, and this ADR will be updated to match.

Events

An event is a state change in the system: Something happened and the service wants to notify interested parties. Delivery matters - the event must arrive - but the emitting service does not care who, or how many, listeners there are.

Interface

A type becomes an event by implementing Event. It binds the type to the subject it lives on, plus the subjects of older versions it can still consume:

pub trait Event {
    /// The subject this event type lives under (versioned; may contain `*`).
    const SUBJECT: &'static str;
    /// Subjects of older compatible versions, consumed and upcast via `From`.
    const COMPATIBLE_SUBJECTS: &'static [&'static str] = &[];
}

Publisher and Subscriber provide the interface to work with the Event:

/// Publisher that can publish message onto their specific subjects
pub trait Publisher {
    fn publish<E>(&self, message: E) -> impl Future<Output = Result<(), Error>>
    where E: Event + Clone + 'static;
}

/// Subscriber will receive on subjects tied to the corresponding type.
pub trait Subscriber {
    fn subscribe<E: Event + 'static>(
        &self,
    ) -> impl Future<Output = Result<
        impl Stream<Item = Result<impl ReceivedEvent<E> + Send + Sync + 'static, Error>>,
        Error,
    >>;
}

/// Helper trait to convey acknowledgment and tracing context.
pub trait ReceivedEvent<E: Event>: Deref<Target = E> {
    fn ack(&self) -> impl Future<Output = Result<(), Error>> + Send;
    fn context(&self) -> &opentelemetry::Context;
}

Example

The #[derive(Event)] macro is purely a convenience: It generates the Event implementation from #[subject(...)] attributes. A type can implement Event by hand just as well.

#[derive(Clone, Serialize, Deserialize, Event)]
#[subject(version = "1")]
struct RoomCreated { room_id: String }

async fn announce(publisher: &impl Publisher, room_id: String) {
    publisher.publish(RoomCreated { room_id }).await.unwrap();
}

async fn consume(subscriber: &impl Subscriber) {
    let stream = subscriber.subscribe::<RoomCreated>().await.unwrap();
    futures::pin_mut!(stream);
    while let Some(Ok(event)) = stream.next().await {
        println!("room created: {}", event.room_id);
        event.ack().await.unwrap();
    }
}

Commands

Commands are requests that require a response from another service. In a microservice architecture services should stay self-reliant and avoid querying others: a service subscribes to the Events it needs and keeps its own copy of that information, so that an outage of another service does not take the whole system down. Sometimes that pattern cannot hold - information must be synced live, or requested because it may not be stored elsewhere - and that is what commands are for.

Interface

A type becomes a command by implementing Command, which binds it to its subject and - crucially - its reply type:

pub trait Command {
    /// The subject this command is sent on (versioned; may contain `*`).
    const SUBJECT: &'static str;
    /// The reply type. `Requester::request` returns exactly this.
    type Reply;

    /// How often the command may reach its handler. Defaults to `AtLeastOnce`.
    const DELIVERY: Delivery = Delivery::AtLeastOnce;
}

Because type Reply lives on the command type, request and response are bound at compile time - a request can never be matched to the wrong reply type. The command declares only the reply type, not where the reply goes. The reply subject is handled by the NATS implementation (a per-request inbox in a header), never by the callsite.

DELIVERY sits on the command type for the same reason. Request-reply gives a command one responder, so every instance of the answering service derives the same guarantee and they cannot disagree about it. Events carry no such constant, because an event has many subscribers and whether handling it twice is safe is a property of each subscriber, not of the event.

The constant decides one thing in the implementation: whether the consumer may hand the command out again. AtLeastOnce leaves redelivery on, so a handler that dies before answering has its command picked up by another attempt. AtMostOnce says the handler has an effect that must not be repeated, so it acknowledges the command before starting work and the transport caps its consumer at a single delivery - an expiring acknowledgement window then hands the command to nobody.

Requester and Responder provide the interface to work with the Command:

/// Requester sends a command and awaits its typed reply.
pub trait Requester {
    fn request<C: Command + Send>(&self, cmd: C)
        -> impl Future<Output = Result<C::Reply, Error>> + Send;
}

/// Responder subscribes to commands of one type and answers each with a typed reply.
pub trait Responder {
    fn respond<C: Command + Send + 'static>(self)
        -> impl Future<Output = Result<
            impl Stream<Item = Result<impl ReceivedCommand<C>, Error>> + Send + 'static,
            Error,
        >> + Send + 'static;
}

/// Helper trait to convey the acknowledgement, response and tracing context
pub trait ReceivedCommand<C: Command>: Deref<Target = C> {
    /// Take responsibility for the command without answering it yet.
    fn ack(&self) -> impl Future<Output = Result<(), Error>> + Send;
    /// Send the response and acknowledge the command.
    fn reply(&self, response: C::Reply) -> impl Future<Output = Result<(), Error>> + Send;
    fn context(&self) -> &opentelemetry::Context;
}

Example

The #[derive(Command)] macro is purely a convenience: it generates the trait Command implementation from #[subject(...)] / #[command(...)] attributes.

/// Request to create a new room and if successful get the generated room id.
#[derive(Serialize, Deserialize, Command)]
#[subject("rooms.create")]
#[command(reply = Result<String, CreateRoomError>)]
pub struct CreateRoomCommand { pub name: String }

async fn create(requester: impl Requester) {
    let cmd = CreateRoomCommand { name: "lobby".into() };
    // `reply` is the newly generated room id.
    let reply: Result<String, CreateRoomError> = requester.request(cmd).await.unwrap();
}

async fn serve(responder: impl Responder) {
    let stream = responder.respond::<CreateRoomCommand>().await.unwrap();
    futures::pin_mut!(stream);
    while let Some(Ok(cmd)) = stream.next().await {
        let result = handle(&*cmd);
        cmd.reply(result).await.unwrap();
    }
}

Errors

Every operation returns the same transport-agnostic Error, so domain code never depends on async_nats error types:

pub enum Error {
    SendFailed { error: String },
    SubscriptionFailed { subject: String, error: String },
    ReceptionFailed { subject: String, error: String },
    AckFailed { error: String },
}

Subject and JetStream Configuration Are Centralized

Because the subject is a property carried by the trait - Event::SUBJECT and Command::SUBJECT - callsites never spell a subject out. They name a type; the type knows its subject.

• Versioned subjects. A subject defaults to <StructName>.v<version>. Per-instance subjects use {field} placeholders: each becomes a * wildcard in SUBJECT (the subscribe/stream pattern) and is interpolated into the concrete subject() (the publish target). For example "rooms.{room_id}.events.v1" gives a subscribe pattern of "rooms.*.events.v1" and a publish target of "rooms.7.events.v1".
• Stream and consumer configuration lives entirely in arminius-lib-nats. Stream naming, retention policy, durable consumer names (the consumer name is the service name, so instances of one service load-balance automatically), and ack windows are set in the implementation. Streams are created idempotently on first use. Callsites configure none of this. Configuration is per-connection, with one exception carried on the message type itself: a command declaring Delivery::AtMostOnce gets a consumer that never redelivers it.
• Schema evolution is also behind the trait. COMPATIBLE_SUBJECTS lets a subscriber consume older versions and upcast them via From, with transitive subject matching across a version chain. The derive’s compatible = [..] attribute is just the convenience that populates this constant and wires up the upcast.

Testing without a Broker

arminius-lib-message-bus ships an in-memory implementation of the traits behind a mock feature, enabled as a dev-dependency so it never ships in release builds. Because service code is written against &impl Publisher / &impl Subscriber / impl ReceivedCommand, the mock substitutes directly:

• MockBus implements both Publisher and Subscriber. Published events are delivered to active subscriptions on a matching subject and recorded for assertions via published_as::<T>(). Clones share backing state.
• MockPublisher<T> captures published events of one type for assertions.
• MockSubscriber wraps a fixed Vec of events into a stream plus parallel “checkers” that assert each event was acknowledged.
• MockReceivedCommand + ReplyChecker let a test run a command handler and read back the captured reply (or observe that the handler dropped the command without replying).

This satisfies the requirement that handlers be unit-testable with no running NATS instance. Tests are run with cargo nextest run for process isolation.

These mocks are deliberately behavioural fakes (a publish really delivers to live subscribers) rather than expectation mocks à la mockall::automock.

Crate Organisation

The abstraction is split across three crates so the interface is decoupled from any concrete transport:

The key property is the layering: callsites depend only on the traits. Service code accepts &impl Publisher, &impl Subscriber, &impl Requester or impl Responder and never names a transport type. (Responder is the exception: respond takes self by value so the returned stream is 'static.) The NATS implementation can therefore be swapped for the in-memory mock with no change to the code under test, and nothing about JetStream leaks above the trait surface.

Alternatives Considered

• Raw NATS subjects + manual serde at each callsite (status quo). Rejected: this is exactly the untyped, transport-coupled, broker-dependent situation the abstraction exists to remove.
• A single untyped envelope/wrapper type (subject field + serde_json::Value body). Loses per-message types and compile-time checking of subjects and - critically - request/reply pairing. Rejected: it does not make the bus type-safe.
• Hand-written trait impls, no derive. Viable, and still supported - but writing SUBJECT/version constants and the payload functions for every type by hand is repetitive and easy to get wrong. The derive is sugar over this, not a replacement for it.
• A runtime subject registry/config file. Rejected: moves subject definitions out of the type and back to runtime, losing the compile-time guarantees and the “no raw subject at the callsite” property the design is built on.

Consequences

Gained

• compile-time–checked subjects and reply types
• a transport-agnostic callsite surface
• unit-testable handlers via mocks
• centralized JetStream configuration
• built-in versioning and schema evolution
• OpenTelemetry context propagation through the trait surface

Cost

• a proc-macro layer to maintain, plus a learning curve for the attribute syntax
• the traits bake in JetStream’s delivery model: ack(), redelivery and durable consumers are assumed by the interface, and Delivery names the two guarantees a transport has to offer, so one without those semantics would not drop in cleanly

ADR: Homeserver Migration Format

When migrating a Matrix homeserver from one deployment to another, operators need a way to export the server state and import it elsewhere. This ADR describes the format produced.

Proposal

The export is a streaming jsonl (newline-delimited json) file. Each line is a self-describing json object with a mandatory "type" field. The file is streamed from the HTTP response with no buffering, so it can be consumed without loading it entirely into memory.

All timestamp fields use milliseconds since the Unix epoch and carry a _ms suffix (e.g. creation_ts_ms). The exported_at field in the header is the only exception: it is an ISO 8601 string (for readability).

The first record is always the header. The last record is always the trailer (to make sure the output wasn’t truncated). Between them is the exported data.

Stream Layout

Record Type Ordering

Tables are exported one at a time in the following order, which respects the dependency chain:

1. private_keys
2. users
3. rooms
4. devices
5. events
6. account_data
7. room_account_data
8. presence
9. receipts
10. device_keys
11. cross_signing_keys
12. cross_signing_signatures
13. one_time_keys
14. fallback_keys
15. device_messages

Chunking

The import/export should support chunking in case an error happens during the process on which we should be able to resume where we stopped. Within each table, rows are sorted by a stable, monotonically-increasing column (the sort key). The importer can keep track of a cursor (the last seen value of the sort key) and pass it as a query to the exporter when resuming.

The name of that query parameter is not fixed by the importer: the exporter advertises it in the header (cursor_param, e.g. from), and the importer echoes the cursor value back under that name (?table=users&from=<cursor>). This keeps the resume contract owned by the exporter, so its parameter name can change on one side alone.

private_keys holds at most one record per key type and is not chunked.

Record Format

Header

{
  "type": "header",
  "format_version": 1,
  "server": "servername",
  "exported_at": "2026-05-21T10:00:00+00:00",
  "table": "events",
  "record_count": 4217,
  "cursor_param": "from"
}

table names the record type contained in this stream. record_count is the total number of data records in this stream (excluding the header and trailer), allowing the importer to track progress and detect truncation. cursor_param is the name of the query parameter the importer sends the resumption cursor back as (see Chunking); the exporter owns this name so it can be changed on one side alone.

Trailer

{ "type": "trailer" }

The trailer carries no additional fields. Its sole purpose is to signal that the stream ended normally and was not truncated.

private_keys

{ "type": "private_keys", "key_id": "ed25519:1", "private_key": "base64..." }

Synapse only has one signing key.

users

{
  "type": "users", "user_id": "@user:testserver", "creation_ts_ms": 1700000000000,
  "admin": false, "is_guest": false, "user_type": null, "appservice_id": null,
  "approved": true, "locked": false, "suspended": false, "shadow_banned": false,
  "deactivated": false, "displayname": "Alice", "avatar_url": "...", "profile_fields": null
}

rooms

{ "type": "rooms", "room_id": "!ubOnvyjOlxOJsnegbq:testserver", "room_version": "10", "is_public": false}

is_public is explicitly for if the room was created as if it was to be published in the public room directory, and not that it was designated as a “public room” from join_rules

events

{ "type": "events", "stream_ordering": 42, "room_id": "!ubOnvyjOlxOJsnegbq:testserver", "event_id": "$abc", "event_json": { ... } }

account_data

{ "type": "account_data", "stream_id": 42, "user_id": "@user:testserver", "data_type": "m.push_rules", "content": { ... } }

room_account_data

{ "type": "room_account_data", "stream_id": 42, "user_id": "@user:testserver", "room_id": "!ubOnvyjOlxOJsnegbq:testserver", "data_type": "m.fully_read", "content": { ... } }

presence

{ "type": "presence", "user_id": "@user:testserver", "state": "online", "status_msg": "...", "last_active_ts_ms": 1700000000000, "manually_set": true }

receipts

{ "type": "receipts", "stream_id": 42, "user_id": "@user:testserver", "room_id": "!ubOnvyjOlxOJsnegbq:testserver", "receipt_type": "m.read", "event_id": "$abc", "ts_ms": 1700000000000, "thread_id": null }

devices

{ "type": "devices", "user_id": "@user:testserver", "device_id": "ABCDEF", "display_name": "Phone", "last_seen_ms": 1700000000000, "ip": "1.2.3.4", "user_agent": "..." }

All devices are exported but in the first version of the migration, only dehydrated devices will get imported and users will have to re-login after the migration is done.

device_keys

{ "type": "device_keys", "user_id": "@user:testserver", "device_id": "ABCDEF", "ts_added_ms": 1700000000000, "key_json": { ... } }

cross_signing_keys

{ "type": "cross_signing_keys", "user_id": "@user:testserver", "key_json": { ... } }

One record per (user_id, key_type). key_type is one of master, self_signing, or user_signing. key_json is the full key object from e2e_cross_signing_keys.keydata (JSON-encoded Matrix cross-signing key, including keys, usage, and any embedded signatures).

cross_signing_signatures

{ "type": "cross_signing_signatures", "signatures": { "@target:testserver": { "DEVICEIDORKEY": { "@user:testserver": { "ed25519:AAAAAAAA": "base64...", "ed25519:BBBBBBBB": "base64..." } } } } }

One record per signed target. signatures uses the same target-keyed nesting as /keys/signatures/upload. Keyed by target user, then target device/key ID whose value is the signer’s key_id signature map, aggregated from the matching rows in e2e_cross_signing_signatures.

one_time_keys

{ "type": "one_time_keys", "user_id": "@user:testserver", "device_id": "ABCDEF", "algorithm": "curve25519", "key_id": "AAAAAAAA", "ts_added_ms": 1700000000000, "key_json": { ... } }

fallback_keys

{ "type": "fallback_keys", "user_id": "@user:testserver", "device_id": "ABCDEF", "algorithm": "signed_curve25519", "key_id": "AAAAAAAA", "key_json": { ... }, "used": false }

There is at most one fallback key per (device_id, algorithm). The used flag indicates whether it has already been handed out to a peer (but not yet replaced by a new upload).

device_messages

{ "type": "device_messages", "stream_id": 42, "user_id": "@user:testserver", "device_id": "ABCDEF", "sender": "@sender:testserver", "event_type": "m.room.key", "content": { ... } }

Import Process

Prerequisites

Both servers must not be in use before the import begins to prevent concurrent writes from interfering with sequence numbering or derived-table consistency.

Validation

The importer validates the header on startup:

• format_version must be a version the importer supports.
• server should match the expected source server name. A mismatch is a warning, not a hard error, since some migrations intentionally rename the server, but the operator must confirm this explicitly.

The importer validates the trailer on completion. A missing trailer means the stream was truncated; any partial data from the last in-progress table should be rolled back.

Resumption

The importer tracks a cursor per table. On restart, it passes the cursor back to the exporter under the query parameter named by the header’s cursor_param, and the exporter resumes the stream from that point. Records within a table are independent of each other, so partial-table imports are always safe to resume.

Per-Record-Type Processing

users

user_id, displayname and avatar_url are stored directly on the user row (user_id, display_name, avatar_id). Arminius has no way to represent a deactivated account, so a deactivated user stops the import at that record (keeping the users imported before it) rather than being imported as active.

devices

Synapse’s last_seen_ms, ip, and user_agent have no equivalent in arminius and are dropped.

events

For each event:

1. Insert into events. The exported stream_ordering maps to events.stream_id.

2. Parse event_json.prev_events and insert corresponding rows into previous_events. Both the current event and each prev-event must already exist; events must therefore be imported in ascending stream_ordering order (which the exporter guarantees).

3. Parse event_json.auth_events and insert corresponding rows into auth_events. The same ordering constraint applies.

4. Call insert_in_ordering to place each event into event_ordering.

5. After all events are inserted, recompute leaf_events per room: an event is a leaf if it has no entry in previous_events.previous_event_nid.

6. Build state_snapshots, state_snapshot_state_events, and events_state_snapshots by replaying state resolution over all events in topological order.

7. Recompute room_state_snapshots from each room’s current resolved state.

Replaying state resolution is only possible where the source server actually computed the state itself. For outlier events Synapse does not always resolve state locally, it instead trusts other homeservers. For this version, rooms whose state is incomplete are not imported, and events that cannot be validated are left out of the imported DAG.

account_data

Maps to global_account_data. user_nid is resolved from user_id.

room_account_data

Maps to room_account_data. Both the user and the room must already exist; room_nid is resolved from room_id.

presence

Field mapping from Synapse to arminius:

receipts

Maps to read_receipts. The event_id must be resolved to event_nid. thread_id: null maps to thread: null. ts_ms maps to ts as timestamptz.

device_keys

Maps to user_keys with key_type = 'device'. The device_nid is resolved from (user_id, device_id). The exported key_json field maps to the key_object column. stream_id is not present in the export; it is assigned from the user_keys_streamids sequence on insert.

cross_signing_keys

Maps to user_keys with the corresponding key_type (master, self_signing, or user_signing). device_nid is NULL for cross-signing keys. user_nid is resolved from user_id. The exported key_json field maps to the key_object column.

cross_signing_signatures

Maps to local_signatures. Only records where target_user_id is a local user are imported.

• signing_user_nid is resolved from user_id.
• signing_key_type is resolved by looking up the user_keys row for the signing user whose key_object contains a key with ID matching key_id; its key_type (master, self_signing, or user_signing) is used.
• signed_key_nid is resolved from user_keys as follows: if target_device_id is a real device ID (a user_keys row exists with key_type = 'device' for the resolved (target_user_id, target_device_id)), use that row’s NID. Otherwise target_device_id is a cross-signing key ID; find the user_keys row where user_nid = target_user_nid, device_nid IS NULL, and the key ID embedded in key_object matches target_device_id.
• The signature field is decoded from base64 into bytea.

one_time_keys

Maps to one_time_keys. device_nid resolved from (user_id, device_id).

fallback_keys

Maps to fallback_keys. device_nid resolved from (user_id, device_id).

device_messages

Maps to to_device_messages. device_nid resolved from (user_id, device_id). sender must be a local user; it is stored as bigint REFERENCES users in arminius, so remote senders are not representable and their messages must be dropped.

Post-Processing

Sequence Reset

All PostgreSQL sequences must be reset after import so that new writes do not collide with imported data:

State Rebuild

The state_snapshots / state_snapshot_state_events / events_state_snapshots / room_state_snapshots tables are derived from the event DAG and are not exported. They must be computed by replaying state resolution over all imported events in topological order. For large rooms this is expensive and may take significant time.

Leaf Events

leaf_events must be recomputed after all events are imported. The leaf set is every event that does not appear in previous_events.previous_event_nid.

IdP Sync

In this first version of the migration, users, and devices will be created automatically, but linking them to the IdP will be a manual operation done by the customer.

Known Gaps

Media files: Events may contain mxc:// URIs. The export carries no media files; only event JSON is exported. Media must be transferred separately at the filesystem level (copy the source server’s media store to the target before or after import). Remote media (hosted on other homeservers) is unaffected as it is re-fetched on demand.

Pushers: Push notification registrations (APNs/FCM tokens) are not exported. Users’ client applications will re-register their push tokens automatically when they first connect to the new server.

Access tokens: Not exported. All existing sessions are invalidated; users must log in again after migration.

User threepids: Will be exported later if needed

Remote-sender to-device messages: to_device_messages in arminius requires the sender to be a local user. Any pending to-device messages from remote senders (e.g. key exchange messages from federated users) are dropped during import. This may cause one-time decryption failures for in-flight E2E sessions; affected users should re-request keys from their peers.

Nuss and IdP at Famedly

Our software has users, and we have to authenticate and authorize them before they can do anything. The standard solution to this problem is identity providers (hereafter referred to as IdP), software that has data about users, allows them to authenticate themselves with a consistent interface, and authorize software to perform actions on their behalf. Contemporary IdPs usually follow the OIDC and SCIM standards, although the latter less consistently and with some caveats.

We need our software to interoperate with such IdPs and provide some Famedly-specific functions for them. Currently Nuss is the piece of software that implements the most custom features related to IdPs, so this document is focused on it.

The goal of the initial version of this document was to reach a consistent approach to dealing with authentication and authorization within Famedly, but that turned out to not be worthwhile, so now it only pertains to features related to Arminius.

Decision

We will develop Nuss as an identity broker geared specifically towards working with Arminius. For now, Arminius will keep synchronizing with it using custom SCIM, however they might become more integrated in the future. Zitadel remains our main IdP solution at Famedly, with no further plans to homogenize our approach.

See the Solution section for more technical details.

Context

Original Problem

When developing the new homeserver the Arminius team stumbled into the problem of login session management. Matrix has long-lived sessions, created for every logged in device, which have to be managed, including the possibility of revocation without access to a device. Our current IdP solution, Zitadel, in no way supported that, so Nuss was created as an identity broker that adds this functionality. At this point Nuss also has other functionalities, but only session handling is absolutely necessary.

Current Design

We use Zitadel as the IdP for all our services. It provides user management functionalities, supports authentication through OIDC and user info fetching through SCIM. It doesn’t quite have all the features needed, so in many places we need various glue services, although only one is relevant to this document.

Nuss

Nuss is an identity broker with additional features that Arminius uses to solve the user management problem in a different way than Synapse – by assuming that it always has an accompanying IdP which keeps all the necessary user data and handles authentication and authorization. Nuss is necessary, because IdPs in general don’t support managing persistent sessions, which is a strict requirement for Matrix. Some technical specification of the interface Arminius expects, and Nuss provides, is present in the SCIM client component documentation.

Nuss interacts with Zitadel by fetching all the user data from it using SCIM and by being an OIDC proxy, capturing authentication requests to remember them as persistent sessions. It should also (although it is not currently properly implemented) handle authorization, both in the sense of ensuring that a given entity is allowed to have certain permissions, as well as requesting user consent to grant these permissions in specific cases.

Constraints

1. We need persistent session data and Matrix-specific permissions in our IdP.
2. Users might be moved between IdPs, so we have to support manually mapping Matrix accounts to users.
3. We have to be able to associate users from the same upstream IdP to different homeservers, both for limiting logins to Matrix accounts, as well as administration rights.
4. We have to support the MLS flow, i.e. users without an associated Matrix account and dynamically created pseudonymous users authenticated through client secrets associated with Matrix accounts.
5. We have a planned release of Arminius in October, so it and all necessary side services have to be ready for then at least as a MVP.

Desiderata

1. We would prefer to keep OIDC login session information for Arminius in a separate service than Arminius.
2. The user should be able to grant different permissions to different Matrix devices, with explicit consent for them.
3. Nuss should be able to run multi-tenant.
4. Arminius should be able to long poll Nuss for user information, to make synchronization simpler and snappier.
5. It should be possible to automatically map information acquired through SCIM to Matrix accounts, in particular (although not limited to):
   1. assign a user ID localpart based on upstream metadata
   2. Create a display name based on potentially several upstream profile fields.
6. Arminius could be a general use Matrix server for self hosting.
7. We would like to support migrating from Synapse to Arminius.

Solution

Proposed Design

Nuss becomes an identity broker geared specifically for Arminius, and always running next to every Arminius instance. It will:

1. synchronize user data from upstream IdPs using SCIM, and proxy OIDC requests to them, injecting persistent session handling and Matrix-specific permission controls in the process
2. map groups from upstream IdPs to available Matrix and Nuss administration permissions in a configurable manner, details in a follow-up ADR
3. have its own frontend for letting the user consent to permissions and some administration tasks as needed
4. support mapping users to Matrix accounts, with the details decided in a separate ADR, but supporting at least
   1. allowing users without associated Matrix accounts, for administration purposes
   2. multiple upstream IdPs
   3. automatically mapping fields from the upstream IdP to a displayname
5. support creating local-only pseudonymous users through client registration, as required by the MLS flow

We are assuming the upstream IdP can use the client ID Nuss provides to limit which users will be able to login on a specific server, so we do not expect to have to implement that functionality on the Nuss side.

Arminius will fetch data from Nuss using SCIM, but there might be tighter integration of the two components in the future, see the Alternatives section for details.

Interaction Diagram

stateDiagram-v2
    nuss: Nuss
    upstreamIdP1: Upstream IdP
    upstreamIdP2: Upstream IdP
    mls: MLS
    arminius: Arminius

    nuss --> upstreamIdP1: SCIM fetch
    nuss --> upstreamIdP2: OIDC redirect

    mls --> nuss: creating pseudonymous users through SCIM
    arminius --> nuss: SCIM fetch, with sessions
    arminius --> nuss: OIDC authentication

The OIDC flow shown in the diagram is standard identity brokerage, only Nuss keeps the data about existing login sessions to facilitate session and device management, and injects consent requests for Matrix-specific permissions.

The two upstream IdPs in the diagram are only for illustration purposes, there can be arbitrarily many and all actions supported by them can be performed on all of them. There can, however, be only one Arminius instance.

Rationale

This approach satisfies all the hard requirements and has most of the desired qualities, while requiring much less development work than alternatives.

We end up with multiple copies of user data in our system, but most ways of dealing with this would require significant architectural changes which we don’t have the resources for currently.

Arminius should be usable as a self-hostable server, although with a relatively complex setup for such a service, requiring also managing Nuss and an upstream IdP.

We are dropping the idea to support multi-tenancy, because it makes little sense for a service that is so tightly connected to Arminius.

Consequences

Roadmap

We have the delivery deadline for Arminius in October, so we need a clear roadmap of what we want to do until then, and what can wait. We expect to be able to implement most of the features described in this document by then in a basic version, but full implementations will follow later.

October Deadline

By October Nuss should support:

1. OIDC:
   1. At least the authorization code flow. [already done]
   2. Proxying the requests to upstream IdPs. [already done]
   3. Saving OIDC session information to allow session management. [already done]
   4. Granting the user only scopes they are authorized to.
   5. Requesting user consent for scopes. [mostly done, but requires human readable descriptions of scopes]
2. Upstream SCIM:
   1. Synchronizing data from a single upstream. [done, although needs a rework to allow multiple replicas]
   2. Mapping some upstream information into standardized, Matrix specific information (user ID localpart, display name), details to be decided in a separate ADR.
3. Downstream SCIM:
   1. Providing user data. [mostly done]
   2. Providing session data. [mostly done]
   3. Long polling of data changes. [already done]
   4. Session management. [already done, deletion only, but we don’t need anything more]
4. Permission control management – we need to define groups in the upstream IdPs and their mapping to Matrix and Nuss permissions, details to be decided in a separate ADR.
5. Remove the dev mode in Nuss and create a test setup with a mock upstream IdP.

In particular we still need two ADRs:

1. Permissions in Arminius and Nuss.
2. User mapping from the upstream IdP to Matrix accounts, with a particular focus on mapping of data.

Longer Term

Longer term we would like to implement the following:

1. MLS:
   1. Users with no associated Matrix account. (with 2.2 likely being a prerequisite)
   2. Registering a client with an associated user authenticated through the client’s secret.
2. User management:
   1. Allow multiple upstream IdPs.
   2. Manage users’ association with Matrix accounts.
3. SCIM, either (the decision will require a separate ADR):
   1. implement filtering of requested resources, metadata endpoints, and all the other minor parts of the specification, documenting which optional parts we are not implementing and why, or
   2. integrate Nuss more tightly with Arminius, removing downstream SCIM altogether, as described in the Alternatives section.

Problems with This Approach

While Zitadel or other upstream IdPs remain the sources of truth for user data, we have multiple copies of this data in many different services, which will become very confusing on any desynchronization.

Arminius will only work if deployed together with Nuss, while Nuss pretends to be a separate component. The communication between these components uses a protocol based on SCIM, but with enough differences that the modularity is only a mirage.

We only support upstream IdPs that use OIDC and SCIM.

Alternatives

This section details two wholly different approaches to the overall design, the first of which we might want to adopt in the future. This is followed by two minor parts of this approach that were discarded as impractical.

Session Management in Arminius

We could implement session management in Arminius. Then Nuss would not be necessary and Arminius would connect to Zitadel, or another IdP supporting standard SCIM and OIDC directly.

Why Not

More unnecessary complexity in Arminius, although we could deploy it with any OIDC/SCIM upstream!

This would essentially mean merging current Nuss functionalities into Arminius, only removing the downstream SCIM protocol.

The Nuss-like component would have to be accessible on a different domain than the rest of Arminius for security reasons. This makes the setup more complex.

To support MLS Arminius would also need to support local, dynamically created users. This is beyond what one would expect from a Matrix server for general use.

There are also issues with populating users before they login. We would have to either mandate SCIM upstreams and proactively update our user list quite often (like Nuss does currently), or ensure users are pushed into Arminius (like with Synapse currently).

Why Maybe

In the future we might want to integrate Nuss and Arminius much more tightly, to remove at least one copy of user data from our system. This should only be done if it’s possible to keep all the user and session data in one copy within Arminius. In the current setup this would mean operating on the single Arminius DB directly, but we might want to rethink how that data is kept anyway.

If Nuss were to be integrated that way, this would also remove the not-quite-standard SCIM interface between Nuss and Arminius, replacing it with data flows through NATS, like with all other Arminius components.

When contemplating such a decision we should try to avoid a situation like with the Matrix Authentication Service, where it’s only partially integrated with Synapse, thus just adding to the confusion rather than clearing stuff up. This means such a change would require very careful consideration of the necessary architectural changes in Arminius.

The details of such a proposal are far beyond the scope of this ADR, and would require a separate one.

Nuss as a Full-Fledged IdP

Make Nuss into a full-fledged IdP, thus eliminating the need for external IdPs, completely centralizing where we keep user data, and perhaps even extending SCIM standards to include our custom modifications.

We could also centralize all frontend in Famedly Control, providing a unified user experience. This standardization would benefit all of Famedly.

Why Not

Supporting a full-fledged IdP is a lot of dev work, and very security sensitive work at that. If we were to do that, it would probably be better to fork an already established IdP, but maintaining a fork is its own can of worms.

At least until the standardization happened, Nuss would still be the only IdP Arminius would work with, making the modularity illusory.

We would still have to support external IdPs, since some customers would use them, so we would have to support identity brokerage anyway.

Also Include Synapse Compatibility

Synapse requires creating accounts on the server itself, in addition to having them in an IdP. Currently Simplified User Management Service plugs into Zitadel to do that, and we could replicate this functionality in Nuss to replace it.

Why Not

This kind of plugin is quite some effort to implement, and Synapse already works with our glue software fine.

This ADR does not mandate such compatibility, but if it turns out it’s for some reason needed and not that much effort no one will be stopped form implementing it somehow. However, it’s more likely that the alternative approach on this branch would be adopted in such a case.

Nuss Pushing Users to Arminius

Arminius is currently pulling users from Nuss to stay synchronized, and it uses the nonstandard long polling to make that reasonable. A natural alternative would be to have Nuss push any user updates to Arminius instead, preferably also through SCIM.

Why Not

Arminius would have to expose an interface for that, and securing that interface would be a nightmare, since Arminius has to be externally accessible at least for clients and often also for federation.

Permissions in Arminius and Nuss

Author: timorl

Some user actions, chiefly account suspensions and locks, on the homeserver require more privileges or additional authentication. Nuss also requires handling permissions for user management. Furthermore, the user might want to consent to only specific action scopes for a given login, e.g. to facilitate read only access from devices, create bots with limited permissions, or just ensure that they are not authorizing a significant action they did not intend. This document describes how we deal with all these authorization issues.

Decision

Nuss will have a configurable mapping from group membership, as indicated by appropriate metadata in the OIDC process, to allowed permission scopes for a given account. Our initial configuration will consist of three privileged groups that users can belong to, two for Matrix administrative and moderation tasks, and another for Nuss. For details, including the definitions of permission scopes, see the Solution section.

Context

Current Setup

Arminius, our Matrix homeserver, delegates user and session management to Nuss. Currently, Nuss only proxies authentication to an upstream IdP, using OIDC. In particular, it does not really proxy authorization, as it completely replaces any request with one that requests only authentication, and later just grants all requested scopes if the authentication is successful. (Yes, this is as insecure as it sounds, at the time of writing this is not remotely production grade software.) We are also synchronizing used data over SCIM, which does contain some group membership information, which might be relevant to permissions, but we are not handling this information in any way.

According to the IdP ADR Nuss will remain just an identity broker for connecting Arminius to upstream IdPs.

Constraints

1. We can create and assign users to groups in upstream IdPs.
2. Upstream IdPs can return all information about group membership in login tokens.
3. Our customers must be able to manage who has access to these groups.
4. We must support the standard Matrix scopes.
5. We must restrict access to the Matrix endpoints spelled out in MSC4484.
6. We must restrict access to Arminius administration endpoints in a manner consistent with Matrix administration endpoints.
7. We must restrict access to the Nuss user management functionality. This includes:
   1. mapping upstream users to Matrix accounts,
   2. mapping user group membership to allowed permission scopes.

Desiderata

1. It would be nice to be able to give permissions only to access specific API endpoint.
2. In particular permissions only for specific rooms.
3. Even more particularly, read-only permissions.

Solution

We introduce a Nuss API for defining a mapping from group membership, as represented in claims returned in ID tokens, to allowed permission scopes.

Scopes

The specific scopes we support are:

1. urn:matrix:client:api:* – the standard Matrix scope granting access to the entire non-admin client API
2. urn:matrix:client:server_administration – the proposed scope for accessing the Matrix admin client API, it will grant access to all endpoints mentioned in the MSC and all Arminius administration endpoints
3. urn:matrix:client:server_moderation – a new scope only allowing for server moderation, it will grant access to all user moderation endpoints, like locks, suspensions, listing users and rooms; it will not grant access to administration endpoints (currently only Arminius key rotation); the server_administration scope expands and overrides this one
4. urn:matrix:client:device:<device_id> – the standard Matrix scope used for associating a login with a device, as defined in the spec
5. urn:nuss:administration – access to Nuss’ administrative endpoints, allowing mapping users to Matrix accounts and changing the mapping of groups to permission scopes
6. urn:matrix:client:api:room:<room_id> – only grant access to operations on the specific room
7. scopes that add read right after urn:matrix:client:api:, and create the same scope, but with read-only access

Redundant requested scopes will be removed by Nuss, only leaving the minimal strongest set – right now this only applies to overriding read-only scopes, removing the room scope if the full API scope is also present, and the Matrix administration scope overriding the moderation scope. However, Arminius will have logic valid for handling any set of scopes anyway.

We might add more scopes in the future, the above list only contains what we already know that we want.

Login Flow

On login, the user’s client will request a set of scopes it needs, like usual. Nuss will first redirect the user to the upstream IdP, where the user will login. Then it will present a consent screen, containing only the scopes that were both requested and are possible to grant based on the user’s group membership. These scopes should be presented as human-readable descriptions of the permissions being granted, but with a possibility to see the scope itself (i.e. through hovering over an UI element). Nuss will return a token granting the scopes if the user granted consent.

Mapping Definitions

A mapping consists of:

1. the ID of the upstream IdP this mapping applies to
2. a group membership description, represented as a JSON containing
   1. the type of mapping, currently either “All” or “IdToken”; the former represents scopes anyone can request
   2. in the case of idToken additionally a specification consisting of
      1. the claim JSON path
      2. a regexp that should be matched by the string under that path
3. a list of allowed scopes; a scope definition ending with a * means all subscopes of that scope can be granted (including the one that literally ends with a *)

This will usually be represented as a JSON, containing additionally an ID to refer to it in some operations. Example:

{
  "id": 13,
  "idpId": 12,
  "group": {
    "IdToken": {
      "path": "roles",
      "match": "\bmatrixAdmin\b"
    }
  },
  "scopes": [
    "urn:matrix:client:server_administration",
    "urn:matrix:client:server_moderation"
  ]
}

Mapping API

The configuration API will be a standard REST API, allowing for listing, adding, modifying, and removing mappings, using the JSON format described above. The detailed definitions of endpoints will be created in the course of implementation, as there is little that isn’t obvious.

Attempting to remove the last mapping granting access to the urn:nuss:administration scope will return an error, otherwise there should be no restrictions.

Initial Configuration

On Nuss setup we will provide a configuration file defining the initial mapping as a JSON array. Minimally the mapping has to include at least one group granting the urn:nuss:administration scope, otherwise the setup should return an error. The initial mapping will be loaded into the database, and any further modifications should happen through the API. Any provided configuration file should be ignored with a warning if the database mapping is already initialized.

Our Mapping

We already mostly know the basic outline of how we would like to map groups to permissions so, while we don’t need to define that at the ADR level due to configurability, the following is both a recommendation, as well as an example.

We would like to define three groups for upstream IdPs: matrixModerator, matrixAdmin and nussAdmin. The first one grants access to the Matrix moderator endpoints (scope 3), the second one to Matrix administrator endpoints (scope 2), and the third one to Nuss administration (scope 5). All other scopes are available for any account that is mapped to a Matrix account. These scopes formatted as a JSON table should look like follows:

[
  {
    "idpId": 0,
    "group": {
      "IdToken": {
        "path": "roles",
        "match": "\bmatrixModerator\b"
      }
    },
    "scopes": [
      "urn:matrix:client:server_moderation"
    ]
  },
  {
    "idpId": 0,
    "group": {
      "IdToken": {
        "path": "roles",
        "match": "\bmatrixAdmin\b"
      }
    },
    "scopes": [
      "urn:matrix:client:server_administration",
      "urn:matrix:client:server_moderation"
    ]
  },
  {
    "idpId": 0,
    "group": {
      "IdToken": {
        "path": "roles",
        "match": "\bnussAdmin\b"
      }
    },
    "scopes": [
      "urn:nuss:administration"
    ]
  },
  {
    "idpId": 0,
    "group": "All",
    "scopes": [
      "urn:matrix:client:api:*",
      "urn:matrix:client:device:*"
    ]
  }
]

These cannot be a complete initial configuration – they refer to an IdP of ID 0, so that has to be configured as well – but this is everything related to permissions. The complete configuration is out of the scope of this document, and will most likely be finalized only during implementation.

Revocations

Revoking administration or moderation permissions, whether by changing user roles in the upstream or changing the mappings, should propagate to all sessions with the appropriate scopes, which in turn will propagate through the SCIM sync to the homeserver. Since login tokens are supposed to represent scopes, they will be marked as revoked, but users will be able to attempt a token refresh.

Other Details

We will ensure authorized access to restricted endpoints, by defining axum extractors for types representing various forms of scoped access.

A note of implementation priorities – we don’t have to implement the last two scopes listed above for the October release. Everything else described here should be there, at least in a MVP version.

We probably will want to propose some of our extended scopes as MSCs, although all of the ones proposed here are entirely optional and shouldn’t break any client compatibility, so this is not high priority.

The mapping of upstream users to Matrix accounts can also be seen as a permission issue, however this topic is left for another ADR since it comes with more problems unrelated to permissions. For the purposes of this document, we assume that before checking for allowed scopes for a specific Matrix account we first checked whether the user is even allowed to log into that account.

Rationale

With this solution we get all the access controls we want in a very flexible and configurable manner.

The solution adds relatively few assumptions about the upstream IdP on top of what we already require. Chiefly the fact that group membership can be represented as substrings in a field in the provided ID token – and even that should be easily extensible in the future if need be, since we already treat “IdToken” and “All” as an enumeration.

Group membership being determined through regexp matches might seem excessive, but unfortunately ID token fields that are lists are usually represented using just spaces as delimiters, so we cannot assume there is any JSON structure there.

We only specify a small number of additional scopes to implement, as their main point is to exemplify the schema how we could specify more. Implementing these should pave the road for any future ones we might want.

Consequences

We will have to implement:

1. Access controls through scope handling on the Arminius and Nuss sides.
2. API for configuring the mapping of group memberships to allowed scopes in Nuss.
3. UI for the above, initially at least a basic version.
4. Executing the above mapping on the Nuss side, including scope deduplication.
5. Initial configuration logic.

This architecture should make it relatively easy to add more granular permissions and mapping to them from arbitrary sources in the future.

Nuss should be able to work with a wide range of IdPs that support just SCIM+OIDC, although initially it would require passing the group membership information in ID tokens, like Zitadel does. Getting this information out of Zitadel requires a custom scope though (urn:zitadel:iam:org:projects:roles), so we will probably have to implement some parts of the support for specific other upstreams manually.

The homeserver will have to keep information about all OIDC sessions, not just devices, to facilitate revoking tokens when the scopes associated to a session change. This will make getting devices from the database somewhat more complex, as it will require parsing some scopes.

Alternatives

Static Mapping of Groups to Allowed Scopes

We could hardcode the mapping of groups to allowed scopes, saving quite a bit of complexity and development time.

Why Not

Having this be configurable is a strict requirement.

This would limit us only to IdPs with configurable group names, and ones where we can require a very specific group assignment.

Group Membership Information in SCIM

Nuss already uses SCIM to get additional information about users, so we could read group membership information there, and use it for the mapping into allowed scope requests. This would let us work with IdPs that don’t support showing group memberships in JWTs.

We might want to implement this at some point anyway, if this turns out to be the best way to support a particular IdP.

Why Not

We currently only use SCIM data for auxiliary info, and this approach would turn it into a mandatory part of authentication.

We only periodically update SCIM data, so we would either have to manually trigger an update on every login, or risk granting outdated permissions. Having all this happen in a single flow leaves less room for authorization problems, and puts less complexity in the code.

Define All Scopes Directly on the Upstream IdP

We could in principle define all scopes in the upstream IdP, and have Nuss only proxy the request, with all the permission logic happening through configuration in the upstream. This would significantly cut down on the complexity of Nuss.

Why Not

Zitadel doesn’t support custom scopes at all, and probably neither do most other OIDC IdPs.

Even if an IdP supported such custom scopes, we would still have to deal with the urn:matrix:client:device:<device_id> scope manually, since its behaviour is very Matrix specific, and related to device and session management.

User Info Mapping in Nuss

Author[s]: timorl

Nuss is the identity broker for Arminius adding functionality on top of what upstream identity providers (IdPs) deliver. It’s supposed to map upstream IdP users into Matrix accounts. This mapping, however, is not trivial, as users might be allowed to only access specific Matrix accounts and user information might map nontrivially to Matrix account properties.

This ADR describes these problems in some details and defines a solution.

Decision

We will assume that several pieces of metadata in the upstream IdP define the necessary information that has to be mapped into Matrix account data. This mapping will happen in Nuss, and the exact way it happens will be configurable.

Whether a user can authenticate to a specific Matrix server will be configured completely on the IdP side, using the fact that a specific Nuss instance acts as a client for the IdP, thus login attempts from a specific server can be identified and rejected or accepted.

Details in the Solution section.

Context

Matrix accounts contain information about the user that is usually also present in identity providers. This information, however, is not structured in the same way in all cases, so we need to map it from the upstream IdP to Arminius. Of particular interest are the Matrix ID localpart, which is necessary for the functioning of the protocol, and display names which might need to be generated from several pieces of information that are separate in the upstream.

Synapse solved this problem with SSO mapping providers, which are Python classes you have to load for a specific deployment.

For localparts specifically, there are several MSCs (e.g. 1, 2), that propose redefining what the localparts represent. For now we assume the localparts will remain arbitrary, with the only restrictions defined in the appropriate part of the Matrix spec.

Other than that, Matrix profiles normally contain 3 fields – the display name, the avatar URL, and the time zone. These can usually be changed through the server API, but this can depend on the server. The avatar URL has to be a Matrix Content URI, so the avatar image itself has to be uploaded to the server somehow.

We are currently populating users in Arminius through a SCIM sync. We are doing this to be able to create and manipulate all user accounts before their first login (as would be the case if we relied on OIDC for this information). We are using SCIM in two places, once when Nuss gets data from the upstream IdP, and once when Arminius gets data from Nuss. The latter use might be replaced in the future, but the former should remain indefinitely.

Constraints

1. We need to work with Zitadel as an upstream IdP.
2. Arminius needs to populate enough Matrix account information to display users before they login for the first time.
3. Nuss has to provide valid localparts to Arminius for users that are supposed to have a Matrix account there.
4. Upstream users might have several associated Matrix accounts on different servers, Nuss has to provide the correct localpart to its associated server.

Desiderata

1. We would like to use SCIM for fetching information from upstream IdPs.
2. Nuss should be able to map arbitrary information from the upstream to a Matrix display name.
3. Nuss should be able to forward the user’s timezone from the upstream IdP to Arminius.
4. Nuss should be able to fetch and upload an avatar from the upstream IdP to Arminius, also setting the correct avatar URL for the downstream user.
5. We would prefer that the upstream IdP were the sole authoritative source of user information.
6. The localparts themselves should be pseudonymous, but easy to associate with upstream users with access to the upstream IdP.
7. We might return login errors on upstream misconfiguration – e.g. two users with identical localparts.
8. An upstream user might be allowed to login to the homeserver without a Matrix profile, e.g. for administration purposes.

Solution

Nuss will continue fetching user data from the upstream IdP through SCIM and provide it to the homeserver both through SCIM, as well as in the OIDC login token where applicable.

Upstream SCIM Data

The upstream has to provide several pieces of data to Nuss through SCIM:

1. the localpart for the specific server
2. the data required to construct a display name
3. the user’s timezone
4. the user’s avatar

We have to treat each of these slightly differently.

Localpart

The localparts of a specific upstream user will be returned as JSON objects in the entitlements field of the SCIM user struct, following the schema:

{
  "type": "localpart-$DOMAIN",
  "value": "$LOCALPART"
}

where $LOCALPART obviously refers to the literal localpart, and $DOMAIN is the domain of the homeserver this localpart is associated with. The upstream might return multiple such objects, and it’s the responsibility of Nuss to find and extract the appropriate one.

Repeated or invalid localparts should result in warnings in logs and the Nuss admin UI, and there should be an interface to view a list of them.

The responsibility of keeping the localparts pseudonymous rests on the upstream IdP data administrator. We might provide some automated tools facilitating this, but such software is out of scope for this document.

Display Name

The display name might need to be constructed out of several pieces of information from the upstream. The particularly relevant ones are:

1. displayName – this should be the default, and in initial implementations might be the only option
2. nickName
3. title
4. all the subfields of name

We might also want to support custom fields put in entitlements, although that would complicate the formatting logic somewhat.

The formatting logic will probably have to be implemented manually, because existing solutions are either too weak to support arbitrary subfields (e.g. new_string_template or are full, feature-rich template engines (e.g. Sailfish, or even minijinja).

The formatting will be configurable through a standard REST API, and might be different for different upstream IdPs. It will be represented by a JSON with two fields, idpId with the ID of the upstream IdP the formatter pertains to and “displayName” containing the formatting string itself. The detailed definition of the endpoint will be created in the course of implementation, as there is little that isn’t obvious.

Timezone

The timezone will be read from the timezone field.

Avatar

The avatar will be fetched from the element in photos that is of type "photo" and is primary. If there is no primary, then the first "photo" should be used. When there is no "photo" then the user will not have an avatar set.

Avatars will be fetched as a separate process from the main user info synchronization process. For what happens next see the appropriate subsection in the following section.

Forwarding the Data

Once the data is available, it has to be forwarded to the homeserver, which again might require special treatment.

Localpart

If the localpart exists, is valid and unique, then it gets forwarded as username in the SCIM interface. Note that this is different than the resource ID, which will be treated as the user ID and also returned in the OIDC token sub field.

If there is no localpart, then the user might still exist and log in, but they will not have any profile, thus they cannot be treated as a Matrix user. They might, however, perform administrative or moderation actions, if they have the appropriate permissions.

If the localpart is not a valid Matrix localpart, the user should be shown an error message telling them to contact their user data administrators. These users might still log in for administrative or moderation purposes, like in the previous paragraph.

If there are multiple users with the same localpart for the given server, then the first user that was synchronized from the upstream should be able to login and normally use their Matrix account. Any subsequent users should be treated as if their username was invalid like in the previous paragraph, but with a different error message mentioning the duplication.

Display Name

The display name will be forwarded in the displayName field.

Timezone

The timezone will be forwarded in the timezone field.

Avatar

Once an avatar is fetched, the same long running process will upload it to Arminius. The resulting Matrix Content URI will be saved into the database and this will be forwarded as the only value in the photos field, of type "photo" and set as primary.

This process should be observable through a Nuss API, so that it can be clear when at least the initial import is done.

Initial Implementation Notes

Due to time constraints we might want to limit the implementation effort. With that in mind, we might initially only support a simple forward of displayName, without configurable composition, and no support for avatars.

Rationale

This approach mostly uses widely adopted standards, with the only stretch being using entitlements for localparts.

Localparts are fully the responsibility of the upstream data administrator. This leaves them flexibility if they need them to have a specific shape. Any mistakes should be easily noticeable with the proposed error reporting.

Avatars are handled separately from other user information, as otherwise preloading all users would take too long. It should be better to have all users loaded quickly, with the avatars only appearing slowly, rather than being limited to waiting for the entire import. However, the other behaviour can also be replicated by observing the photo import process and waiting for it to finish before letting anyone login.

The REST API for display name formatting should be consistent with the one proposed for permission handling.

Consequences

Upstream IdP administrators will have to put all the profile information into the IdPs. We might want to implement helper software to make this simpler.

Only upstream IdP administrators will be able to change profile information of users, unless they explicitly grant the users permissions to change the appropriate fields.

Arminius will have to support avatar uploads with Nuss as a source.

Avatar change propagation might be lag, and will always require changing the upstream avatar URI.

Matrix endpoints for manipulating profile information will all be disabled, all operations will have to happen in the upstream.

Alternatives

Upstream Channels Other Than SCIM

Using Zitadel’s other endpoints, we might be able to get profile information of users in a more natural way, and with more flexibility than with SCIM. In particular we wouldn’t have to stretch the meaning of entitlements.

Why Not?

This would limit us to Zitadel as the only upstream, while using SCIM should make supporting other upstreams much simpler.

Forbid Anyone with Overlapping Localparts from Login

When a duplicated localpart is detected for different users we could stop all users with the localpart from logging in. This behaviour would be more consistent and predictable.

Why Not?

Adding a new user might prevent an old user from logging in. This is a much bigger problem than a new user being able to log in – some friction is expected when introducing new users, and fixing it then at least shouldn’t impact any ongoing conversations.

Use Matrix Endpoints for Changing Profile Information

Clients support Matrix endpoints, so changing information through them would be a free feature.

Why Not?

In the context Arminius will be used we don’t want users to be able to change their own profiles anyway.

We could support this as an interface for users with permissions to change their profiles, but this should be rare enough that the engineering effort is not worth it. This is a feature that might one day show up though.

Putting the Domain for the Localpart in the $ref Field

The $ref field usually contains URIs so it would be a relatively natural place to put the domain, rather than in type.

Why Not?

This domain doesn’t really represent a resource, so it doesn’t quite fit. The spec also discourages returning the same (type, value) pairs more than once, which might happen if we have several with the only difference being $ref.

Homeserver Configuration with Runtime API

We will have 2 primary ways of configuration. Primarily configuration should be specified in a config file. But we may need to change certain configuration at runtime. As such we define a general scheme on how to configure options in either place. Each config option should then document if it can be changed at runtime or not.

Levels of Configuration

• Built-in default (if there can be a default, otherwise startup error)
• default section of the configuration file: Values can be overwritten at runtime.
• Configuration file: Can not be overwritten at runtime!
• API+NATS KV: Runtime configuration/modification via the API

This allows both specifying defaults during deployment as well as locking down values from modification at runtime. Configuration is ordered. If a configuration value is specified at a higher level, it replaces the lower level. However, if something is specified outside the default section in the configuration file, it cannot be overwritten and attempts to modify the value are rejected at runtime.

Since the configuration is dynamic, processes need to be able to react to changes to the configuration. The processes get notified of changes via the NATS buckets. Handlers need to be able to react to such changes, but only at the start of a request. It is fine for a request to finish with the previous state of the configuration as long as the next request starts with the right configuration.

The replacement could be done based on just the toplevel config key or on a lower level, if we want to increase readability of the config file.

We will not support locking values to the server builtin default at the start. Any such values would need to be specified explicitly. But long term we may want to introduce an easier way to lock such configuration to their builtin values, if our configuration becomes long and complex.

The configuration file will be using yaml as this is the most familiar format to the expected users of the config file according to the discussions we had.

Built-in Defaults

Some values can be safely defaulted to a specific value without having to block startup. For example the list of blocked servers can easily be defaulted to an empty list without impacting functionality. Such defaults can always be overwritten at at least one higher level, otherwise they wouldn’t be configuration values.

default Section in the Config File

To ease kubernetes deployments, there is a default section in the config file. This section mirrors the normal configuration. The only difference is that this configuration can be overwritten at runtime (and is “indented” one level further because it is below the default key).

If this turns out to not be needed in the end and add too much support overhead, we might remove this section in the future to simplify our configuration handling. For now this seems like the easiest option to address all the requirements.

Configuration File

The configuration file is using yaml as the format. Its location can be specified via a command line parameter (--config-file/-c). Otherwise the location defaults to /etc/arminius/config.yaml. Each process individually reads this file and there is currently no support for recognizing diverging configuration between processes (this hasn’t been an issue with other applications so far either).

Secrets would be problematic if they were specified in the configuration file directly as it makes leaking secrets by accident more likely. Instead such values require configuring a path to the secret in the config file and the actual secret is stored in the file at that path. Those files are treated as binary files and read as is without any postprocessing. Relative paths are assumed to be relative to the config file. Paths are assumed to use Unix file separators, but this could change if there was ever support for Windows. The config key in the config file always has _path appended to it to make it explicit, that the actual secret value is in a different file and providing it inline would be an error.

Config keys use snake_case.

The config file is currently not supposed to be reread on changes, because certain configuration can not be changed at runtime and any such configuration is only configurable in the config file. For now such configuration should be done via the API instead. This may change if there is a need for it.

Dynamic Configuration via the API

For dynamic configuration at runtime the homeserver exports an API which allows reading, writing as well as deleting configuration values. In the future “patching” might also be supported. Writes are then persisted in a NATS bucket named config based on the key used for their overwrite semantics.

The API is in the admin namespace:

GET /arminius/admin/config
GET /arminius/admin/config/key/<overwrite-key>
POST /arminius/admin/config/key/<overwrite-key>
DELETE /arminius/admin/config/key/<overwrite-key>

The GET method can be used to query the full or a subset of the configuration. The values are returned as JSON and wrapped in the following object:

{
  "value": /* The actual value, unless it is unset and there is no default */,
  "level": "builtin" /* or "file-default", "file", "api" or "unset" */,
  "locked": false /* or true to indicate it can't be modified via the API */,
}

Any secrets will not be returned. Instead they will only show if they are set.

Nested secrets are currently not supported via the API. Instead their keys will simply be missing from the response for now and dedicated APIs would be introduced to query or modify them.

POST allows replacing a specific value completely. It accepts a raw JSON value for config values and binary value for secrets. Certain configuration options are documented as “not configurable at runtime” and modification attempts will always yield an error. Similarly trying to modify a value from the configuration file (apart from inside the default section) will always yield an error.

DELETE unsets the value runtime configured value and makes it default to any higher configuration level as if the runtime value was never configured originally. Deleting a value is always okay, since the value would not have present at the first startup anyway.

Specific configuration options may introduce dedicated APIs to modify lists more easily or deal with nested secrets. Such APIs would be below config/ next to key/. This is the reason for the extra key/ component in the path to prevent clashes or confusion between custom config APIs and config values. For example the federation block/allow lists might have a dedicated API to add or remove entries like PUT /arminius/admin/config/federation.allow_list/entry and POST /arminius/admin/config/federation.allow_list/entry/<idx>.

The config API tries to immediately modify the value via the NATS key-value API. If that modification fails because of a concurrent update, an error is returned and the user has to retry. This is because we can’t safely resolve conflicts between users changing the configuration at the same time. This is another reason some configurations may want to have dedicated APIs, that explicitly modify a subset. The configuration is NOT written to the config files on the file system! Those are treated as read-only. Currently the API does not offer any explicit CAS (compare and swap), but this could be added in the future.

Every service listens to config changes on the NATS KV bucket. If such a change is detected, it updates the config provided in State for any future HTTP requests as well as for any next iteration of background jobs or similar. Updates do not affect requests or tasks already in progress, unless they are particularly long running, in which case such tasks need to manually implement config reload logic at reasonable locations.

Dynamic configuration can only overwrite defaults, builtins or previous dynamic configuration. It can not overwrite other configuration from the config file.

Configuration Merging

Since there are several levels of configuration, they need to be merged in order. Merges are done based on the overwrite key, that is defined for the configuration option. Any configuration at a higher level completely overwrites the lower level (unless overwriting is blocked for the file -> dynamic case). No merging of values is done apart from collecting all overwrite keys at their highest respective level.

The configuration is read from disk and NATS before serving any endpoints and may terminate based on config errors in the configuration file (missing or wrong format). NATS values should not cause startup errors, but might be ignored with an explicit warning in the logs. This is to allow moving a config value to the config file without having to first unset it, but also because config values should always have been valid at some point before. The NATS config needs to be fully read before startup to prevent races where a config has a different value for a short period during startup.

Configuration Options

This list is the initial set of configuration options, that are either currently in use or will be needed at a later date. The actual list should live in an explicit page of the documentation and the below list will not be updated.

Any . in the overwrite key indicates a multi-level mapping in yaml. E.g. server.name would be represented as the following yaml:

server:
  name: "something"

The other NUSS configuration options are skipped for simplicity in the list above.

Maintenance mode may also be converted to a configuration in the future

Opentelemetry

Opentelemetry specifies standardized environment variables. Because these might be used by kubernetes operators, we should support them as well. However, they can be overwritten by either an explicitly empty opentelemetry configuration to disable all telemetry or by specifying the full OpenTelemetry config in that section.

More information on OTel config options can be found here:

• https://opentelemetry.io/docs/specs/otel/configuration/sdk/#in-memory-configuration-model
• https://opentelemetry.io/docs/specs/otel/protocol/exporter/#configuration-options
• https://opentelemetry.io/docs/specs/otel/configuration/sdk-environment-variables/

Implementation Ideas

Since we want the config to be cheaply available to request handlers without having to lock a mutex or otherwise stall on cache lines a lot, it makes sense to hook into the Clone done by axum before each request to reduce the amount of atomic or locking operations we need to do.

This should be achievable by manually implementing Clone in State and having the read value be cloned cheaply when the state is cloned, but doing a more expensive replacement of the write value when there is an update on NATS. A Mutex would work for the write value, but would require a lock during clone(), as such an alternative like ArcSwap is likely better. Ideally we would just have stable atomics for Box/Arc.

ADR: Simplified Sliding Sync

Issue #1741.

At the highest level, sliding sync works by the client being able to filter which rooms they are interested in having updates from (instead of the server fetching every possible room update) using “lists” and “subscriptions”. If none of the filtered rooms have any update, then long-polling starts by the server holding that connection until new data arises.

Terminology Note: In the following document, the Matrix mechanism “Simplified Sliding Sync” (MSC4186) is referred to as “Sync v4”. Code implementations, structs, and database schemas related to this mechanism will use the syncv4 identifier.

Author: Pierre

Co-Authors: Michael Schleicher, Flori Ava Star Weber

Resources

Here are some resources useful to understand simplified sliding sync:

• MSC4186: Simplified Sliding Sync
• A talk about simplified sliding sync
• MSC3575: Sliding Sync (aka Sync v3)

These are Synapse code-parts implementing Sync v4:

• Session Storage
• Session tables: migrations 87 to 93
• Handler
• Types

These are the MSCs defining extensions:

• ToDevice
• Account Data.
• Receipts.
• E2EE.

Acceptance Criteria

This ADR must provide an answer to the following problems:

• How events flow from the room server to the client (via the client server)
• How wakeups are handled for long polling requests
• How connections/sessions are stored/resumed
• How to handle the “sync config” somewhat efficiently
• How this relates to legacy sync and what aspects can be shared

Solution

Storage

In Sync v4, a session is tied to its (user, device, connection id) triplet.

Here is an overview of all the various database tables involved:

erDiagram
  direction LR

  SYNCV4_CONNECTIONS {
    BIGINT nid PK

    BIGINT device_nid FK

    STRING connection_id

    TIMESTAMP last_used_at
  }

  SYNCV4_POSITIONS {
    BIGINT nid PK

    BIGINT connection_nid FK

    BIGINT prev_position_nid FK
  }

  SYNCV4_ROOM_CONFIGS {
    BIGINT nid PK

    BIGINT position_nid FK
    BIGINT room_nid FK

    BIGINT required_state_nid FK

    INT timeline_limit
  }

  SYNCV4_REQUIRED_STATES {
    BIGINT nid PK

    BIGINT connection_nid FK

    JSON required_states
  }

  SYNCV4_ROOM_STREAMS {
    BIGINT nid PK
    BIGINT position_nid FK
    BIGINT room_nid FK

    STREAM_TYPE stream_type

    BIGINT up_to_stream_id
  }

  SYNCV4_CONNECTIONS ||--|{ SYNCV4_POSITIONS: "stores"
  SYNCV4_ROOM_CONFIGS }o--|| SYNCV4_POSITIONS: "full per-room accumulated config"
  SYNCV4_REQUIRED_STATES }o--|| SYNCV4_CONNECTIONS: "deduped per connection"
  SYNCV4_ROOM_STREAMS ||--|{ SYNCV4_POSITIONS: "stores"
  SYNCV4_ROOM_CONFIGS }o--|| SYNCV4_REQUIRED_STATES: "points at deduped list"

Tip

Required states are stored separately so that they can be de-duplicated (i.e. multiple room configs pointing to the same required states table entry).

Note

For the syncv4_rooms_streams, the various streams are:

• events: The highest events.stream_id that has already been sent back to the client.
• receipts: The highest read_receipts.stream_id that has already been sent back to the client.
• todevice: The highest todevice_messages.stream_id that has already been sent back to the client.
• user_keys: The highest user_keys.stream_id that has already been sent back to the client.
• room_account_data: The highest room_account_data.stream_id that has already been sent back to the client.
• global_account_data: The highest global_account_data.stream_id that has already been sent back to the client.

CREATE TABLE syncv4_connections (
  nid BIGINT GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY,

  user_nid int NOT NULL REFERENCES users(nid),
  device_nid int NOT NULL REFERENCES devices(nid),
  -- Connection ID set by the client, can be null.
  conn_id TEXT,

  last_used_at timestamp with time zone NOT NULL DEFAULT now(),

  UNIQUE NULLS NOT DISTINCT(user_id, device_nid, conn_id)
);

CREATE TABLE syncv4_positions (
  nid bigint GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY,

  -- Link to Sync v4 connections/sessions
  connection_nid bigint NOT NULL REFERENCES syncv4_connections(nid) ON DELETE SET NULL,
);

-- The various streams we have in our system
CREATE TYPE stream_type AS ENUM (
  'events',
  'receipts',
  'todevice',
  'user_keys',
  'room_account_data',
  'global_account_data'
);

CREATE TABLE syncv4_rooms_streams (
  position_nid bigint NOT NULL REFERENCES syncv4_positions(nid) ON DELETE CASCADE,

  -- Name of the stream this applies to
  stream_type STREAM_TYPE NOT NULL,

  -- Up to where we have already sent down this stream
  up_to_stream_id bigint,
);

CREATE TABLE syncv4_required_states (
  nid bigint GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY,

  connection_nid bigint NOT NULL REFERENCES syncv4_connections(nid) ON DELETE CASCADE,

  -- The list of `(event_type, state_key)` tuples that were already sent
  required_states JSON[] NOT NULL, -- FIXME: JSON array, or just JSON object ?
);

At the highest level, the general course of actions that happens for every sync request is:

1. Figure out where we are in the session.
2. We get the server’s current stream IDs, this will be the response’s pos.
3. Identify which rooms are filtered in by the request’s config.
4. Check if there is data eligible between the stored room streams and the current server’s streams.
   1. If not, enter long-polling and do notified sync.
5. If yes, generate the response, and advance the individual rooms streams.
6. Answer, setting the pos to the current server’s stream.

The rest of this document should help understanding each of these steps.

Handling Sessions and Positions

Clients are supposed to start by sending a first request without any pos set, and then to, in each request, set the pos to whatever was the last response’s value, like so:

sequenceDiagram

  participant Client
  participant Server

  Note over Client,Server: Initial request should have an empty pos
  Client->>Server: Request 1: pos=None
  Server->>Client: Response: pos=1

  Note over Client,Server: Second request should use the last response's pos
  Client->>Server: Request 2: pos=1
  Server->>Client: Response: pos=2

When receiving a new request, we can construct the (User ID, Device ID, Connection ID) tuple and check in our database to tell in which situation we are:

• Not present: We now need to look at the request’s pos.
  • pos is empty: The user is starting a session, we simply handle that request, and store its state.
  • pos is not empty: We send back a 400 - M_UNKNOWN_POS to the user so that they start a new session with an empty pos.
• Present: We now check which pos we got, based on our stored ones.
  • Last response’s one: We can now assume the client correctly received the previous response. We need to combine the room configs and respond with the response of this combined request. After having answered, we store a new entry for it.
  • Penultimate response’s one: It’s either a duplication, or a cancelled+resent request. We answer to the new request, and we store the new response’s one (See forking).
  • Any other value: It is invalid, we send back a 400 - M_UNKNOWN_POS.

Here is the same decision tree as a diagram:

stateDiagram-v2

checkConn: Check for an entry for this session in our storage
checkPosConn: Check if the **pos** field of the request is set
checkPosNoConn: Check if the **pos** field of the request is set

newSession: New session
resetSession: Session reset
comparePos: Compare with the **pos** we have in storage

redoRequest: Retried position
nextRequest: Next linear position
invalidPos: Answer with <br/>400 - M_UNKNOWN_POS

%% Tree root
[*] --> checkConn: We receive a request

%% First level
checkConn --> checkPosNoConn: No
checkConn --> checkPosConn: Yes

%% Second level - Left
checkPosNoConn --> newSession: No
checkPosNoConn --> invalidPos: Yes

%% Second level - Right
checkPosConn --> resetSession: No
checkPosConn --> comparePos: Yes

%% Second level Right - Sublevel
comparePos --> nextRequest: Equal to last response's new one
comparePos --> redoRequest: Equal to penultimate response's one
comparePos --> invalidPos: Equal to any other value

%% Links to output
newSession --> [*]: New connection with a first position
invalidPos --> [*]
resetSession --> [*]: Delete all positions in session and store first one
nextRequest --> [*]: Store new position, delete penultimate one
redoRequest --> [*]: New forked position

Handling Retries (Aka Forking)

Here is a scenario that will happen, and hence that we have to be able to handle:

1. Client sends the sync request with pos=2 -> We answer it and store the one with next_pos=3
2. Client re-sends again the sync request with pos=2 -> We answer it and store the one with next_pos=3'
3. We receive either pos=3 or pos=3' -> We answer it, delete the now irrelevant one, and store the one with next_pos=4

Note

The solution we come up with needs to work even if the two pos=2 requests hit different sync server instances (i.e. multi-tenancy support).

This is why the syncv4_positions table is shaped this way:

erDiagram
  SYNCV4_POSITIONS {
    BIGINT pos PK
    BIGINT connection_nid FK
    BIGINT prev_position FK
  }

Or in SQL:

CREATE TABLE syncv4_positions (
  -- Auto-incremented unique ID. Part of the `pos` token.
  pos BIGINT NOT NULL AUTO_INCREMENT,

  -- Link to the connection this response belongs to.
  connection_nid BIGINT NOT NULL REFERENCES syncv4_connections(nid),

  -- Self-referencing link to the previous position.
  prev_position BIGINT REFERENCES syncv4_positions(pos) ON DELETE CASCADE
);

And now, how the above scenario plays:

sequenceDiagram
    participant C as Client
    participant CS1 as Client API #35;1
    participant CS2 as Client API #35;2
    participant DB as Database

    C->>CS1: sync (pos=2)
    CS1->>DB: SELECT <CONNECTION_STATE> WHERE syncv4_positions.pos = 2
    Note over DB: DELETE syncv4_positions.prev_position deletes pos=1 here
    DB-->>CS1: per-connection state
    CS1->>CS1: diff vs live room state
    CS1->>DB: INSERT new position (copy data from pos=2)
    DB-->>CS1: pos=3
    CS1-->>C: next_pos=3

    Note over C,DB: Response lost in transit -> Client redoes request with pos=2
    C->>CS2: sync (pos=2) [retry]
    CS2->>DB: SELECT <CONNECTION_STATE> WHERE syncv4_positions.pos = 2
    Note over DB: DELETE syncv4_positions.prev_position deletes nothing this time.
    DB-->>CS2: same per-connection state
    CS2->>CS2: diff vs live room state
    CS2->>DB: INSERT new position (copy data from pos=2)
    DB-->>CS2: pos=3'
    CS2-->>C: next_pos=3'

    NOTE over C,DB: There are now 3 states stored: pos=2, pos=3 and pos=3'

    C->>CS1: sync (pos=3) [client decides to go with pos=3]
    CS1->>DB: SELECT <CONNECTION_STATE> WHERE syncv4_positions.pos = 3

    Note over DB: DELETE syncv4_positions.prev_position deletes pos=2 and pos=3' thanks to cascading.

    DB-->>CS1: per-connection state
    CS1->>CS1: diff vs live room state
    CS1->>DB: INSERT new position (copy data from pos=3)
    DB-->>CS1: pos=4
    CS1-->>C: next_pos=4

Filtering & Activity Ordering

Within sliding sync, a few per-room values are needed to either compute responses, or satisfy filters. These values would very quickly become too costly to compute on each sync request and as such should be stored in pre-populated tables. This section describes such tables, the values they hold, and what is their usage during synchronization.

• syncv4_rooms_info: Stores whether rooms are encrypted or not, and which space they are part of. This is used to satisfy the is_encrypted and spaces filters.

  SQL declaration

  CREATE TABLE syncv4_rooms_info (
    -- The room this entry belongs to.
    room_nid int NOT NULL REFERENCES rooms.nid,
  
    -- Whether this room has a `m.room.encryption` event.
    is_encrypted bool NOT NULL DEFAULT FALSE,
  
    -- The list of **joined** space rooms that contain a
    -- `m.space.child` state event referring to this room.
    spaces TEXT[]
  );
  

• syncv4_max_rooms_streams: This table stores two variables (per room):

  • The highest room event’s stream ID. This is used to compute rooms activity ordering.
  • The highest room bump event’s stream ID. This is used to compute responses bump_stamp field.
  SQL declaration

  CREATE TABLE syncv4_max_room_streams (
    -- The room this entry belongs to.
    room_nid int NOT NULL REFERENCES rooms.nid,
  
    -- Stream ID of the latest event of the room.
    max_event_stream_id bigint NOT NULL,
  
    -- Stream ID of the latest **bump** state event of the room.
    max_bump_event_stream_id bigint NOT NULL,
  );
  

  Note

  When constructing responses, we should sort rooms by their highest bump event stream ID and convert these to their sorted indexes (instead of leaking the actual stream_id of events). As an example, this list of room’s max_bump_event_stream_id: [12, 4, 8, 14, 33] is sorted as [33, 14, 12, 8, 4] (most recent activity first) and then the response’s rooms will get their bump_stamp respectively set to 5, 4, 3, 2 and 1.

Tracking Rooms States

This section will go into details explaining how table syncv4_room_streams works. Here is its declaration:

CREATE TABLE syncv4_room_streams (
  -- Link to the room
  room_nid NOT NULL REFERENCES rooms(nid),

  -- The stream this applies to (e.g. `events` or `receipts`, etc)
  stream_type STREAM_TYPE NOT NULL,

  -- Either `PREVIOUSLY` or `LIVE`
  room_status ROOM_STATUS NOT NULL,

  -- The stream ID up to which this room was last synced. Set when `room_status = PREVIOUSLY`.
  stream_id BIGINT
)

A room status of LIVE means it is up to date with the given sync-token event_stream_position, and a room status of PREVIOUSLY(stream_id) means it was synced up to the stored stream_id.

Now, take the following three rooms, and their respective events. In our scenario, we will sync at events F and I (an initial sync occurred at event A before). The sync at event F will filter in rooms 1 and 2, and the sync at event F will filter in rooms 2 and 3.

flowchart LR
  %% Nodes colors
  classDef blackOutline stroke:#000000,stroke-width:2px
  classDef greenOutline stroke:#00f000,stroke-width:2px
  classDef blueOutline stroke:#0000ff,stroke-width:2px

  %% Room #1 events
  subgraph R1 ["**Room 1**"]
    B@{ shape: circle, class: blue }
    class B blueOutline

    G@{ shape: circle }
    class G blueOutline
  end

  %% Room #2 events
  subgraph R2 ["**Room 2**"]
    C@{ shape: circle }
    class C blueOutline

    D@{ shape: circle }
    class D blueOutline

    I@{ shape: dbl-circ, label: "**I**" }
    class I greenOutline
  end

  %% Room #3 events
  subgraph R3 ["**Room 3**"]
    A@{ shape: dbl-circ, label: "**A**" }
    class A blackOutline

    E@{ shape: circle }
    class E greenOutline

    F@{ shape: circle }
    class F greenOutline

    H@{ shape: dbl-circ, label: "**H**" }
    class H greenOutline
  end

  A --> B
  B --> C
  C --> D
  D --> E
  E --> F
  F --> G
  G --> H
  H --> I

Here is what would be the respective entries for each room in the syncv4_room_streams (for the ‘events’ stream_type):

Tip

Notice how, if we sync again at event I, we won’t need to write anything to the database, as each room is still LIVE. This is intended as to make “syncing nothing” as cheap as possible.

And for each sync request what happens:

• @H (rooms 1 and 2): For each room, the server’s current stream ID (which is H) is higher than the room’s current highest stream ID (respectively G and D). We therefore return the latest timeline_limit events (down to at most pos’s value which is A here) in each room to be responded with (so events [B, C, D, G]), and set the room as being live with the response’s next-pos (i.e. here H). We also store that room 3 has fallen behind from LIVE to PREVIOUSLY(A).
• @I (rooms 2 and 3): For each room, the server’s current stream ID (which is I) is higher or equal than the room’s current highest stream ID (respectively I and H). We therefore return the latest timeline_limit events (down to at most pos’s value which is H here) in each room to be responded with (so events [E, F, H, I]) and set the rooms as being live with the response’s next-pos (i.e. here I).

Sync Token and pos

Sync v4 still requires us to have a sync_token in order to a) handle extensions, and b) tell which rooms are eligible to be in a response. Additionally, we also need to get the new pos value that will be used to tell what is the current connection state and what has already been sent through it.

1. Wrapping: We wrap the existing sync token with the new connection position token like so: ss_<SYNC_TOKEN>_<POSITION_ID>. Which in turn gives: ss_s_<EVENT_POS>_<TODEVICE_POS>_<ACCOUNT_DATA_POS>_<USER_KEYS_POS>_<READ_RECEIPTS_POS>_<POSITION_ID>

2. Prefixing: We prefix the existing sync token with the connection position ID (this is what Synapse chose btw) like so: <POSITION_ID>/<SYNC_TOKEN>. Which in turn gives: <POSITION_ID>/s_<EVENT_POS>_<TODEVICE_POS>_<ACCOUNT_DATA_POS>_<USER_KEYS_POS>_<READ_RECEIPTS_POS>

Notified Sync

Similarly to normal sync, sliding sync must support long-polling requests. As a reminder: these consist in clients specifying a timeout in their requests, and us having to watch if we have new data for them for as long as this timeout lasts before giving up.

In order to do so, a first initial bad idea would be to be busy looping on the database before finally, sometimes, get a little data to send back. Here is instead an hopefully better design for it.

First, each of the handler that produces synchronizable data will also publish NATS notification payloads related to newly written data, and on their dedicated subjects:

flowchart LR
    %% Services
    RS[Roomserver]
    acc[Account Data Handlers]
    e2ee[E2EE Keys Handlers]
    receipts[Receipts Handlers]
    ToDev[ToDevice Messages Handler]

    %% Actors
    SYNC[Sync Endpoint]
    Client

    RS -- events.<ROOM_ID> --> SYNC
    acc -- account.<USER_ID> --> SYNC
    e2ee -- e2ee_keys.<USER_ID>.<TYPE> --> SYNC
    receipts -- receipts.<ROOM_ID> --> SYNC
    ToDev -- todevice.<USER_ID> --> SYNC

    Client -- GET /sync long poll --> SYNC
    SYNC -- response --> Client

Tip

I suggest that we change the existing federation output topic to the new roomserver one, so that a single NATS message can be used to both federate and sync new events.

These notifications need to contain just enough data for the sync endpoint to be able to determine, based on the evaluated sync configuration, if the new data would be returned by the ongoing request:

Caution

FIXME: Do we need to self-contain messages here ? I do not see the benefit of doing so, and I see the downside of increased payload sizes, but maybe some of y’all have other opinion on this.

enum NotificationSubjectType {
    Events,
    AccountData,
    E2EEKeys,
    Receipts,
    ToDeviceMessage,
}

enum E2eeActionType {
    OneTimeKeysCount,
    DeviceLists,
    UnusedFallbackKeysTypes,
}

/// (de)serializes as `<user_id>.<e2ee_action>`
struct E2eeKeysSubject {
    user_id: OwnedUserId,
    e2ee_action: E2eeActionType,
}

enum Subject {
    Events(OwnedRoomId),
    AccountData(OwnedUserId),
    E2eeKeys(E2eeKeysSubject),
    Receipts(OwnedRoomId),
    ToDevice(OwnedUserId),
}

struct NotificationSubject {
    notification_type: NotificationSubjectType,
    subject: Subject,
}

enum E2eeNotificationContent {
    OneTimeKeysCount(usize),
    DeviceLists {
        changed: Vec<OwnedUserId>,
        left: Vec<OwnedUserId>,
    },
    UnusedFallbackKeysTypes(Vec<OneTimeKeyAlgorithm>),
}

enum NotificationContent {
    Events(StreamId),
    AccountData {
        stream_id: StreamId,
        room_id: Option<OwnedRoomId>,
    },
    E2eeKeys(E2eeNotificationContent),
    Receipts(StreamId),
    ToDevice(StreamId),
}

Then, ongoing long-polling sync endpoints would just fetch, from the database, the result for their sync request, knowing they should now find some eligible data thanks to the notification they saw pass by.

Note

The only change we would need to do with this plan is that we need to store a single stream ID per user to track changes to their e2ee keys. This stream ID however would not need to use the complicated double-lock stream ID mechanism, and instead a simple SELECT FOR UPDATE query would give us what we need. Note that once implemented and proven to work, we should be able to extend this mechanism to all the other user-scoped data (i.e. todevice messages and account data).

Further Work

These are stuff to go look back at once we are done implementing the above solution. They contribute to us having a final clean version over a proof of concept one.

Optimisations

The proposed solution here is heavily based on Synapse implementation of Sync v4. However, I stripped many optimisation tricks that they did, but that we might want to consider in the future:

• Decide if want to delegate populating syncv4_joined_rooms to a background job instead of it being updated (and hence awaited) in actual handlers.
• Other pre-populated table: sliding_sync_connection_lazy_members: This table would contain the membership status of every user in regards to every room. This would greatly decrease the cost of computing lazy membership as well as shared encrypted rooms for the E2EE keys changes extension.
• Add a configurable UPDATE_LAST_USED_INTERVAL that prevents syncv4_connections.last_used_at column to be updated on every single sync request.
• Introduce some cache mechanisms

Expired Sessions Cleanup

The above session storage could obviously clutter our server’s database, especially since there is no way to tell if a session is over (i.e. the client will start a new one in that connection the next time they sync) or not (i.e. the client will just use the latest pos we gave them in their next sync request). This is why we might decide later on to run a scheduled task, or to implement an admin feature to remove sessions based on relevant criteria (e.g. last_used_at > 7 days, etc). The good news is that these deletion simply need to happen on the syncv4_connections table, and will trickle down deleting all the other entries that are linked to these sessions.

Tip

We should also configure a UPDATE_LAST_USED_INTERVAL on our server that determines how often we’ll update syncv4_connections.last_used_at to avoid updating on every single sync request.

Rejecting Too-Big Requests

The Security considerations section of the MSC states:

Servers MAY decide to expire the sync connection if the generated response on an incremental request is likely very large or expensive to compute.

We could therefore define an algorithm and a bunch of metrics that will make us detect that a request’s configuration will result in too-big of a response and we therefore decide not to fulfill it but instead expire its associated session (by simply removing the session from our storage).

I however think this is to be determined after having fully implemented Sync v4, so that we can benchmark relevant such metrics. This also need to be fully configurable so that administrators can tune them, and so that we can provide different relevant defaults for different deployments (Cloud, Raspberry Pi, single server, etc).

We should nonetheless define a few defaults for these limits, just to prevent nuisance:

• MAX_TIMELINE_LIMIT: 100
• MAX_ROOMS: 50

Tip

We should consider adding a list of metrics to be computable from the get go as part of the implementation’s acceptance criteria, so that we can later on reflect on this problem.

Implementation TODOs

This is just a small list of stuff that I have noticed whilst reading the MSCs, and that we should not forget to do when actually implementing Sync v4.

• Limit the number of connections per user’s device

  As mentioned in the MSC’s Security considerations section:

  Servers SHOULD limit the amount of data that they store per-user to guard against resource consumption, e.g. limiting the number of connections a device can have active. This protects against malicious clients creating large numbers of connections that get persisted to the database.

  This means we should first get the number of known sessions for each (user_id, device_id, conn_id) before creating and storing a new one when being requested to. Note that we can always decide on a policy for what happens when a device reached the maximum number of connections (e.g. delete the oldest one).

• URL to use for Sync v4

  The unstable URL for Simplified Sliding Sync is /_matrix/client/unstable/org.matrix.simplified_msc3575/sync. Nothing special to do about this, but writing it here will avoid having to look it up time and time again.

• Version flag for Sync v4

  The flag to set in /_matrix/client/versions to indicate that we support Sync v4 is org.matrix.simplified_msc3575.

• Expired sessions response

  The spec specifies that we have to answer with a 400 - Bad Request HTTP status code and a M_UNKNOWN_POS Matrix error code when sessions are expired. See Expired sessions cleanup for when this can happen.

Matrix Specification Deviations

Arminius targets compliance with the Matrix specification. This section records intentional differences between arminius behavior and normative spec requirements.

Each entry describes the deviation, links to the spec where possible, and explains why arminius behaves differently.

Deviations

• GET /members additionally accepts sync tokens

GET /members Additionally Accepts Sync Tokens

Spec Requirement

Client-Server § GET /_matrix/client/v3/rooms/{roomId}/members defines the optional at query parameter as a pagination token for the point in time whose membership state should be returned. The spec says this token comes from a room’s prev_batch field in a /sync response.

Arminius represents pagination tokens as MessagesToken values (m_<event_id>), as described in the message pagination tokens ADR. That format matches the spec’s intent, which is to be an opaque token tied to a position in the room timeline.

Deviation

Arminius also accepts SyncToken values (s_<event_stream_position>_…, the next_batch format from /sync) when resolving at. If the parameter is a sync token, arminius maps its event stream position to the latest room event at that position and uses that event as the snapshot point.

Strictly speaking, this is not what the spec describes. Unlike GET /messages, which explicitly allows both prev_batch and next_batch, /members does not mention next_batch at all.

Why

Element Web and Element Android pass sync tokens (next_batch) as at, not room prev_batch tokens. Without accepting them, /members requests from Element fail and encrypted rooms break when the client needs historical membership for encryption setup.

This behavior is tracked in arminius#1765 and “discussed” upstream in matrix-spec#1302, where it remains unclear whether the spec text or the Element clients are wrong.

We accept sync tokens in addition to the spec-mandated behavior as a pragmatic compatibility measure until the spec clarifies the parameter or Element changes its behavior.

Implementation

Implemented in resolve_at_token in arminius-server-rooms (get_member_events.rs) (Related Pull Request).

Frequently Asked Questions

Why Does Arminius Exist?

Arminius is developed for the requirements of BundesMessenger and enterprise Matrix deployments: maintainable protocol code, integration with existing identity infrastructure, predictable operations, and explicit service boundaries. It also provides a foundation for integrating MLS and pseudonymous identities.

Why Build Another Matrix Homeserver in Rust?

An independent implementation gives the project control over its architecture and how new protocol capabilities fit into it. Rust’s ownership and type system help prevent classes of memory and concurrency errors, while traits support clear boundaries between protocol logic and infrastructure.

The value of Arminius depends on how well it meets these requirements. Compatibility, performance, and operational reliability must be demonstrated through tests and deployment experience. Relation to other great existing Homeservers:

• Conduit and Continuwuity - great for small appliances, hard to scale for HA setups
• Tuwunel - even greater for small appliances, hard to scale for HA setups
• Dendrite - cool thing, architecture was mostly the blueprint for Arminius

Can I Run Arminius Locally with Reasonable Effort?

We provide also a docker-compose setup but the most supported local deployment is based on Nix derivations. Yes, being new to Nix makes this hard and somehow alien but it’s really worth the effort and let’s you run full setup with only a few commands and also makes it possible to do most of the CI Pipeline locally in a reproducible way. Oh, did we mention that Nix makes everything reproducible? See this for running it with Nix

Is Arminius Ready for Production?

Arminius is under active development and also the necessary Helmchart for cloud-native Provisioning and Operation of Arminius is under active Development. We’re planning for the first productive release in Summer 2027.

Is Arminius Feature-Complete?

Complete Matrix specification coverage is not claimed. The current blacklist of Complement Tests gives a good pointer about the already supported features:

• https://gitlab.com/bundesmessenger/backend/neo/arminius/arminius/-/tree/main/nix/packages/complement_utils

Is It Possible to Migrate from Other Homeservers to Arminius

While most homeserver implementations doesn’t plan or cover migration from other homeserver implementations, we will provide this functionality. We also attempt to provide a standardized migration format that could be used for generic migrations between Homeservers. See this ADR for more information: Homeserver Migration

Can I Use Existing Matrix Clients, Bridges, and Bots?

We are targeting especially compatibility with existing Matrix Clients. Our priority for client compatibility are right now Element-derived Client-Apps like the BundesMessenger.

The Arminius Application Service engine will be called Extension API and will strictly following the existing Matrix Spec Application Service API. Like with Synapse an Extension can be created in any programming language, as long it uses the correct API Specs. Therefore existing Application Services should also work with Arminius.

How Is Compatibility Tested?

Arminius exercises protocol behavior with the Matrix Complement test suite. Additionally we’re doing a extensive set of own E2E Tests using the Matrix-Rust-SDK (also with Federation between different Homeservers (Arminius and Synapse). See the tests section in the docs for more detailed information about the tests and how to run them.

Which Infrastructure Does Arminius Use and Will It Support Kubernetes?

Additional Components

The current design uses PostgreSQL for persistent data and NATS JetStream for internal messaging. The architecture guide describes service boundaries of each component.

Cloud-Native Design (Kubernetes)

The Arminius components are developed for an cloud-native Kubernetes environment with a Helm Chart. We will provide all necessary configuration and components for running Arminius in a Kubernetes Cluster. Albeit we won’t provide productive deployments for the additional components NATS and PostgreSQL. The included provisioning for these components will only cover testing purposes.

What Means Support of Enterprise Authentication?

The architecture aims to keep authentication separate from Matrix protocol processing and message storage, allowing integration with enterprise identity infrastructure. Consult the configuration guide for the supported authentication service, client configuration, and provisioning requirements. This is still work in progress

Can I Migrate from Another Homeserver?

Use only an explicitly documented migration path in the migration guide. Matrix protocol compatibility does not imply database compatibility. Reusing another homeserver’s database requires a migration procedure that covers its schema and data.

Where Are Upgrades, Backups, and Monitoring Documented?

The documentation for this is still work-in-progress. We already provide documentation for the incorporation and usage of OpenTelemetry Traces:

• Telemetry Tracing Docs

Is Arminius Certified for Handling Classified Information?

No. Arminius is regarded now as an ordinary Matrix Homeserver that is capable of providing the same security clearance as other Homeserver Implementations. We are targeting with specific MLS-driven Client-SDK and server-side components for a certification for VS-NfD.

Style Guide

This document describes the style guide used for technical writing in the Arminius project.

Tone and Style

Language

• Write all documentation in English.
• Use simple, clear language.
• Replace idioms with plain phrasing.
• Keep humour and cultural references light and easy to ignore.
• Prioritise clarity over elegance.
• Use short, direct sentences.
• Spell and format consistently—use British English spelling (“colour”, “authorise”, “behaviour”).

Voice and Tense

• Use active voice: “The API returns an error” instead of “An error is returned.”
• Use present tense: describe the system’s current state, not what it “will” or “should” do.

Precision and Brevity

• Include only what the reader needs to complete a task or understand a concept.
• Be specific: “The process runs every 10 minutes” instead of “The process runs regularly.”
• Avoid filler phrases like “It should be noted that…” or “Basically…”.

Consistency and Neutrality

• Use consistent terminology across all documents. If a term exists in the glossary, use it exactly.
• Maintain a neutral, professional tone.
• Avoid first-person pronouns (I, we, ours).
• Address the reader in the second person (you, your).
• Use singular they as a gender-neutral pronoun.
• Use full words instead of abbreviations unless the abbreviation is widely recognised. Spell out abbreviations on their first use in a document.

Structure and Format

General Layout

• Use mdBook flavoured markdown.
• Start each document with a one-line summary of its contents. Example: “This document describes the deployment and configuration of the user service.”
• Keep paragraphs short—3 to 5 sentences each.
• Use bulleted lists for unordered information and numbered lists for sequential steps.
• Avoid nesting lists more than one level deep.

Prose Linting

Vale checks the Markdown under docs/ against the Vale and Google style packages. A pre-commit hook and CI both run it.

• Run vale docs in the devshell to see the same results locally. Editors get them through the vale-ls language server, which the devshell also provides. Leave its configPath unset, or it loses the project styles.
• Vale flags every word its dictionary lacks. Correct the prose when the word is a genuine misspelling.
• Add a term Vale should accept to nix/vale/words.txt, or to nix/vale/names.txt when only one capitalisation is correct. Both files describe their own format.

Vale also checks the comments in Rust files, parsed as Markdown like rustdoc does. Rules for Rust are enabled one at a time in the [*.rs] section of nix/vale/vale.ini.

• Run vale --glob='*.rs' crates in the devshell to check them locally.
• Put code and identifiers in backticks, and URLs in angle brackets (<https://…>), so Vale skips them.

Glossary

Common terms we use to make sure we use the same language

admin-component

TODO

Appservices

An interface of the homeserver to third-party applications that can extend functionality in many ways. Currently mostly used for bridging to other protocols.

See Application Services API in the Matrix spec.

Component

TODO

Database

TODO

Extensions

An Arminius-specific concept of extended appservices. Extensions are planned to have more control over the homeserver and potentially be consulted synchronously when processing certain events or requests. While it is based mostly on the appservice API, it serves more the purpose of modules from other homeservers.

Homeserver

A server, that multiple users can use to send messages (over Matrix currently) to other homeservers.

IdP

“Identity Provider”

A service consulted during authentication via OIDC. It usually serves as the source of users and potentially their displaynames and other profile data.

NATS

TODO

nehwgeneration, ngen

nehwgeneration is a custom-built OpenAPI 3.1 to Rust Axum Server code generator. “ngen” is a shortened form of the term “nehwgeneration”.

OIDC

Basically the one standard for authentication nowadays.

For more information you can read the standard or the readable explanations of it.

SCIM

“System for Cross-domain Identity Management”

Basically our way of syncing changes to user data without requiring the user to do a new OIDC login.

For more information you can read the standard or the readable explanations of it.

Server

TODO

Service

TODO