Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

Architecture

This document outlines the high-level architecture of Arminius and provides some context on how that architecture was reached. In the future some of these decisions may be revisited.

Context

The architecture in Arminius follows a few core principles:

  • To provide potentially infinite scalability, there should not be any central process. Instead everything should be potentially redundant and be able to fail over.
  • There are external interfaces: Client traffic, federation traffic, administration and application services/extensions. The architecture has to route data between those interfaces with various constraints. Ideally reverse proxy configurations just need to route prefixes, not specific endpoints, to those components.

To simplify architectural decisions there are some assumptions:

  • A single database can scale far enough: This has mostly worked out on the largest homeserver currently in existence and it avoids synchronising separate sources of truth like in a more decentralised microservice system. Currently this database is PostgreSQL.
  • All internal traffic can be done over an abstract messaging fabric with potential database lookups. Currently this fabric is NATS.

This leads us to the following basic set of components:

  • A client service to deal with requests from clients, (potentially) application services, and (potentially) a more powerful extension system based on the latter. The application service API is largely based on the client API with only a few additions. Usually the client service only receives requests from clients. Potentially the best way to shard the client service is based on the user ID or access token.
  • A federation output service to send queued up transactions as well as queries to other homeservers. There is a shared retry and backoff logic for external services, so this may potentially shard based on the destination. This is, apart from appservice transactions, the only real outbound connection.
  • A federation input service to deal with incoming requests from other homeservers. Rate limits are usually applied based on the incoming service, so it may be beneficial to shard based on the incoming server name or IP.
  • To bind those services together and because rooms are the central decentralisation mechanism in Matrix, there is a room service. Room data needs to be processed in order in some cases and the room service should generally be the single source of truth about a room. As a result sharding based on the rooms may be the best solution.
  • Then there is another API for administration. To ensure availability and to potentially restrict access to only the internal network, the admin service is a separate component. It also potentially deals with cross-cutting concerns.
  • Another component is reserved for authentication. On the one hand we need to sync users to the homeserver database, which is done by the SCIM service. On the other hand we also need to deal with OIDC authentication. It is good practice to separate this out into a separate domain/url. As such we have 2 components to deal with identity and authentication related topics. A SCIM service for syncing and the intermediate IdP-like service NUSS to connect to an upstream IdP and add Matrix-specific OIDC features.

External to the homeserver we have clients, other homeservers, a potential admin frontend as well as application services/extensions. Traffic between our homeserver and such components may route through the public internet and may include forward or reverse proxies in between.

High-Level Architecture

The following diagram shows the high-level architecture of Arminius. Some flows have been simplified to make the diagram more readable.

---
config:
  flowchart:
    curve: step
  theme: "redux-color"
  look: neo
  layout: elk
---

flowchart LR
  %% Define the various components of the system
  HS[External Homeservers]@{shape: processes}
  Clients[Clients]
  %% FIXME: this shape is only supported after mermaid 1.17 @{shape: browser}
  FedIn[Federation Input Component]@{shape: processes}
  FedOut[Federation Output Component]@{shape: processes}
  ROOM[Roomserver]@{shape: processes}
  ClientServ[Client Component]@{shape: processes}
  SCIM[SCIM Component]@{shape: processes}
  NUSS[NUSS/IDP]@{shape: processes}
  AppServ[Extensions/Appservices]@{shape: processes}
  Admin[Admin Component]@{shape: processes}
  AdminFE[Admin Frontend]
  %% FIXME: this shape is only supported after mermaid 1.17 @{shape: browser}
  DB[Database]@{shape: database}

  %% Dotted border class
  classDef dotted stroke-dasharray: 5 5
  class AdminFE dotted
  class AppServ dotted

  %% Declare links between components
  subgraph Internal components
    direction TB
    
    
    ClientServ <== NATS ==> ROOM
    ClientServ <==> DB

    ROOM <==> DB

    FedOut <===> DB

    FedIn <===> DB
    FedIn <== NATS ==> ROOM

    FedOut <== NATS ==> ROOM

    Admin <-...->|administration to all components| DB
    %% invisible link for putting admin and scim next to each other.
    Admin ~~~~ SCIM

    SCIM ====> DB
  end

  subgraph EE[External Entities]
    direction LR
    
    Clients
    AppServ
    AdminFE
    NUSS
    HS
  end

  %% Invisible link for alignment
  HS ~~~ FedOut 
  FedOut e1@---> HS
  HS e2@---> FedIn
  Clients --> ClientServ

  AppServ <-.-> ClientServ
  AdminFE -.-> Admin
  NUSS --> SCIM

  %% Animate certain links
  e1@{ animate: true }
  e2@{ animate: true }

Note

Dotted components (Admin Frontend and Extensions) are not yet implemented.

Arminius is structured as a distributed monolith. The services are split up to allow us to scale them independently, but we’re not interested in having services developed or deployed independently.

Therefore, all services also share a single database. Based on other homeservers in the ecosystem, we think that a single database will be able to scale sufficiently for our needs.

Communication between components is done via NATS.

Components

Detailed information on the individual components can be found in their dedicated pages.

Folder Structure

All rust code is contained in the crates/ directory. The crates/ directory is split up into the directories libs/, servers/, tests/ and tools/.

  • servers/ - contains our binaries, which run parts of the Arminius homeserver
  • libs/ - libraries used in our other crates
  • tests/ - crates that contain tests
  • tools/ - developer tooling (e.g. nehwgeneration)

We decided on this hierarchical folder structure, instead of a flat crates/ directory due to the high amount of crates in our project. All crates are prefixed with arminius in order to not conflict with external crates. The servers, libs and tests path is also included in crate names, in order to make the module path match the file path.

📚 Other Pages