Federation Output Component

The federation output component implements the outbound parts of Matrix’s federation algorithm defined in the Server-Server API. It is responsible for delivering events or messages to remote homeservers and handling their responses. In the context of this project, the component serves as the outbound gateway for cross-server communication.

Scope

The federation output component is responsible for federating room events (PDUs and EDUs) to other homeservers. Additionally, other components, such as the federation input component, can request the output component to fetch data from remote homeservers.

Note

In summary, the responsibilities of the federation output component are:

• pushing events from federated rooms to other homeservers, and
• requesting data from other homeservers when required by other components.

Message Delivery

Warning

Message delivery from other components to the federation output component MUST be handled via a message broker.

In the current implementation, NATS is used. However, the underlying structs and traits are generic enough to support any other message broker.

By using JetStream instead of core NATS, we ensure durable message persistence, retry support, and message ordering within a stream.

A single JetStream stream named federation-output-stream is configured to capture all subjects using a wildcard: federation.output.*.

Prioritization is achieved using two worker pools: a transaction worker pool and a miscellaneous (misc) worker pool. More details are provided in the Message Processing section.

Assumptions about the Message Broker

The only assumptions made about the message broker are:

• messages not acknowledged within a specific period of time are redelivered, and
• messages explicitly not acknowledged are also redelivered.

These assumptions are not strict requirements—other brokers can be used—but this may weaken the guarantees provided to the server.

Message Processing

Messages are processed in an asynchronous event loop. After receiving a message, the bundler groups it into a task.

• Transaction tasks: multiple messages are bundled into one task.
• Misc tasks: a single message forms the task content.

This abstraction allows the misc and transaction task managers to remain generic over the message broker’s implementation details.

While the bundler fetches events from the broker, HTTP workers continuously request new tasks from the bundler.

The HTTP workers are responsible for performing the actual HTTP requests. They report task results back to the bundler, which acknowledges all messages associated with the task.

To prevent starvation of misc requests, two worker pools are maintained:

• The misc worker pool prioritizes misc tasks.
• The transaction worker pool prioritizes transaction tasks.

The number of workers in each pool is configurable via environment variables, allowing adaptation to different use cases. If a pool has no available tasks, its workers will assist the other pool to avoid wasted resources.

The current design of the message processing pipeline is illustrated in the following diagram:

Communication between the Bundler and HTTP Workers

In the current implementation, communication between the bundler and workers is coordinated through shared mutable state. Access is synchronized via a lock, ensuring requests to the pool are queued in FIFO order.

Further implementation details can be found in the Federation output crate.

Task and Message Lifecycle

The lifecycle of messages can be described by the following diagram:

1. Another component sends a request to the federation output component via the message broker.
2. The bundler receives the message and either creates a new task or associates it with an existing task.
3. An HTTP worker fetches a new task from the task manager.
4. The HTTP worker sends the request to the remote homeserver.
5. The HTTP worker reports the outcome to the manager.
6. The manager acknowledges or does not acknowledge the messages associated with the task.
   • Acknowledged → message is considered delivered.
   • Not acknowledged → message broker redelivers the message to retry the task.
7. (Optional) The outcome of the HTTP request is reported back to the requesting component.

Guarantees

The guarantees of this component depend on the underlying message broker infrastructure. Each guarantee is justified with its dependencies:

• Transaction prioritization: Transactions are prioritized by size.
• Restart safety: The component maintains internal transaction pools.
  • On restart, these pools are lost.
  • If the broker redelivers unacknowledged messages, tasks are reconstructed.
  • If the broker lacks redelivery, this guarantee is lost.
• Exclusivity: For Matrix transactions, one active transaction per homeserver marks the server as occupied. Two transactions cannot be sent simultaneously to the same server.
• Infallibility: Task processing is fail-safe. Errors are logged, messages are not acknowledged, and unaffected tasks continue execution.
• No starvation of misc tasks: Dedicated pools ensure misc tasks receive processing power. Prioritization can be tuned via pool size configuration.

What the Component Does Not Guarantee

• In-order processing: New messages may be processed later if existing tasks have higher priority.