Homeserver Configuration with Runtime API

We will have 2 primary ways of configuration. Primarily configuration should be specified in a config file. But we may need to change certain configuration at runtime. As such we define a general scheme on how to configure options in either place. Each config option should then document if it can be changed at runtime or not.

Levels of Configuration

• Built-in default (if there can be a default, otherwise startup error)
• default section of the configuration file: Values can be overwritten at runtime.
• Configuration file: Can not be overwritten at runtime!
• API+NATS KV: Runtime configuration/modification via the API

This allows both specifying defaults during deployment as well as locking down values from modification at runtime. Configuration is ordered. If a configuration value is specified at a higher level, it replaces the lower level. However, if something is specified outside the default section in the configuration file, it cannot be overwritten and attempts to modify the value are rejected at runtime.

Since the configuration is dynamic, processes need to be able to react to changes to the configuration. The processes get notified of changes via the NATS buckets. Handlers need to be able to react to such changes, but only at the start of a request. It is fine for a request to finish with the previous state of the configuration as long as the next request starts with the right configuration.

The replacement could be done based on just the toplevel config key or on a lower level, if we want to increase readability of the config file.

We will not support locking values to the server builtin default at the start. Any such values would need to be specified explicitly. But long term we may want to introduce an easier way to lock such configuration to their builtin values, if our configuration becomes long and complex.

The configuration file will be using yaml as this is the most familiar format to the expected users of the config file according to the discussions we had.

Built-in Defaults

Some values can be safely defaulted to a specific value without having to block startup. For example the list of blocked servers can easily be defaulted to an empty list without impacting functionality. Such defaults can always be overwritten at at least one higher level, otherwise they wouldn’t be configuration values.

default Section in the Config File

To ease kubernetes deployments, there is a default section in the config file. This section mirrors the normal configuration. The only difference is that this configuration can be overwritten at runtime (and is “indented” one level further because it is below the default key).

If this turns out to not be needed in the end and add too much support overhead, we might remove this section in the future to simplify our configuration handling. For now this seems like the easiest option to address all the requirements.

Configuration File

The configuration file is using yaml as the format. Its location can be specified via a command line parameter (--config-file/-c). Otherwise the location defaults to /etc/arminius/config.yaml. Each process individually reads this file and there is currently no support for recognizing diverging configuration between processes (this hasn’t been an issue with other applications so far either).

Secrets would be problematic if they were specified in the configuration file directly as it makes leaking secrets by accident more likely. Instead such values require configuring a path to the secret in the config file and the actual secret is stored in the file at that path. Those files are treated as binary files and read as is without any postprocessing. Relative paths are assumed to be relative to the config file. Paths are assumed to use Unix file separators, but this could change if there was ever support for Windows. The config key in the config file always has _path appended to it to make it explicit, that the actual secret value is in a different file and providing it inline would be an error.

Config keys use snake_case.

The config file is currently not supposed to be reread on changes, because certain configuration can not be changed at runtime and any such configuration is only configurable in the config file. For now such configuration should be done via the API instead. This may change if there is a need for it.

Dynamic Configuration via the API

For dynamic configuration at runtime the homeserver exports an API which allows reading, writing as well as deleting configuration values. In the future “patching” might also be supported. Writes are then persisted in a NATS bucket named config based on the key used for their overwrite semantics.

The API is in the admin namespace:

GET /arminius/admin/config
GET /arminius/admin/config/key/<overwrite-key>
POST /arminius/admin/config/key/<overwrite-key>
DELETE /arminius/admin/config/key/<overwrite-key>

The GET method can be used to query the full or a subset of the configuration. The values are returned as JSON and wrapped in the following object:

{
  "value": /* The actual value, unless it is unset and there is no default */,
  "level": "builtin" /* or "file-default", "file", "api" or "unset" */,
  "locked": false /* or true to indicate it can't be modified via the API */,
}

Any secrets will not be returned. Instead they will only show if they are set.

Nested secrets are currently not supported via the API. Instead their keys will simply be missing from the response for now and dedicated APIs would be introduced to query or modify them.

POST allows replacing a specific value completely. It accepts a raw JSON value for config values and binary value for secrets. Certain configuration options are documented as “not configurable at runtime” and modification attempts will always yield an error. Similarly trying to modify a value from the configuration file (apart from inside the default section) will always yield an error.

DELETE unsets the value runtime configured value and makes it default to any higher configuration level as if the runtime value was never configured originally. Deleting a value is always okay, since the value would not have present at the first startup anyway.

Specific configuration options may introduce dedicated APIs to modify lists more easily or deal with nested secrets. Such APIs would be below config/ next to key/. This is the reason for the extra key/ component in the path to prevent clashes or confusion between custom config APIs and config values. For example the federation block/allow lists might have a dedicated API to add or remove entries like PUT /arminius/admin/config/federation.allow_list/entry and POST /arminius/admin/config/federation.allow_list/entry/<idx>.

The config API tries to immediately modify the value via the NATS key-value API. If that modification fails because of a concurrent update, an error is returned and the user has to retry. This is because we can’t safely resolve conflicts between users changing the configuration at the same time. This is another reason some configurations may want to have dedicated APIs, that explicitly modify a subset. The configuration is NOT written to the config files on the file system! Those are treated as read-only. Currently the API does not offer any explicit CAS (compare and swap), but this could be added in the future.

Every service listens to config changes on the NATS KV bucket. If such a change is detected, it updates the config provided in State for any future HTTP requests as well as for any next iteration of background jobs or similar. Updates do not affect requests or tasks already in progress, unless they are particularly long running, in which case such tasks need to manually implement config reload logic at reasonable locations.

Dynamic configuration can only overwrite defaults, builtins or previous dynamic configuration. It can not overwrite other configuration from the config file.

Configuration Merging

Since there are several levels of configuration, they need to be merged in order. Merges are done based on the overwrite key, that is defined for the configuration option. Any configuration at a higher level completely overwrites the lower level (unless overwriting is blocked for the file -> dynamic case). No merging of values is done apart from collecting all overwrite keys at their highest respective level.

The configuration is read from disk and NATS before serving any endpoints and may terminate based on config errors in the configuration file (missing or wrong format). NATS values should not cause startup errors, but might be ignored with an explicit warning in the logs. This is to allow moving a config value to the config file without having to first unset it, but also because config values should always have been valid at some point before. The NATS config needs to be fully read before startup to prevent races where a config has a different value for a short period during startup.

Configuration Options

This list is the initial set of configuration options, that are either currently in use or will be needed at a later date. The actual list should live in an explicit page of the documentation and the below list will not be updated.

Any . in the overwrite key indicates a multi-level mapping in yaml. E.g. server.name would be represented as the following yaml:

server:
  name: "something"

The other NUSS configuration options are skipped for simplicity in the list above.

Maintenance mode may also be converted to a configuration in the future

Opentelemetry

Opentelemetry specifies standardized environment variables. Because these might be used by kubernetes operators, we should support them as well. However, they can be overwritten by either an explicitly empty opentelemetry configuration to disable all telemetry or by specifying the full OpenTelemetry config in that section.

More information on OTel config options can be found here:

• https://opentelemetry.io/docs/specs/otel/configuration/sdk/#in-memory-configuration-model
• https://opentelemetry.io/docs/specs/otel/protocol/exporter/#configuration-options
• https://opentelemetry.io/docs/specs/otel/configuration/sdk-environment-variables/

Implementation Ideas

Since we want the config to be cheaply available to request handlers without having to lock a mutex or otherwise stall on cache lines a lot, it makes sense to hook into the Clone done by axum before each request to reduce the amount of atomic or locking operations we need to do.

This should be achievable by manually implementing Clone in State and having the read value be cloned cheaply when the state is cloned, but doing a more expensive replacement of the write value when there is an update on NATS. A Mutex would work for the write value, but would require a lock during clone(), as such an alternative like ArcSwap is likely better. Ideally we would just have stable atomics for Box/Arc.