Telemetry

We aim to have the server be fully observable to developers and administrators without compromising data protection. The exact approach is still getting refined, but we have some basic rules in the mean time.

Having different versions of the otel tracing crates in the dependency tree can cause problems with the tracing infrastructure like configuration not applying properly.

You can test for this using `cargo tree -i tracing-opentelemetry -v`.

Traces

• We collect traces for every incoming http request. This is handled automatically using an Axum layer. This also accepts external tracing headers allowing for cross-service traces.
• We generally try to trace every database call to give us basic performance insights. Long term we might lower that for performance reasons or to reduce data ending up in traces.
• We trace requests across NATS by inserting and extracting tracing headers. Necessary utilities can be found in arminius-lib-nats in the opentelemetry module.

Logs

Logs are automatically forwarded to OTel when configured.

Currently logs are not automatically attached to tracing spans, but they will be once we depend on tracing-opentelemetry 0.32 or higher: https://github.com/tokio-rs/tracing-opentelemetry/releases/tag/v0.32.0

Metrics

TBD. We currently don’t have a structured approach to metrics yet.