Rate Limiting

Overview

The goal of rate limiting is to control the amount of incoming and outgoing traffic to protect system stability, prevent resource exhaustion, and defend against certain types of attacks.

Key Concepts

Requests per Second (RPS)

The number of API calls allowed within a one-second timeframe.

Rate Limit Window

The time period during which requests are counted.

Throttling

Temporarily slowing down or delaying requests when limits are approached.

Quota

The maximum number of requests allowed in a longer period (e.g., daily/monthly).

Common Implementation Methods

Token Bucket

A token bucket algorithm allows for bursts of requests while maintaining an average rate limit. Tokens are added to a bucket at a constant rate, and each request consumes a token. When the bucket is empty, further requests are throttled.

Leaky Bucket

Similar to token bucket but processes requests at a constant rate, removing bursts. Requests enter a queue and are processed at a fixed rate.

Fixed Window

Divides time into fixed windows (e.g., 1-minute intervals) and allows a certain number of requests per window. Simple but can allow request spikes at window boundaries.

Sliding Window

Considers a rolling time period rather than fixed intervals, providing smoother rate limiting. More complex to implement but avoids boundary problems.

HTTP Response Headers

{
  "X-RateLimit-Limit": "100",       // Total requests allowed in period
  "X-RateLimit-Remaining": "75",    // Requests remaining in current period
  "X-RateLimit-Reset": "1627834096" // Timestamp when the limit resets
}

Rate Limiting Strategies

IP-Based

Limits requests based on client IP address.

User-Based

Limits requests based on authenticated user.

Room-Based

Room based. What did you expect?

File-Based

Limit too much access on a certain file or resource.

Service-Based

Limits for specific services or microservices → Rate limit if a stream is falling behind.

Federation Transactions

Deny transactions, if federating servers act maliciously (because of malicious users, for example).

Handling Rate Limit Exceeding

When a client exceeds rate limits, the API should respond with a 429 (Too Many Requests) status code and provide information about when the client can retry.

HTTP/1.1 429 Too Many Requests
Content-Type: application/json
Retry-After: 30

{
  "errcode": "M_LIMIT_EXCEEDED",
  "error": "Too many requests",
  "retry_after_ms": 30000
}

Best Practices

• Implement exponential backoff in clients for graceful handling of rate limits
• Provide clear documentation of rate limits for API consumers
• Consider different tiers of rate limits for different customer segments
• Monitor and analyze rate limit hits to adjust limits as needed
• Implement soft limits with warnings before hard limits are reached

Tools and Libraries

• NATS: High-performance messaging system for distributed rate limiting
• Rust libraries: axum, tower, tower-http
• Custom implementation: Building rate limiting directly in Rust services using NATS for coordination
• Alternative options: Rate limit (Lyft), Envoy for API gateway rate limiting

How Do We Make Rate Limits Configurable? What Are Our Requirements?

• Per power level
• scalable, reliable: emergency announcements should be retrievable
• Some delay is acceptable, but we don’t want an interruption of the service.
  • General limits (x requests per second) may have a delay before kicking in
  • Blocking a foreign server should be instant