Configure rate limits and streams
Rate limiting enforces quotas at the Gateway and/or AIGatewayRoute, giving you precise control over LLM resource usage per user and per organization. You can limit by request count, by actual token consumption, or both.
Token-based rate limiting (mode: TOKEN_RATE) measures actual token consumption, which reflects real GPU resource usage — unlike request-count limits.
Requests exceeding the configured quota receive a 429 Too Many Requests response before the request reaches the LLM backend.
Important:
Rate limits are the primary control for capping LLM spend. Without a token-based limit (
mode: TOKEN_RATE), a single user or organization can consume unlimited tokens against a paid provider, with no upper bound on cost. Configure aTOKEN_RATElimit per user and per organization to cap spend.
Per Gateway Vs Per Model rate limits
Rate limits can be applied globally at the Gateway level (across all model routes) or to a specific AI model through an AIGatewayRoute.
| Type | How it binds | Scope |
|---|---|---|
| Per-model |
filters[].extensionRef in AIGatewayRoute rule |
One model route only |
| Per-gateway |
targetRef pointing to the Gateway resource |
All routes on the Gateway |
Note:
Both per-model and global rate limits can be simultaneously active. The per-model limit is evaluated first. The global limit applies across all models regardless of per-model quota state.
Configuration
Apply per-model rate limits
These RateLimit CRDs apply per AI model and are referenced from AIGatewayRoute rules via filters[].extensionRef.
The following examples enforce rate limits per organization and per user, keyed on the X-org-id and X-user-id request headers. These are only examples — you can key a rate limit on any attribute that the ratelimit CRD supports.
Per organization rate limits
A per-organization limit caps how much every organization can consume, identified by the X-org-id header. For example, this sample configuration sets two quotas per organization over a 60-second window: a request-count limit (req_threshold: 20) and a token limit (mode: TOKEN_RATE, req_threshold: 4000). Requests over either quota are blocked (throttle_action: RESPOND). For all parameters, see the ratelimit CRD.
kubectl apply -f - <<EOF
apiVersion: citrix.com/v1beta1
kind: ratelimit
metadata:
name: orgaimodel
namespace: default
spec:
gatewayClassName: "aigwclass"
selector_keys:
basic:
header_name: "X-org-id"
ratelimits:
- req_threshold: 20
alertsintimeslice: 40
timeslice: 60000
throttle_action: "RESPOND"
- mode: "TOKEN_RATE"
req_threshold: 4000
alertsintimeslice: 40
timeslice: 60000
throttle_action: "RESPOND"
EOF
<!--NeedCopy-->
Per user rate limits
A per-user limit caps how much each individual user can consume, identified by the X-user-id header. For example, this sample configuration allows each user 20 requests and 4000 tokens per minute. For all parameters, see the ratelimit CRD.
kubectl apply -f - <<EOF
apiVersion: citrix.com/v1beta1
kind: ratelimit
metadata:
name: useraimodel
namespace: default
spec:
gatewayClassName: "aigwclass"
selector_keys:
basic:
header_name: "X-user-id"
ratelimits:
- req_threshold: 20
alertsintimeslice: 40
timeslice: 60000
throttle_action: "RESPOND"
- mode: "TOKEN_RATE"
req_threshold: 4000
alertsintimeslice: 40
timeslice: 60000
throttle_action: "RESPOND"
EOF
<!--NeedCopy-->
Streams for observability
Streams collect and export consumption metrics from the Gateway, so you can see how requests and tokens are being used against the configured limits. You add a streams section to the ratelimit spec, and the same configuration applies to both per-user and per-organization rate limits.
For example, this sample configuration defines two streams over a 10-minute interval (interval: 10, in minutes): one that tracks the top consumers by request count (sort: REQUESTS) and one by token count (sort: TOKENS). Each stream logs up to 20 entries (logLimit: 20) at a logInterval of 10 minutes. The logInterval must be greater than or equal to the stream interval.
For these streams to be useful, you must also apply the analytics configuration — see Observability.
streams:
- logInterval: 10
logLimit: 20
sort: "REQUESTS"
interval: 10
- logInterval: 10
logLimit: 20
sort: "TOKENS"
interval: 10
<!--NeedCopy-->
Apply AIGatewayRoute with rate limit filters
The AIGatewayRoute binds the rate limits defined earlier to a model route through its filters section. For example, this sample references both the orgaimodel and useraimodel rate limits (defined above) as ExtensionRef filters, so both quotas apply to the matched model.
kubectl apply -f - <<EOF
apiVersion: citrix.com/v1
kind: aigatewayroute
metadata:
name: ai-route-model
spec:
parentRefs:
- name: aigw
namespace: default
aigatewayprofiles:
frontend:
name: lbaigwyfp
namespace: default
default-backend:
name: aigwybp
namespace: default
rules:
- filters:
- type: ExtensionRef
extensionRef:
group: "citrix.com"
kind: "ratelimit"
name: "orgaimodel"
- type: ExtensionRef
extensionRef:
group: "citrix.com"
kind: "ratelimit"
name: "useraimodel"
matches:
- policyExpression: "HTTP.REQ.URL.PATH.AFTER_STR(\"/openai/deployments/\").BEFORE_STR(\"/\") == \"gpt\""
backendRefs:
- name: gpt-model-svc
port: 80
EOF
<!--NeedCopy-->
Each rule in the AIGatewayRoute references the RateLimit CRDs via filters of type ExtensionRef — binding the rate limit policy to that specific model route.
Apply Gateway-level rate limits
These RateLimit CRDs use targetRef to reference the Gateway directly — enforcing quotas across all model routes regardless of which model is matched.
Per org Gateway level rate limits
kubectl apply -f - <<EOF
apiVersion: citrix.com/v1beta1
kind: ratelimit
metadata:
name: orgcs
namespace: default
spec:
gatewayClassName: "aigwclass"
targetRef:
- name: aigw
namespace: default
kind: Gateway
group: gateway.networking.k8s.io
selector_keys:
basic:
header_name: "X-org-id"
ratelimits:
- req_threshold: 20
alertsintimeslice: 40
timeslice: 60000
throttle_action: 'RESPOND'
- mode: "TOKEN_RATE"
req_threshold: 4000
alertsintimeslice: 40
timeslice: 60000
throttle_action: 'RESPOND'
streams:
- logInterval: 10
logLimit: 20
sort: "REQUESTS"
interval: 10
- logInterval: 10
logLimit: 20
sort: "TOKENS"
interval: 10
EOF
<!--NeedCopy-->
Per user Gateway level rate limits
kubectl apply -f - <<EOF
apiVersion: citrix.com/v1beta1
kind: ratelimit
metadata:
name: usercs
namespace: default
spec:
gatewayClassName: "aigwclass"
targetRef:
- name: aigw
namespace: default
kind: Gateway
group: gateway.networking.k8s.io
selector_keys:
basic:
header_name: "X-user-id"
ratelimits:
- req_threshold: 20
alertsintimeslice: 40
timeslice: 60000
throttle_action: "RESPOND"
- mode: "TOKEN_RATE"
req_threshold: 4000
alertsintimeslice: 40
timeslice: 60000
throttle_action: "RESPOND"
streams:
- logInterval: 10
logLimit: 20
sort: "REQUESTS"
interval: 10
- logInterval: 10
logLimit: 20
sort: "TOKENS"
interval: 10
EOF
<!--NeedCopy-->