Configure spillover to an alternate model
When a primary model’s token quota is exhausted, NetScaler® AI Gateway automatically routes requests to a configured backup model, maintaining service continuity without any client-side changes.
Warning:
Spillover forwards overflow traffic to the backup model you configure. If that backup is a paid external provider (Azure OpenAI, OpenAI, Anthropic, or Google Vertex AI), every request that exceeds the primary model’s token quota is billed by that provider. Size the primary
tokenquotaand choose the spillover backend deliberately to avoid unexpected charges.
How it works
Each rule in an AIGatewayRoute lists a primary backendRef and a spillover backendRef marked with is_spillover: true. The spillover backend uses its own backend-aigatewayprofile for independent quota tracking.
A RewritePolicy rewrites the model name in the URL path so the backup backend receives the correct model identifier it expects.
Key configuration elements
| Element | Purpose |
|---|---|
is_spillover: true on backendRef
|
Marks the backend as spillover. This element is used only when the primary quota is exhausted. |
backend-aigatewayprofile on the spillover backend |
Separate backend profile giving the spillover backend its own independent quota tracking. If not specified, the default-backend profile under spec.aigatewayprofiles is used. |
spillover-frontend under spec.aigatewayprofiles
|
Frontend profile always used for the spillover backend. |
RewritePolicy through ExtensionRef filter |
Rewrites the model name in the URL to match what the backup service expects. |
Configuration
Spillover backend profile
Create a backend AI Gateway profile for the spillover model so it gets its own independent quota tracking. The route references this profile as aigwybp-spill.
Create a Kubernetes Secret containing the authentication token or API key for the spillover AI Gateway endpoint. Replace <auth-token> with the authentication token for the spillover endpoint.
kubectl create secret generic aigw-auth-spill --from-literal=authToken=<auth-token>
<!--NeedCopy-->
kubectl apply -f - <<EOF
apiVersion: citrix.com/v1
kind: aigatewayprofile
metadata:
name: aigwybp-spill
spec:
gatewayClassName: "aigwclass"
config:
authtoken:
name: aigw-auth-spill
namespace: default
endpointtype: "azureopenai"
profiletype: "backend"
tokenquota: 25000
quotarefreshfrequency: 1
EOF
<!--NeedCopy-->
Spillover rewrite policy
kubectl apply -f - <<EOF
apiVersion: citrix.com/v1
kind: rewritepolicy
metadata:
name: spilloverrewrite
spec:
gatewayClassName: "aigwclass"
rewrite-policies:
- goto-priority-expression: "NEXT"
rewrite-policy:
operation: replace
target: 'http.req.url.path.get(3)'
modify-expression: '"gpt2.5"'
comment: 'Replace model name in URL path for spillover backend'
direction: REQUEST
rewrite-criteria: 'http.req.is_valid'
EOF
<!--NeedCopy-->
This RewritePolicy replaces the model name in the URL path (third segment) with the backup model name. For example, a request originally targeting /openai/deployments/gpt/chat/completions is rewritten to /openai/deployments/gpt2.5/chat/completions before forwarding to the backup service.
AI Gateway routes with spillover service
The AIGatewayRoute defines primary and spillover backends per model rule. The spillover backend (is_spillover: true) attaches the spillover rewrite policy through filters[].extensionRef.
The backend-aigatewayprofile on the spillover backend defines a separate backend profile so the spillover model gets its own independent quota tracking. If backend-aigatewayprofile is not specified, the spillover backend falls back to the default-backend profile defined under spec.aigatewayprofiles.
The frontend profile for the spillover backend is always taken from spec.aigatewayprofiles.spillover-frontend.
kubectl apply -f - <<EOF
apiVersion: citrix.com/v1
kind: aigatewayroute
metadata:
name: ai-route-model
spec:
parentRefs:
- name: aigw
namespace: default
aigatewayprofiles:
frontend:
name: lbaigwyfp
namespace: default
default-backend:
name: aigwybp
namespace: default
spillover-frontend:
name: lbaigwyfp
namespace: default
rules:
- matches:
- policyExpression: "HTTP.REQ.URL.PATH.AFTER_STR(\"/openai/deployments/\").BEFORE_STR(\"/\") == \"gpt\""
backendRefs:
- name: gpt-model-svc
port: 80
- name: gpt-2-5-model-svc
port: 80
is_spillover: true
backend-aigatewayprofile:
name: aigwybp-spill
namespace: default
filters:
- type: ExtensionRef
extensionRef:
group: "citrix.com"
kind: "rewritepolicy"
name: "spilloverrewrite"
EOF
<!--NeedCopy-->
Reference
Configure spillover route requests to a different language model