Configure model routing
NetScaler® AI Gateway routes incoming inference requests to the correct Large Language Model (LLM) backend based on the model name in the request. This feature provides a single Gateway endpoint for all models. Clients never need to know the backend service addresses or topology.
Model upgrades, backend changes, and new model additions don’t require application code changes.
Prerequisites
- NetScaler Kubernetes Gateway Controller deployed with desired configurations.
- GatewayClass created and bound to the NetScaler instance.
- AI inference services deployed as external services in Kubernetes.
Configuration
AI Gateway profiles
An AI Gateway profile defines the settings that the NetScaler applies when it processes AI traffic between your applications and LLM backends. A profile can be of type frontend (applied to client-facing virtual servers) or backend (applied to the service groups that connect to the LLM backends). You can refer to the aigatewayprofile CRD documentation for more details.
Note:
Once created, modifying an AI Gateway profile, or changing the AI Gateway profile attached to a Gateway or AIGatewayRoute, causes the associated NetScaler entities to be recreated. As a result, traffic impacted.
Frontend AI Gateway profile
The front-end profile defines the AI Gateway endpoint type.
Front-end AIGateway profile for Gateway
kubectl apply -f - <<EOF
apiVersion: citrix.com/v1
kind: aigatewayprofile
metadata:
name: csaigwyfp
spec:
gatewayClassName: "aigwclass"
config:
endpointtype: "azureopenai"
profiletype: "frontend"
EOF
<!--NeedCopy-->
Front-end AIGateway profile for AIGatewayRoute
kubectl apply -f - <<EOF
apiVersion: citrix.com/v1
kind: aigatewayprofile
metadata:
name: lbaigwyfp
spec:
gatewayClassName: "aigwclass"
config:
endpointtype: "azureopenai"
profiletype: "frontend"
EOF
<!--NeedCopy-->
Backend AI Gateway profile
The backend profile defines the per-model quota and authentication configuration.
Create a Kubernetes Secret containing the Authentication token/API key for the AI Gateway endpoint. Replace <auth-token> with the authentication token for AI Gateway endpoint.
kubectl create secret generic aigw-auth --from-literal=authToken=<auth-token>
<!--NeedCopy-->
kubectl apply -f - <<EOF
apiVersion: citrix.com/v1
kind: aigatewayprofile
metadata:
name: aigwybp
spec:
gatewayClassName: "aigwclass"
config:
authtoken:
name: aigw-auth
namespace: default
endpointtype: "azureopenai"
profiletype: "backend"
tokenquota: 25000
quotarefreshfrequency: 1
---
EOF
<!--NeedCopy-->
Kubernetes Gateway with AI Gateway profile
The Kubernetes Gateway defines the entry point for the AI incoming traffic. To add the frontend AI Gateway profile to the Gateway, add the name of the frontend profile under the infrastructure reference. This frontend profile is applied to the content switching (CS) virtual server that fronts the AI traffic in NetScaler.
kubectl apply -f - <<EOF
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
name: aigw
spec:
gatewayClassName: aigwclass
infrastructure:
parametersRef:
group: citrix.com
kind: aigatewayprofile
name: csaigwyfp
listeners:
- name: https
protocol: HTTPS
port: 443
tls:
mode: Terminate
certificateRefs:
- kind: Secret
name: web-ingress-secret
group: ""
allowedRoutes:
namespaces:
from: All
---
EOF
<!--NeedCopy-->
Refer to Gateway Documentation for the fields of the Gateway Object.
Notes:
Only frontend AI Gateway profile can be applied with a Gateway object.
Create the TLS secret
web-ingress-secretreferenced by the Gateway listener before applying the Gateway. Replace the certificate and key paths with your own:kubectl create secret tls web-ingress-secret --cert=<path-to-tls.crt> --key=<path-to-tls.key> <!--NeedCopy-->
NetScaler AI Gateway route
Define routing rules that match requests by model name and forward each rule to the corresponding backend service. In the AIGatewayRoute, the frontend AI Gateway profile is applied to the load balancing (LB) virtual server, and the backend AI Gateway profile is applied to the service group that connects to the LLM backend. You can refer to the aigatewayroute CRD documentation for more details.
Complex routing with NetScaler PolicyExpression
Use a NetScaler policyExpression in matches to route on any part of the request. For example, this sample inspects the request body and routes to gpt-model-svc when the body contains gpt.
kubectl apply -f - <<EOF
apiVersion: citrix.com/v1
kind: aigatewayroute
metadata:
name: ai-route-model
spec:
parentRefs:
- name: aigw
aigatewayprofiles:
frontend:
name: lbaigwyfp
default-backend:
name: aigwybp
rules:
- matches:
- policyExpression: HTTP.REQ.BODY(30).CONTAINS("gpt")
backendRefs:
- name: gpt-model-svc
port: 80
---
EOF
<!--NeedCopy-->
Simple routing in Kubernetes native way
Use the standard Gateway API path match for straightforward routing. For example, this sample routes all requests under the / path prefix to gpt-model-svc.
kubectl apply -f - <<EOF
apiVersion: citrix.com/v1
kind: aigatewayroute
metadata:
name: ai-route-model
spec:
parentRefs:
- name: aigw
aigatewayprofiles:
frontend:
name: lbaigwyfp
default-backend:
name: aigwybp
rules:
- matches:
- path:
type: PathPrefix
value: "/"
backendRefs:
- name: gpt-model-svc
port: 80
---
EOF
<!--NeedCopy-->