AI Gateway
NetScaler® AI Gateway enables enterprises to govern AI traffic flowing between applications and large language model (LLM) backends. NetScaler Kubernetes Gateway Controller extends AI Gateway capabilities to Kubernetes environments, allowing you to configure and manage AI Gateway policies using Kubernetes-native resources without manual CLI or GUI configuration on the NetScaler instance.
Using NetScaler Kubernetes Gateway Controller, you can automate AI Gateway configuration through CRDs and annotation-based policies along with your existing application workloads.
Overview
When to use AI Gateway
- Your applications run on Kubernetes and call Azure OpenAI LLM.
- You need a single governed entry point for all AI traffic, with token-aware routing, rate limiting, prompt inspection, and spillover.
- You want an AI Gateway policy defined as Kubernetes manifests and managed through GitOps workflows.
- External applications or legacy systems outside the cluster need governed access to LLM backends running inside Kubernetes.
Deployment options
The Gateway Controller supports two NetScaler form factors depending on where your traffic originates:
| Deployment | Form factor | Use when |
|---|---|---|
| In-cluster | NetScaler CPX | Client applications run inside the Kubernetes cluster. Handles both east-west traffic to self-hosted LLM backends and egress to external providers |
| External | NetScaler VPX | Client applications run outside the cluster — external apps, legacy systems, or other clusters reaching LLM backends |
Architecture
The Gateway Controller sits in the control plane. It watches Kubernetes resources and programs NetScaler — it is not in the traffic path. Inference requests flow directly from the client application to NetScaler and on to the LLM backend.
Key Kubernetes resources
| Resource | Defined by | Purpose |
|---|---|---|
GatewayClass |
Gateway API (upstream) | Binds the Gateway to the NetScaler controller (CPX or VPX) |
Gateway |
Gateway API (upstream) | Defines the listener with AI Gateway front end profile |
AIGatewayRoute |
NetScaler CRD | Model routing rules — matches by model name, routes to backend service |
RateLimit |
NetScaler CRD | Token-based quota — per-user, per-org, per-model, or per-gateway |
RewritePolicy |
NetScaler CRD | Prompt modification and URL rewrite for spillover |
GatewayClass and Gateway are standard Gateway API resources. AIGatewayRoute, RateLimit, and RewritePolicy are NetScaler-specific CRDs installed with the Gateway Controller and are not portable to other Gateway API implementations.
Note:
AI traffic sent to external providers (Azure OpenAI) and any spillover to a backup provider is billed by that provider per request. Use token quotas (
aigatewayprofile) and rate limits (Configure rate limits and streams) to control spend.
Related topics
- Configure model routing
- Configure rate limits and streams
- Configure prompt management
- Configure spillover to an alternate model
- Observability
Reference
For AI Gateway configuration at NetScaler level, see AI capabilities.