Product Documentation
NetScaler® Kubernetes Gateway Controller
View PDF

Observability

August 12, 2026
Contributed by: 
C

NetScaler® AI Gateway exports per-request AI metrics to Splunk, including token counts, model identity, client identity, latency, and cost attribution once observability is configured. No application instrumentation is required.

NetScaler exports these metrics to Splunk through the Splunk HTTP Event Collector (HEC). You configure the HEC connection and analytics profiles as described in this page.

Prerequisites

  • NetScaler Kubernetes Gateway Controller already deployed. See Deploy Kubernetes Gateway Controller with AI capabilities.
  • A reachable Splunk instance with the HTTP Event Collector (HEC) enabled, plus a HEC token and a target index.
  • The Splunk HEC endpoint host and port (the HEC default port is 8088), reachable from the cluster and NetScaler.
  • kubectl and Helm 3.x, with permission to create secrets and ConfigMaps in the target namespace.

Note:

Create the Splunk token secret and the ConfigMap for NetScaler CPX custom metrics in the same namespace as the controller or CPX, and set authToken.namespace to that namespace.

Configuration overview

Observability is configured in two parts:

Part Where Required
Splunk server connection analyticsConfig in Helm values.yaml Mandatory
Per-route analytics profile aigatewayroute.citrix.com/analytics-profile annotation on AIGatewayRoute Optional — overrides global config per route

Global versus route-level profiles

Profile type Configured in Scope
Time series metrics analyticsConfig.splunk.servers[].profiles.timeseries in values.yaml Global only — cannot be per-route
Stream insight analyticsConfig.splunk.servers[].profiles.streaminsight in values.yaml Global default
Per-route override aigatewayroute.citrix.com/analytics-profile annotation Per AIGatewayRoute

Create the Splunk token secret

kubectl create secret generic splunktoken \
  --from-literal=splunk-token=<your-splunk-token> -n <namespace>
<!--NeedCopy-->

The chart reads the token from the splunk-token key of this secret. Use your Splunk HEC token as the value. The authToken reference in analyticsConfig (below) points to this secret by name and namespace.

Configure the Splunk connection in Helm values (mandatory)

The analyticsConfig section in values.yaml of Helm chart creates the Splunk server connection and default analytics profiles:

analyticsConfig:
  required: true
  logAllJsonFields: "true"
  splunk:
    enable: "true"
    servers:
      - name: "my-splunk"
        endpoint: "<splunk-endpoint>"
        port: 8088   # Splunk HEC port (default 8088); match your HEC listener
        authToken:
          name: splunktoken
          namespace: <namespace>
        profiles:
          timeseries:
            metricsexportfrequency: 60   # seconds
            analyticsendpointurl: "netscaler/metric"
          streaminsight:
            analyticsendpointurl: "/services/collector/event"
            analyticsendpointcontenttype: "application/json"
<!--NeedCopy-->

Note:

Setting logAllJsonFields: "true" enables all counters for transaction-type metrics, ensuring AI Gateway-specific details are included in the exported data. However, it exports all JSON fields on every transaction and can significantly increase Splunk ingest and licensing cost. Enable it when you need full AI metrics; otherwise leave it disabled.

After editing values.yaml, apply it to update the deployment:

helm upgrade <release> <chart> -f values.yaml
<!--NeedCopy-->

Alternatively, add -f values.yaml to the initial helm install command.

Override the analytics profile per route

Use the aigatewayroute.citrix.com/analytics-profile annotation on an AIGatewayRoute to customize the collector and insight type for specific model routes. Add the annotation to the AIGatewayRoute metadata, as shown in this complete example:

apiVersion: citrix.com/v1
kind: aigatewayroute
metadata:
  name: ai-route-model
  annotations:
    aigatewayroute.citrix.com/analytics-profile: |
      [
        {
          "collectors": {"splunk": ["my-splunk"]},
          "type": "webinsight",
          "analyticsEndpointUrl": "/services/collector/event",
          "allHttpHeaders": "enabled"
        },
        {
          "collectors": {"splunk": ["my-splunk"]},
          "type": "streaminsight",
          "analyticsEndpointUrl": "/services/collector/event",
          "analyticsEndpointContentType": "application/json",
          "topn": "enabled"
        }
      ]
spec:
  parentRefs:
    - name: aigw
  aigatewayprofiles:
    frontend:
      name: lbaigwyfp
    default-backend:
      name: aigwybp
  rules:
    - matches:
      - path:
          type: PathPrefix
          value: "/"
      backendRefs:
        - name: gpt-model-svc
          port: 80
<!--NeedCopy-->

Important:

The server name in the annotation’s collectors field (for example, "my-splunk") must match the server name defined in analyticsConfig in values.yaml. The analyticsConfig section in values.yaml creates the server connection; the annotation references it by name.

Custom AI Gateway metrics

To export AI Gateway-specific metrics counters, create a custom metrics configuration file named metrics.json. Build this file using the AI Gateway counters listed in AI Gateway observability. The way you add this file to NetScaler depends on the form factor (NetScaler CPX, the containerized NetScaler that runs in the cluster, or NetScaler VPX/MPX/BLX, the virtual appliance that runs outside the cluster).

NetScaler CPX

For NetScaler CPX (the containerized NetScaler), add the metrics.json file to the CPX container using a ConfigMap. The metrics.json file is available at https://raw.githubusercontent.com/netscaler/netscaler-k8s-ingress-controller/refs/heads/master/example/metrics.json.

Create a ConfigMap from the metrics.json file:

kubectl create configmap my-custom-metrics-configmap \
  --from-file=metrics.json -n <namespace>
<!--NeedCopy-->

Configure extraVolumeMounts and extraVolumes in values.yaml to mount the ConfigMap into the CPX container:

extraVolumeMounts:
  - name: custom-metrics-conf
    mountPath: /cpx/custom_metrics_conf

extraVolumes:
  - name: custom-metrics-conf
    configMap:
      name: "my-custom-metrics-configmap"
      defaultMode: 420
<!--NeedCopy-->

After editing values.yaml, apply it to update the deployment:

helm upgrade <release> <chart> -f values.yaml
<!--NeedCopy-->

NetScaler VPX/MPX/BLX

For NetScaler VPX/MPX/BLX (the virtual appliance running outside the cluster), place the metrics.json file directly inside NetScaler VPX/MPX/BLX as described in AI Gateway observability. The mertrics.json file is available at https://raw.githubusercontent.com/netscaler/netscaler-k8s-ingress-controller/refs/heads/master/example/metrics.json.

Verify

After you apply the configuration, confirm that metrics are reaching Splunk:

  • In Splunk, search the configured index or endpoint (for example, a source containing netscaler/metric).
  • Within one metricsexportfrequency interval (60 seconds in the example), you should see per-request AI events that include token counts and model identity.

Reference

AI Gateway observability

Observability
August 12, 2026
Contributed by: 
C
August 12, 2026
Contributed by: 
C

In this article

Site feedback | Your privacy choices footer linkYour Privacy Choices | Privacy and legal terms | Cookie preferences | Consent settings | docs.cloud.com
© 1999- Cloud Software Group, Inc. All rights reserved.