Use NetScaler credentials stored in a Vault server with NetScaler Ingress Controller

Last published : Nov 14, 2026
In most organizations, tier-1 NetScaler devices and Kubernetes clusters are managed by separate teams, and storing the NetScaler user name and password directly as Kubernetes secrets is not always acceptable. You can instead store the NetScaler credentials in a HashiCorp Vault server and have a Vault Agent sidecar deliver them to NetScaler Ingress Controller as files.
NetScaler Ingress Controller also hot-reloads those file-based credentials: when the credential rotates in Vault, the controller validates the new credential against the NetScaler and switches to it live, without a pod restart. If a rotation delivers an invalid credential, the controller does not use it and keeps the last-good credential. If you already changed the password on NetScaler, the last-good credential no longer works either, so the controller cannot log in to NetScaler until you fix the value in Vault (see Step 7). The controller checks for rotated credentials every 30 seconds by default. To change the interval, set the NS_CRED_RELOAD_INTERVAL environment variable.
This feature needs NetScaler Ingress Controller 4.3.22 or later.
The other NetScaler controllers support the same feature. See the page for your controller:
  • Use NetScaler credentials stored in a Vault server with NetScaler Kubernetes Gateway Controller
  • Use NetScaler credentials stored in a Vault server with NetScaler GSLB controller
  • Use NetScaler credentials stored in a Vault server with NetScaler Node Controller
For the metrics exporter that runs in the NetScaler Ingress Controller pod, see Metrics exporter.
Related information
Use cases

How it works

A Vault Agent sidecar in the controller pod authenticates to Vault (Kubernetes auth) and renders two files — username and password — into an in-memory volume the Helm chart mounts at vaultCredentials.mountPath (default /etc/citrix). The controller reads these files first (falling back to the NS_USER/NS_PASSWORD environment variables if the files are absent or empty), and a built-in watcher re-reads them on change, validates the new credential against the NetScaler, and adopts it without a restart.
  1. Store — an administrator (or a pipeline) puts the NetScaler user name and password into Vault under a KV path (for example, secret/netscaler/adc).
  2. Deliver — a Vault Agent sidecar in the controller pod logs in to Vault using the pod's Kubernetes ServiceAccount token and writes username and password into a shared in-memory volume, re-writing them whenever the secret changes.
  3. Consume — the controller reads those files (file-first — see precedence); a built-in watcher notices any change, validates the new credential against the NetScaler before adopting it, and switches over live.
  4. Rotate — update the NetScaler password and the Vault secret; the sidecar re-renders the files and the controller hot-reloads — no pod restart (see Step 7).
Follow the steps below in order: install Vault (Step 0) → set up Kubernetes authentication (Steps 1–2) → store the credential and allow the controller to read it (Step 3) → add the Vault Agent configuration (Step 4) → deploy the controller (Step 5) → verify (Step 6) → rotate (Step 7).

Credential precedence (vaultCredentials vs adcCredentialSecret)

The Vault (file) source and the Kubernetes-secret source (adcCredentialSecret, delivered as the NS_USER/NS_PASSWORD environment variables) are not mutually exclusive. When both are present, the controller resolves each field (user name and password) independently, file-first:
  1. NS-IP-keyed file — <mountPath>/auth/<NS_IP>/<field>
  2. flat file — <mountPath>/<field> (rendered by the Vault Agent / a mounted Secret)
  3. environment variable — NS_USER / NS_PASSWORD (from adcCredentialSecret)
So a file that has a value always takes precedence over adcCredentialSecret. If a file does not exist, is empty, or contains only white space, the controller skips it and uses the next source in the list. Because resolution is per field, mixed sources are allowed (for example, user name from a file and password from adcCredentialSecret).
Whether adcCredentialSecret is injected at all is controlled by vaultCredentials.suppressEnvCredentials:
vaultCredentials.enabled suppressEnvCredentials Credentials in the pod Effective source
false (n/a) adcCredentialSecret env only environment — legacy behavior, no hot-reload
true true (default) Vault files only; NS_USER/NS_PASSWORD omitted Vault files; adcCredentialSecret becomes optional
true false both Vault files and adcCredentialSecret env Vault files win; adcCredentialSecret is a static fallback
Important: only the file source is hot-reloaded, and only when vaultCredentials.enabled is true. The chart then turns on the reload watcher. adcCredentialSecret (the environment variable) is read once at start-up and never reloaded, so in the "both" mode above it serves only as a static fallback for a missing/empty file. For a clean Vault-only deployment keep the default suppressEnvCredentials=true (then adcCredentialSecret is not required); set it to false only when you deliberately want adcCredentialSecret as a start-up fallback.

Prerequisites

  • A Kubernetes or OpenShift cluster (v1.24 or later) and kubectl/oc with admin access.
  • helm, plus the NetScaler Helm charts: helm repo add netscaler https://netscaler.github.io/netscaler-helm-charts/.
  • The NetScaler management IP and a system-user name/password for the controller to log in with.
  • A HashiCorp Vault server. If you do not have one, install it in the cluster in Step 0 below.

One-time Vault setup

Steps 0, 1, and 2 install Vault and set up its Kubernetes authentication. You do them once for each Vault server and cluster. The Vault pages of the GSLB controller and the Node Controller use the same setup. If you already did these steps for another NetScaler controller in this cluster, go to Step 3.

Step 0: Install HashiCorp Vault (skip if you already have one)

If you already run Vault, just make sure the key-value (KV v2) secrets engine is enabled and note its address (for example https://<VAULT URL>:8200), then go to Step 1. Otherwise, the simplest option is to run Vault in the cluster with its official Helm chart.
  1. Add the HashiCorp Helm repository and install Vault:
    helm repo add hashicorp https://helm.releases.hashicorp.com
    helm repo update
    helm install vault hashicorp/vault -n vault --create-namespace
    Note: For a quick proof of concept you can add --set "server.dev.enabled=true". Dev mode keeps all data in memory, starts unsealed, and uses the root token root. It is not secure, so use it only with test credentials, never in production. For production, leave it out and complete item 2.
  2. Production only — initialize and unseal Vault once, and save the unseal keys + root token somewhere safe:
    kubectl -n vault exec -it vault-0 -- vault operator init
    # run the following once per unseal key that `init` printed (default: 3 of 5);
    # it asks for the key without showing it
    kubectl -n vault exec -it vault-0 -- vault operator unseal
  3. Open a shell in the Vault pod, log in, and enable the KV v2 engine at path secret/ (dev mode already has it):
    kubectl -n vault exec -it vault-0 -- sh
    export VAULT_ADDR=http://127.0.0.1:8200
    vault login   # asks for the root token without showing it
    vault secrets enable -path=secret -version=2 kv   # skip if "secret/" already exists
    The address the controller's sidecar will use is http://vault.vault:8200 for this in-cluster install (namespace vault), or your external Vault URL. Keep this handy for Step 5.
    This install serves Vault over plain HTTP. For production, turn on TLS in the Vault Helm chart, use an https:// address, and give the Vault Agent the Vault CA certificate (mount the CA file and set the VAULT_CACERT environment variable).
    Important: The root token can do anything in Vault. Use it only for this initial setup. For later changes, such as the rotation in Step 7, use an administrator login with limited rights, and revoke the root token when you no longer need it. For more information, see Tokens in the Vault documentation.

Step 1: Create a service account for Kubernetes authentication

Vault uses this service account to check the tokens that the controller pods present (the Kubernetes TokenReview API).
  1. Create the service account cic-k8s-role and bind it to the system:auth-delegator ClusterRole. Save the following as cic-k8s-role-service-account.yml, replace <namespace> with your namespace, and apply it:
    apiVersion: v1
    kind: ServiceAccount
    metadata:
      name: cic-k8s-role
      namespace: <namespace>
    ---
    apiVersion: rbac.authorization.k8s.io/v1
    kind: ClusterRoleBinding
    metadata:
      name: cic-k8s-role-tokenreview
    roleRef:
      apiGroup: rbac.authorization.k8s.io
      kind: ClusterRole
      name: system:auth-delegator
    subjects:
      - kind: ServiceAccount
        name: cic-k8s-role
        namespace: <namespace>    kubectl apply -f cic-k8s-role-service-account.yml
  2. On Kubernetes 1.24 and later, a ServiceAccount no longer gets an auto-generated token Secret. Create one explicitly (the older jsonpath="{.secrets[*].name}" lookup returns empty on 1.24+). Save the following as cic-k8s-role-token.yaml:
    apiVersion: v1
    kind: Secret
    metadata:
      name: cic-k8s-role-token
      annotations:
        kubernetes.io/service-account.name: cic-k8s-role
    type: kubernetes.io/service-account-token
    Apply it in the same namespace and export the token and CA certificate:
    kubectl apply -n &lt;namespace> -f cic-k8s-role-token.yaml
    export VAULT_SA_NAME=cic-k8s-role-token
    export SA_JWT_TOKEN=$(kubectl get secret -n &lt;namespace> $VAULT_SA_NAME -o jsonpath="{.data.token}" | base64 --decode)
    export SA_CA_CRT=$(kubectl get secret -n &lt;namespace> $VAULT_SA_NAME -o jsonpath="{.data['ca\.crt']}" | base64 --decode)
    Note:
    When Vault runs inside the same cluster you can instead omit token_reviewer_jwt in Step 2 and let Vault use its own pod token (the in-cluster local reviewer token). This avoids storing a JWT that is revoked if the source pod/ServiceAccount is deleted.

Step 2: Set up Kubernetes authentication on Vault

Log in to the Vault server and run the following:
  1. Enable Kubernetes authentication. If it is already enabled, Vault reports that the path is already in use; skip this command.
    vault auth enable kubernetes
  2. Configure how Vault reaches the cluster:
    vault write auth/kubernetes/config

    token_reviewer_jwt="$SA_JWT_TOKEN"

    kubernetes_host="https://&lt;K8S_CLUSTER_URL>:&lt;API_SERVER_PORT>"

    kubernetes_ca_cert="$SA_CA_CRT"

Step 3: Store the credentials and allow NetScaler Ingress Controller to read them

Log in to the Vault server and run the following:
  1. Create a read-only policy citrix-adc-kv-ro. First write the policy file (KV v2 stores data under secret/data/... and metadata under secret/metadata/..., so grant read on both):
    citrix-adc-kv-ro.hcl:
    path "secret/data/netscaler/adc" {
      capabilities = ["read"]
    }
    path "secret/metadata/netscaler/adc" {
      capabilities = ["read"]
    }
    Then load it:
    vault policy write citrix-adc-kv-ro citrix-adc-kv-ro.hcl
  2. Store the NetScaler credentials (KV v2 shown):
    vault kv put secret/netscaler/adc username='&lt;NetScaler user name>' password='&lt;password>'
    Note:
    • A KV secret does not rotate by itself. You rotate it in Step 7. The ttl field on a KV entry has no effect.
    • A password that you type in a command stays in your shell history. To avoid that, first read the password into a variable without showing it (for example, read -rs PASSWORD in bash), and then use password="$PASSWORD" in the command.
  3. Map the controller's service account to the policy. The Vault Agent sidecar logs in with the token of the service account that the controller pod runs as. This example uses the service account nsic-vault, which you set with serviceAccount.name in Step 5. Replace <controller-namespace> with the namespace of the controller.
    vault write auth/kubernetes/role/netscaler-adc

    bound_service_account_names=nsic-vault

    bound_service_account_namespaces=&lt;controller-namespace>

    policies=citrix-adc-kv-ro

    ttl=24h
    Use a role name that no other controller uses. Writing a role with the same name replaces it, so another controller that uses that role can no longer log in to Vault.

Step 4: Create the Vault Agent configuration

Create a Vault Agent config that authenticates and continuously renders the two credential files. Note exit_after_auth = false (so the agent keeps running and re-renders on rotation), static_secret_render_interval (so the agent checks Vault for changes every 30 seconds; the default is 5 minutes), and the two template stanzas that write username and password (not a single .env):
vault-agent-config.hcl:
exit_after_auth = false
pid_file        = "/home/vault/pidfile"

auto_auth {
  method "kubernetes" {
    mount_path = "auth/kubernetes"
    config = { role = "netscaler-adc" }
  }
  sink "file" { config = { path = "/home/vault/.vault-token" } }
}

template_config {
  static_secret_render_interval = "30s"
}

template {
  destination = "/etc/citrix/username"
  perms       = "0444"
  contents    = "{{ with secret \"secret/data/netscaler/adc\" }}{{ .Data.data.username }}{{ end }}"
}
template {
  destination = "/etc/citrix/password"
  perms       = "0444"
  contents    = "{{ with secret \"secret/data/netscaler/adc\" }}{{ .Data.data.password }}{{ end }}"
}
The templates write the files with mode 0444, because the Vault Agent runs as a different user than the controller. Only the containers that mount the citrix-creds volume can read the files: the agent, the controller, and the metrics exporter when it shares the volume (exporter.required: true with vaultCredentials.shareWithExporter: true).
Create a ConfigMap from it:
kubectl create configmap vault-agent-config --from-file=./vault-agent-config.hcl
Note: Pin the images — use hashicorp/vault:<version> (the bare vault Docker image is deprecated). If you prefer consul-template, use hashicorp/consul-template:0.42.1 (the :alpine tag no longer exists), set renew_token = true, and run it as a long-running sidecar without -once.

Step 5: Deploy NetScaler Ingress Controller with Vault credentials

Enable the feature with vaultCredentials.enabled=true and add the Vault Agent as a sidecar via extraContainers. The chart creates the shared citrix-creds in-memory volume and, because its ServiceAccount disables automount, you project a token for the agent via extraVolumes. Set serviceAccount.name to the service account that the Vault role in Step 3 allows. If you also run the metrics exporter in the pod (exporter.required: true), add a Vault Agent init container as well, as described in Metrics exporter.
vault-values.yaml:
serviceAccount:
  name: nsic-vault             # the service account that the Vault role in Step 3 allows

vaultCredentials:
  enabled: true                # opt-in; adcCredentialSecret becomes optional

extraContainers:
  - name: vault-agent
    image: hashicorp/vault:1.21
    args: ["agent", "-config=/etc/vault/vault-agent-config.hcl", "-log-level=info"]
    env:
      - name: VAULT_ADDR
        value: "https://<VAULT URL>:8200"
      - name: SKIP_SETCAP      # needed when the container does not run as root, for example on OpenShift
        value: "true"
    volumeMounts:
      - { name: vault-config, mountPath: /etc/vault }
      - { name: vault-token,  mountPath: /home/vault }
      - { name: citrix-creds, mountPath: /etc/citrix }
      - { name: sa-token,     mountPath: /var/run/secrets/kubernetes.io/serviceaccount, readOnly: true }

extraVolumes:
  - name: vault-config
    configMap: { name: vault-agent-config }
  - name: vault-token
    emptyDir: { medium: Memory }
  - name: sa-token
    projected:
      sources:
        - serviceAccountToken: { path: token, expirationSeconds: 3600, audience: vault }
        - configMap: { name: kube-root-ca.crt, items: [{ key: ca.crt, path: ca.crt }] }
        - downwardAPI: { items: [{ path: namespace, fieldRef: { fieldPath: metadata.namespace } }] }
The projected token has the audience vault, so it works for logging in to Vault but not against the Kubernetes API.
Install NetScaler Ingress Controller (add its mandatory values):
helm install nsic netscaler/netscaler-ingress-controller -f vault-values.yaml --set nsIP=<NSIP>,license.accept=yes,ingressClass[0]=<class>
Note: The Vault Agent sidecar logs in to Vault with the token of the controller pod's service account. That service account must be the one in bound_service_account_names of the Vault role (Step 3). Otherwise, the Vault login fails with a permission error.
The new variables (all default to the values shown; set them only to change behavior):
Variable Default Purpose
vaultCredentials.enabled false Turn Vault credential hot-reload on.
vaultCredentials.mountPath /etc/citrix Where the sidecar must render username/password.
vaultCredentials.suppressEnvCredentials true Omit NS_USER/NS_PASSWORD env (makes adcCredentialSecret optional).
vaultCredentials.credentialWaitSeconds 120 Startup wait for the sidecar to render creds before failing.
vaultCredentials.shareWithExporter true Also mount the creds volume into the metrics exporter at /mnt/nslogin. See Metrics exporter.
extraContainers / extraVolumes / extraVolumeMounts / extraEnv [] Extra containers, volumes, volume mounts, and environment variables, such as the Vault Agent sidecar, its token, and its configuration.
initContainers [] Init containers, such as the Vault Agent init container that writes the credential files before the controller and the metrics exporter start. See Metrics exporter.
The following example sets every vaultCredentials value explicitly and adds NS_CRED_RELOAD_INTERVAL through extraEnv, so that the controller checks for rotated credentials every 60 seconds:
vaultCredentials:
  enabled: true
  mountPath: /etc/citrix         # where the sidecar renders the username and password files
  credentialWaitSeconds: 300     # wait up to 5 minutes at start-up for the sidecar to render the files
  suppressEnvCredentials: true   # set to false to keep adcCredentialSecret as a start-up fallback
  shareWithExporter: true        # see Metrics exporter
extraEnv:
  - name: NS_CRED_RELOAD_INTERVAL
    value: "60"

Step 6: Verify

Check the controller logs (no need to log in to the NetScaler). Find the deployment name with kubectl get deploy, then run:
kubectl logs deploy/<deployment-name> -c nsic | grep -iE "hot-reload|CredentialWatcher|credentials"
Expected:
  • Credential hot-reload ENABLED
  • Starting CredentialWatcher ... for <NSIP>
  • The pod becomes Ready (it logged in with the Vault-rendered credentials).

Step 7: Rotate the credentials

  1. Change the NetScaler user's password: set system user <user> <newpassword>.
  2. Right after that, update the Vault secret to the same value: vault kv put secret/netscaler/adc username=<user> password=<newpassword>.
  3. Within 30 seconds, the Vault Agent picks up the change and re-renders the files. The controller validates the new credential and switches to it at its next check (NS_CRED_RELOAD_INTERVAL, default 30 seconds), so the switch takes about a minute. There is no restart, and traffic is not affected.
  4. If the new value is wrong, the controller does not use it. It keeps the last-good credential and logs Credential hot-reload: probe FAILED. Because you changed the password on NetScaler in item 1, the last-good credential no longer works either, so the controller cannot log in to NetScaler. The controller keeps testing the new value, but after repeated failures it waits longer between tests, up to 10 minutes. Fix the value in Vault right away.
  5. After you fix the value, the controller switches to it at its next test, which can take 10 minutes or more.
If you run the metrics exporter in the pod, restart the pod after the controller has switched to the new password (it logs Credential hot-reload: promoted), so that the exporter gets the new password too: kubectl rollout restart deployment/<deployment-name>. The controller does not need this restart, but the exporter reads the password only when it starts. For more information, see Metrics exporter.

Deploy with NetScaler Operator

To deploy with NetScaler Operator, see Deploy NetScaler Ingress Controller using NetScaler Operator.

Metrics exporter

The metrics exporter in the NetScaler Ingress Controller pod (exporter.required: true) uses the same NetScaler credentials as the controller. With vaultCredentials.shareWithExporter: true (the default), the chart mounts the citrix-creds volume into the exporter container at /mnt/nslogin, so the exporter reads the same username and password files. It does not read the auth/<NS_IP>/ layout, so write the files directly under vaultCredentials.mountPath.
When the exporter shares the Vault files (vaultCredentials.shareWithExporter: true, the default), also add a Vault Agent init container to the pod. Like the sidecar, it runs the HashiCorp Vault image (hashicorp/vault) with the agent command, so it is a Vault client, not a Vault server. It logs in to Vault with the pod's service account token, writes the username and password files to the shared citrix-creds volume once, before the controller and the exporter start, and then exits. Keep the vault-agent sidecar from Step 5: it rewrites the files after each rotation so that the controller can hot-reload them. In vault-values.yaml, add the init container under initContainers. Use the same image, environment, and volume mounts as the sidecar, and add the -exit-after-auth flag. This flag overrides exit_after_auth = false in the configuration from Step 4, so the agent exits after it writes the files:
initContainers:
  - name: vault-agent-init
    image: hashicorp/vault:1.21   # the same image as the sidecar, run as a Vault Agent, not a Vault server
    args: ["agent", "-config=/etc/vault/vault-agent-config.hcl", "-exit-after-auth"]   # write the files once, then exit
    env:
      - name: VAULT_ADDR
        value: "https://<VAULT URL>:8200"
      - name: SKIP_SETCAP      # needed when the container does not run as root, for example on OpenShift
        value: "true"
    volumeMounts:
      - { name: vault-config, mountPath: /etc/vault }
      - { name: vault-token,  mountPath: /home/vault }
      - { name: citrix-creds, mountPath: /etc/citrix }
      - { name: sa-token,     mountPath: /var/run/secrets/kubernetes.io/serviceaccount, readOnly: true }
Note:
  • If you set a securityContext for the sidecar, set the same one for the init container. Both must write the files as the same user, so that the sidecar can replace them after a rotation.
  • If the init container cannot log in to Vault or read the secret, it keeps retrying, and the whole pod, including the controller, stays in the Init state (see kubectl get pods). To see the error, run kubectl logs <pod-name> -c vault-agent-init.
  • The exporter logs in to NetScaler at its first scrape, not when it starts. To check that it logged in, run kubectl logs deploy/<deployment-name> -c exporter and look for ADC Session Login Successful.
  • After each credential rotation, restart the pod. See Step 7.
The init container is required because the exporter reads /mnt/nslogin/username and /mnt/nslogin/password only once, when it starts. It does not wait for the files and never reads them again. Kubernetes starts the containers of a pod one after another, without waiting for one to be ready before it starts the next, and the chart lists the containers from extraContainers, such as the sidecar, after the exporter. So without the init container, the exporter is likely to look for the files before the sidecar writes them. It then cannot log in to NetScaler. It might log no error, but its citrixadc_probe_success metric is 0. The controller does not have this problem, because it waits for the files (vaultCredentials.credentialWaitSeconds).
Restarting the pod does not reliably fix this, because the new pod starts with a new, empty credential volume. Kubernetes starts the other containers only after the init containers finish, so with the init container the files exist before the controller and the exporter start.
The pod restart after a rotation is needed for the same reason. The controller switches to the new password, but the exporter keeps the old one. It keeps working while its current NetScaler session lasts. When the session ends, the exporter can no longer log in, because it still uses the old password. In the restarted pod, the init container writes the new password before the exporter starts.