Use NetScaler credentials stored in a Vault server with NetScaler Kubernetes Gateway Controller

• Use NetScaler credentials stored in a Vault server with NetScaler Node Controller
• Use NetScaler credentials stored in a Vault server with NetScaler Ingress Controller
• Use NetScaler credentials stored in a Vault server with NetScaler GSLB controller

How it works

1. Store: an administrator or a pipeline puts the NetScaler user name and password into Vault under a KV path, for example secret/netscaler/gateway.
2. Deliver: the Vault Agent sidecar logs in to Vault with the pod's service account token and writes username and password into the shared in-memory volume.
3. Consume: the controller reads those files (see Credential precedence). Its watcher notices any change, validates the new credential, and switches over live.
4. Rotate: update the NetScaler password and the Vault secret. The sidecar rewrites the files and the controller hot-reloads the credential (see Step 7).

Credential precedence (vaultCredentials vs netscaler.adcCredentialSecret)

1. <mountPath>/auth/<NS_IP>/<field>
2. <mountPath>/<field> (written by the Vault Agent)
3. NS_USER or NS_PASSWORD (from netscaler.adcCredentialSecret)

Prerequisites

• The prerequisites for deploying NetScaler Kubernetes Gateway Controller. For more information, see Deploy NetScaler Kubernetes Gateway Controller.
• helm, plus the NetScaler Helm charts: helm repo add netscaler https://netscaler.github.io/netscaler-helm-charts/.
• The NetScaler management IP address and a NetScaler system user for the controller. You store the user name and password of that user in Vault.
• A HashiCorp Vault server. If you do not have one, install it in the cluster in Step 0.

One-time Vault setup

Step 0: Install HashiCorp Vault (skip if you already have one)

1. Add the HashiCorp Helm repository and install Vault:
   helm repo add hashicorp https://helm.releases.hashicorp.com
helm repo update
helm install vault hashicorp/vault -n vault --create-namespace
   Note: For a quick proof of concept, you can add --set "server.dev.enabled=true". Dev mode keeps all data in memory, starts unsealed, and uses the root token root. It is not secure, so use it only with test credentials, never in production. For production, leave it out and complete item 2.

2. Production only: initialize and unseal Vault once, and save the unseal keys and the root token somewhere safe:
   kubectl -n vault exec -it vault-0 -- vault operator init
# run the following once per unseal key that `init` printed (default: 3 of 5);
# it asks for the key without showing it
kubectl -n vault exec -it vault-0 -- vault operator unseal
3. Open a shell in the Vault pod, log in, and enable the KV v2 engine at path secret/ (dev mode already has it):
   kubectl -n vault exec -it vault-0 -- sh
export VAULT_ADDR=http://127.0.0.1:8200
vault login   # asks for the root token without showing it
vault secrets enable -path=secret -version=2 kv   # skip if "secret/" already exists
   For this in-cluster install (namespace vault), the address that the Vault Agent sidecar uses is http://vault.vault:8200. Otherwise, use the URL of your Vault server. You need it in Step 5.
   This install serves Vault over plain HTTP. For production, turn on TLS in the Vault Helm chart, use an https:// address, and give the Vault Agent the Vault CA certificate (mount the CA file and set the VAULT_CACERT environment variable).
   Important: The root token can do anything in Vault. Use it only for this initial setup. For later changes, such as the rotation in Step 7, use an administrator login with limited rights, and revoke the root token when you no longer need it. For more information, see Tokens in the Vault documentation.

Step 1: Create a service account for Kubernetes authentication

1. Create the service account vault-auth and bind it to the system:auth-delegator ClusterRole. Save the following as vault-auth-service-account.yaml, replace <namespace> with your namespace, and apply it:
   apiVersion: v1
kind: ServiceAccount
metadata:
  name: vault-auth
  namespace: <namespace>
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: vault-auth-tokenreview
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:auth-delegator
subjects:
  - kind: ServiceAccount
    name: vault-auth
    namespace: <namespace>kubectl apply -f vault-auth-service-account.yaml
2. On Kubernetes 1.24 and later, a service account no longer gets a token secret automatically. Create one. Save the following as vault-auth-token.yaml:
   apiVersion: v1
kind: Secret
metadata:
  name: vault-auth-token
  annotations:
    kubernetes.io/service-account.name: vault-auth
type: kubernetes.io/service-account-token
   Apply it in the same namespace and export the token and the CA certificate:
   kubectl apply -n <namespace> -f vault-auth-token.yaml
export SA_JWT_TOKEN=$(kubectl get secret -n <namespace> vault-auth-token -o jsonpath="{.data.token}" | base64 --decode)
export SA_CA_CRT=$(kubectl get secret -n <namespace> vault-auth-token -o jsonpath="{.data['ca\.crt']}" | base64 --decode)
   Note:
   When Vault runs in the same cluster, you can leave out token_reviewer_jwt in Step 2. Vault then uses its own pod token to check tokens, and you do not store a token that stops working when its service account is deleted.

Step 2: Set up Kubernetes authentication on Vault

1. Enable Kubernetes authentication. If it is already enabled, Vault reports that the path is already in use; skip this command.
   vault auth enable kubernetes
2. Configure how Vault reaches the cluster:
   vault write auth/kubernetes/config

token_reviewer_jwt="$SA_JWT_TOKEN"

kubernetes_host="https://<K8S_CLUSTER_URL>:<API_SERVER_PORT>"

kubernetes_ca_cert="$SA_CA_CRT"

Step 3: Store the credentials and allow the gateway controller to read them

1. Store the NetScaler credentials of the gateway controller:
   vault kv put secret/netscaler/gateway username='<NetScaler user name>' password='<password>'
   Note:
   • A KV secret does not rotate by itself. You rotate it in Step 7.
   • A password that you type in a command stays in your shell history. To avoid that, first read the password into a variable without showing it (for example, read -rs PASSWORD in bash), and then use password="$PASSWORD" in the command.

2. Create a read-only policy for this secret. KV v2 stores data under secret/data/... and metadata under secret/metadata/..., so grant read on both.
   netscaler-gateway-kv-ro.hcl:
   path "secret/data/netscaler/gateway" {
  capabilities = ["read"]
}
path "secret/metadata/netscaler/gateway" {
  capabilities = ["read"]
}
   Then load it:
   vault policy write netscaler-gateway-kv-ro netscaler-gateway-kv-ro.hcl
3. Create a role that lets the gateway controller's service account use the policy. The Vault Agent sidecar logs in with the token of the service account that the controller pod runs as. This example uses the service account nsgw-vault, which you set with gatewayController.serviceAccount.name in Step 5. Replace <gateway-namespace> with the namespace of the gateway controller.
   vault write auth/kubernetes/role/netscaler-gateway

bound_service_account_names=nsgw-vault

bound_service_account_namespaces=<gateway-namespace>

policies=netscaler-gateway-kv-ro

ttl=24h
   Use a role name that no other controller uses. Writing a role with the same name replaces it, so another controller that uses that role can no longer log in to Vault.

Step 4: Create the Vault Agent configuration

Step 5: Deploy the gateway controller with Vault credentials

Step 6: Verify

• Credential hot-reload ENABLED
• Starting CredentialWatcher ... for <NSIP>
• The pod becomes Ready.

Step 7: Rotate the credentials

1. On the NetScaler, change the password of the user: set system user <user> <newpassword>.
2. Right after that, update the Vault secret to the same value: vault kv put secret/netscaler/gateway username='<user>' password='<newpassword>'.
3. Within 30 seconds, the Vault Agent picks up the change and rewrites the files. The controller validates the new credential and switches to it at its next check (NS_CRED_RELOAD_INTERVAL, default 30 seconds), so the switch takes about a minute. There is no restart, and traffic is not affected.
4. If the new value is wrong, the controller does not use it. It keeps the last-good credential and logs Credential hot-reload: probe FAILED. Because you changed the password on NetScaler in item 1, the last-good credential no longer works either, so the controller cannot log in to NetScaler. The controller keeps testing the new value, but after repeated failures it waits longer between tests, up to 10 minutes. Fix the value in Vault right away.
5. After you fix the value, the controller switches to it at its next test, which can take 10 minutes or more.

Metrics exporter

1. Create a service account for the exporter in the namespace of the gateway controller. Run the exporter in that namespace, so that it can use the ConfigMap from Step 4.
   kubectl create serviceaccount <exporter-service-account> -n <gateway-namespace>
2. Allow this service account in the Vault role. Run the vault write auth/kubernetes/role/netscaler-gateway command from Step 3 again with both service accounts: bound_service_account_names=<controller-service-account>,<exporter-service-account>. The controller's service account is nsgw-vault in the Step 5 example. The command replaces the role, so if you leave out the controller's service account, the gateway controller can no longer log in to Vault.
3. Add the following to the pod spec (spec.template.spec) of your exporter deployment:
   automountServiceAccountToken: false              # the exporter does not need a token; the init container mounts its own
serviceAccountName: <exporter-service-account>   # the service account that you allowed in the Vault role
initContainers:
  - name: vault-agent-init
    image: hashicorp/vault:1.21   # run as a Vault Agent (a Vault client), not a Vault server
    args: ["agent", "-config=/etc/vault/vault-agent-config.hcl", "-exit-after-auth"]   # write the files once, then exit
    env:
      - name: VAULT_ADDR
        value: "https://<VAULT URL>:8200"
      - name: SKIP_SETCAP      # needed when the container does not run as root, for example on OpenShift
        value: "true"
    volumeMounts:
      - { name: vault-config, mountPath: /etc/vault }
      - { name: vault-token,  mountPath: /home/vault }
      - { name: citrix-creds, mountPath: /etc/citrix }
      - { name: sa-token,     mountPath: /var/run/secrets/kubernetes.io/serviceaccount, readOnly: true }
containers:
  - name: exporter
    image: quay.io/netscaler/netscaler-adc-metrics-exporter:1.7.0
    # keep the args and other settings of your exporter, but remove any other mount at /mnt/nslogin, such as the nslogin secret
    volumeMounts:
      - { name: citrix-creds, mountPath: /mnt/nslogin, readOnly: true }
volumes:
  - name: citrix-creds
    emptyDir: { medium: Memory }
  - name: vault-config
    configMap: { name: gateway-vault-agent-config }
  - name: vault-token
    emptyDir: { medium: Memory }
  - name: sa-token
    projected:
      sources:
        - serviceAccountToken: { path: token, expirationSeconds: 3600, audience: vault }
        - configMap: { name: kube-root-ca.crt, items: [{ key: ca.crt, path: ca.crt }] }
        - downwardAPI: { items: [{ path: namespace, fieldRef: { fieldPath: metadata.namespace } }] }