NetScaler ingress controller

Unified ingress

The Unified ingress deployment is used to perform load balancing of North-South traffic. North-South traffic is the traffic that enters or exits Kubernetes cluster, that is, the traffic that flows between the client and the front-end microservices. In this deployment, NetScaler Ingress Controller automates the configuration of NetScaler, which load balances traffic to microservices in the Kubernetes environment. NetScaler Ingress Controller runs as a pod that monitors the Kubernetes API server and configures NetScaler MPX or NetScaler VPX.

Deploy NetScaler Ingress Controller as a standalone pod in the Kubernetes cluster

Use the netscaler-ingress-controller Helm chart to deploy NetScaler Ingress Controller as a pod in your Kubernetes cluster.

The installation involves NetScaler Ingress Controller pod creation, cluster role creation, cluster role bindings, and service account creation.

When you deploy NSIC using the Helm chart, you can use a values.yaml file to specify the values of the configurable parameters instead of providing each parameter as an argument.

Note:

NSIC can also work with RBAC roles.

Prerequisites

  • Determine the NS_IP (NetScaler IP) address using which NetScaler Ingress Controller communicates with NetScaler. The IP address might be any one of the following IP addresses depending on the type of NetScaler deployment:

    • NSIP (for standalone appliances) - The management IP address of a standalone NetScaler appliance. For more information, see IP Addressing in NetScaler.

    • SNIP (for appliances in High Availability mode) - The subnet IP address. For more information, see IP Addressing in NetScaler.

    • CLIP (for appliances in Cluster mode) - The cluster management IP (CLIP) address for a cluster NetScaler deployment. For more information, see IP addressing for a cluster.

  • The user name and password of a system user account of NetScaler MPX or NetScaler VPX used as the Ingress device. NetScaler needs to have a system user account (non-default) with certain privileges so that the NetScaler Ingress Controller can configure NetScaler MPX or NetScaler VPX. For instructions to create the system user account on NetScaler, see Create System User Account for NetScaler Ingress Controller in NetScaler.

    You can directly pass the user name and password or use Kubernetes secrets. If you want to use Kubernetes secrets, create a secret for the user name and password using the following command:

     kubectl create secret  generic nslogin --from-literal=username=<username> --from-literal=password=<password>
    
  • Version 1.21 and later in the Kubernetes environment.

  • Version 4.8 and later in the OpenShift platform.

  • Ensure that you have installed Helm version 3.x or later. To install Helm, see here.

Create a system user account for NetScaler Ingress Controller in NetScaler

NetScaler Ingress Controller configures NetScaler using a system user account of NetScaler. The system user account should have certain privileges so that NetScaler Ingress Controller has permission to configure the following on NetScaler:

  • Add, delete, or view content switching (CS) virtual server
  • Configure CS policies and actions
  • Configure Load Balancing (LB) virtual server
  • Configure service groups
  • Configure SSL certkeys
  • Configure routes
  • Configure user monitors
  • Add system file (for uploading SSL certkeys from Kubernetes)
  • Configure virtual IP address (VIP)
  • Check the status of the NetScaler appliance

To create the system user account, perform the following:

  1. Log on to NetScaler. Perform the following steps:
    1. Use an SSH client, such as PuTTy, to open an SSH connection to NetScaler.

    2. Log on to NetScaler using the administrator credentials.

  2. Create the system user account using the following command:

    add system user <username> <password>
    

    For example:

    add system user cic mypassword
    
  3. Create a policy to provide required permissions to the system user account. Use the following command:

    add cmdpolicy cic-policy ALLOW '^(\?!shell)(\?!sftp)(\?!scp)(\?!batch)(\?!source)(\?!.*superuser)(\?!.*nsroot)(\?!install)(\?!show\s+system\s+(user|cmdPolicy|file))(\?!(set|add|rm|create|export|kill)\s+system)(\?!(unbind|bind)\s+system\s+(user|group))(\?!diff\s+ns\s+config)(\?!(set|unset|add|rm|bind|unbind|switch)\s+ns\s+partition).*|(^install\s*(wi|wf))|(^\S+\s+system\s+file)^(\?!shell)(\?!sftp)(\?!scp)(\?!batch)(\?!source)(\?!.*superuser)(\?!.*nsroot)(\?!install)(\?!show\s+system\s+(user|cmdPolicy|file))(\?!(set|add|rm|create|export|kill)\s+system)(\?!(unbind|bind)\s+system\s+(user|group))(\?!diff\s+ns\s+config)(\?!(set|unset|add|rm|bind|unbind|switch)\s+ns\s+partition).*|(^install\s*(wi|wf))|(^\S+\s+system\s+file)'

    Note:

    The system user account has privileges based on the command policy that you define.

    The command policy mentioned in step 3 is similar to the built-in sysAdmin command policy with additional permission to upload files.

    In the command policy specification provided, special characters which need to be escaped are already omitted to easily copy-paste into the NetScaler command line.

    For configuring the command policy from NetScaler configuration wizard (GUI), use the following command policy specification.

    ^(?!shell)(?!sftp)(?!scp)(?!batch)(?!source)(?!.*superuser)(?!.*nsroot)(?!install)(?!show\s+system\s+(user|cmdPolicy|file))(?!(set|add|rm|create|export|kill)\s+system)(?!(unbind|bind)\s+system\s+(user|group))(?!diff\s+ns\s+config)(?!(set|unset|add|rm|bind|unbind|switch)\s+ns\s+partition).*|(^install\s*(wi|wf))|(^\S+\s+system\s+file)^(?!shell)(?!sftp)(?!scp)(?!batch)(?!source)(?!.*superuser)(?!.*nsroot)(?!install)(?!show\s+system\s+(user|cmdPolicy|file))(?!(set|add|rm|create|export|kill)\s+system)(?!(unbind|bind)\s+system\s+(user|group))(?!diff\s+ns\s+config)(?!(set|unset|add|rm|bind|unbind|switch)\s+ns\s+partition).*|(^install\s*(wi|wf))|(^\S+\s+system\s+file)

  4. Bind the policy to the system user account using the following command:

    bind system user cic cic-policy 0
    

Deploy NetScaler Ingress Controller as a standalone pod in the Kubernetes cluster

To deploy NetScaler Ingress Controller as a standalone pod:

  1. Add the NetScaler Helm chart repository to your local registry using the following command.

    helm repo add netscaler https://netscaler.github.io/netscaler-helm-charts/   
    <!--NeedCopy-->
    
  2. Install NetScaler CPX with NetScaler Ingress Controller using the following command.

    helm install nsic netscaler/netscaler-ingress-controller --set nsIP=<NSIP>,license.accept=yes,adcCredentialSecret=<Secret-for-NetScaler-credentials>
    <!--NeedCopy-->
    

Note:

You can also use a values.yaml file in the helm install command to specify the values of the configurable parameters instead of providing each parameter as a command-line argument. For example, helm install nsic netscaler/netscaler-ingress-controller -f values.yaml.

The following table lists the mandatory and optional parameters that you can configure during NSIC installation:

Parameters Mandatory or Optional Default value Description
license.accept Mandatory no Set yes to accept the NSIC end user license agreement.
imageRegistry Optional quay.io Specify the NetScaler Ingress Controller image registry.
imageRepository Mandatory netscaler/netscaler-ingress-controller Specify NetScaler Ingress Controller image repository
imageTag Optional Latest version available The NetScaler Ingress Controller image tag.
pullPolicy Optional IfNotPresent The NSIC image pull policy.
imagePullSecrets Optional N/A Provide a list of Kubernetes secrets to be used for pulling the images from a private Docker registry or repository. For more information on how to create this secret please see Pull an Image from a Private Registry.
nameOverride Optional N/A String to partially override the deployment full name template (prepends the release name).
fullNameOverride Optional N/A String to fully override deployment full name template
resources Optional {} CPU/Memory resource requests/limits for NetScaler Ingress Controller container.
adcCredentialSecret Mandatory N/A The secret key to log on to NetScaler VPX or NetScaler MPX.
secretStore.enabled Optional False Set to “True” for deploying other Secret Provider classes.
secretStore.username Optional N/A if secretStore.enabled, username of NetScaler is fetched from the Secret Provider.
secretStore.password Optional N/A if secretStore.enabled, password of NetScaler is fetched from the Secret Provider.
nsIP Mandatory N/A Specify the IP address of NetScaler.
nsVIP Optional N/A Specify the virtual IP address on NetScaler.
nsSNIPS Optional N/A The list of subnet IP addresses on NetScaler, which are used to create PBR routes instead of static routes PBR support
nsPort Optional 443 The port used by NSIC to communicate with NetScaler. You can use port 80 for HTTP.
nsProtocol Optional HTTPS The protocol used by NSIC to communicate with NetScaler. You can also use HTTP on port 80.
nsEnableLabel Optional True Set to true for plotting service graph. Ensure analyticsConfig are set.
nitroReadTimeout Optional 20 The duration, in seconds, that NSIC has to wait to receive a response from NetScaler before terminating the connection.
logLevel Optional INFO The loglevel to control the logs generated by NSIC. The supported log levels are: CRITICAL, ERROR, WARNING, INFO, DEBUG and TRACE. For more information, see Logging.
jsonLog Optional false Set this argument to true if log messages are required in JSON format.
nsConfigDnsRec Optional false Enable or disable DNS address record addition in NetScaler through Ingress.
nsSvcLbDnsRec Optional false Enable or disable DNS address Record addition in NetScaler through Type Load Balancer service.
nsDnsNameserver Optional N/A To add DNS Nameservers in NetScaler.
optimizeEndpointBinding Optional false Enable or disable binding of back-end endpoints to servicegroup in a single API-call. Recommended when there are a large number of endpoints (pods) per application. Applicable only for NetScaler version 13.0-45.7 and later.
kubernetesURL Optional N/A The kube-apiserver url that NSIC uses to register the events. If the value is not specified, NSIC uses the internal kube-apiserver IP address.
clusterName Optional N/A The unique identifier of the kubernetes cluster on which the NSIC is deployed. Used in gslb-controller deployments.
ingressClass Optional N/A If multiple ingress load balancers are used to load balance different ingress resources, you can use this parameter to specify NSIC to configure NetScaler associated with a specific ingress class. For more information on Ingress class, see Ingress class support. For Kubernetes version 1.19 and later, this parameter creates an IngressClass object with the name specified here.
setAsDefaultIngressClass Optional False Set the IngressClass object as default ingress class. New Ingresses without an “ingressClassName” field specified are assigned the class specified in IngressClass. Applicable only for Kubernetes versions 1.19 and later.
serviceClass Optional N/A By default, ingress controller configures all TypeLB service on NetScaler. You can use this parameter to finetune this behavior by specifying NSIC to only configure TypeLB service with specific service class. For more information on service class, see Service class support.
nodeWatch Optional false Use the argument if you want to automatically configure a network route from NetScaler VPX or NetScaler MPX to the pods in the Kubernetes cluster. For more information, see Automatically configure route on the NetScaler instance.
cncPbr Optional False Use this argument to inform NSIC that NetScaler Node Controller(NSNC) is configuring Policy Based Routes (PBR) on NetScaler. For more information, see NSNC-PBR-SUPPORT
defaultSSLCertSecret Optional N/A Provide Kubernetes secret name that needs to be used as a default non-SNI certificate in NetScaler.
podIPsforServiceGroupMembers Optional False By default, NetScaler Ingress Controller adds NodeIP and NodePort as service group members while configuring type LoadBalancer services and NodePort services. This variable if set to True changes the behavior to add pod IP and pod port instead of nodeIP and nodePort. Users can set this to True if there is a route between NetScaler and K8s clusters internal pods either using feature-node-watch argument or using NetScaler Node Controller.
ignoreNodeExternalIP Optional False While adding NodeIP as service group members for type LoadBalancer services or NodePort services, NetScaler Ingress Controller has a selection criteria where it chooses Node ExternalIP if available and Node InternalIP, if Node ExternalIP is not present. But some users may want to use Node InternalIP over Node ExternalIP even if Node ExternalIP is present. If this variable is set to True, then it prioritizes the Node Internal IP to be used for service group members even if node ExternalIP is present.
nsHTTP2ServerSide Optional OFF Set this argument to ON for enabling HTTP2 for NetScaler service group configurations.
nsCookieVersion Optional 0 Specify the persistence cookie version (0 or 1).
profileSslFrontend Optional N/A Specify the front-end SSL profile. For more information, see Configuration using FRONTEND_SSL_PROFILE
profileTcpFrontend Optional N/A Specify the front-end TCP profile. For more information, see Configuration using FRONTEND_TCP_PROFILE
profileHttpFrontend Optional N/A Specify the front-end HTTP profile. For more information, see Configuration using FRONTEND_HTTP_PROFILE
ipam Optional False Set this argument if you want to use the IPAM controller to automatically allocate an IP address to the service of type LoadBalancer.
disableAPIServerCertVerify Optional False Set this parameter to True for disabling API Server certificate verification.
logProxy Optional N/A Provide Elasticsearch or Kafka or Zipkin endpoint for NetScaler observability exporter.
entityPrefix Optional k8s The prefix for the resources on NetScaler VPX or NetScaler MPX.
updateIngressStatus Optional True Set this argument if Status.LoadBalancer.Ingress field of the Ingress resources managed by NetScaler Ingress Controller needs to be updated with allocated IP addresses. For more information see this.
routeLabels Optional N/A Use this parameter to provide the route labels selectors to be used by NetScaler Ingress Controller for routeSharding in OpenShift cluster.
namespaceLabels Optional N/A Use this parameter to provide the namespace labels selectors to be used by NetScaler Ingress Controller for routeSharding in OpenShift cluster.
podAnnotations Optional N/A Map of annotations to add to the pods.
affinity Optional N/A Affinity labels for pod assignment.
exporter.required Optional false Use the parameter, if you want to run the Exporter for NetScaler Stats along with NSIC to pull metrics for NetScaler VPX or NetScaler MPX.
exporter.imageRegistry Optional quay.io The Exporter for NetScaler Stats image registry.
exporter.imageRepository Optional citrix/citrix-adc-metrics-exporter The Exporter for NetScaler Stats image repository.
exporter.imageTag Optional 1.4.9 The Exporter for NetScaler Stats image tag.
exporter.pullPolicy Optional IfNotPresent The Exporter image pull policy.
exporter.ports.containerPort Optional 8888 The Exporter container port.
exporter.resources Optional {} CPU/Memory resource requests/limits for Metrics exporter container.
exporter.extraVolumeMounts Optional [] Specify the additional VolumeMounts to be mounted in Exporter container. Specify the volumes in extraVolumes.
exporter.serviceMonitorExtraLabels Optional   Specify the extra labels for the service monitor when NetScaler-adc-metrics-exporter is enabled.
openshift Optional false Set this argument if the OpenShift environment is being used.
disableOpenshiftRoutes Optional false By default, OpenShift routes are processed in the OpenShift environment. This variable can be used to disable Ingress controller processing the OpenShift routes.
nodeSelector.key Optional N/A Specify the node label key to be used for the nodeSelector option in NSIC deployment.
nodeSelector.value Optional N/A Specify the node label value to be used for the nodeSelector option in NSIC deployment.
tolerations Optional N/A Specify the tolerations for the NSIC deployment.
crds.install Optional False Unset this argument if you don’t want to installCustomResourceDefinitions which are consumed by NSIC.
crds.retainOnDelete Optional false Set this argument if you want to retain CustomResourceDefinitions even after uninstalling NSIC. This avoids data-loss of Custom Resource Objects created before uninstallation.
analyticsConfig.required Mandatory false Set this parameter to true if you want to configure NetScaler to send metrics and transaction records to analytics.
analyticsConfig.distributedTracing.enable Optional false Set this value to true to enable OpenTracing in NetScaler.
analyticsConfig.distributedTracing.samplingrate Optional 100 Specify the OpenTracing sampling rate in percentage.
analyticsConfig.endpoint.server Optional N/A Set this value as the IP address or DNS address of the analytics server.
analyticsConfig.endpoint.service Optional N/A Set this value as the IP address or service name with namespace of the analytics service deployed in k8s environment. Format: namespace/servicename
analyticsConfig.timeseries.port Optional 30002 Specify the port used to expose analytics service outside the cluster for timeseries endpoint.
analyticsConfig.timeseries.metrics.enable Optional False Set this value to true to enable sending metrics from NetScaler.
analyticsConfig.timeseries.metrics.mode Optional avro Specify the mode of metric endpoint.
analyticsConfig.timeseries.metrics.exportFrequency Optional 30 Specify the time interval for exporting time-series data. Possible values range from 30 to 300 seconds.
analyticsConfig.timeseries.metrics.schemaFile Optional schema.json Specifies the name of a schema file with the required NetScaler counters to be added and configured for metricscollector to export. A reference schema file reference_schema.json with all the supported counters is also available under the path /var/metrics_conf/. This schema file can be used as a reference to build a custom list of counters.
analyticsConfig.timeseries.metrics.enableNativeScrape Optional false Set this value to true for native export of metrics.
analyticsConfig.timeseries.auditlogs.enable Optional false Set this value to true to export audit log data from NetScaler.
analyticsConfig.timeseries.events.enable Optional false Set this value to true to export events from NetScaler.
analyticsConfig.transactions.enable Optional false Set this value to true to export transactions from NetScaler.
analyticsConfig.transactions.port Optional 30001 Specify the port used to expose analytics service outside the cluster for transaction endpoint.
nsLbHashAlgo.required Optional false Set this value to specify the LB consistent hashing algorithm.
nsLbHashAlgo.hashFingers Optional 256 Specify the number of fingers to be used for hashing algorithm. Possible values are from 1 to 1024. Default value is 256.
nsLbHashAlgo.hashAlgorithm Optional ‘default’ Specifies the supported algorithm. Supported algorithms are “default”, “jarh”, “prac”. Default value is ‘default’.
extraVolumeMounts Optional [] Specify the additional VolumeMounts to be mounted in an NSIC container.
extraVolumes Optional [] Specify the additional volumes for additional volumeMounts.
rbacRole Optional false To deploy NSIC with RBAC role, set rbacRole=true; by default, NSIC gets installed with RBAC ClusterRole (rbacRole=false).

Warning:

Do not create NSIC pod replicas in the Kubernetes cluster. When you create replicas of an NSIC pod by specifying replicatset>1, an unexpected behavior is observed with NSIC/VPX because the same APP_PREFIX is shared among the NSIC PODs and each instance tries to configure the same resources.

Route addition in NetScaler MPX or NetScaler VPX

For seamless functioning of services deployed in the Kubernetes cluster, NetScaler should be able to reach the underlying overlay network over which the pods are running. The feature-node-watch argument of NetScaler Ingress Controller can be used for automatic route configuration on NetScaler towards the pod network. Refer to Static Routing for more information. By default, feature-node-watch is set to false; set to true to enable automatic route configuration.

If your deployment uses a single NetScaler to load balance between multiple k8s clusters, there is a possibility of CNI subnets to overlap, causing the static routing to fail due to route conflicts. In such deployments, Policy Based Routing (PBR) can be used. PBR requires you to provide one or more subnet IP addresses unique for each kubernetes cluster either using an environment variable or ConfigMap, see PBR Support.

Use the following command to provide subnet IP addresses (SNIPs) to configure Policy Based Routes (PBR) on NetScaler.

helm install my-release netscaler/netscaler-ingress-controller --set nsIP=<NSIP>,license.accept=yes,adcCredentialSecret=<Secret-for-NetScaler-credentials>,nsSNIPS='[<NS_SNIP1>\, <NS_SNIP2>\, ...]'
<!--NeedCopy-->

Note:

PBR can also be achieved by deploying NetScaler Node Controller. Netscaler Node Controller, by default, adds static routes while creating the VXLAN tunnel. To use Policy Based Routing (PBR) to avoid static route clash, both NetScaler Node Controller and NetScaler Ingress Controller have to work in conjunction and have to be started with specific arguments. For more information, refer Configure PBR using the NetScaler node controller.

Use the nsncPbr=<True/False> parameter in the helm install command of NSIC to inform NSIC that NetScaler Node Controller is configuring Policy Based Routes (PBR) on NetScaler.

helm install my-release netscaler/netscaler-ingress-controller --set nsIP=<NSIP>,license.accept=yes,adcCredentialSecret=<Secret-for-NetScaler-credentials>,clusterName=<unique-cluster-identifier>,nsncPbr=<True/False>
<!--NeedCopy-->
Unified ingress