Unified Ingress
The Unified Ingress deployment is used to perform load balancing of North-South traffic. North-South traffic is the traffic that enters or exits Kubernetes cluster, that is, the traffic that flows between the client and the front-end microservices. In this deployment, NetScaler Ingress Controller automates the configuration of NetScaler, which load balances traffic to microservices in the Kubernetes environment. NetScaler Ingress Controller runs as a pod that monitors the Kubernetes API server and configures NetScaler MPX or NetScaler VPX.
Deploy NetScaler Ingress Controller as a standalone pod in the Kubernetes cluster
Use the netscaler-ingress-controller Helm chart to deploy NetScaler Ingress Controller as a pod in your Kubernetes cluster.
The installation involves NetScaler Ingress Controller pod creation, cluster role creation, cluster role bindings, and service account creation.
When you deploy NSIC using the Helm chart, you can use a values.yaml file to specify the values of the configurable parameters instead of providing each parameter as an argument.
Note:
NSIC can also work with RBAC roles.
Prerequisites
-
Determine the NS_IP (NetScaler IP) address using which NetScaler Ingress Controller communicates with NetScaler. The IP address might be any one of the following IP addresses depending on the type of NetScaler deployment:
-
NSIP (for standalone appliances) - The management IP address of a standalone NetScaler appliance. For more information, see IP Addressing in NetScaler.
-
SNIP (for appliances in High Availability mode) - The subnet IP address. For more information, see IP Addressing in NetScaler.
-
CLIP (for appliances in Cluster mode) - The cluster management IP (CLIP) address for a cluster NetScaler deployment. For more information, see IP addressing for a cluster.
-
-
The user name and password of a system user account of NetScaler MPX or NetScaler VPX used as the Ingress device. NetScaler needs to have a system user account (non-default) with certain privileges so that the NetScaler Ingress Controller can configure NetScaler MPX or NetScaler VPX. For instructions to create the system user account on NetScaler, see Create System User Account for NetScaler Ingress Controller in NetScaler.
You can directly pass the user name and password or use Kubernetes secrets. If you want to use Kubernetes secrets, create a secret for the user name and password using the following command:
kubectl create secret generic nslogin --from-literal=username=<username> --from-literal=password=<password> -
Version 1.21 and later in the Kubernetes environment.
-
Version 4.8 and later in the OpenShift platform.
-
Ensure that you have installed Helm version 3.x or later. To install Helm, see here.
Create a system user account for NetScaler Ingress Controller in NetScaler
NetScaler Ingress Controller configures NetScaler using a system user account of NetScaler. The system user account should have certain privileges so that NetScaler Ingress Controller has permission to configure the following on NetScaler:
- Add, delete, or view content switching (CS) virtual server
- Configure CS policies and actions
- Configure Load Balancing (LB) virtual server
- Configure service groups
- Configure SSL certkeys
- Configure routes
- Configure user monitors
- Add system file (for uploading SSL certkeys from Kubernetes)
- Configure virtual IP address (VIP)
- Check the status of the NetScaler appliance
To create the system user account, perform the following:
- Log on to NetScaler. Perform the following steps:
-
Use an SSH client, such as PuTTy, to open an SSH connection to NetScaler.
-
Log on to NetScaler using the administrator credentials.
-
-
Create the system user account using the following command:
add system user <username> <password>For example:
add system user cic mypassword -
Create a policy to provide required permissions to the system user account. Use the following command:
add cmdpolicy cic-policy ALLOW '^(\?!shell)(\?!sftp)(\?!scp)(\?!batch)(\?!source)(\?!.*superuser)(\?!.*nsroot)(\?!install)(\?!show\s+system\s+(user|cmdPolicy|file))(\?!(set|add|rm|create|export|kill)\s+system)(\?!(unbind|bind)\s+system\s+(user|group))(\?!diff\s+ns\s+config)(\?!(set|unset|add|rm|bind|unbind|switch)\s+ns\s+partition).*|(^install\s*(wi|wf))|(^\S+\s+system\s+file)^(\?!shell)(\?!sftp)(\?!scp)(\?!batch)(\?!source)(\?!.*superuser)(\?!.*nsroot)(\?!install)(\?!show\s+system\s+(user|cmdPolicy|file))(\?!(set|add|rm|create|export|kill)\s+system)(\?!(unbind|bind)\s+system\s+(user|group))(\?!diff\s+ns\s+config)(\?!(set|unset|add|rm|bind|unbind|switch)\s+ns\s+partition).*|(^install\s*(wi|wf))|(^\S+\s+system\s+file)'Note:
The system user account has privileges based on the command policy that you define.
The command policy mentioned in step 3 is similar to the built-in
sysAdmincommand policy with additional permission to upload files.In the command policy specification provided, special characters which need to be escaped are already omitted to easily copy-paste into the NetScaler command line.
For configuring the command policy from NetScaler configuration wizard (GUI), use the following command policy specification.
^(?!shell)(?!sftp)(?!scp)(?!batch)(?!source)(?!.*superuser)(?!.*nsroot)(?!install)(?!show\s+system\s+(user|cmdPolicy|file))(?!(set|add|rm|create|export|kill)\s+system)(?!(unbind|bind)\s+system\s+(user|group))(?!diff\s+ns\s+config)(?!(set|unset|add|rm|bind|unbind|switch)\s+ns\s+partition).*|(^install\s*(wi|wf))|(^\S+\s+system\s+file)^(?!shell)(?!sftp)(?!scp)(?!batch)(?!source)(?!.*superuser)(?!.*nsroot)(?!install)(?!show\s+system\s+(user|cmdPolicy|file))(?!(set|add|rm|create|export|kill)\s+system)(?!(unbind|bind)\s+system\s+(user|group))(?!diff\s+ns\s+config)(?!(set|unset|add|rm|bind|unbind|switch)\s+ns\s+partition).*|(^install\s*(wi|wf))|(^\S+\s+system\s+file) -
Bind the policy to the system user account using the following command:
bind system user cic cic-policy 0
Deploy NetScaler Ingress Controller as a standalone pod in the Kubernetes cluster
To deploy NetScaler Ingress Controller as a standalone pod:
-
Add the NetScaler Helm chart repository to your local registry using the following command.
helm repo add netscaler https://netscaler.github.io/netscaler-helm-charts/ <!--NeedCopy--> -
Install NetScaler CPX with NetScaler Ingress Controller using the following command.
helm install nsic netscaler/netscaler-ingress-controller --set nsIP=<NSIP>,license.accept=yes,adcCredentialSecret=<Secret-for-NetScaler-credentials> <!--NeedCopy-->
Note:
You can also use a
values.yamlfile in thehelm installcommand to specify the values of the configurable parameters instead of providing each parameter as a command-line argument. For example,helm install nsic netscaler/netscaler-ingress-controller -f values.yaml.
The following table lists the mandatory and optional parameters that you can configure during NSIC installation:
| Parameters | Mandatory or Optional | Default value | Description |
|---|---|---|---|
| license.accept | Mandatory | no | Set yes to accept the NSIC end user license agreement. |
| imageRegistry | Optional | quay.io |
Specify the NetScaler Ingress Controller image registry. |
| imageRepository | Mandatory | netscaler/netscaler-ingress-controller |
Specify NetScaler Ingress Controller image repository |
| imageTag | Optional | Latest version available |
The NetScaler Ingress Controller image tag. |
| pullPolicy | Optional | IfNotPresent | The NSIC image pull policy. |
| imagePullSecrets | Optional | N/A | Provide a list of Kubernetes secrets to be used for pulling the images from a private Docker registry or repository. For more information on how to create this secret please see Pull an Image from a Private Registry. |
| nameOverride | Optional | N/A | String to partially override the deployment full name template (prepends the release name). |
| fullNameOverride | Optional | N/A | String to fully override deployment full name template |
| resources | Optional | {} | CPU/Memory resource requests/limits for NetScaler Ingress Controller container. |
| adcCredentialSecret | Mandatory | N/A | The secret key to log on to NetScaler VPX or NetScaler MPX. |
| secretStore.enabled | Optional | False | Set to “True” for deploying other Secret Provider classes. |
| secretStore.username | Optional | N/A | if secretStore.enabled, username of NetScaler is fetched from the Secret Provider. |
| secretStore.password | Optional | N/A | if secretStore.enabled, password of NetScaler is fetched from the Secret Provider. |
| nsIP | Mandatory | N/A | Specify the IP address of NetScaler. |
| nsVIP | Optional | N/A | Specify the virtual IP address on NetScaler. |
| nsSNIPS | Optional | N/A | The list of subnet IP addresses on NetScaler, which are used to create PBR routes instead of static routes PBR support |
| nsPort | Optional | 443 | The port used by NSIC to communicate with NetScaler. You can use port 80 for HTTP. |
| nsProtocol | Optional | HTTPS | The protocol used by NSIC to communicate with NetScaler. You can also use HTTP on port 80. |
| nsEnableLabel | Optional | True | Set to true for plotting service graph. Ensure analyticsConfig are set. |
| nitroReadTimeout | Optional | 20 | The duration, in seconds, that NSIC has to wait to receive a response from NetScaler before terminating the connection. |
| logLevel | Optional | INFO | The loglevel to control the logs generated by NSIC. The supported log levels are: CRITICAL, ERROR, WARNING, INFO, DEBUG and TRACE. For more information, see Logging. |
| jsonLog | Optional | false | Set this argument to true if log messages are required in JSON format. |
| nsConfigDnsRec | Optional | false | Enable or disable DNS address record addition in NetScaler through Ingress. |
| nsSvcLbDnsRec | Optional | false | Enable or disable DNS address Record addition in NetScaler through Type Load Balancer service. |
| nsDnsNameserver | Optional | N/A | To add DNS Nameservers in NetScaler. |
| optimizeEndpointBinding | Optional | false | Enable or disable binding of back-end endpoints to servicegroup in a single API-call. Recommended when there are a large number of endpoints (pods) per application. Applicable only for NetScaler version 13.0-45.7 and later. |
| kubernetesURL | Optional | N/A | The kube-apiserver url that NSIC uses to register the events. If the value is not specified, NSIC uses the internal kube-apiserver IP address. |
| clusterName | Optional | N/A | The unique identifier of the kubernetes cluster on which the NSIC is deployed. Used in gslb-controller deployments. |
| ingressClass | Optional | N/A | If multiple ingress load balancers are used to load balance different ingress resources, you can use this parameter to specify NSIC to configure NetScaler associated with a specific ingress class. For more information on Ingress class, see Ingress class support. For Kubernetes version 1.19 and later, this parameter creates an IngressClass object with the name specified here. |
| setAsDefaultIngressClass | Optional | False | Set the IngressClass object as default ingress class. New Ingresses without an “ingressClassName” field specified are assigned the class specified in IngressClass. Applicable only for Kubernetes versions 1.19 and later. |
| serviceClass | Optional | N/A | By default, ingress controller configures all TypeLB service on NetScaler. You can use this parameter to finetune this behavior by specifying NSIC to only configure TypeLB service with specific service class. For more information on service class, see Service class support. |
| nodeWatch | Optional | false | Use the argument if you want to automatically configure a network route from NetScaler VPX or NetScaler MPX to the pods in the Kubernetes cluster. For more information, see Automatically configure route on the NetScaler instance. |
| cncPbr | Optional | False | Use this argument to inform NSIC that NetScaler Node Controller(NSNC) is configuring Policy Based Routes (PBR) on NetScaler. For more information, see NSNC-PBR-SUPPORT |
| defaultSSLCertSecret | Optional | N/A | Provide Kubernetes secret name that needs to be used as a default non-SNI certificate in NetScaler. |
| podIPsforServiceGroupMembers | Optional | False | By default, NetScaler Ingress Controller adds NodeIP and NodePort as service group members while configuring type LoadBalancer services and NodePort services. This variable if set to True changes the behavior to add pod IP and pod port instead of nodeIP and nodePort. Users can set this to True if there is a route between NetScaler and K8s clusters internal pods either using feature-node-watch argument or using NetScaler Node Controller. |
| ignoreNodeExternalIP | Optional | False | While adding NodeIP as service group members for type LoadBalancer services or NodePort services, NetScaler Ingress Controller has a selection criteria where it chooses Node ExternalIP if available and Node InternalIP, if Node ExternalIP is not present. But some users may want to use Node InternalIP over Node ExternalIP even if Node ExternalIP is present. If this variable is set to True, then it prioritizes the Node Internal IP to be used for service group members even if node ExternalIP is present. |
| nsHTTP2ServerSide | Optional | OFF | Set this argument to ON for enabling HTTP2 for NetScaler service group configurations. |
| nsCookieVersion | Optional | 0 | Specify the persistence cookie version (0 or 1). |
| profileSslFrontend | Optional | N/A | Specify the front-end SSL profile. For more information, see Configuration using FRONTEND_SSL_PROFILE |
| profileTcpFrontend | Optional | N/A | Specify the front-end TCP profile. For more information, see Configuration using FRONTEND_TCP_PROFILE |
| profileHttpFrontend | Optional | N/A | Specify the front-end HTTP profile. For more information, see Configuration using FRONTEND_HTTP_PROFILE |
| ipam | Optional | False | Set this argument if you want to use the IPAM controller to automatically allocate an IP address to the service of type LoadBalancer. |
| disableAPIServerCertVerify | Optional | False | Set this parameter to True for disabling API Server certificate verification. |
| logProxy | Optional | N/A | Provide Elasticsearch or Kafka or Zipkin endpoint for NetScaler observability exporter. |
| entityPrefix | Optional | k8s | The prefix for the resources on NetScaler VPX or NetScaler MPX. |
| updateIngressStatus | Optional | True | Set this argument if Status.LoadBalancer.Ingress field of the Ingress resources managed by NetScaler Ingress Controller needs to be updated with allocated IP addresses. For more information see this. |
| routeLabels | Optional | N/A | Use this parameter to provide the route labels selectors to be used by NetScaler Ingress Controller for routeSharding in OpenShift cluster. |
| namespaceLabels | Optional | N/A | Use this parameter to provide the namespace labels selectors to be used by NetScaler Ingress Controller for routeSharding in OpenShift cluster. |
| podAnnotations | Optional | N/A | Map of annotations to add to the pods. |
| affinity | Optional | N/A | Affinity labels for pod assignment. |
| exporter.required | Optional | false | Use the parameter, if you want to run the Exporter for NetScaler Stats along with NSIC to pull metrics for NetScaler VPX or NetScaler MPX. |
| exporter.imageRegistry | Optional | quay.io |
The Exporter for NetScaler Stats image registry. |
| exporter.imageRepository | Optional | citrix/citrix-adc-metrics-exporter |
The Exporter for NetScaler Stats image repository. |
| exporter.imageTag | Optional | 1.4.9 |
The Exporter for NetScaler Stats image tag. |
| exporter.pullPolicy | Optional | IfNotPresent | The Exporter image pull policy. |
| exporter.ports.containerPort | Optional | 8888 | The Exporter container port. |
| exporter.resources | Optional | {} | CPU/Memory resource requests/limits for Metrics exporter container. |
| exporter.extraVolumeMounts | Optional | [] | Specify the additional VolumeMounts to be mounted in Exporter container. Specify the volumes in extraVolumes. |
| exporter.serviceMonitorExtraLabels | Optional | Specify the extra labels for the service monitor when NetScaler-adc-metrics-exporter is enabled. | |
| openshift | Optional | false | Set this argument if the OpenShift environment is being used. |
| disableOpenshiftRoutes | Optional | false | By default, OpenShift routes are processed in the OpenShift environment. This variable can be used to disable Ingress controller processing the OpenShift routes. |
| nodeSelector.key | Optional | N/A | Specify the node label key to be used for the nodeSelector option in NSIC deployment. |
| nodeSelector.value | Optional | N/A | Specify the node label value to be used for the nodeSelector option in NSIC deployment. |
| tolerations | Optional | N/A | Specify the tolerations for the NSIC deployment. |
| crds.install | Optional | False | Unset this argument if you don’t want to installCustomResourceDefinitions which are consumed by NSIC. |
| crds.retainOnDelete | Optional | false | Set this argument if you want to retain CustomResourceDefinitions even after uninstalling NSIC. This avoids data-loss of Custom Resource Objects created before uninstallation. |
| analyticsConfig.required | Mandatory | false | Set this parameter to true if you want to configure NetScaler to send metrics and transaction records to analytics. |
| analyticsConfig.distributedTracing.enable | Optional | false | Set this value to true to enable OpenTracing in NetScaler. |
| analyticsConfig.distributedTracing.samplingrate | Optional | 100 | Specify the OpenTracing sampling rate in percentage. |
| analyticsConfig.endpoint.server | Optional | N/A | Set this value as the IP address or DNS address of the analytics server. |
| analyticsConfig.endpoint.service | Optional | N/A | Set this value as the IP address or service name with namespace of the analytics service deployed in k8s environment. Format: namespace/servicename |
| analyticsConfig.timeseries.port | Optional | 30002 | Specify the port used to expose analytics service outside the cluster for timeseries endpoint. |
| analyticsConfig.timeseries.metrics.enable | Optional | False | Set this value to true to enable sending metrics from NetScaler. |
| analyticsConfig.timeseries.metrics.mode | Optional | avro | Specify the mode of metric endpoint. |
| analyticsConfig.timeseries.metrics.exportFrequency | Optional | 30 | Specify the time interval for exporting time-series data. Possible values range from 30 to 300 seconds. |
| analyticsConfig.timeseries.metrics.schemaFile | Optional | schema.json | Specifies the name of a schema file with the required NetScaler counters to be added and configured for metricscollector to export. A reference schema file reference_schema.json with all the supported counters is also available under the path /var/metrics_conf/. This schema file can be used as a reference to build a custom list of counters. |
| analyticsConfig.timeseries.metrics.enableNativeScrape | Optional | false | Set this value to true for native export of metrics. |
| analyticsConfig.timeseries.auditlogs.enable | Optional | false | Set this value to true to export audit log data from NetScaler. |
| analyticsConfig.timeseries.events.enable | Optional | false | Set this value to true to export events from NetScaler. |
| analyticsConfig.transactions.enable | Optional | false | Set this value to true to export transactions from NetScaler. |
| analyticsConfig.transactions.port | Optional | 30001 | Specify the port used to expose analytics service outside the cluster for transaction endpoint. |
| nsLbHashAlgo.required | Optional | false | Set this value to specify the LB consistent hashing algorithm. |
| nsLbHashAlgo.hashFingers | Optional | 256 | Specify the number of fingers to be used for hashing algorithm. Possible values are from 1 to 1024. Default value is 256. |
| nsLbHashAlgo.hashAlgorithm | Optional | ‘default’ | Specifies the supported algorithm. Supported algorithms are “default”, “jarh”, “prac”. Default value is ‘default’. |
| extraVolumeMounts | Optional | [] | Specify the additional VolumeMounts to be mounted in an NSIC container. |
| extraVolumes | Optional | [] | Specify the additional volumes for additional volumeMounts. |
| rbacRole | Optional | false | To deploy NSIC with RBAC role, set rbacRole=true; by default, NSIC gets installed with RBAC ClusterRole (rbacRole=false). |
Route addition in NetScaler MPX or NetScaler VPX
For seamless functioning of services deployed in the Kubernetes cluster, NetScaler should be able to reach the underlying overlay network over which the pods are running. The feature-node-watch argument of NetScaler Ingress Controller can be used for automatic route configuration on NetScaler towards the pod network. Refer to Static Routing for more information. By default, feature-node-watch is set to false; set to true to enable automatic route configuration.
If your deployment uses a single NetScaler to load balance between multiple k8s clusters, there is a possibility of CNI subnets to overlap, causing the static routing to fail due to route conflicts. In such deployments, Policy Based Routing (PBR) can be used. PBR requires you to provide one or more subnet IP addresses unique for each kubernetes cluster either using an environment variable or ConfigMap, see PBR Support.
Use the following command to provide subnet IP addresses (SNIPs) to configure Policy Based Routes (PBR) on NetScaler.
helm install my-release netscaler/netscaler-ingress-controller --set nsIP=<NSIP>,license.accept=yes,adcCredentialSecret=<Secret-for-NetScaler-credentials>,nsSNIPS='[<NS_SNIP1>\, <NS_SNIP2>\, ...]'
<!--NeedCopy-->
Note:
PBR can also be achieved by deploying NetScaler Node Controller. Netscaler Node Controller, by default, adds static routes while creating the VXLAN tunnel. To use Policy Based Routing (PBR) to avoid static route clash, both NetScaler Node Controller and NetScaler Ingress Controller have to work in conjunction and have to be started with specific arguments. For more information, refer Configure PBR using the NetScaler node controller.
Use the nsncPbr=<True/False> parameter in the helm install command of NSIC to inform NSIC that NetScaler Node Controller is configuring Policy Based Routes (PBR) on NetScaler.
helm install my-release netscaler/netscaler-ingress-controller --set nsIP=<NSIP>,license.accept=yes,adcCredentialSecret=<Secret-for-NetScaler-credentials>,clusterName=<unique-cluster-identifier>,nsncPbr=<True/False>
<!--NeedCopy-->