NetScaler ingress controller

Dual-tier ingress

In a dual-tier deployment, NetScaler VPX or NetScaler MPX is deployed outside the Kubernetes cluster (tier-1) and NetScaler CPXs are deployed inside the Kubernetes cluster (tier-2).

NetScaler MPX or NetScaler VPX devices in tier-1 proxy the traffic (North-South) from the client to NetScaler CPXs in tier-2. The tier-2 NetScaler CPX then routes the traffic to the microservices in the Kubernetes cluster. NetScaler Ingress Controller deployed as a standalone pod configures the tier-1 NetScaler. And, the sidecar NetScaler Ingress Controller in one or more NetScaler CPX pods configures the associated NetScaler CPX in the same pod.

The dual-tier deployment can be set up on Kubernetes in a bare metal environment or on public clouds such as AWS, GCP, or Azure.

dual-tier

Dual-tier ingress deployment

Prerequisites

  • Create a Kubernetes cluster in cloud or on-premises. The Kubernetes cluster in the cloud can be a managed Kubernetes (for example: GKE, EKS, or AKS) or a custom created Kubernetes deployment.

Dual-tier Ingress deployment

  1. Deploy NetScaler MPX or NetScaler VPX on a multi-NIC deployment mode outside the Kubernetes cluster.

  2. Deploy a standalone NetScaler Ingress Controller to configure NetScaler VPX to direct the client requests to a CPX service in the Kubernetes cluster.

    For information on how to deploy a standalone NetScaler Ingress Controller, see Unified Ingress.

  3. In cloud deployments, enable MAC-Based Forwarding mode on the tier-1 NetScaler VPX. As NetScaler VPX is deployed in multi-NIC mode, it does not have the return route to reach the POD CNI network or the client network. Hence, you must enable MAC-Based Forwarding mode on the tier-1 NetScaler VPX to handle this scenario.

  4. Deploy a sample microservice by using the following command:

    ---
    kubectl apply -f - <<EOF
    apiVersion: apps/v1
    kind: Deployment  
    metadata:
      name: frontend-colddrinks
    spec:
      selector:
        matchLabels:
          app: frontend-colddrinks
      replicas: 2
      template:
        metadata:
          labels:
            app: frontend-colddrinks
        spec:
          containers:
            - name: frontend-colddrinks
              image: quay.io/sample-apps/colddrinks:v1
              ports:
                - name: colddrinks-80
                  containerPort: 80
    
    ---
    apiVersion: v1
    kind: Service
    metadata:
      name: frontend-colddrinks
    spec:
      ports:
        - name: colddrinks-80
          port: 80
          targetPort: 80
      selector:
        app: frontend-colddrinks
    
    ---
    null
    EOF
    <!--NeedCopy-->
    
  5. Deploy NetScaler CPX as tier-2 ingress with NetScaler Ingress Controller as a sidecar.

    1. Add the NetScaler Helm chart repository to your local registry by using the following command.

      helm repo add netscaler https://netscaler.github.io/netscaler-helm-charts/   
      <!--NeedCopy-->
      
    2. Install NetScaler CPX with NetScaler Ingress Controller by using the following command.

      helm install netscaler-cpx-with-ingress-controller netscaler/netscaler-cpx-with-ingress-controller --set license.accept=yes,serviceType.nodePort.enabled=True
      <!--NeedCopy-->
      

    Note:

    You can also use a values.yaml file in the helm install command to specify the values of the configurable parameters instead of providing each parameter as a command-line argument. For example, helm install netscaler-cpx-with-ingress-controller netscaler/netscaler-cpx-with-ingress-controller -f values.yaml.

    The following table lists the mandatory and optional parameters that you can configure during CPX with NSIC installation:

    Parameters Mandatory or Optional Default value Description
    license.accept Mandatory no Set yes to accept the NSIC end user license agreement.
    imageRegistry Optional quay.io Specify NetScaler CPX image registry.
    imageRepository Mandatory netscaler/netscaler-ingress-controller Specify NetScaler CPX image repository
    imageTag Optional Latest version available Specify NetScaler CPX image tag.
    pullPolicy Optional IfNotPresent Specify NetScaler CPX image pull policy.
    daemonSet Optional False Set this parameter to “true” if NetScaler CPX needs to be deployed as a DaemonSet.
    nsic.imageRegistry Mandatory quay.io Specify the NetScaler Ingress Controller image registry.
    nsic.imageRepository Mandatory netscaler/netscaler-k8s-ingress-controller Specify the NetScaler Ingress Controller image repository.
    nsic.imageTag Mandatory 1.41.5 Specify the NetScaler CPX image tag.
    nsic.pullPolicy Mandatory IfNotPresent Specify the NetScaler Ingress Controller image pull policy.
    nsic.required Mandatory true Specify whether the NSIC runs as a sidecar with NetScaler CPX.
    nsic.resources Optional {} CPU/memory resource requests/limits for NetScaler Ingress Controller container.
    nsic.rbacRole Optional false To deploy NSIC with RBAC Role, set rbacRole=true; by default NSIC gets installed with RBAC ClusterRole(rbacRole=false).
    nsic.prometheusCredentialSecret Optional N/A Specify the secret key required to create a read-only user for native export of metrics using Prometheus.
    imagePullSecrets Optional N/A Provide a list of Kubernetes secrets to be used for pulling the images from a private Docker registry or repository. For more information on how to create this secret, see Pull an Image from a Private Registry.
    nameOverride Optional N/A String to partially override the deployment full name template (prepends the release name).
    fullNameOverride Optional N/A String to fully override the deployment full name template with a string.
    resources Optional {} CPU/Memory resource requests/limits for NetScaler CPX container.
    nitroReadTimeout Optional 20 The duration, in seconds, that NSIC has to wait to receive a response from NetScaler before terminating the connection.
    logLevel Optional INFO The loglevel to control the logs generated by NSIC. The supported log levels are: CRITICAL, ERROR, WARNING, INFO, DEBUG and TRACE. For more information, see Logging.
    jsonLog Optional false Set this parameter to true if log messages are required in JSON format.
    nsConfigDnsRec Optional false Enable or disable DNS address record addition in NetScaler through Ingress.
    nsSvcLbDnsRec Optional false Enable or disable DNS address record addition in NetScaler through type LoadBalancer service.
    nsDnsNameserver Optional N/A To add DNS Nameservers in NetScaler.
    optimizeEndpointBinding Optional false Enable or disable binding of back-end endpoints to servicegroup in a single API-call. Recommended when there are a large number of endpoints (pods) per application. Applicable only for NetScaler version 13.0-45.7 and later.
    defaultSSLCertSecret Optional N/A Provide Kubernetes secret name that needs to be used as a default non-SNI certificate in NetScaler.
    defaultSSLSNICertSecret Optional N/A Provide Kubernetes secret name that needs to be used as a default SNI certificate in NetScaler.
    nsHTTP2ServerSide Optional OFF Set this argument to ON for enabling HTTP2 for NetScaler service group configurations.
    cpxLicenseAggregator Optional N/A IP/FQDN of the CPX License Aggregator if it is being used to license NetScaler CPX.
    nsCookieVersion Optional 0 Specify the persistence cookie version (0 or 1).
    profileSslFrontend Optional N/A Specify the front-end SSL profile. For more information, see Configuration using FRONTEND_SSL_PROFILE
    profileTcpFrontend Optional N/A Specify the front-end TCP profile. For more information, see Configuration using FRONTEND_TCP_PROFILE
    profileHttpFrontend Optional N/A Specify the front-end HTTP profile. For more information, see Configuration using FRONTEND_HTTP_PROFILE
    logProxy Optional N/A Provide Elasticsearch or Kafka or Zipkin endpoint for NetScaler Observability Exporter.
    nsProtocol Optional HTTPS The protocol used by NSIC to communicate with NetScaler. You can also use HTTP on port 80.
    nsEnableLabel Optional True Set to true for plotting service graph. Ensure analyticsConfig are set.
    cpxBgpRouter Optional false If set to true, this CPX is deployed as a daemonset in BGP controller mode wherein BGP advertisements are done for attracting external traffic to Kubernetes clusters.
    replicaCount Optional 1 Number of CPX-NSIC pods to be deployed. With cpxBgpRouter : true, replicaCount is 1 because CPX will be deployed as DaemonSet.
    nsIP Optional 192.168.1.2 NSIP used by CPX for internal communication when run in Host mode, that is, when cpxBgpRouter is set to true. A /24 internal network is created in this IP range which is used for internal communications within the network namespace.
    nsGateway Optional 192.168.1.1 Gateway used by CPX for internal communication when run in Host mode, that is, when cpxBgpRouter is set to true. If not specified, first IP in the nsIP network is used as a gateway. It must be in the same network as nsIP.
    bgpPort Optional 179 BGP port used by CPX for BGP advertisement if cpxBgpRouter is set to true.
    ingressIP Optional N/A External IP address to be used by ingress resources if not overridden by ingress.com/frontend-ip annotation in Ingress resources. This is also advertised to external routers when pxBgpRouter is set to true.
    entityPrefix Optional k8s The prefix for the resources on NetScaler CPX.
    ingressClass Optional N/A If multiple ingress load balancers are used to load balance different ingress resources, you can use this parameter to specify NSIC to configure NetScaler associated with a specific ingress class. For more information on Ingress class, see Ingress class support. For Kubernetes version 1.19 and later, this parameter creates an IngressClass object with the name specified here.
    setAsDefaultIngressClass Optional False Set the IngressClass object as default ingress class. New Ingresses without an “ingressClassName” field specified are assigned the class specified in IngressClass. Applicable only for Kubernetes versions 1.19 and later.
    updateIngressStatus Optional False Set this argument if you want to update ingress status of the ingress resources exposed through CPX. This is only applicable if the service type of CPX service is LoadBalancer.
    disableAPIServerCertVerify Optional False Set this parameter to True for disabling API Server certificate verification.
    openshift Optional false Set this parameter if the OpenShift environment is being used.
    disableOpenshiftRoutes Optional false By default, OpenShift routes are processed in the OpenShift environment. This parameter can be used to disable the ingress controller processing the OpenShift routes.
    routeLabels Optional proxy in () Use this parameter to provide the route-labels selectors to be used by NetScaler Ingress Controller for routeSharding in OpenShift cluster.
    namespaceLabels Optional N/A Use this parameter to provide the namespace-labels selectors to be used by NetScaler Ingress Controller for routeSharding in OpenShift cluster.
    sslCertManagedByAWS Optional False Set this argument if SSL certs used is managed by AWS while deploying NetScaler CPX in AWS.
    nodeSelector.key Optional N/A Specify the node label key to be used for the nodeSelector option in CPX-NSIC deployment.
    nodeSelector.value Optional N/A Specify the node label value to be used for the nodeSelector option in CPX-NSIC deployment.
    podAnnotations Optional N/A Map of annotations to add to the pods.
    affinity Optional N/A Affinity labels for pod assignment.
    tolerations Optional N/A Specify the tolerations for the CPX-NSIC deployment.
    serviceType.loadBalancer.enabled Optional False Set this parameter if you want servicetype of CPX service to be LoadBalancer.
    serviceType.nodePort.enabled Optional False Set this parameter if you want servicetype of CPX service to be NodePort.
    serviceType.nodePort.httpPort Optional N/A Specify the HTTP nodeport to be used for NodePort CPX service.
    serviceType.nodePort.httpsPort Optional N/A Specify the HTTPS nodeport to be used for NodePort CPX service.
    serviceAnnotations Optional N/A Dictionary of annotations to be used in CPX service. Key in this dictionary is the name of the annotation and Value is the required value of that annotation.
    serviceSpec.externalTrafficPolicy Optional Cluster Use this parameter to provide externalTrafficPolicy for CPX service of type LoadBalancer or NodePort. serviceType.loadBalancer.enabled or serviceType.nodePort.enabled should be set to true according to your use case for using this parameter.
    serviceSpec.loadBalancerIP Optional N/A Use this parameter to provide LoadBalancer IP to CPX service of type LoadBalancer. serviceType.loadBalancer.enabled should be set to true for using this parameter.
    serviceSpec.loadBalancerSourceRanges Optional N/A Provide the list of IP Address or range which should be allowed to access the Network Load Balancer. serviceType.loadBalancer.enabled should be set to true for using this parameter. For details, see Network Load Balancer support on AWS.
    servicePorts Optional N/A List of port. Each element in this list is a dictionary that contains information about the port. For example, see this.
    ADMSettings.licenseServerIP Optional N/A Provide the NetScaler Application Delivery Management (ADM) IP address to license NetScaler CPX. For more information, see Licensing.
    ADMSettings.licenseServerPort Optional 27000 NetScaler ADM port if non-default port is used.
    ADMSettings.ADMIP Optional N/A NetScaler Application Delivery Management (ADM) IP address.
    ADMSettings.loginSecret Optional N/A The secret key to login to the ADM. For information on how to create the secret keys, see Prerequisites.
    ADMSettings.bandWidthLicense Optional False Set to true if you want to use bandwidth based licensing for NetScaler CPX.
    ADMSettings.bandWidth Optional 1000 Desired bandwidth capacity to be set for NetScaler CPX in Mbps.
    ADMSettings.vCPULicense Optional N/A Set to true if you want to use vCPU based licensing for NetScaler CPX.
    ADMSettings.licenseEdition Optional PLATINUM License edition that can be Standard, Platinum and Enterprise . By default, Platinum is selected.
    ADMSettings.cpxCores Optional 1 Desired number of vCPU to be set for NetScaler CPX.
    exporter.required Optional false Use the parameter, if you want to run the Exporter for NetScaler Stats along with NSIC to pull metrics for NetScaler VPX or NetScaler MPX.
    exporter.imageRegistry Optional quay.io The Exporter for NetScaler Stats image registry.
    exporter.imageRepository Optional citrix/citrix-adc-metrics-exporter The Exporter for NetScaler Stats image repository.
    exporter.imageTag Optional 1.4.9 The Exporter for NetScaler Stats image tag.
    exporter.pullPolicy Optional IfNotPresent The Exporter image pull policy.
    exporter.resources Optional {} CPU/Memory resource requests/limits for Metrics exporter container.
    exporter.ports.containerPort Optional 8888 The Exporter container port.
    exporter.serviceMonitorExtraLabels Optional   Specify the extra labels for the service monitor when NetScaler-adc-metrics-exporter is enabled.
    analyticsConfig.required Mandatory false Set this parameter to true if you want to configure NetScaler to send metrics and transaction records to analytics.
    analyticsConfig.distributedTracing.enable Optional false Set this value to true to enable OpenTracing in NetScaler.
    analyticsConfig.distributedTracing.samplingrate Optional 100 Specify the OpenTracing sampling rate in percentage.
    analyticsConfig.endpoint.server Optional N/A Set this value as the IP address or DNS address of the analytics server.
    analyticsConfig.endpoint.service Optional N/A Set this value as the IP address or service name with namespace of the analytics service deployed in k8s environment. Format: namespace/servicename
    analyticsConfig.timeseries.port Optional 30002 Specify the port used to expose analytics service outside the cluster for timeseries endpoint.
    analyticsConfig.timeseries.metrics.enable Optional False Set this value to true to enable sending metrics from NetScaler.
    analyticsConfig.timeseries.metrics.mode Optional avro Specify the mode of metric endpoint.
    analyticsConfig.timeseries.metrics.exportFrequency Optional 30 Specify the time interval for exporting time-series data. Possible values range from 30 to 300 seconds.
    analyticsConfig.timeseries.metrics.schemaFile Optional schema.json Specifies the name of a schema file with the required NetScaler counters to be added and configured for metricscollector to export. A reference schema file reference_schema.json with all the supported counters is also available under the path /var/metrics_conf/. This schema file can be used as a reference to build a custom list of counters.
    analyticsConfig.timeseries.metrics.enableNativeScrape Optional false Set this value to true for native export of metrics.
    analyticsConfig.timeseries.auditlogs.enable Optional false Set this value to true to export audit log data from NetScaler.
    analyticsConfig.timeseries.events.enable Optional false Set this value to true to export events from NetScaler.
    analyticsConfig.transactions.enable Optional false Set this value to true to export transactions from NetScaler.
    analyticsConfig.transactions.port Optional 30001 Specify the port used to expose analytics service outside the cluster for transaction endpoint.
    bgpSettings.required Optional false Set this argument if you want to enable BGP configurations for exposing service of Type Loadbalancer through BGP fabric.
    bgpSettings.bgpConfig Optional N/A This represents BGP configurations in YAML format. For the description about individual fields, please refer to the documentation.
    nsLbHashAlgo.required Optional false Set this value to specify the LB consistent hashing algorithm.
    nsLbHashAlgo.hashFingers Optional 256 Specify the number of fingers to be used for hashing algorithm. Possible values are from 1 to 1024. Default value is 256.
    nsLbHashAlgo.hashAlgorithm Optional ‘default’ Specify the supported algorithm. Supported algorithms are “default”, “jarh”, “prac”. Default value is ‘default’.
    cpxCommands Optional N/A This parameter accepts user-provided NetScaler bootup config that is applied as soon as the CPX is instantiated. Please note that this is not a dynamic config, and any subsequent changes to the configmap don’t reflect in the CPX config unless the pod is restarted.
    cpxShellCommands Optional N/A This parameter accepts user-provided bootup config that is applied after NetScaler CPX is instantiated. Note that this is not a dynamic config, and any subsequent changes to the configmap don’t reflect in the CPX configuration unless the pod is restarted.
  6. Create a Kubernetes secret using a certificate to copy the certificate to NetScaler when an ingress object is deployed. For information on how to generate Kubernetes secret, see Generate Kubernetes secret.

  7. Create an ingress object to route traffic from NetScaler VPX or NetScaler MPX to NetScaler CPX in the Kubernetes cluster by using the following command:

    kubectl apply -f - <<EOF 
    apiVersion: networking.k8s.io/v1
    kind: Ingress
    metadata:
      annotations:
        ingress.citrix.com/frontend-ip: <NSVIP>
        ingress.citrix.com/secure-service-type: "ssl_tcp"
        ingress.citrix.com/secure_backend: '{"frontend-colddrinks": "True"}'
      name: vpx-ingress
      spec:
        ssl_tcp:
        - secretName: frontend-secret
        ingressClassName: nsic-vpx
        rules:
        - host: frontend-colddrinks.com
          http:
            paths:
            - backend:
                service:
                  name: cpx-service
                  port:
                    number: 80
              path: /
              pathType: Prefix
    ---
    apiVersion: networking.k8s.io/v1
    kind: IngressClass
    metadata:
      name: nsic-vpx
    spec:
      controller: citrix.com/ingress-controller
    <!--NeedCopy-->
    
  8. Create an ingress object for the tier-2 NetScaler CPX to route the traffic to the microservice application by using the following command:

kubectl apply -f - <<EOF 
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    ingress.citrix.com/secure-service-type: "ssl_tcp"
    ingress.citrix.com/secure-backend: '{"frontendcolddrinks":"True"}'
  name: colddrinks-ingress
spec:
  ingressClassName: colddrink
  defaultBackend:
    service:
    name: frontend-colddrinks
    port:
        number: 443
  ssl_tcp:
  - secretName: "colddrink-secret"
---
apiVersion: networking.k8s.io/v1
kind: IngressClass
metadata:
  name: colddrink
spec:
  controller: citrix.com/ingress-controller
EOF
<!--NeedCopy-->

Warning:

Do not create NSIC pod replicas in the Kubernetes cluster. When you create replicas of an NSIC pod by specifying replicatset>1, an unexpected behavior is observed with NSIC/VPX because the same APP_PREFIX is shared among the NSIC PODs and each instance tries to configure the same resources.

Dual-tier ingress