NetScaler CPX troubleshooting

This document explains how to troubleshoot issues that you may encounter while using NetScaler CPX. Using this document, you can collect logs to determine the causes and apply workarounds for some of the common issues related to the installation and configuration of NetScaler CPX.

• Why is the memory usage percentage always high in NetScaler CPX?

The memory usage percentage is determined by comparing the current memory usage of NetScaler CPX to the total memory NetScaler CPX allocates itself from the operating system (OS). NetScaler CPX starts by allocating a minimum necessary memory from the OS for booting and basic operations. Initially, because NetScaler CPX allocates the minimum necessary memory for its operations, the memory consumption percentage is high. Eventually, as the need for more resources grows with the scaling of NetScaler CPX operations, NetScaler CPX incrementally allocates additional memory from the OS and the memory usage is also high, resulting in high memory usage percentage.

Thus, due to NetScaler CPX’s dynamic memory allocation strategy, NetScaler CPX typically shows a high memory usage percentage, reflecting its efficient use of resources rather than a lack of available memory.

• How can I view NetScaler CPX logs?

  You can view NetScaler CPX logs using the kubectl logs command if NetScaler CPX is deployed with the tty:true option. You can run the following command to display the logs:

   kubectl logs <pod-name> [-c <container-name>] [-n <namespace-name>]
  

  Example,

   kubectl logs cpx-ingress1-69b9b8c648-t8bgn -c cpx -n citrix-adc
  

  The following is an example of the NetScaler CPX pod deployment with the tty:true option:

     containers:
       - name: cpx-ingress
         image: "quay.io/citrix/citrix-k8s-cpx-ingress:13.0-58.30"
         tty: true
         securityContext:
            privileged: true
         env:
  
   <!--NeedCopy-->
  

  You can find more boot logs in the /cpx/log/boot.log file of NetScaler CPX file system.

  Note: To get the pod name, run the kubectl get pods -o wide command.

• How can I collect the technical support bundle from NetScaler CPX?

  You can run the following command on the shell interface of the Kubernetes master node to collect the NetScaler CPX technical support bundle:

   kubectl exec <cpx-pod-name> [-c <cpx-container-name>] [-n <namespace-name>] /var/netscaler/bins/cli_script.sh "show techsupport"
  

  You can view the technical support bundle in the /var/tmp/support directory of the NetScaler CPX’s file system. Use scp or kubectl cp to copy the technical support bundle from NetScaler CPX to the desired destination.

  Example:

   root@localhost# kubectl exec cpx-ingress1-55b9b6fc75-t5kc6 -c cpx -n citrix-adc /var/netscaler/bins/cli_script.sh "show techsupport"
   exec: show techsupport
     Scope:  NODE
   Done
   root@localhost# kubectl cp cpx-ingress1-55b9b6fc75-t5kc6:var/tmp/support/collector_P_192.168.29.232_31Aug2020_07_30.tar.gz /tmp/collector_P_192.168.29.232_31Aug2020_07_30.tar.gz -c cpx
   root@localhost# ll /tmp/collector_P_192.168.29.232_31Aug2020_07_30.tar.gz
   -rw-r--r-- 1 root root 1648109 Aug 31 13:23 /tmp/collector_P_192.168.29.232_31Aug2020_07_30.tar.gz
  

• Why is NetScaler CPX pod stuck while booting?

  You can check the pod status using the kubectl describe pods command. Run the following command to know the pod status:

   kubectl describe pods <pod-name> [-c <container-name>] [-n <namespace-name>]
  

  Example:

   kubectl describe pods cpx-ingress1-69b9b8c648-t8bgn
  

  If the pod events show that container is started, then you must check the pod logs.

• How do I copy files between the NetScaler CPX pod and the Kubernetes master node?

  It is recommended to use the volume mount feature of docker to mount the /cpx directory to the file system of the host. If a NetScaler CPX container exits core-dumps, logs and other important data are available on the mount point.

  You can use any one of the following commands to copy files between the NetScaler CPX pod and the Kubernetes master node:

  kubectl cp: You can run the following command to copy files from pod to node:

   kubectl cp <pod-name>:<absolute-src-path> <dst-path> [-c <container-name>] [-n <namespace-name>]
  

  Example:

   root@localhost:~# kubectl cp cpx-ingress-596d56bb6-zbx6h:cpx/log/boot.log /tmp/cpx-boot.log -c cpx-ingress
   root@localhost:~# ll /tmp/cpx-boot.log
   -rw-r--r-- 1 root root 7880 Sep 11 00:07 /tmp/cpx-boot.log
  

  scp: You can use the command to copy files between the NetScaler CPX pod and the Kubernetes node. Run the following command to copy files from pod to node. When it prompts for the password, provide the password for the SSH user:

   scp <user>@<pod-ip>:<absolute-src-path> <dst-path>
  

  Example:

   root@localhost:~# scp nsroot@192.168.29.198:/cpx/log/boot.log /tmp/cpx-boot.log
   nsroot@192.168.29.198's password:
   boot.log
   100% 7880     5.1MB/s   00:00
   root@localhost:~#
  

• How do I capture packets on NetScaler CPX?

  To capture packets on NetScaler CPX, launch the shell interface of NetScaler CPX using the kubectl exec command. Run the following command to launch the shell interface of the NetScaler CPX pod:

       kubectl exec -it pod-name [-c container-name] [-n namespace-name] bash
  

  Example:

       kubectl exec -it cpx-ingress1-69b9b8c648-t8bgn -c cpx -n citrix-adc bash
  

  And, run the following command to begin packet capture:

       cli_script.sh “start nstrace -size 0”
  

  If you want to stop the ongoing packet capture, run the following command:

       cli_script.sh “stop nstrace”
  

  You can view the packets captured in a .cap file in the /cpx/nstrace/time-stamp directory on the NetScaler CPX file system.

• Why is the license server not configured even when NetScaler CPX is deployed with the LS_IP=<ADM-IP> environment variable?

  Ensure that the license server is accessible from the node on which NetScaler CPX is deployed. You can use the ping <ADM-IP> command to verify the connectivity from the NetScaler CPX node to NetScaler ADM.

  If NetScaler ADM is accessible from the node, then you must verify the license server configuration logs in the /cpx/log/boot.log file. You can also check for license server configuration using the following command on the shell interface of the NetScaler CPX pod:

   cli_script.sh "show licenseserver"
  

  Example:

   root@cpx-ingress-596d56bb6-zbx6h:/cpx/log# cli_script.sh "show licenseserver"
   exec: show licenseserver
   ServerName:  10.106.102.199Port:  27000 Status:  1   Grace:  0  Gptimeleft:  720
   Done
  

• Why is pooled license not configured on NetScaler CPX even after a successful license server configuration on NetScaler CPX?

  Verify the license configuration logs in the /cpx/log/boot.log file. You can also verify the configured pooled license on NetScaler CPX using the following command on the shell interface of the NetScaler CPX pod:

   cli_script.sh “show capacity”
  

  Example,

   root@cpx-ingress-596d56bb6-zbx6h:/cpx/log# cli_script.sh "show capacity"
   exec: show capacity
   Actualbandwidth:  1000  MaxVcpuCount:  2  Edition:  Platinum      Unit:  Mbps   Bandwidth:  0`  `Maxbandwidth:  40000        Minbandwidth:  20       Instancecount:  1
   Done
  

  Also, ensure that the required license files are uploaded in the license server. You can also verify the available licenses on the license server once it is successfully configured on NetScaler CPX by using the following command. Run the command on the shell interface of NetScaler CPX pod:

   cli_script.sh “sh licenseserverpool”
  

  Example:

   root@cpx-ingress-596d56bb6-zbx6h:/cpx/log# cli_script.sh "show licenseserverpool"
   exec: show licenseserverpool
       Instance Total                     : 5
       Instance Available                 : 4
       Standard Bandwidth Total           : 0 Mbps
       Standard Bandwidth Availabe        : 0 Mbps
       Enterprise Bandwidth Total         : 0 Mbps
       Enterprise Bandwidth Available     : 0 Mbps
       Platinum Bandwidth Total           : 10.00 Gbps
       Platinum Bandwidth Available       : 9.99 Gbps
       CP1000 Instance Total              : 100
       CP1000 Instance Available          : 100
     Done
   <!--NeedCopy-->
  

• Why do NITRO API calls get Connection Refused response from NetScaler CPX?

  The default port for NITRO APIs is 9080 (unsecure) and 9443 (secure) from the NetScaler CPX release 12.1 onwards. Ensure that the NITRO port of NetScaler CPX you try to access is exposed on the pod. You can run the kubectl describe command to view the exposed and mapped port of the NetScaler CPX container in the NetScaler CPX container section:

   kubectl describe pods <pod-name> | grep -i port
  

  Example:

        ng472 | grep -i port
           Ports:          80/TCP, 443/TCP, 9080/TCP, 9443/TCP
           Host Ports:     0/TCP, 0/TCP, 0/TCP, 0/TCP
             NS_HTTP_PORT:            9080
             NS_HTTPS_PORT:           9443
           Port:          <none>
           Host Port:     <none>
             NS_PORT:                 80
   <!--NeedCopy-->
  

• Why does the NSPPE process in NetScaler CPX consume most of the CPU usage even when there is no or little traffic?

  If NetScaler CPX is deployed with the CPX_CONFIG=’{“YIELD”:”NO”}' environment variable, the NSPPE process consumes 100 percent CPU usage even when there is no or little traffic. If you want the NSPPE process not to consume the CPU usage, you must deploy NetScaler CPX without the CPX_CONFIG=’{“YIELD”:”NO”} environment variable. By default, the NSPPE process in CPX is configured not to hog or consume the CPU usage.

• Why is NetScaler CPX not listed in NetScaler ADM even when it was deployed with the required environment variables for registration with NetScaler ADM?

  You can find the logs for NetScaler CPX registration with NetScaler ADM in the /cpx/log/boot.log file on the NetScaler CPX file system.

  You can verify the accessibility of the NetScaler ADM IP address from the NetScaler CPX pod using the ping command. Also, ensure that all the required environment variables for NetScaler ADM registration are configured for the NetScaler CPX container.

  • NS_MGMT_SERVER: Specifies the ADM-IP address or FQDN.
  • HOST: Specifies the node IP address.
  • NS_HTTP_PORT: Specifies the mapped HTTP- port on node.
  • NS_HTTPS_PORT: Specifies the mapped HTTPS port on node.
  • NS_SSH_PORT: Specifies the mapped SSH port on node.
  • NS_SNMP_PORT: Specifies the mapped SNMP port on node.
  • NS_ROUTABLE: NetScaler CPX pod IP address is not routable from outside.
  • NS_MGMT_USER: Specifies the ADM username.
  • NS_MGMT_PASS: Specifies the ADM password.

• Why does cli_script.sh show Invalid user name or password error message after changing the password for nsroot user?

  The command cli_script.sh is a wrapper utility for NSCLI on NetScaler CPX. It runs the first argument as command string or file path and the second argument is optional which is credentials. If the password for the nsroot user is changed, you need to provide credentials as the second argument to cli_script.sh. You can run the following command to run NSCLI with credentials:

   cli_script.sh “<command>” “:<username>:<password>”
  

  Example:

   root@087a1e34642d:/# cli_script.sh "show ns ip"
   exec: show ns ip
  
   ERROR: Invalid username or password
  
   root@087a1e34642d:/# cli_script.sh "show ns ip" ":nsroot:nsroot123"
  
   exec: show ns ip
  
   Ipaddress        Traffic Domain         Type             Mode     Arp     Icmp     Vserver  State        
   --------------   ----                   ----             ---     ----     -------  ------   -----
   172.17.0.3       0                   NetScaler IP     Active   Enabled  Enabled  NA       Enabled
   192.0.0.1        0                   SNIP             Active   Enabled  Enabled  NA       Enabled
   Done
  

• Why does SSH to NetScaler CPX fail with root and nsroot user?

  From 13.0-64.35 release onwards, NetScaler CPX generates a default password and updates it for SSH users - root and nsroot. If you have not changed the password manually, password for SSH users can be found in /var/deviceinfo/random_id on NetScaler CPX’s file-system.