Citrix ADC CPX

NetScaler CPX troubleshooting

October 17, 2023

Contributed by:

This document explains how to troubleshoot issues that you may encounter while using NetScaler CPX. Using this document, you can collect logs to determine the causes and apply workarounds for some of the common issues related to the installation and configuration of NetScaler CPX.

How can I view NetScaler CPX logs?

You can view NetScaler CPX logs using the kubectl logs command if NetScaler CPX is deployed with the tty:true option. You can run the following command to display the logs:
```
 kubectl logs <pod-name> [-c <container-name>] [-n <namespace-name>]
```
Example,
```
 kubectl logs cpx-ingress1-69b9b8c648-t8bgn -c cpx -n citrix-adc
```
The following is an example of the NetScaler CPX pod deployment with the tty:true option:
```
   containers:
     - name: cpx-ingress
       image: "quay.io/citrix/citrix-k8s-cpx-ingress:13.0-58.30"
       tty: true
       securityContext:
          privileged: true
       env:

 
```
You can find more boot logs in the /cpx/log/boot.log file of NetScaler CPX file system.

Note: To get the pod name, run the kubectl get pods -o wide command.

How can I collect the technical support bundle from NetScaler CPX?

You can run the following command on the shell interface of the Kubernetes master node to collect the NetScaler CPX technical support bundle:

 kubectl exec <cpx-pod-name> [-c <cpx-container-name>] [-n <namespace-name>] /var/netscaler/bins/cli_script.sh "show techsupport"

You can view the technical support bundle in the /var/tmp/support directory of the NetScaler CPX’s file system. Use scp or kubectl cp to copy the technical support bundle from NetScaler CPX to the desired destination.

Example:

 root@localhost# kubectl exec cpx-ingress1-55b9b6fc75-t5kc6 -c cpx -n citrix-adc /var/netscaler/bins/cli_script.sh "show techsupport"
 exec: show techsupport
   Scope:  NODE
 Done
 root@localhost# kubectl cp cpx-ingress1-55b9b6fc75-t5kc6:var/tmp/support/collector_P_192.168.29.232_31Aug2020_07_30.tar.gz /tmp/collector_P_192.168.29.232_31Aug2020_07_30.tar.gz -c cpx
 root@localhost# ll /tmp/collector_P_192.168.29.232_31Aug2020_07_30.tar.gz
 -rw-r--r-- 1 root root 1648109 Aug 31 13:23 /tmp/collector_P_192.168.29.232_31Aug2020_07_30.tar.gz

Why is NetScaler CPX pod stuck while booting?

You can check the pod status using the kubectl describe pods command. Run the following command to know the pod status:
```
 kubectl describe pods <pod-name> [-c <container-name>] [-n <namespace-name>]
```
Example:
```
 kubectl describe pods cpx-ingress1-69b9b8c648-t8bgn
```
If the pod events show that container is started, then you must check the pod logs.
How do I copy files between the NetScaler CPX pod and the Kubernetes master node?

It is recommended to use the volume mount feature of docker to mount the /cpx directory to the file system of the host. If a NetScaler CPX container exits core-dumps, logs and other important data are available on the mount point.

You can use any one of the following commands to copy files between the NetScaler CPX pod and the Kubernetes master node:

kubectl cp: You can run the following command to copy files from pod to node:
```
 kubectl cp <pod-name>:<absolute-src-path> <dst-path> [-c <container-name>] [-n <namespace-name>]
```
Example:
```
 root@localhost:~# kubectl cp cpx-ingress-596d56bb6-zbx6h:cpx/log/boot.log /tmp/cpx-boot.log -c cpx-ingress
 root@localhost:~# ll /tmp/cpx-boot.log
 -rw-r--r-- 1 root root 7880 Sep 11 00:07 /tmp/cpx-boot.log
```
scp: You can use the command to copy files between the NetScaler CPX pod and the Kubernetes node. Run the following command to copy files from pod to node. When it prompts for the password, provide the password for the SSH user:
```
 scp <user>@<pod-ip>:<absolute-src-path> <dst-path>
```
Example:
```
 root@localhost:~# scp nsroot@192.168.29.198:/cpx/log/boot.log /tmp/cpx-boot.log
 nsroot@192.168.29.198's password:
 boot.log
 100% 7880     5.1MB/s   00:00
 root@localhost:~#
```
How do I capture packets on NetScaler CPX?

To capture packets on NetScaler CPX, launch the shell interface of NetScaler CPX using the kubectl exec command. Run the following command to launch the shell interface of the NetScaler CPX pod:
```
     kubectl exec -it pod-name [-c container-name] [-n namespace-name] bash
```
Example:
```
     kubectl exec -it cpx-ingress1-69b9b8c648-t8bgn -c cpx -n citrix-adc bash
```
And, run the following command to begin packet capture:
```
     cli_script.sh “start nstrace -size 0”
```
If you want to stop the ongoing packet capture, run the following command:
```
     cli_script.sh “stop nstrace”
```
You can view the packets captured in a .cap file in the /cpx/nstrace/time-stamp directory on the NetScaler CPX file system.
Why is the license server not configured even when NetScaler CPX is deployed with the LS_IP=<ADM-IP> environment variable?

Ensure that the license server is accessible from the node on which NetScaler CPX is deployed. You can use the ping <ADM-IP> command to verify the connectivity from the NetScaler CPX node to NetScaler ADM.

If NetScaler ADM is accessible from the node, then you must verify the license server configuration logs in the /cpx/log/boot.log file. You can also check for license server configuration using the following command on the shell interface of the NetScaler CPX pod:
```
 cli_script.sh "show licenseserver"
```
Example:
```
 root@cpx-ingress-596d56bb6-zbx6h:/cpx/log# cli_script.sh "show licenseserver"
 exec: show licenseserver
 ServerName:  10.106.102.199Port:  27000 Status:  1   Grace:  0  Gptimeleft:  720
 Done
```

Why is pooled license not configured on NetScaler CPX even after a successful license server configuration on NetScaler CPX?

Verify the license configuration logs in the /cpx/log/boot.log file. You can also verify the configured pooled license on NetScaler CPX using the following command on the shell interface of the NetScaler CPX pod:

 cli_script.sh “show capacity”

Example,

 root@cpx-ingress-596d56bb6-zbx6h:/cpx/log# cli_script.sh "show capacity"
 exec: show capacity
 Actualbandwidth:  1000  MaxVcpuCount:  2  Edition:  Platinum      Unit:  Mbps   Bandwidth:  0`  `Maxbandwidth:  40000        Minbandwidth:  20       Instancecount:  1
 Done

Also, ensure that the required license files are uploaded in the license server. You can also verify the available licenses on the license server once it is successfully configured on NetScaler CPX by using the following command. Run the command on the shell interface of NetScaler CPX pod:

 cli_script.sh “sh licenseserverpool”

Example:

 root@cpx-ingress-596d56bb6-zbx6h:/cpx/log# cli_script.sh "show licenseserverpool"
 exec: show licenseserverpool
     Instance Total                     : 5
     Instance Available                 : 4
     Standard Bandwidth Total           : 0 Mbps
     Standard Bandwidth Availabe        : 0 Mbps
     Enterprise Bandwidth Total         : 0 Mbps
     Enterprise Bandwidth Available     : 0 Mbps
     Platinum Bandwidth Total           : 10.00 Gbps
     Platinum Bandwidth Available       : 9.99 Gbps
     CP1000 Instance Total              : 100
     CP1000 Instance Available          : 100
   Done
 <!--NeedCopy-->

Why do NITRO API calls get Connection Refused response from NetScaler CPX?

The default port for NITRO APIs is 9080 (unsecure) and 9443 (secure) from the NetScaler CPX release 12.1 onwards. Ensure that the NITRO port of NetScaler CPX you try to access is exposed on the pod. You can run the kubectl describe command to view the exposed and mapped port of the NetScaler CPX container in the NetScaler CPX container section:
```
 kubectl describe pods <pod-name> | grep -i port
```
Example:
```
      ng472 | grep -i port
         Ports:          80/TCP, 443/TCP, 9080/TCP, 9443/TCP
         Host Ports:     0/TCP, 0/TCP, 0/TCP, 0/TCP
           NS_HTTP_PORT:            9080
           NS_HTTPS_PORT:           9443
         Port:          <none>
         Host Port:     <none>
           NS_PORT:                 80
 
```
Why does the NSPPE process in NetScaler CPX consume most of the CPU usage even when there is no or little traffic?

If NetScaler CPX is deployed with the CPX_CONFIG=’{“YIELD”:”NO”}' environment variable, the NSPPE process consumes 100 percent CPU usage even when there is no or little traffic. If you want the NSPPE process not to consume the CPU usage, you must deploy NetScaler CPX without the CPX_CONFIG=’{“YIELD”:”NO”} environment variable. By default, the NSPPE process in CPX is configured not to hog or consume the CPU usage.
Why is NetScaler CPX not listed in NetScaler ADM even when it was deployed with the required environment variables for registration with NetScaler ADM?

You can find the logs for NetScaler CPX registration with NetScaler ADM in the /cpx/log/boot.log file on the NetScaler CPX file system.

You can verify the accessibility of the NetScaler ADM IP address from the NetScaler CPX pod using the ping command. Also, ensure that all the required environment variables for NetScaler ADM registration are configured for the NetScaler CPX container.
- NS_MGMT_SERVER: Specifies the ADM-IP address or FQDN.
- HOST: Specifies the node IP address.
- NS_HTTP_PORT: Specifies the mapped HTTP- port on node.
- NS_HTTPS_PORT: Specifies the mapped HTTPS port on node.
- NS_SSH_PORT: Specifies the mapped SSH port on node.
- NS_SNMP_PORT: Specifies the mapped SNMP port on node.
- NS_ROUTABLE: NetScaler CPX pod IP address is not routable from outside.
- NS_MGMT_USER: Specifies the ADM username.
- NS_MGMT_PASS: Specifies the ADM password.

Why does cli_script.sh show Invalid user name or password error message after changing the password for nsroot user?

The command cli_script.sh is a wrapper utility for NSCLI on NetScaler CPX. It runs the first argument as command string or file path and the second argument is optional which is credentials. If the password for the nsroot user is changed, you need to provide credentials as the second argument to cli_script.sh. You can run the following command to run NSCLI with credentials:

 cli_script.sh “<command>” “:<username>:<password>”

Example:

 root@087a1e34642d:/# cli_script.sh "show ns ip"
 exec: show ns ip

 ERROR: Invalid username or password

 root@087a1e34642d:/# cli_script.sh "show ns ip" ":nsroot:nsroot123"

 exec: show ns ip

 Ipaddress        Traffic Domain         Type             Mode     Arp     Icmp     Vserver  State        
 --------------   ----                   ----             ---     ----     -------  ------   -----
 172.17.0.3       0                   NetScaler IP     Active   Enabled  Enabled  NA       Enabled
 192.0.0.1        0                   SNIP             Active   Enabled  Enabled  NA       Enabled
 Done

Why does SSH to NetScaler CPX fail with root and nsroot user?

From 13.0-64.35 release onwards, NetScaler CPX generates a default password and updates it for SSH users - root and nsroot. If you have not changed the password manually, password for SSH users can be found in /var/deviceinfo/random_id on NetScaler CPX’s file-system.

The official version of this content is in English. Some of the Cloud Software Group documentation content is machine translated for your convenience only. Cloud Software Group has no control over machine-translated content, which may contain errors, inaccuracies or unsuitable language. No warranty of any kind, either expressed or implied, is made as to the accuracy, reliability, suitability, or correctness of any translations made from the English original into any other language, or that your Cloud Software Group product or service conforms to any machine translated content, and any warranty provided under the applicable end user license agreement or terms of service, or any other agreement with Cloud Software Group, that the product or service conforms with any documentation shall not apply to the extent that such documentation has been machine translated. Cloud Software Group will not be held responsible for any damage or issues that may arise from using machine-translated content.

NetScaler CPX troubleshooting

October 17, 2023

Contributed by:

October 17, 2023

Contributed by:

NetScaler CPX troubleshooting

In this article