Product Documentation
ADC
14.1-Current Release 13.1
View PDF

Tracing the packets of a NetScaler cluster

May 2, 2023
Contributed by: 
S C

The NetScaler operating system provides a utility called ns trace to get a dump of the packets that are received and sent out by an appliance. The utility stores the packets in trace files. You can use these files to debug problems in the flow of packets to the cluster nodes. The trace files must be viewed with the Wireshark application.

Some salient aspects of the ns trace utility are:

  • Can be configured to trace packets selectively by using classic expressions and default expressions.
  • Can capture the trace in multiple formats: ns trace format (.cap) and TCP dump format (.pcap).
  • Can aggregate the trace files of all cluster nodes on the configuration coordinator.
  • Can merge multiple trace files into a single trace file (only for.cap files).

You can use the ns trace utility from the NetScaler command line or the NetScaler shell.

To trace packets of a standalone appliance

Run the start ns trace command on the appliance. The command creates trace files in the /var/nstrace/<date-timestamp> directory. The trace file names are of the form nstrace<id\ >.cap.

You can view the status by running the show ns trace command. You can stop tracing the packets by running the stop ns trace command.

Note

You can also run the ns trace utility from the NetScaler shell by running the nstrace.sh file. However, it is recommended that you use the ns trace utility through the NetScaler command line interface.

To trace packets of a cluster

You can trace the packets on all the cluster nodes and obtain all the trace files on the configuration coordinator.

Run the start ns trace command on the cluster IP address. The command is propagated and run on all the cluster nodes. The trace files are stored in individual cluster nodes in the /var/nstrace/<date-timestamp> directory. The trace file names are of the form nstrace<id>_node<id\ >.cap.

You can use the trace files of each node to debug the nodes operations. But if you want the trace files of all cluster nodes in one location, you must run the stop ns trace command on the cluster IP address. The trace files of all the nodes are downloaded on the cluster configuration coordinator in the /var/nstrace/<date-timestamp> directory as follows:

Cluster trace

Merge multiple trace files

You can prepare a single file from the trace files (supported only for. Cap files) obtained from the cluster nodes. The single trace files give you a cumulative view of the trace of the cluster packets. The trace entries in the single trace file are sorted based on the time the packets were received on the cluster.

To merge the trace files, at the NetScaler shell, type:

> nstracemerge.sh -srcdir \<DIR\> -dstdir \<DIR\> -filename \<name\> -filesize \<num\>

Where,

  • srcdir is the directory from which the trace files are merged. All trace files within this directory are merged into a single file.
  • dstdir is the directory where the merged trace file is created.
  • Filename is the name of the trace file that is created.
  • Filesize is the size of the trace file.

Examples

Following are some examples of using the ns trace utility to filter packets.

  • To trace the packets on the backplane interfaces of three nodes:

    Using classic expressions:

     > start nstrace -filter "INTF == 0/1/1 && INTF == 1/1/1 && INTF == 2/1/1"

    Using default expressions:

     > start nstrace -filter "CONNECTION.INTF.EQ("0/1/1") && CONNECTION.INTF.EQ("1/1/1") && CONNECTION.INTF.EQ("2/1/1")"

  • To trace the packets from a source IP address 10.102.34.201 or from a system whose source port is greater than 80 and the service name is not “s1”:

    Using classic expressions

     > start nstrace -filter \"SOURCEIP == 10.102.34.201 \|| \(SVCNAME != s1 && SOURCEPORT > 80)\"

    Using default expressions

     > start nstrace -filter \"CONNECTION.SRCIP.EQ\(10.102.34.201) \|| \(CONNECTION.SVCNAME.NE\(\"s1\") && CONNECTION.SRCPORT.GT\(80))\"

Note

For more information about filters used in ns trace, see ns trace.

Capturing SSL Session Keys During a Trace

When you run the “start ns trace” command, you can set the new capsslkeys parameter to capture the SSL master keys for all SSL sessions. If you include this parameter, a file named nstrace.sslkeys is generated along with the packet trace. This file can be imported into Wireshark to decrypt the SSL traffic in the corresponding trace file.

This functionality is similar to web browsers exporting session keys that can later be imported into Wireshark for decrypting SSL traffic.

Advantages of using SSL session keys

Following are the advantages of using SSL session keys:

  1. Generates smaller trace files that do not include the extra packets created by the SSLPLAIN mode of capturing.
  2. Provides the ability to view plaintext [SP(1] from the trace and choose whether to share the master keys file or protect sensitive data by not sharing it.

Limitations of using SSL session keys

Following are the limitations of using SSL session keys:

  1. SSL sessions cannot be decrypted if the initial packets of the session are not captured.
  2. SSL sessions cannot be captured if the Federal Information Processing Standard (FIPS) mode is enabled.

To capture SSL session keys by using the command line interface (CLI)

At the command prompt, type the following commands to enable or disable SSL session keys in a trace file and verify trace operation.

> start nstrace -capsslkeys ENABLED
> show nstrace
Example
> start nstrace -capsslkeys ENABLED
> show nstrace
      State:  RUNNING          Scope:  LOCAL            TraceLocation:  "/var/nstrace/04May2016_17_51_54/..."
      Nf:  24                  Time:  3600              Size:  164               Mode:  TXB NEW_RX
      Traceformat:  NSCAP      PerNIC:  DISABLED        FileName:  04May2016_17_51_54 Link:  DISABLED
      Merge:  ONSTOP           Doruntimecleanup:  ENABLED TraceBuffers:  5000      SkipRPC:  DISABLED
      SkipLocalSSH:  DISABLED  Capsslkeys:  ENABLED     InMemoryTrace:  DISABLED
 Done

To configure SSL session keys by using the NetScaler GUI

  1. Navigate to Configuration > System > Diagnostics > Technical Support Tools and click Start new Trace to start tracing encrypted packets on an appliance.
  2. On the Start Trace page, select the Capture SSL Master Keys check box.
  3. Click OK and Done.

To import the SSL Master Keys into Wireshark

On the Wireshark GUI, navigate to Edit > Preferences > Protocols > SSL > (Pre)-Master-Secret log filename and specify the master key files obtained from the appliance.

Tracing the packets of a NetScaler cluster
May 2, 2023
Contributed by: 
S C
May 2, 2023
Contributed by: 
S C

In this article

Site feedback | Your privacy choices footer linkYour Privacy Choices | Privacy and legal terms | Cookie preferences | Consent settings | docs.cloud.com
© 1999- Cloud Software Group, Inc. All rights reserved.