ADC

How to record a packet trace on Citrix ADC

You can record a packet trace using the Citrix ADC GUI. The trace is stored in nstrace.cap.

  1. Navigate to System > Diagnostics.
  2. Click Start new trace under Technical Support Tools.
  3. In the Start Trace page update the following fields:

    1. Packet Size - Enter the size of the packet to capture during the trace. Enter 0 for full packet trace.

      • Default value: 164
      • Minimum value: 0
      • Maximum value: 1514
    2. Capture trace in .pcap format - You have the option to capture a packet trace in nstrace (.cap) or TCP dump (.pcap) format. By default, the packet trace is captured in nstrace format (.cap) and it is the recommended format. To capture trace in TCP dump format, select Capture trace in .pcap format.

    3. Capture SSL Master keys - To analyze the traces better, enable the Capture SSL Master keys option. This setting captures SSL keys for the current session, which are necessary to decrypt encrypted data. The SSL keys are stored in a file named nstrace.sslkeys.

      • When you click Start to initiate the trace with the Capture SSL Master keys option enabled, a security warning message is displayed. Acknowledge this warning to proceed.
      • The SSL session keys generated are an alternative to the private key and can be used when the private key is either unavailable or cannot be shared. You can use these keys to decrypt only the NS trace. This method can decrypt RSA and Diffie-Hellman-based key exchanges (such as DHE or ECDHE).
    4. Number of trace files - Enter the number of trace files to be generated during the trace.

      • Default value: 24
      • Minimum value: 1
      • Maximum value: 100
    5. Trace File name - Enter the name for the trace file.
    6. Trace File ID - Enter the file ID for the trace file.
    7. Duration of data per file (seconds) - Enter the time (in seconds) to capture the data for each trace file.

      • Default value: 3600
      • Minimum value: 1
    8. File size - Enter the file size (in MB) for each trace file.

      • Default value: 1024
      • Minimum value: 0
      • Maximum value: 10240

      When the trace reaches the specified file size, a new trace starts. If the free disk space is less than 2 GB, the trace stops.

    9. Trace Buffers - Enter the number of trace buffers (temporary storage) to store the packet capture. Each buffer is about 16 KB.

      • Default value: 5000
      • Minimum value: 1000
  4. Enter the Filter Expression.

    Adding filter expressions for IP address, port, VLAN, or interface ensures capturing only the relevant traffic and reduces the load on Citrix ADC during the packet trace.

  5. Select the Merge option from the list.

    • ONSTOP - The temporary trace files are merged into a single trace file.
    • NOMERGE - The trace files are not merged.
    • ONTHEFLY - The trace files are merged without creating any temporary file.

    Default value: ONSTOP

  6. Select the relevant option from the additional packet capture options available.

    Default value: Do Runtime cleanup

  7. Select the required options in Capturing Mode.

    By default, Packets buffered for Transmission (TXB) and Receive packets after NIC pipelining (NEW_RX) are selected. To decrypt the trace without a private key, select Decrypted SSL packets (SSLPLAIN).

  8. Click Start to start recording the network packet trace.

  9. In the Stop Trace page, click Stop and Download to stop recording the network packet trace after the test is complete.

  10. In the Delete/Download Trace files page, select the file, click Download, and then click Close.

    Open the trace file with the Wireshark utility to display the content of the file.

We recommend you to use the recent Wireshark version from the automated build section available in the following webpage: http://www.wireshark.org/download/automated.

Use case to capture a packet trace with virtual server IP filter (both front-end and back end)

Using a filter of the virtual server IP address and enabling the option “–link” in CLI or selecting the option “Trace filtered connection peer traffic” in GUI, you can capture both the front-end and back-end traffic for the IP address.

start nstrace -size 0 -filter "CONNECTION.IP.EQ(1.1.1.1)" -link ENABLED

show nstrace
        State:  RUNNING          Scope:  LOCAL            TraceLocation:  "/var/nstrace/24Mar2017_16_00_19/..." Nf:  24                  Time:  3600              Size:  0                 Mode:  TXB NEW_RX
        Traceformat:  NSCAP      PerNIC:  DISABLED        FileName:  24Mar2017_16_00_19 Filter:  "CONNECTION.IP.EQ(1.1.1.1)" Link:  ENABLED           Merge:  ONSTOP           Doruntimecleanup:  ENABLED
        TraceBuffers:  5000      SkipRPC:  DISABLED       Capsslkeys:  DISABLED    InMemoryTrace:  DISABLED
<!--NeedCopy-->

Capturing cyclic traces

It is always challenging to troubleshoot an intermittent issue. Cyclic tracing is best suited for issues which are intermittent. The traces can be run over a span of few hours or days before the issue occurs. Also, you can use a specific filter and evaluate the size of the trace files that are generated before you run it for a longer time.

Run the following command from the CLI:

start nstrace -nf 60 -time 30 -size 0
This particular trace will create 60 files each of them for 30 sec. This means the files will start getting overwritten after 60 trace files or 30 mins
Show nstrace - To check the status of the nstrace
Stop nstrace - To stop the nstrace.

<!--NeedCopy-->

Best Practices

On a unit handling GB of traffic per second, capturing traffic is a very resource intensive process. The impact to resources is mainly in terms of the CPU and the disk space. Disk space impact can be reduced by using filtering expressions. However, the impact on the CPU remains and sometimes causes a slight increase as the appliance now needs to process packets according to the filter before capturing them.

The best practice with regard to tracing is:

  1. The duration for which the trace is run must be as limited as possible when you still ensure the packets of interest are captured.
  2. Schedule the tracing activity to happen at a time when the number of users (and hence the traffic) is greatly reduced, such as during off hours.
How to record a packet trace on Citrix ADC