Troubleshoot HDX Insight issues
If the HDX Insight solution is not functioning as expected, the issue might be with one of the following. Refer to the checklists in the respective sections for troubleshooting.
-
HDX Insight configuration.
-
Connectivity between NetScaler and NetScaler ADM.
-
Record generation for HDX/ICA traffic in NetScaler.
-
Population of records in NetScaler ADM.
HDX Insight configuration checklist
-
Make sure that the AppFlow feature is enabled in NetScaler. For details, see Enabling AppFlow.
-
Check HDX Insight configuration in the NetScaler running configuration.
Run the
show running | grep -i <appflow_policy>
command to check the HDX Insight configuration. Make sure that the bind type is ICA REQUEST. For example;bind vpn vserver afsanity -policy afp -priority 100 -type ICA_REQUEST
For transparent mode, the bind type must be ICA_REQ_DEFAULT. For example;
bind appflow global afp 100 END -type ICA_REQ_DEFAULT
-
For single-hop/Access Gateway or double-hop deployment, make sure that HDX Insight AppFlow policy is bound to the VPN virtual server, where HDX/ICA traffic is flowing.
-
For Transparent mode or LAN user mode make sure the ICA ports 1494 and 2598 are set.
-
Check
appflowlog
parameter in NetScaler Gateway or VPN virtual server is enabled for Access Gateway or double-hop deployment. For details, see Enabling AppFlow for Virtual Servers. -
Check “Connection Chaining” is enabled in double-hop NetScaler. For details see, Configuring NetScaler Gateway appliances to export data.
-
After HA Failover if the HDX Insight details are Skip parsed, check ICA param “enableSRonHAFailover” is enabled. For details, see Session Reliability on NetScaler High Availability Pair.
Connectivity between NetScaler and NetScaler ADM checklist
-
Check AppFlow collector status in NetScaler. For details, see How to check the status of connectivity between NetScaler and AppFlow Collector.
-
Check HDX Insight AppFlow policy hits.
Run the command
show appflow policy <policy_name>
to check the AppFlow policy hits.You can also navigate to Settings > AppFlow > Policies in the GUI to check the AppFlow policy hits.
-
Validate any firewall blocking AppFlow ports 4739 or 5557.
Record generation for HDX/ICA traffic in NetScaler checklist
Run the command tail -f /var/log/ns.log | grep -i "default ICA Message"
for log validation. Based on the logs that are generated, you can use this information for troubleshooting.
-
Log: Skipped parsing ICA connection - HDX Insight not supported for this host
Cause: Unsupported Citrix Virtual Apps and Desktops versions
Workaround: Upgrade the Citrix Virtual Apps and Desktops servers to a supported version.
-
Log: Client type received 0x53, NOT SUPPORTED
Cause: Unsupported version of Citrix Workspace
Solution: Upgrade Citrix Workspace to a supported version. For details, see Citrix Workspace app.
-
Log: Error from Expand Packet - Skipping all hdx processing for this flow
Cause: Issue with uncompressing ICA traffic
Solution: No reports are available for this ICA session until a new session is established.
-
Log: Invalid transition: NS_ICA_ST_FLOW_INIT/NS_ICA_EVT_INVALID -> NS_ICA_ST_UNINIT”
Cause: Issue with parsing the ICA handshake
Solution: No reports are available for this particular ICA session until a new session is established.
-
Log: Missing EUEM ICA RTT
Cause: Unable to parse End-User Experience Monitoring channel data
Solution: Make sure End-User Experience Monitoring service in started on the Citrix Virtual Apps and Desktops servers. Make sure you are using the supported versions of Citrix Workspace App.
-
Log: Invalid Channel Header
Cause: Unable to identify channel header
Solution: No reports are available for this particular ICA session until a new session is established.
-
Log: Skip code
If you see any of the following values for skip code, then the Insight details are skip parsed.
Skip code 0 indicates that the record is successfully exported from NetScaler.
Skip Code | Error message | Cause of error |
---|---|---|
100 | NS_ICA_ERR_NULL_FRAG | Error handling ICA fragments, likely due to memory conditions |
101 | NS_ICA_ERR_INVALID_HS_CMD | Invalid handshake command received |
102 | NS_ICA_ERR_REDUC_PARAM_CNT | Invalid parameter specified for V3 expander initialization |
103 | NS_ICA_ERR_REDUC_INIT | Unable to initialize the V3 expander correctly |
104 | NS_ICA_ERR_REDUC_PARAM_BYTES | Insufficient bytes to assign a coder to a channel |
105 | NS_ICA_ERR_INVALID_CHANNEL | Invalid ICA channel number |
106 | NS_ICA_ERR_INVALID_DECODER | Invalid decoder specified for a channel |
107 | NS_ICA_ERR_INVALID_TW_PARAM | Invalid parameter count specified on Thinwire channel |
108 | NS_ICA_ERR_INVALID_TW_DECODER | Invalid decoder for Thinwire channel |
109 | NS_ICA_ERR_REDUC_NO_DECODER | No decoder defined for channel |
110 | NS_ICA_ERR_REDUC_V3_EXPANDER | Failed to expand channel data |
111 | NS_ICA_ERR_REDUC_BYTES_V3_OOR | Expander error: Bytes consumed more than bytes available |
112 | NS_ICA_ERR_REDUC_BYTES_OOR | Error: Uncompressed data overrun |
113 | NS_ICA_ERR_REDUC_INVALID_CMD | Undefined Expander command |
114 | NS_ICA_ERR_CGP_FILL_HOLE | Error while handling split CGP frames |
115 | NS_ICA_ERR_MEM_NSB_ALLOC | NSB allocation error – due to low memory conditions |
116 | NS_ICA_ERR_MEM_REDUC_CTX_ALLOC | Memory allocation error for expander context |
117 | NS_ICA_ERR_ICA_OLD_SERVER | Old server, capability blocks not supported |
118 | NS_ICA_ERR_PIR_MANY_FRAG | Packet Init request is fragmented, unable to process |
119 | NS_ICA_ERR_INIT_ICA_CAPS | ICA capability initialization error |
120 | NS_ICA_ERR_NO_MSI_SUPPORT | Host does not support MSI feature. Indicates for XenApp version lower than 6.5 or XenDesktop versions lower than 5.0 |
121 | NS_ICA_ERR_CGP_INVALID_CMD | Invalid CGP command encountered |
122 | NS_ICA_ERR_INSUFFICENT_CHANNEL_BYTES | Insufficient bytes over channel |
123 | NS_ICA_ERR_CHANNEL_DATA | Incorrect data on EUEM, CONTROL, or SEAMLESS channel |
124 | NS_ICA_ERR_INVALID_PURE_CMD | Invalid command received while processing pure ICA channel data |
125 | NS_ICA_ERR_INVALID_PURE_LEN0 | Invalid length encountered while processing pure ICA channel data |
126 | NS_ICA_ERR_INVALID_PURE_LEN | Invalid length encountered while processing PURE ICA channel data |
127 | NS_ICA_ERR_INVALID_CLNT_DATA | Invalid data length received from client |
128 | NS_ICA_ERR_MSI_GUID_SZ | Error in MSI GUID size |
129 | NS_ICA_ERR_INVALID_CHANNEL_HEADER | Detected invalid channel header |
130 | NS_ICA_ERR_CGP_PARSE_RECONNECT_ID | Retrieval of reconnected session failed |
131 | NS_ICA_ERR_DISABLE_SR_NON_NS_RECONNECT | Error in disabling SR |
132 | NS_ICA_ERR_REDUC_NOT_V3 | Unsupported ICA Reducer version |
133 | NS_ICA_ERR_HS_COMPRESSION_DISABLED | Compression disabled, not honored by host |
134 | NS_ICA_ERR_IDENT_PROTO | Unable to identify ICA or CGP protocol, seen with incorrect workspaces |
135 | NS_ICA_ERR_INVALID_SIGNATURE | Incorrect ICA signature or magic string |
136 | NS_ICA_ERR_PARSE_RAW | Error while parsing the ICA handshake packet |
137 | NS_ICA_ERR_INCOMPLETE_PKT | Incomplete packet received in handshake |
138 | NS_ICA_ERR_ICAFRAME_TOO_LARGE | ICA frame is too large, exceeds 1460 bytes |
139 | NS_ICA_ERR_FORWARD | Error while forwarding the ICA data |
140 | NS_ICA_ERR_MAX_HOLES | Unable to process CGP command as it is split beyond supported limit |
141 | NS_ICA_ERR_ASSEMBLE_FRAME | Unable to reassemble ICA frame correctly |
142 | NS_ICA_ERR_UNSUPPORTED_RECEIVER_VERSION | Skipped ICA parsing for this workspace (client) as it is not in the allow list |
143 | NS_ICA_ERR_LOOKUP_RECONNECT_ID | Unable to detect parsing state for client reconnect cookie |
144 | NS_ICA_ERR_SYNCUP_RECONNECT_ID | Invalid reconnect cookie length detected post client reconnect |
145 | NS_ICA_ERR_INVALID_RECONNECT_ID | Client reconnects cookie missed the needed constraint |
146 | NS_ICA_ERR_INVALID_CLIENT_VERSION | Invalid workspace version string received from client |
147 | NS_ICA_ERR_UNKNOWN_CLIENT_PRODUCT_ID | Invalid product ID received from client |
148 | NS_ICA_ERR_V3_HDR_CORRUPT_LEN | Invalid channel length post expansion |
149 | NS_ICA_ERR_SPECIAL_THINWIRE | Decompression error |
150 | NS_ICA_ERR_SEAMLESS_INSUFFBYTE | Encountered insufficient bytes for seamless command |
151 | NS_ICA_ERR_EUEM_INSUFFBYTE | Encountered insufficient bytes for EUEM command |
152 | NS_ICA_ERR_SEAMLESS_INVALID_EVENT | Invalid event for seamless channel parsing |
153 | NS_ICA_ERR_CTRL_INVALID_EVENT | Invalid event for CTRL channel parsing |
154 | NS_ICA_ERR_EUEM_INVALID_EVENT | Invalid event for EUEM channel parsing |
155 | NS_ICA_ERR_USB_INVALID_EVENT | Invalid event for USB channel parsing |
156 | NS_ICA_ERR_PURE_INVALID_EVENT | Invalid event for pure channel parsing |
157 | NS_ICA_ERR_VCP_INVALID_EVENT | Invalid event for virtual channel parsing |
158 | NS_ICA_ERR_ICAP_INVALID_EVENT | Invalid event for ICA data parsing |
159 | NS_ICA_ERR_CGPP_INVALID_EVENT | Invalid event for CGP data parsing |
160 | NS_ICA_ERR_BASICCRYPT_INVALIDSTATE | Invalid state for a crypt command in basic encryption |
161 | NS_ICA_ERR_BASICCRYPT_INVALIDCRYPTCMD | Invalid crypt command in basic encryption |
162 | NS_ICA_ERR_ADVCRYPT_INVALIDSTATE | Invalid state for a crypt command in RC5 encryption |
163 | NS_ICA_ERR_ADVCRYPT_INVALIDCRYPTCMD | Invalid crypt command in RC5 encryption |
164 | NS_ICA_ERR_ADVCRYPT_ENC | Error in RC5 encryption/decryption |
165 | NS_ICA_ERR_ADVCRYPT_DEC | Error in RC5 encryption/decryption |
166 | NS_ICA_ERR_SERVER_NOT_REDUCER_V3 | VDA does not support Reducer Version 3 |
167 | NS_ICA_ERR_CLIENT_NOT_REDUCER_V3 | Workspace does not support Reducer Version 3 |
168 | NS_ICA_ERR_ICAP_INSUFFBYTE | Unexpected number of bytes in ICA handshake |
169 | NS_ICA_ERR_HIGHER_RECONSEQ | Higher CGP resumption sequence number from peer post reconnects |
170 | NS_ICA_ERR_DESCSRINFO_ABSENT | Unable to restore ICA parsing state post reconnect |
171 | NS_ICA_ERR_NSAP_PARSING | Error while parsing Insight channel data |
172 | NS_ICA_ERR_NSAP_APP | Error while parsing app details from Insight channel data |
173 | NS_ICA_ERR_NSAP_ACR | Error while parsing ACR details from Insight channel data |
174 | NS_ICA_ERR_NSAP_SESSION_END | Error while parsing session end details from Insight channel data |
175 | NS_ICA_ERR_NON_NSAP_SN | Skipped ICA parsing on service node due to the absence of Insight channel support |
176 | NS_ICA_ERR_NON_NSAP_CLIENT | NSAP is not supported by client |
177 | NS_ICA_ERR_NON_NSAP_SERVER | NSAP is not supported by VDA |
178 | NS_ICA_ERR_NSAP_NEG_FAIL | Error while NSAP data negotiation |
179 | NS_ICA_ERR_SN_RECONNECT_TKT_FETCH | Error in fetching service reconnects ticket in service node |
180 | NS_ICA_ERR_SN_HIGHER_RECONSEQ | Error when receiving higher reconnect sequence number in service node |
181 | NS_ICA_ERR_DISABLE_HDXINSIGHT_NONNSAP | Error while disabling HDX Insight for non-NSAP connections |
Sample logs:
Jan 9 22:57:02 <local0.notice> 10.106.40.223 01/09/2020:22:57:02 GMT ns-223 0-PPE-2 : default ICA Message 1234 0 : "Session setup data send: Session GUID [57af35043e624abab409f5e6af7fd22c], Client IP/Port [10.105.232.40/52314], Server IP/Port [10.106.40.215/2598], MSI Client Cookie [Non-MSI], Session setup time [01/09/2020:22:56:49 GMT], Client Type [0x0052], Receiver Version [19.12.0.23], User [user1], Client [10.105.232.40], Server [WIN2K12-215], Ctx Flags [0x8820220228], Track Flags [0x1775010c3fc], Skip Code [0]"
Jan 9 22:55:41 <local0.notice> 10.106.40.223 01/09/2020:22:55:41 GMT ns-223 0-PPE-0 : default ICA Message 156 0 : "Skipping ICA flow: Session GUID [4e3a91175ebcbe686baf175eec7e0200], Client IP/Port [10.105.232.40/60059], Server IP/Port [10.106.40.219/2598], MSI Client Cookie [Non-MSI], Session setup time [01/09/2020:22:55:39 GMT], Client Type [0x0052], Receiver Version [19.12.0.23], User [user1], Client [10.105.232.40], Server [10.106.40.219], Ctx Flags [0x8820220008], Track Flags [0x1600010c040], Skip Code [171]"
Error counters
Various counters are captured ICA parsing. The following table lists the various counters for ICA parsing.
Run the command nsconmsg –g hdx –d statswt0
for viewing the counter details.
HDX counter name | Purpose | Category(Stats/Error/Diagnostics) |
---|---|---|
hdx_tot_ica_conn | Indicates total number of Pure ICA connections detected by NS. Incremented whenever an ICA connection based on the ICA signature on a client PCB is detected. | Stats |
hdx_tot_cgp_conn | Indicates total number of CGP connections detected by NS (Session Reliability ON). Incremented whenever a CGP connection based on the CGP signature on a client PCB is detected. | Stats |
hdx_dbg_tot_udt_conn | Indicates total number of UDP ICA connections detected by NS | Stats |
hdx_dbg_tot_nsap_conn | Indicates total number of NSAP supported connections detected by NS | Stats |
hdx_tot_skip_conn | Indicates how many ICA connections were skipped by parser due to invalid ICA or CGP signature. | Stats |
hdx_dbg_active_conn | Total Active EDT/CGP/ICA connections at that instant. | Stats |
hdx_dbg_active_nsap_conn | Total Active EDT/CGP/ICA NSAP connections at that instant. | Stats |
hdx_dbg_skip_appflow_disabled | Total number of instances where AppFlow was detached from a session because of disabling AppFlow | Stats/Diagnostics |
hdx_dbg_transparent_user | Total number of transparent user access | Stats/Diagnostics |
hdx_dbg_ag_user | Total number of Access Gateway user access | Stats/Diagnostics |
hdx_dbg_lan_user | Total number of LAN user mode access | Stats/Diagnostics |
hdx_basic_enc | Indicates the number of ICA connections using basic encryption | Stats/Diagnostics |
hdx_advanced_enc | Indicates the number of ICA connections using advanced RC5 based encryption | Stats/Diagnostics |
hdx_dbg_reconnected_session | Total number of reconnect requests from client without any NetScaler error | Stats/Diagnostics |
hdx_dbg_host_rejected_ns_reconnect | Total number of hosts rejected reconnects requests by client | Stats/Diagnostics |
hdx_euem_available | Indicates the number of connections having the End User Experience Monitoring channel available. End User Experience Monitoring channel is required to collect statistics such as ICA RTT. | Stats/Diagnostics |
hdx_err_disabled_sr | Session Reliability is disabled using nsapimgr knob. Session does not work for this session. |
Error |
hdx_err_skip_no_msi | XA/XD server is Missing MSI capability. This indicates an older server version and HDX Insight skips this connection. | Error |
hdx_err_skip_old_server | Old unsupported server version | Error |
hdx_err_clnt_not_whitelist | Client workspace not in allow list, HDX Insight skips this connection | Error |
hdx_sm_ica_cam_channel_disabled | Total number of NS_ICA_CAM_CHANNEL disabled via SmartAccess policy | Diagnostics |
hdx_sm_ica_usb_channel_disabled | Total number of NS_ICA_USB_CHANNEL disabled via SmartAccess policy | Diagnostics |
hdx_sm_ica_clip_channel_disabled | Total number of NS_ICA_CLIP_CHANNEL disabled via SmartAccess policy | Diagnostics |
hdx_sm_ica_ccm_channel_disabled | Total number of NS_ICA_CCM_CHANNEL disabled via SmartAccess policy | Diagnostics |
hdx_sm_ica_cdm_channel_disabled | Total number of NS_ICA_CDM_CHANNEL disabled via SmartAccess policy | Diagnostics |
hdx_sm_ica_com1_channel_disabled | Total number of NS_ICA_COM1_CHANNEL disabled via SmartAccess policy | Diagnostics |
hdx_sm_ica_com2_channel_disabled | Total number of NS_ICA_COM2_CHANNEL disabled via SmartAccess policy | Diagnostics |
hdx_sm_ica_cpm_channel_disabled | Total number of NS_ICA_CPM_CHANNEL disabled via SmartAccess policy | Diagnostics |
hdx_sm_ica_lpt1_channel_disabled | Total number of NS_ICA_LPT1_CHANNEL disabled via SmartAccess policy | Diagnostics |
hdx_sm_ica_lpt2_channel_disabled | Total number of NS_ICA_LPT2_CHANNEL disabled via SmartAccess policy | Diagnostics |
dx_dbg_sm_ica_msi_disabled | Total number of cases where MSI is disabled via SmartAccess policy | Diagnostics |
hdx_sm_ica_file_channel_disabled | Total number of NS_ICA_FILE_CHANNEL is disabled via SmartAccess policy | Diagnostics |
hdx_dbg_usb_accept_device | Total number of USB devices accepted | Diagnostics |
hdx_dbg_usb_reject_device | Total number of USB devices rejected | Diagnostics |
hdx_dbg_usb_reset_endpoint | Total number of USB endpoints reset | Diagnostics |
hdx_dbg_usb_reset_device | Total number of USB devices reset | Diagnostics |
hdx_dbg_usb_stop_device | Total number of USB devices stopped | Diagnostics |
hdx_dbg_usb_stop_device_response | Total number of responses from stopped USB devices | Diagnostics |
hdx_dbg_usb_device_gone | Total number of USB devices gone | Diagnostics |
hdx_dbg_usb_device_stopped | Total number of USB devices stopped | Diagnostics |
nstrace validation
Check for CFLOW protocol to see all AppFlow records going out of NetScaler.
Population of records in NetScaler ADM checklist
-
Run the command
tail -f /var/mps/log/mps_afdecoder.log | grep -i "Data Record: ica_"
and check logs to confirm NetScaler ADM is receiving AppFlow records. - Confirm NetScaler instance is added to NetScaler ADM.
- Validate NetScaler Gateway/VPN virtual server is licensed in NetScaler ADM.
- Make sure multi-hop parameter setting is enabled for double-hop.
- Make sure NetScaler Gateway is cleared for second-hop in double-hop deployment.
Before contacting Citrix technical support
For a speedy resolution, make sure that you have the following information before contacting Citrix technical support:
-
Details of the deployment and network topology.
- NetScaler and NetScaler ADM versions.
- Citrix Virtual Apps and Desktops server versions.
- Client Workspace versions.
- Number of Active ICA sessions when the issue occurred.
- Tech support bundle captured by running the
show techsupport
command at the NetScaler command prompt. - Tech support bundle captured for NetScaler ADM.
- Packet traces captured on all NetScaler.
To start a packet trace, type,
start nstrace -size 0'
To stop a packet trace, type,stop nstrace
- Collect entries in the system’s ARP table by running the
show arp
command.
Known Issues
Refer ADC release notes for known issues on HDX Insight.