Application Delivery Management

Troubleshoot Gateway Insight issues

If the Gateway Insight solution is not functioning as expected, the issue might be with one of the following. Refer to the checklists in the respective sections for troubleshooting.

  • Gateway Insight configuration.
  • Connectivity issue between NetScaler and NetScaler Console.
  • Record generation in NetScaler.
  • Validations in NetScaler Console.

Gateway Insight configuration checklist

  • Make sure that the AppFlow feature is enabled in the NetScaler appliance. For details, see Enabling AppFlow.

  • Check the Gateway Insight configuration in the NetScaler running configuration.

    Run the show running | grep -i <appflow_policy> command to check the Gateway Insight configuration. Make sure that the bind type is REQUEST. For example;

     bind vpn vserver afsanity -policy afp -priority 100 -type REQUEST
     <!--NeedCopy-->
    

    Bind type OTHERTCP_REQUEST is also required for Gateway Insight.

     bind vpn vserver afsanity -policy afp -priority 100 -type OTHERTCP_REQUEST
     <!--NeedCopy-->
    
  • For single-hop, Access Gateway, or Unified Gateway deployment, make sure that Gateway Insight AppFlow policy is bound to the VPN virtual server, where VPN traffic is flowing. For details, see Enabling HDX Insight data collection.
  • For double-hop, Gateway Insight must be configured on both the hops.
  • Check appflowlog parameter in NetScaler Gateway/VPN virtual server. For details, see Enabling AppFlow for Virtual Servers.

Connectivity between NetScaler and NetScaler Console checklist

  • Check AppFlow collector status in NetScaler. For details, see How to check the status of connectivity between NetScaler and AppFlow Collector.

  • Check Gateway Insight AppFlow policy hits.

    Run the command show appflow policy <policy_name> to check the AppFlow policy hits.

    You can also navigate to Settings > AppFlow > Policies in the GUI to check the AppFlow policy hits.

  • Validate any firewall blocking AppFlow ports 4739 or 5557.

Record generation in NetScaler checklist

  • Run the nsconmsg -d stats -g ai_tot command and check for the stats increments in NetScaler.
  • Capture nstrace logs and check for CFLOW packets to confirm NetScaler exports AppFlow records.

    Note:

    The nstrace logs are required only for IPFIX. For Logstream, nstrace logs do not confirm if the NetScaler appliance exported the AppFlow records.

Validation of records in NetScaler Console

  • Run the tail -f /var/mps/log/mps_afdecoder.log | grep -i "Data Record: vpn_" command to check the logs to confirm NetScaler Console is receiving AppFlow records.
  • Make sure that the NetScaler instance is added to the NetScaler Console.
  • Make sure that the NetScaler Gateway/VPN virtual server is licensed in NetScaler Console.

Validation of Logstream logs in NetScaler Console

Validation of Logstream data received by NetScaler Console can be done using the following methods:

  • Enabling data record logging in NetScaler Console

    Once enabled, the logs can be seen in the /var/mps/log/mps_afdecoder.log

  • Enabling ULFD library logging

    Run the command /mps/decoder_enable_debug

    The logs are captured in/var/ulflog/libulfd.log

    You can disable logging by using the command /mps/decoder_disable_debug

Gateway Insight counters

The following Gateway Insight counters are available.

  • ai_tot_preauth_epa_export
  • ai_tot_auth_export
  • ai_tot_auth_session_id_update_export
  • ai_tot_postauth_epa_export
  • ai_tot_vpn_update_export
  • ai_tot_ica_fileinfo_export
  • ai_tot_app_launch_failure
  • ai_tot_logout_export
  • ai_tot_skip_appflow_export
  • ai_tot_sso_appflow_export
  • ai_tot_authz_appflow_export
  • ai_tot_appflow_pol_eval_failure
  • ai_tot_vpn_export_state_mismatch
  • ai_tot_appflow_disabled
  • ai_tot_appflow_pol_eval_in_gwinsight
  • ai_tot_app_launch_success

AppFlow records in NetScaler log

Starting from release 13.0 build 71.x, you can check the NetScaler logs to confirm if the AppFlow records are exported. The default log level of syslogparams captures all the error and information logs. In case you do not find a clue about the errors, enable all log levels including DEBUG in syslogparams to capture even the DEBUG logs.

Sample logs

<local0.info> … GMT 0-PPE-0 : default SSLVPN Message 147 0 : "GwInsight: Sent auth record Func=ns_sslvpn_export_auth_data Username=<name> Clientip=<ip>:<port> Destip=0:80 SessSeq=0 Sessid=<sessid> Gwip=<ip>:443 StatusCode=0 CSappid=0 CSAppname=(null) VPNfqdn=<vpnfqdn> Authtype=3 EPAid=(null) AuthStage=1 AuthDuration=309 AuthAgent=<auth_server_ip> Groupname= Policyname=<name> CurfactorPolname=<name> NextfactorPolname= CSecExpr= Devicetype=16777219 Deviceid=0 email="
<local0.err> … GMT 0-PPE-0 : default SSLVPN Message 143 0 : "GwInsight: Func=ns_aaa_copy_email_id_to_vpn_record input hash_attrs_len is zero"
<local0.err> … GMT 0-PPE-0 : default SSLVPN Message 148 0 : "GwInsight: Func=update_session_appflow_collector pcb or session is NULL"
<local0.info> … GMT 0-PPE-0 : default SSLVPN Message 165 0 : "GwInsight: Sent session update record Func=ns_sslvpn_send_update_record Username=<> Clientip=<ip>:<port> Destip=<ip>:80 SessSeq=1 Sessid=<sessid> Gwip=<ip>:443 StatusCode=0 CSappid=0 CSAppname=(null) VPNfqdn=<fqdn> SSOAuthMethod=0 SessState=2 SessMode=2 IIP=0 AppByteCount=0 ReqURL=/Citrix/Store
Web BackendServername= SSOurl= email="
SSO logs:
<!--NeedCopy-->
<local0.info> … GMT 0-PPE-0 : default SSLVPN Message 463 0 : "GwInsight: Sent session update record Func=ns_sslvpn_send_update_record Username=<name> Clientip=<ip>:<port> Destip=<ip>:80 SessSeq=2 Sessid=<sessid> Gwip=<ip>:443 StatusCode=150 CSappid=0 CSAppname=(null) VPNfqdn=<fqdn> SSOAuthMethod=1 SessState=4 SessMode=3 IIP=0 AppByteCount=0 ReqURL= BackendServername=<> SSOurl= email="
<!--NeedCopy-->
<local0.info> … GMT 0-PPE-0 : default SSLVPN Message 582 0 : "GwInsight: Sent session update record Func=ns_sslvpn_send_update_record Username=<name> Clientip=<ip>:<port> Destip=<ip>:80 SessSeq=2 Sessid=<sessid> Gwip=<ip>:443 StatusCode=150 CSappid=0 CSAppname=(null) VPNfqdn=<fqdn> SSOAuthMethod=3 SessState=4 SessMode=3 IIP=0 AppByteCount=0 ReqURL= BackendServername=<> SSOurl= email="
<!--NeedCopy-->
<local0.info> … GMT 0-PPE-0 : default SSLVPN Message 513 0 : "GwInsight: Sent session update record Func=ns_sslvpn_send_update_record Username=<name> Clientip=<ip>:<port> Destip=<ip>:80 SessSeq=2 Sessid=<sessid> Gwip=<ip>:443 StatusCode=150 CSappid=0 CSAppname=(null) VPNfqdn=<fqdn> SSOAuthMethod=2 SessState=4 SessMode=3 IIP=0 AppByteCount=0 ReqURL= BackendServername=<> SSOurl= email="
<!--NeedCopy-->
<local0.info> … GMT 0-PPE-0 : default SSLVPN Message 29796 0 : "GwInsight: Sent session update record Func=ns_sslvpn_send_update_record Username=<name> Clientip=<ip>:<port> Destip=<ip>:443 SessSeq=c Sessid=<sessid> Gwip=<ip>:443 StatusCode=155 CSappid=0 CSAppname=(null) VPNfqdn=<fqdn> SSOAuthMethod=6 SessState=4 SessMode=3 IIP=0 AppByteCount=0 ReqURL= BackendServername=<> SSOurl= email="
<!--NeedCopy-->

Contact Citrix technical support

For a speedy resolution, make sure that you have the following information before contacting Citrix technical support:

  • Details of the deployment and network topology.
  • NetScaler and NetScaler Console versions.
  • Tech support bundle for NetScaler and NetScaler Console.
  • nstrace capture during the issue.

Known Issues

Refer NetScaler release notes for known issues on Gateway Insight.