Product Documentation
ADC
14.1 - Current Release 13.1 13.0 12.1
View PDF

Logs

October 24, 2024
Contributed by: 
C

Logs refer to the software-generated data that contains information about usage patterns, activities and operations within an operating system, application, server, or another device. The files that store these logs are called the log files. Log files are the primary data source for network observability.

This topic covers the following:

Audit logs

Auditing is a methodical examination or review of a condition or situation. The audit logging feature enables you to log the NetScaler states and status information collected by various modules. The log information can be in the kernel and in the user-level daemons.

You can configure NetScaler to store the log information locally on NetScaler or export it to a remote server or both. For more information on audit logs, see Audit logging.

Configure audit logs

For information on how to configure NetScaler for audit logging, see Configuring NetScaler appliance for audit logging.

Sample audit log

The following is a sample audit log:

May 31 02:05:23 <local0.info> 10.106.44.21  05/31/2024:06:05:23 GMT HA1 0-PPE-0 : default API CMD_EXECUTED 7825 0 :  User admuser -
ADM_User NONE - Remote_ip 10.106.44.21 - Command "show ns license" - Status "Success"
<!--NeedCopy-->

The following image describes the audit log:

Audit log format

Export audit logs

You can export audit logs and events from NetScaler to industry standard log aggregator platforms such as Splunk and get meaningful insights. For more information, see Export audit logs and events directly from NetScaler to Splunk.

You can export audit logs securely over an SSL connection. For more information, see Export syslog securely over SSL.

Management logs

Management logs are the logs that are generated by the processes that are running on the NetScaler management core. (For example, bash, httpd). The messages that are part of the following log files are exported as part of management logs:

  • Shell or platform logs

    • bash.log: The log file is created by the bash process. It records the commands processed by the bash shell.
    • sh.log: The log file is created by the sh process. It records the commands processed by the sh shell.

Access logs

  • auth.log: The log file records all SSHd activity and Console activity.

  • nsvpn.log: The log file records detailed information about Virtual Private Network (VPN) activities and events.

  • httpaccess.log: The log file records any HTTP activity directed to the management IP address is logged. Typically, this log file includes NetScaler GUI access, NITRO API and VPN access.

  • httperror.log: The log file records any HTTP activity directed to the management IP address and resulting in an error due to any timeout (example, PHP maximum timeout reached) or any programming error in the backend. Periodic Apache restarts also get logged into this file.

  • httpaccess-vpn.log: The log file consists of gateway portal endpoints data served by apache that are accessed by users along with the date and time.

  • httperror-vpn.log: The log file consists of gateway portal endpoints data served by apache that are accessed by users along with the date and time.

NetScaler management logs

  • notice.log: The log file is a syslog data stream and it records the syslog records that are generated according to the “notice” datastream of syslogd.

  • ns.log: The log file contains the NetScaler system log. The logs generated by several modules of NetScaler (for example, load balancing, SSL, app firewall) are written into this file. During export, only the content of ns.log which is written by the management processes are exported. The logs written by the NSPPE processes are not exported.

Management logs are logged to respective log files by the syslogd running on NetScaler which are generated by various processes running on management core. Management logs are exported periodically, typically every 60 seconds.

Configure export of management logs

You can configure the export of management logs using one of the following methods:

  • Configure export of management logs to a syslog server
  • Configure export of management logs to an HTTP server or HEC endpoint such as Splunk

Note:

When configuring syslog action and analytics profiles, it is recommended not to use NSMGMT for management log. Using NSMGMT might negatively impact the performance of NetScaler.

For example, in add audit syslogAction test1 <IP address> -serverPort <port> -logLevel ALL -managementlog <ACCESS | ALL | NONE | NSMGMT | SHELL> and add analytics profile <profile-name> -type timeseries -managementlog <ACCESS | ALL | NONE | NSMGMT | SHELL> commands, avoid using NSMGMT for -managementlog.

Configure export of management logs to Syslog server

To configure the export of management logs, you must perform the following steps:

  1. Create an audit syslog action on NetScaler with the management log option.
  2. Create a syslog audit policy with the syslog action.
  3. Bind the syslog audit policy with the system global entity to enable logging of all NetScaler system events.

Note:

  • Management logs are not supported in non-default admin partitions.
  • If the transport is TCP, then management logs are not exported out of secondary nodes by default. Load balancing configuration can be used to enable secondary export for syslog when the destination is reachable using NSIP address and secondary export is required.
  • Starting with NetScaler release 14.1-29.x, the management logs exported by NetScaler comply with RFC 5424.
Configure syslog audit action

To configure syslog audit action on NetScaler using CLI, run the following command:

add audit syslogAction <name> \(<serverIP> \[-serverPort <port>] -logLevel <logLevel> ... \[-managementlog <managementlog> ...] ... \[-mgmtloglevel <managementloglevel> ...]\[-transport \( TCP | UDP )])
<!--NeedCopy-->

Example:

add audit syslogAction test 10.106.186.102 -serverPort 514 -logLevel ALL -managementlog SHELL NSMGMT -mgmtloglevel ALL -transport TCP
<!--NeedCopy-->

In this configuration:

  • name: Name of the syslog action
  • serverIP: IP address of the syslog server.
  • serverPort: Port on which the syslog server accepts connections.
  • logLevel: Audit log level.
  • managementlog: Types of management logs that you must export.
  • mgmtloglevel: Management log levels that you want to set for export.
  • transport: The transport type used to send audit logs to the syslog server.

To configure syslog audit action on NetScaler using the GUI, perform the following steps:

  1. Navigate to the System > Auditing > Syslog > Server tab and click Add.
  2. On the Create Auditing Server page, provide the following details:
    1. Enter a name for the Syslog server.
    2. Select Server IP from the Server Type list and enter the syslog server IP address and port.
    3. Choose the required log levels from the Log Levels, Management Logs and Management Log Levels sections.
  3. Click Create.
Load balance management logs

In releases prior to NetScaler 14.1-29.x, when management logs are enabled, the syslogAction configuration supports only server IP address and port configuration. Domain Based Service (DBS) and load balancing virtual server name configurations are not supported. For load balancing management logs, use the following sample configuration:

add service syslog_server <server_ip> UDP <port>
add service syslog_server1 1.3.4.4 UDP 514
add service syslog_server2 1.3.4.5 UDP 514
add lb vserver lb1 UDP <lb_vip> <lb_port>
bind lb vserver lb1 syslog_server1
bind lb vserver lb1 syslog_server2
<!--NeedCopy-->

In syslogAction, configure the following:

add syslogAction sys1 <server_ip> -serverPort <server_port> -transport UDP -loglevel <loglevel>
add syslogAction sys1 lb_vip -serverPort lb_port -transport UDP -loglevel <loglevel>
<!--NeedCopy-->

In releases NetScaler 14.1-29.x and later, when management logs are enabled, the syslogAction configuration supports load balancing virtual server name. For load balancing management logs, use the following sample configuration:

add service syslog_server1 10.9.8.7 UDP 514
add service syslog_server2 10.9.8.8 UDP 514
add lb vserver syslog_lb SYSLOGUDP
bind lb vserver syslog_lb syslog_server1
bind lb vserver syslog_lb syslog_server2
<!--NeedCopy-->

Configure syslog to use the above virtual server, and enable management logs on the action.

add syslogAction syslog_act -lbVserverName syslog_lb -logLevel NONE -managementLog ALL -mgmtlogLevel ALL
add syslogPolicy syslog_pol true syslog_act
bind syslogGlobal -policyName syslog_pol -priority 1
<!--NeedCopy-->

You can also configure domain name on syslog action, instead of IP address. The following is a sample configuration:

add syslogAction syslog_act my.syslog.server -logLevel NONE -managementLog ALL -mgmtlogLevel ALL
<!--NeedCopy-->
Configure syslog audit policy

To configure a syslog audit policy using the CLI, run the following command:

add audit syslogPolicy <name> TRUE <syslogAction>

Example:

add audit syslogPolicy test-policy TRUE test

To configure a syslog audit policy using the GUI, perform the following steps:

  1. Navigate to the System > Auditing > Syslog > Policies tab and click Add.
  2. On the Create Auditing Syslog Policy page, enter a name, select Advanced Policy, and then select the audit syslog server that you created from the Server list.
Bind audit log policy

To bind the syslog audit log policy to the bind point SYSTEM_GLOBAL using CLI, run the following command:

bind audit syslogGlobal <policyname> -globalBindType SYSTEM_GLOBAL

Example:

bind audit syslogGlobal test-policy -globalBindType SYSTEM_GLOBAL

To bind the syslog audit log policy globally using the GUI, run the following command:

  1. Navigate to the System > Auditing > Syslog > Policies tab and select the syslog audit policy that you created.
  2. Right-click on the selected syslog audit policy and click Advanced Policy Global Bindings.
  3. Select the syslog audit policy that you created from the Select Policy list.
  4. Enter the priority in the Priority field.
  5. Select SYSTEM_GLOBAL from the Global Bind Type field and click Bind.
  6. On the Syslog Auditing page, select the syslog audit policy and click Done.

Once the configuration is successful, the management logs are sent as syslog payloads to the external syslog server.

Configure export of management logs to Splunk configured as HTTP server

You can configure Splunk as an HTTP server. In the HTTP server configuration, you can use the HTTP event collector to send management logs over HTTP (or HTTPS) directly to the Splunk platform from your NetScaler. For more information, see Export management logs to Splunk configured as an HTTP server.

Sample management log

The following is a sample management log:

10.222.4.55 10.217.11.103 [04/Oct/2023:18:38:34 +0000] [43533] "GET / HTTP/1.1" 200 20783 "Go-http-client/l.1" "Time: 38848 microsecs"

127.0.0.1 [04/0ct/2023:16:30:00 +0000] [16510] "GET / HTTP/1.1" 200 20783 “curl/7.87.0" "Time: 42015 microsecs"

[Wed Oct 04 16:00:00.093670 2023] [authz_core:error] [pid 7487] [client 127.0.0.1:16161] AH01630: client denied by server configuration: /netscaler/ns_gui/var
<!--NeedCopy-->

The following image describes the management log:

Management log format

The following is a sample bash log:

Apr 24 11:21:00 <local7.notice> ns bash[3893]: root on /dev/pts/0 shell_command="PATH=/netscaler:/bin:/sbin:/usr/bin:/usr/sbin:/usr/libexec:/usr/local/bin:/usr/local/sbin:."
Apr 24 11:21:00 <local7.notice> ns bash[3893]: root on /dev/pts/0 shell_command="EDITOR=/usr/bin/vi"
Apr 24 11:21:00 <local7.notice> ns bash[3893]: root on /dev/pts/0 shell_command="[ -f /var/python/bin/python ]"
Apr 24 11:21:00 <local7.notice> ns bash[3893]: root on /dev/pts/0 shell_command="export PATH=$PATH:/var/python/bin"
Apr 24 11:21:00 <local7.notice> ns bash[3893]: root on /dev/pts/0 shell_command="[ -f /var/golang/bin/go ]"
Apr 24 11:21:00 <local7.notice> ns bash[3893]: root on /dev/pts/0 return_code="1"
Apr 24 11:21:00 <local7.notice> ns bash[3893]: root on /dev/pts/0 shell_command=""
Apr 24 11:21:12 <local7.notice> ns bash[3893]: root on /dev/pts/0 shell_command="cat /var/log/bash.log"
<!--NeedCopy-->

The following is a sample httpaccess log:

10.102.201.155 -> 10.102.201.155 - - [10/Jun/2024:23:07:11 +0530] [1571] "GET /nitro/v1/config/route6?format=json&sessionid=[FILTERED] HTTP/1.1" 200 1162 "-" "-" "Time: 9797 microsecs"

10.102.201.155 -> 10.102.201.155 - - [10/Jun/2024:23:07:11 +0530] [1571] "GET /nitro/v1/config/nslicense?format=json&sessionid=[FILTERED] HTTP/1.1" 200 1118 "-" "-" "Time: 7774 microsecs"

10.102.201.155 -> 10.102.201.155 - - [10/Jun/2024:23:07:11 +0530] [1571] "GET /nitro/v1/config/appflowparam?format=json&sessionid=[FILTERED] HTTP/1.1" 200 1643 "-" "-" "Time: 8098 microsecs"

10.102.201.155 -> 10.102.201.155 - - [10/Jun/2024:23:07:12 +0530] [1571] "GET /nitro/v1/config/snmpmib?format=json&sessionid=[FILTERED] HTTP/1.1" 200 347 "-" "-" "Time: 8185 microsecs"

10.102.201.155 -> 10.102.201.155 - - [10/Jun/2024:23:11:16 +0530] [1570] "GET /nitro/v1/config/nslicense?format=json&sessionid=[FILTERED] HTTP/1.1" 200 1118 "-" "-" "Time: 23195 microsecs"

10.102.201.155 -> 10.102.201.155 - - [10/Jun/2024:23:11:17 +0530] [1570] "GET /nitro/v1/config/nslicenseserver?format=json&sessionid=[FILTERED] HTTP/1.1" 200 57 "-" "-" "Time: 8992 microsecs"
<!--NeedCopy-->

The following is a sample httperror log:

[Mon Jun 10 15:00:01.016898 2024] [mpm_prefork:notice] [pid 47106] AH00171: Graceful restart requested, doing restart

[Mon Jun 10 15:00:01.323746 2024] [mpm_prefork:notice] [pid 47106] AH00163: Apache/2.4.56 (Unix) mod_perl/2.0.10 Perl/v5.30.2 configured -- resuming normal operations

[Mon Jun 10 15:00:01.323756 2024] [core:notice] [pid 47106] AH00094: Command line: '/bin/httpd -f /etc/httpd.conf'

[Mon Jun 10 15:30:00.211099 2024] [authz_core:error] [pid 80696] [client 127.0.0.1:31128] AH01630: client denied by server configuration: /netscaler/ns_gui/var

[Mon Jun 10 15:38:06.205307 2024] [authz_core:error] [pid 80696] [client 127.0.0.2:24903] AH01630: client denied by server configuration: /netscaler/ns_gui/var
<!--NeedCopy-->

The following is a sample auth log:

Jun 13 12:03:33 <auth.info> ns login: login on pts/1 as root
Jun 13 12:03:33 <auth.info> ns sshd[17798]: Received disconnect from 10.102.201.155 port 33145:11: Normal Shutdown
Jun 13 12:03:33 <auth.info> ns sshd[17798]: Disconnected from user admuser 10.102.201.155 port 33145
Jun 13 12:03:33 <auth.info> ns sshd[17803]: rexec line 19: Deprecated option UsePrivilegeSeparation
Jun 13 12:03:34 <auth.info> ns sshd[17803]: Accepted keyboard-interactive/NS for admuser from 10.102.201.155 port 65127 ssh2
<!--NeedCopy-->

The following is a sample nsvpn.log:

Jun  6 09:18:16 <local1.info> ns [1289]: (0-15) extract_ldap_attribute: retrieved mail value mail@citrix.com for Complex1, length is 15
Jun  6 09:18:16 <local1.info> ns [1289]: (0-15) receive_ldap_user_search_event: extracted attribute, name: mail, value: mail@citrix.com
Jun  6 09:18:16 <local1.info> ns [1289]: (0-15) receive_ldap_user_search_event: built group string for Complex1 of:group5 group4 group3 g1
Jun  6 09:18:16 <local1.info> ns [1289]: (0-15) receive_ldap_user_search_event: User search succeeded, attempting user authentication(Bind) for <Complex1>
Jun  6 09:18:16 <local1.info> ns [1289]: (0-15) receive_ldap_user_bind_event: Got user bind event.
Jun  6 09:18:16 <local1.info> ns [1289]: (0-15) receive_ldap_user_bind_event: User authentication (Bind event) for user Complex1 succeeded
Jun  6 09:18:16 <local1.notice> ns [1289]: (0-15) send_accept: sending accept to kernel for : Complex1
Jun  6 09:18:16 <local1.info> ns [1289]: (0-15) aaad_alloc_serialize_keyValue_attrs:  2 custom attribute key_len 4  value_len 15
Jun  8 06:21:48 <local1.info> ns [1289]: (0-62) aaad_remove_user_login_failure_info:  sqlite3_step is SUCCESS
<!--NeedCopy-->

The following is a sample httperror-vpn log:

[Thu Jun 06 17:43:38.184137 2024] [core:error] [pid 63478] (63)File name too long: [client 127.0.0.2:42117] AH00036: access to /logon/L                                                                                                                                                       ogonPoint/aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa                                                                                                                                                       aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa                                                                                                                                                       aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa                                                                                                                                                       aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa                                                                                                                                                       aaaaaaaaaResources/List failed (filesystem path '/var/netscaler/logon/LogonPoint/aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa                                                                                                                                                       aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa                                                                                                                                                       aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa                                                                                                                                                       aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa                                                                                                                                                       aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaResources')
[Thu Jun 06 17:44:23.338278 2024] [core:info] [pid 63475] [client 127.0.0.2:65214] AH00128: File does not exist: /var/netscaler/logon/L                                                                                                                                                       ogoqqqqnPoint/Resources/List
[Thu Jun 06 17:44:41.245211 2024] [core:info] [pid 63476] [client 127.0.0.2:56821] AH00128: File does not exist: /var/netscaler/logon/L                                                                                                                                                       og@
[Tue Jun 11 10:10:55.360797 2024] [core:info] [pid 37517] [client 127.0.0.2:53996] AH00128: File does not exist: /netscaler/ns_gui/vpn/                                                                                                                                                       pluginCustomization.json
[Mon Jun 17 14:11:15.816726 2024] [core:info] [pid 63188] [client 127.0.0.2:27594] AH00128: File does not exist: /netscaler/ns_gui/vpn/                                                                                                                                                       pluginCustomization.json
<!--NeedCopy-->

The following is a sample httpaccess-vpn log:

127.0.0.2 - - [20/Jun/2024:08:37:47 +0000] [1508] "GET /logon/LogonPoint/custom/strings.en.js?_=1718869081399 HTTP/1.1" 304 -          "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:120.0) Gecko/20100101 Firefox/120.0" "Time: 529 microsecs"
127.0.0.2 - - [20/Jun/2024:08:37:47 +0000] [1508] "GET /logon/LogonPoint/plugins/ns-gateway/nsg-epa.js HTTP/1.1" 304 - "-" "Mo         zilla/5.0 (Windows NT 10.0; Win64; x64; rv:120.0) Gecko/20100101 Firefox/120.0" "Time: 303 microsecs"
127.0.0.2 - - [20/Jun/2024:08:37:47 +0000] [1508] "GET /logon/LogonPoint/plugins/ns-gateway/nsg-setclient.js HTTP/1.1" 304 - "         -" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:120.0) Gecko/20100101 Firefox/120.0" "Time: 311 microsecs"
127.0.0.2 - - [20/Jun/2024:08:37:47 +0000] [1508] "GET /logon/LogonPoint/plugins/ns-gateway/ns-nfactor.js HTTP/1.1" 304 - "-"          "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:120.0) Gecko/20100101 Firefox/120.0" "Time: 315 microsecs"
127.0.0.2 - - [20/Jun/2024:08:37:47 +0000] [1508] "GET /vpn/media/citrixgateway_logo_white.png HTTP/1.1" 304 - "-" "Mozilla/5.         0 (Windows NT 10.0; Win64; x64; rv:120.0) Gecko/20100101 Firefox/120.0" "Time: 358 microsecs"
<!--NeedCopy-->

Export management logs

You can export management logs from NetScaler to industry standard log aggregator platforms such as Splunk. For more information, see Export management logs directly from NetScaler to Splunk.

Even after you configure export of management logs, there might be some issues that prevent management logs getting exported to the observability endpoints. This section provides tips for troubleshooting issues related to the following scenarios:

Management logs are not getting exported to the syslog server

When management logs are not getting exported to the syslog server, do the following:

  1. Check if you have enabled some management logs on the syslog action.

    Scenario

    John, a system administrator, ran the show syslogaction ns-mgmt-logs-act command. In the output, he found managementlog: NONE and mgmtloglevel: NONE, which confirmed that management logs were not enabled.

    Show syslogaction command output

    To fix this issue, he enabled management logs and management log levels by running the following command:

    add syslogaction ns-mgmt-logs-act 9.9.9.9 -loglevel none -mgmtlogLevel all -managementlog all
<!--NeedCopy-->

    After enabling, he ran the show syslogaction ns-mgmt-logs-act command and verified that the management logs were enabled as highlighted in the following output.

    Show syslogaction command output enabled

  2. Check if the configured logs are generated locally. For example, if you’ve configured management logs with INFO severity, check if /var/log/ns.log contains management logs (lines that do not contain ‘-PPE-’) generated with INFO severity. The following are a couple of scenarios:

    Scenario 1

    Cassidy, a system administrator, added the following action that had only NSMGMT warning logs enabled and found that the logs were not getting generated.

    add syslogaction ns-mgmt-logs-act 9.9.9.9 -loglevel none -mgmtlogLevel warning -managementlog NSMGMT
<!--NeedCopy-->

    Cassidy looked at the local log files ns.log and notice.log and found that warning level messages were not generated.

    root@ns# cat /var/log/ns.log | grep -v -e '-PPE-'
Jun 26 05:00:00 ns newsyslog[74791]: logfile turned over due to size>100K
rootQns# cat /var/log/notice.log | grep warn
Jun 26 05:08:34 <local7.notice> ns bash[81962]: root on /dev/pts/5 shell_command="cat /var/log/notice.log | grep warning"
Jun 26 05:08:40 <local7.notice> ns bash[81962]: root on /dev/pts/5 shell_command="cat /var/log/notice.log | grep warn"
Jun 26 05:09:15 <local7.notice> ns bash[81962]: root on /dev/pts/5 shell_command="cat /var/log/notice.log | grep warn"
<!--NeedCopy-->

    Scenario 2

    John, a system administrator, added the following action that had all management logs enabled for all levels.

    add syslogaction ns-mgmt-logs-act 9.9.9.9 -loglevel none -mgmtlogLevel all -managementlog all
<!--NeedCopy-->

    The logs were getting generated locally in the bash.log and therefore, the same appeared at their observability endpoint as well.

    root@ns# tail -f /var/log/bash.log
Jun 26 05:16:54 <local7.notice> ns bash[4278]: root on (null) return_code="l"
Jun 26 05:16:54 <local7.notice> ns bash[5909]: root on (null) shell_command="true"
Jun 26 05:16:54 <local7.notice> ns bash[5909]: root on (null) shell_command="curl 10.102.154.54; sleep 0.5" 
Jun 26 05:16:54 <local7.notice> ns bash[5909]: root on (null)shell_command="curl 10.102.154.54"
<!--NeedCopy-->

  3. Check and confirm the following:

    • If you are using the correct transport protocol
    • If you are using the correct IP address or domain name
    • If you are using the correct load balancing configuration
    • If the domain name is resolved
    • If the service or the virtual server is UP

  4. Check and confirm that the policy is created with ‘true’ rule and bound to system global. If the binding is not present, the logs are not exported.

    Scenario

    Cassidy, a system administrator, ran the show syslogglobal command and noticed that the policy was not bound to system global.

    > show syslogglobal
Done
<!--NeedCopy-->

    Cassidy ran the following command to bind the policy to system global:

    bind syslogGlobal -policyName ns-mgmt-logs-pol -priority 1 Done
<!--NeedCopy-->

    After binding, the show syslogglobal command showed the binding details.

    > sh syslogglobal
1)      Policy Name: ns-mgmt-logs-pol
        Priority: 1
        GlobalType: SYSTEM_GLOBAL
Done
<!--NeedCopy-->

  5. Check management log counters. You can run shell nsconmsg -g mgmt_push -g mgmtlog -d current. For example, if you see mgmtlog_tot_send_syslog_failures_due_to_server_not_up increment in the output, it indicates that the logs were not sent because the server was not up.

  6. Run nstrace to check the UDP or TCP flow for any abnormalities. Check if the endpoint is configured to use TLS or SSL but NetScaler is actually sending unencrypted data or conversely.

Management logs not seen on Splunk (HEC)

When management logs are not getting exported to Splunk (HEC), perform the following:

  1. Check if you have enabled management logs on a time series profile. To check, run the show analyticsprofile <profilename> command and see the output.

    Scenario

    Cassidy, a system administrator, realized that she actually needed the JSON format and not the syslog format. Therefore, she removed all management syslog configurations and added analytics configurations. However, she saw that the logs were not going out of NetScaler. When she ran the show analyticsprofile <name> command, she saw Managementlog: NONE in the output and realized that she had forgotten to enable management logs on the analytics profile.

    To fix the issue, she enabled management logs using the set analytics profile <name> -managementlog ACCESS. With this fix, she saw that the logs were sent out of the NetScaler and also the show analyticsprofile <name> command showed Managementlog: ACCESS.

    Show analytics Profile Management log set to none

  2. Check if the configured logs are actually being generated locally. For example, if you’ve configured NSMGMT management logs, you can check if /var/log/ns.log has management logs (lines that do not contain ‘-PPE-’) generated.

  3. Check and confirm the following:

    • If you are using the correct profile. You must use the time series profile.
    • If the service is resolved and is UP.
    • If metadata (-analyticsEndpointMetadata) is properly configured and is a valid JSON and also if all the fields contained in it are present in the logs exported to Splunk.
    • If the Splunk Auth Token is correct. Because the auth token is not displayed, you have to ensure it is entered correctly in the format -analyticsAuthToken ‘Splunk <token>’.

  4. Run the shell nsconmsg -g mgmt_push -g mgmtlog -d current command and check for management log counters. For example, if you see mgmtlog_tot_json_non_2xx_http_responses increment, then it indicates that the logs failed to reach Splunk because it responded with non-2XX response.

  5. Run nstrace to check the TCP or HTTP flow for any abnormalities. Check if the endpoint is configured to use TLS or SSL but NetScaler is sending unencrypted data or conversely. You can further check if the exported JSON is not malformed.

Common troubleshooting tips when management logs are not seen on any observability endpoint

If the tips provided in the Management logs are not getting exported to syslog server and Management Logs not seen on Splunk (HEC) sections do not resolve your issue, perform the following:

  1. Look at the following log files related to the process that exports management logs. These log files might give you an idea about the issue:

    • /var/log/export_mgmtlog_status.log
    • /var/log/export_mgmtlog_cron.log

    Scenario: Cassidy, a system administrator, saw broken pipe errors in export_mgmtlog_status.log and found that their collector was sending resets because of firewall issues, which was rectified at their end.

    Fri May 10 04:02:15 2024   do_logexport.py   ERROR   Traceback (most recent call last): 
   File "/netscaler/do_logexport.py", line 505, in netio_send
   pe_socket.sendall(data)
  BrokenPipeError: [Errno 32] Broken pipe
<!--NeedCopy-->

  2. Check if the configuration file /nsconfig/mgmtlogcfg.json is properly generated and also check if it hints to contain Syslog or JSON related configuration.

    For example, when both Syslog and JSON are configured, the /nsconfig/mgmtlogcfg.json file looks as shown below. The “syslog” field confirms that syslog is configured, while the “json” field confirms that json is configured.

    root@ns# cat /nsconfig/mgmtlogcfg.json
{"syslog":{"typemask":7,"levelmask":127,"ip":"127.0.0.2","port" :5566},"json":[{"name":"ns-mgmt-log-json-profile","typemask":2,"ip":"127.0.0.2","port":5563,"metadata":"{\"index\":\"ns-mgmt-log-syslog-index\"}"}]}
<!--NeedCopy-->

  3. Ensure that you are looking at the correct index and the correct timestamp.

    Scenario

    Cassidy, a system administrator, realized that she configured syslog index named ns-mgmt-log-syslog-index instead of ns-mgmt-log-json-index on Splunk, which was later rectified.

    show analyticsprofile correct index

  4. On cluster deployments, if the logs are not going out of non-CCO nodes, try running the force cluster sync command.

  5. HA deployments: You might notice that the secondary node is not sending logs by default. If you are exporting JSON, then you can enable the appflow param TimeseriesOverNSIP to allow a secondary node to send logs using the NSIP address to the collector, if the latter is reachable using the NSIP address (SNIP addresses cannot export logs out of secondary nodes). If you are exporting Syslog, then you can use a load balancing virtual server instead to export information out of secondary nodes.

  6. Netprofiles: It is also possible to use netprofiles. For syslog, ensure that you have set the ‘-netprofile’ on the syslog action. If you are load balancing syslogs or you are exporting them in JSON format, then ensure that you have set the netprofile on the service.

  7. Partitions: Management logs feature is not supported on non-default Partitions.

  8. Timestamps: Management logs generated by the processes are periodically exported. Hence, the time you receive the management log might not be the same as the time it was generated. Also, the date format might not comply with the date format configured on the syslog action, because management syslog follows RFC-5424 format.

Logs
October 24, 2024
Contributed by: 
C
October 24, 2024
Contributed by: 
C

In this article

Site feedback | Your privacy choices footer linkYour Privacy Choices | Privacy and legal terms | Cookie preferences | Consent settings | docs.cloud.com
© 1999- Cloud Software Group, Inc. All rights reserved.