Product Documentation
ADC
14.1 - Current Release 13.1 13.0 12.1
View PDF

Processed insights

September 3, 2024
Contributed by: 
C

NetScaler records the application traffic flow events such as HTTP requests and responses in the form of counters and transactions. When this data is sent to NetScaler Console or other observability tools, the receiving tool processes the data and provides actionable insights. For example, NetScaler Console processes all transactions and generates Web Insight reports. However, this method poses challenges as significant data must be transferred from NetScaler to observability tools and logics must be created to obtain insights.

From NetScaler release 14.1-29.x onwards, NetScaler can process the raw data and generate processed insights that can be sent outside NetScaler. The following are the advantages of NetScaler generating processed insights:

  • The data transferred from NetScaler to NetScaler Console or any other observability tool is minimum.
  • Aggregation and filtering of data can be done according to the use case from the source itself. For example, aggregate data based on a client or a URL.
  • The data transferred is easily consumable on observability tools without any processing.
  • Dashboards on NetScaler Console or other observability tools can be built efficiently.

Processed insights from NetScaler using stream analytics

The performance of a website or an application depends on how well you optimize the delivery of the content. Stream analytics aggregates the real-time statistics of the website or application traffic based on the expressions provided in an entity called, selector. For example, a selector can be a client IP or a URL. You can configure a stream identifier that specifies the selector to be used, the statistics collection interval, the sample count, and the field on which the records must be sorted. You can sort the data based on the requests and create a list of top records, such as, top URLs, top clients. These top records are known as top N, where N is the number or count of records listed.

NetScaler processes the top N records that are generated by the stream analytics and provides insights. You can also export the stream analytics data to NetScaler Console or any other observability tool.

Based on the data generated by the stream identifier, you can take the required actions, such as optimizing the traffic, limiting bandwidth consumption for user or client. For more information, see Configure policy for optimizing traffic and How to limit bandwidth consumption for user or client device.

Notes:

  • You can bind one stream identifier to multiple virtual servers and collect statistics from multiple virtual servers for the same stream selector expressions.
  • You can export a maximum of 1000 Top N records per stream identifier.
  • Stream identifier aggregates result data as stream sessions, based on the evaluation of the selector expression. The total length of the stream session is limited to 60 characters and the length is equally shared among the number of selector expressions. For example, if the selector has 5 expressions, each expression output gets 12 characters.
  • You can configure export of stream analytics data through syslog and through analytics profile at the same time.

Configure stream analytics using the CLI

To configure stream analytics using the CLI, perform the following steps:

  1. Configure stream selector.

    add stream selector <name> <rule> ...
<!--NeedCopy-->

    Example: To collect statistics aggregated at client IP and URL, run the following command:

    > add stream selector myselector HTTP.REQ.URL CLIENT.IP.SRC
 Done
> show stream selector myselector
    Name: myselector
    Expressions:
        1) HTTP.REQ.URL
        2) CLIENT.IP.SRC
 Done
>
<!--NeedCopy-->

  2. Configure an identifier for the selector.

    add stream identifier <identifier name> <selector name> -interval <int in mins> -sort <req> -Log SYSLOG -LogInterval <int in mins> -LogLimit <int>
<!--NeedCopy-->

    Example: To get statistics for each Client IP and URL and export them to SYSLOG, run the following command:

    add stream identifier TopTalkers myselector -interval 5  -log SYSLOG -loginterval 10 -loglimit 10
<!--NeedCopy-->

  3. Create a responder policy to collect stats for the given identifier.

    add responder policy <resp policy name> "analytics.stream(\"<identifier name>\").collect_stats" NOOP
<!--NeedCopy-->

    Example:

    add responder policy TopTalkers "analytics.stream(\"TopTalkers\").collect_stats" NOOP
<!--NeedCopy-->

  4. Bind the responder policy to the target virtual server for which the traffic is to be analyzed by the identifier. To enable the same stream identifier to process traffic from multiple virtual servers, bind the responder policy to all the virtual servers.

    bind lb vserver dnslb -policyName <resp policy name> -priority 1 -gotoPriorityExpression NEXT -type REQUEST
<!--NeedCopy-->

    Example:

    bind lb vserver dnslb -policyName TopTalkers -priority 1 -gotoPriorityExpression NEXT -type REQUEST
<!--NeedCopy-->

Configure stream analytics using the GUI

To configure stream analytics through the GUI, perform the following steps:

  1. Navigate to AppExpert > Action Analytics > Selectors.

  2. In the details pane, click Add and set the Name and Expressions parameter.

  3. Click Insert and then click Create.

  4. Navigate to AppExpert > Action Analytics > Stream Identifiers.

  5. In the details pane, click Add. In the Configure Stream Identifier page, set the following parameters, and click Create.

    • Name
    • Selector
    • Interval
    • Sample Count
    • Sort

  6. Navigate to AppExpert > Responder > Policies. On the Responder Policies page, click Add.

  7. On the Create Responder Policy page, enter a name for the responder policy.

  8. In the Expression editor, configure the condition for the responder policy. For example, ‘sys.check_limit(“limit_one_login”)’ and click Create.

  9. Navigate to Traffic Management > Load Balancing > Virtual Servers.

  10. On the Load Balancing Virtual Servers page, select the virtual server to which you want to bind the responder policy, and then click Open.

  11. In the Configure Virtual Server (Load Balancing) dialog box, select the Policies tab. Select the checkbox next to the name of the policy that you want to bind to this virtual server and click OK.

Export of stream analytics data using syslog through CLI

To export the stream analytics data using syslog, perform the following steps by using the NetScaler CLI:

  1. Create a syslog server endpoint. You can create multiple syslog servers for multiple log levels.

    add audit syslogAction <syslog servername> <server ip address>  -logLevel ALL -managementlog ALL -mgmtlogLevel ALL -dateFormat YYYYMMDD -userDefinedAuditlog YES -transport TCP -streamAnalytics <ALL/NONE>
<!--NeedCopy-->

    Example:

    add audit syslogAction topk_syslog_server 10.106.44.6 -logLevel ALL -managementlog ALL -mgmtlogLevel ALL -dateFormat YYYYMMDD -userDefinedAuditlog YES -transport TCP -streamAnalytics  <enabled/disabled>
<!--NeedCopy-->

  2. Create a syslog policy for the syslog server created in step 5.

    add audit syslogPolicy <syslog policy> <syslog servername>
<!--NeedCopy-->

    Example:

    add audit syslogPolicy topk_syslog true topk_syslog_server
<!--NeedCopy-->

  3. Bind the syslog policy at the right level.

    bind syslogGlobal -policyName <syslog policy> -priority 1
<!--NeedCopy-->

    Example:

    bind syslogGlobal -policyName topk_syslog -priority 1
<!--NeedCopy-->

Export of stream analytics data using syslog through GUI

To export the stream analytics data using syslog, perform the following steps by using the NetScaler GUI:

  1. Navigate to Configuration > System > Auditing > Syslog.

  2. In the Policies tab, click Add.

  3. Enter a name for the policy, select Advanced Policy and click Add to create an auditing server.

  4. Enter the required details in the Create Auditing server, select Stream Analytics, and click Create.

  5. On the Syslog Auditing page, select the policy, and click Advanced Policy Global Bindings.

  6. Select the policy, priority, and SYSTEM_GLOBAL from the Global Bind Type drop-down list. Click Bind.

Export stream analytics data to Observability tools using the CLI

You can export the stream analytics data in JSON format to third-party observability tools, such as Splunk. To export stream analytics data by using the NetScaler CLI, perform the following steps:

  1. Create an analytics profile of type streaminsight and enable the topn parameter.

    add analytics profile <profile-name> -collectors Splunk -type streaminsight -topn ENABLED
<!--NeedCopy-->

    Example:

    add analytics profile Splunk_Prof -collectors Splunk -type streaminsight -topn ENABLED
<!--NeedCopy-->

  2. Bind the stream identifier to the analytics profile.

    bind stream identifier <identifier-name> -analyticsprofile <profile-name>
<!--NeedCopy-->

    Example

    bind stream identifier TopTalkers -analyticsprofile Splunk_Prof
<!--NeedCopy-->

  3. Add a responder policy.

    add responder policy <policy-name> <expression>
<!--NeedCopy-->

    Example:

    add responder policy TopTalkers "analytics.stream(\"TopTalkers\").collect_stats" NOOP
<!--NeedCopy-->

  4. Bind the responder policy to the load balancing virtual server.

    bind lb vserver <vserver-name> -policyName <policy-name> -priority 1 -gotoPriorityExpression NEXT -type REQUEST
<!--NeedCopy-->

    Example:

    bind lb vserver dnslb -policyName TopTalkers -priority 1 -gotoPriorityExpression NEXT -type REQUEST
<!--NeedCopy-->

  5. Verify the configuration by running the show analytics profile <name> and show stream identifier globalidentifier commands.

    Example 1: In the following example, the reference count indicates the stream identifiers associated with the analytics profile.

    >   sh analytics profile Splunk_Prof 
1)          Name: Splunk_Prof
            Collector: Splunk_ssl
            Profile-type: streaminsight
                    TopN: ENABLED
                     Data Format File: splunk_format.txt
            Authentication Token : ffc4a9d8c60f238e89eafcc9827991fal3302c41b2el455e4e2e262
            Endpoint URL: /services/collector/event
            Endpoint Content-type: application/json
            Reference Count: 2
            Managementlog: NONE
Done
<!--NeedCopy-->

    Example 2: The following is a sample output of sh stream identifier globalidentifier:

    > sh stream identifier globalidentifier
Name: globalidentifier
        Selector: clientipsrc
        Expressions:
                1) CLIENT.IP.SRC
        Sample Count: 1 Interval: 1 min
        Log : SYSLOG Log Interval : 1 min Log Limit : 500
        Sort by : BANDWIDTH
        Appflow logging: DISABLED
        SNMP trap: DISABLED
        Track Ack only pkts: DISABLED
<!--NeedCopy-->

Export of stream analytics data to Observability tools using the GUI

You can export the stream analytics data in JSON format to third-party observability tools, such as Splunk. To export the stream analytics data by using the NetScaler GUI, perform the following steps:

  1. Create an analytics profile of type streaminsight and enable the topN parameter.

    1. Navigate to System > Profiles > Analytics Profiles and click Add.

    2. On the Create Analytics Profile page, provide the following details:

      1. Enter a name for the profile.
      2. From the Collectors list, select the service that you created.
      3. Select a STREAM INSIGHT from the Type list.
      4. Enter the Analytics Auth Token that you received from the Observability endpoint such as Splunk.
      5. Enter the details for Analytics Endpoint URL, Analytics Endpoint Content Type, and Analytics Endpoint Metadata.
      6. Selecct topN.
      7. Select the Management Logs that you want to export and also the Output Mode in which you want to export.
      8. Click Create.

  2. Navigate to Configuration > AppExpert > Action Anlytics > Stream Identifiers.

  3. Select one or more required stream identifiers and click Show Bindings.

  4. Click Bind. Select the analytics profile of type streaminsight and click Bind.

The stream analytics data gets exported through the collector configured on the analytics profile of type streaminsight.

Sample stream analytics data sent in syslog 

New audit log message module type for Stream Analytics = SA.

Apr  2 15:30:14 10.106.44.24  04/02/2024:09:58:28 GMT HA1 0-PPE-0 : default SA Message 3285 0 : "stream_iden_name: TopKTalkers, stream_sess_name: 1.1.1.1, Requests: 5836754, Bandwidth: 1635886, RespTime: 51, Connections: 9"
Apr  2 15:30:14 10.106.44.24  04/02/2024:09:58:28 GMT HA1 0-PPE-0 : default SA Message 3286 0 : "stream_iden_name: TopKTalkers, stream_sess_name: 2.2.2.2, Requests: 4939608, Bandwidth: 1384438, Resptime: 120, Connections: 7"
Apr  2 15:30:14 10.106.44.24  04/02/2024:09:58:28 GMT HA1 0-PPE-0 : default SA Message 3287 0 : "stream_iden_name: TopKTalkers, stream_sess_name: 3.3.3.3, Requests: 4293067, Bandwidth: 1203231, RespTime: 73, Connections: 8"
Apr  2 15:30:14 10.106.44.24  04/02/2024:09:58:28 GMT HA1 0-PPE-0 : default SA Message 3289 0 : "stream_iden_name: TopKTalkers, stream_sess_name: 4.4.4.4, Requests: 3834040, Bandwidth: 1074577, ResptTime: 109, Connections: 5"
<!--NeedCopy-->

The following image explains the stream analytics data sent in syslog:

stream analytics data

Sample stream analytics data sent in JSON format to observability tools

The following image explains the stream analytics data sent in JSON format to Splunk:

stream analytics data sent to Splunk

Processed insights
September 3, 2024
Contributed by: 
C
September 3, 2024
Contributed by: 
C

In this article

Site feedback | Your privacy choices footer linkYour Privacy Choices | Privacy and legal terms | Cookie preferences | Consent settings | docs.cloud.com
© 1999- Cloud Software Group, Inc. All rights reserved.