Product Documentation
ADC
14.1 - Current Release 13.1 13.0 12.1
View PDF

Monitoring NetScaler, applications, and application security using Prometheus

August 11, 2023
Contributed by: 
C

Metrics are a numeric representation of data that are measured over a certain period of time. Metric data is useful to track the health of a system over time. Prometheus is an open-source monitoring tool that collects metrics data and stores that data with a time stamp at which the data was recorded.

By monitoring and analyzing metrics, you can track the health of your applications, detect any anomalies, create alerts, and take necessary corrective actions to ensure robust software delivery.

NetScaler now supports directly exporting metrics to Prometheus. You can use the rich set of metrics provided by NetScaler ADC to monitor NetScaler health as well as application health. For example, you can gather metrics on CPU and memory usage to know the NetScaler health. Similarly, you can use metrics like the number of HTTP requests received per second or the number of active clients to monitor application health.

Export of metrics from NetScaler to Prometheus

NetScaler supports the Prometheus pull mode as well as push mode. In the pull mode, you need to configure a time series profile that Prometheus queries at regular intervals and pulls the metrics data directly without an exporter resource in between. With pull mode, you can enable read-only access for a user without superuser privileges to export metrics to Prometheus. Using Grafana, you can visualize the NetScaler metrics exported to Prometheus for easier interpretation and understanding.

The following diagram shows a Prometheus and Grafana integration with NetScaler.

NetScaler integration with Prometheus and Grafana

Configure the export of metrics from NetScaler to Prometheus and visualization using Grafana

You must perform the followings steps to configure the export of metrics from NetScaler to Prometheus and visualize it using Grafana.

  1. Configure NetScaler with a time series analytics profile for exporting metrics to Prometheus.
  2. Install prometheus and configure it with the NetScaler specific parameters.
  3. Add Prometheus as a data source in Grafana.
  4. Create visualization in Grafana

Configure a time series analytics profile on NetScaler to support the Prometheus pull mode

Do the following steps for configuring pull mode using the NetScaler CLI:

  1. Create an analytics profile with type as time series. Specify the schema file with the required NetScaler metrics.

    add analytics profile <timeseries_profile_name> -type timeseries -schemaFile <name_of_schema_file>
-outputMode Prometheus -serveMode PULL -metrics ENABLED

    In this command:

    • timeseries_profile_name: Specify the time series profile name.

    • schemaFile: Specify the name of the schema file with NetScaler counters. By default, a schema file /var/metrics_conf/schema.json with a list of counters is configured. A reference schema file reference_schema.json with all the supported counters is also available under the path /var/metrics_conf/. This schema file can be used as a reference to build a custom list of counters. When you specify the schema file, the path of the schema file /var/metrics_conf/ is automatically added and you need to mention only the schema file name. For example, if you have created a schema file schema1.json with a custom list of counters at /var/metrics_conf/, you need to specify only the file name as schema1.json.

    • outputMode: Set the output mode as Prometheus.
    • serveMode: Specify Prometheus pull mode.
    • metrics: Enable collecting metrics from NetScaler.

Note:

You can configure an analytics profile with all the necessary parameters using the add command. If you need to make changes after creating the profile, you can use the set command to take the appropriate action such as disabling metrics and changing the server mode. You can configure read-only Prometheus access for a non-super user. For more information, see configure read-only Prometheus access for a non-superuser.

Install and configure Prometheus for metrics export from NetScaler

You can download Prometheus from repositories such as DockerHub or Quay or the official Prometheus repository.

To run Prometheus as a Docker container use the following command:

    docker run -dp 39090:9090 -v /tmp/prometheus.yml:/etc/prometheus/prometheus.yml --name native_prom prom/prometheus:latest  > **Note:** > > Here, `/tmp/prometheus.yml` is used as the path to the `prometheus.yml` file. Instead of that, you can specify the path on your virtual machine.

You must edit the prometheus.yml with the NetScaler parameters.

To export metrics from NetScaler, you must specify the following NetScaler specific parameters in the Prometheus YAML scrape configuration section. The scrape configuration section specifies a set of targets and configuration parameters describing how to scrape them.

  • metrics_path : Specify the HTTP resource path in NetScaler (/nitro/v1/config/systemfile) to fetch metrics.
  • username: Specify the NetScaler user name.
  • password: Specify the NetScaler password.
  • targets: Specify the IP address of the NetScaler from which you need to export metrics the metrics and the port you want to expose.
  • filename : Specify the name of the configured time series profile in place of timeseries_profile_name at the metrics_prom_<timeseries_profile_name>.log file.
  • filelocation: Specify the file location as /var/nslog.

Following is the scrap configuration section of the Prometheus YAML to add the NetScaler IP address as a target on Prometheus to export metrics. Here, HTTP is used as the scheme. You can use either HTTP or HTTPS.

scrape_configs:
  - job_name: 'vpx2_metrics_direct'
    metrics_path: /nitro/v1/config/systemfile
    params:
      args: ['filename:metrics_prom_ns_analytics_time_series_profile.log,filelocation:/var/nslog']
      format: ['prometheus']
    basic_auth:
     username: 'prom_user'
     password: 'user_password'
    scheme: http
    scrape_interval: 30s
    static_configs:
     - targets: ['10.102.34.231:80']
<!--NeedCopy-->

Add Prometheus as a data source in Grafana

If you need visualization of metrics using Grafana dashboards, you need to add Prometheus as a data source in Grafana. For more information, see add Prometheus as a data source in Grafana.

Create the visualization of metrics in Grafana

You can create a Grafana dashboard and select the key metrics and the appropriate visualization-type.

The following procedure shows adding a metric to the Grafana panel and creating a sample visualization dashboard.

  1. Specify the Panel Title.
  2. In the Query tab, for the query A, specify the required metric.
  3. In the Settings tab, select the Visualization type.

You can modify the data and its representation in Grafana. For more information, see the Grafana Documentation.

Following is a sample Grafana dashboard with a few NetScaler metrics:

Sample Grafana dash board

In this dashboard, you can see graphs for different NetScaler metrics such as:

  • vsvr_tot_Hits: Shows the number of requests received by the virtual server.
  • cc_cpu_use: Shows the CPU utilization percentage.
  • http_tot_Requests: Shows HTTP requests received.
  • serv_tot_serviced: Shows the request being serviced.
  • mem_cur_used_size: Shows the currently used memory of the NetScaler appliance.

Sample Prometheus graphs

Using the Prometheus expression browser, you can display the time series metrics collected by the Prometheus server. You can access the expression browser by pointing to prometheu-server-ip-address/graph in your browser. You can enter an expression and see the result either as a table or graph over time. Specify which exact metric that you want to display by typing in the metric name into the Expression field. You can specify multiple counters using different panels.

The following diagram shows Prometheus graphs for two NetScaler metrics cpu_use and http_tot_requests.

Sample Prometheus dash board

Sample Grafana dashboard

You can download the sample dashboards from NetScaler downloads page.

The following is a sample Grafana dashboard with the option to see the various metrics of overall infrastructure at one place like NetScaler health, virtual server status, application health (HTTP and TCP apps), and application security (SSL apps).

Sample Grafana dash board

You can expand the corresponding section in the dashboard to view the detailed visualization of each section such as HTTP Apps, SSL Apps, TCP Apps, virtual server stats (vStats), and NetScaler stats.

The following diagram shows a sample Grafana dashboard with NetScaler stats expanded:

Sample Grafana dash board

Additional information

Schema with the required NetScaler counters to export

Metrics collector exports counters present in the configured schema file. The /var/metrics_conf/schema.json file is the default schema file configured in the analytics profile.

The schema file is a list of entity types and associated counters. In the schema, all global or system level counters are grouped under the entity type netscaler. Some of the global counters are CPU usage (cpu_use), management CPU usage (mgmt_cpu_use), and total number of HTTP requests received (http_tot_Requests). The counters specific to service groups, lbvserver, csvserver, and so on are listed under the respective entity types.

Following is a sample of counters in the schema.json file for the authentication virtual server (vserver_authn) entity.

"vserver_authn":
  [
      {"name":"si_tot_Requests","rate":"True"},
      {"name":"si_tot_Responses","rate":"True"},
      {"name":"si_tot_RequestBytes","rate":"True"},
      {"name":"si_cur_state","rate":"False"},
      {"name":"si_tot_ResponseBytes","rate":"True"},
      {"name":"si_peer_port","rate":"True"},
      {"name":"vsvr_Protocol","rate":"False"}
  ]

The following table explains the counters mentioned in this sample:

Counter name Description
si_tot_Requests Total number of requests received on this service or virtual server.
si_tot_Responses Total number of responses received on this service or virtual server.
si_tot_RequestBytes Total number of request bytes received on this service or virtual server.
si_cur_state Current state of the virtual server.
si_tot_ResponseBytes Total number of response bytes received on this service or virtual server.
si_peer_port The port on which the service is running.
vsvr_Protocol Protocol associated with the virtual server.

The rate field can be set as True if the rate value a counter needs to be exported. For example, the rate of si_tot_Requests is exported if the rate is set to True for si_tot_Requests.

Following is a sample of counters from the netscaler entity.

  "netscaler":
    [   
        {"name":"cpu_use","rate":"False"},
        {"name":"mgmt_cpu_use","rate":"False"},
        {"name":"tcp_tot_rxpkts","rate":"True"},
        {"name":"tcp_tot_rxbytes","rate":"True"},
        {"name":"tcp_tot_txpkts","rate":"True"},
        {"name":"tcp_tot_txbytes","rate":"True"},
        {"name":"tcp_cur_ClientConnEst","rate":"False"},
        {"name":"tcp_cur_ServerConnEst","rate":"False"},
        {"name":"tcp_cur_ClientConn","rate":"False"},
        {"name":"tcp_cur_ClientConnClosing","rate":"False"},
        {"name":"tcp_tot_ClientOpen","rate":"True"},
        {"name":"tcp_cur_ServerConn","rate":"False"},
        {"name":"tcp_cur_ServerConnClosing","rate":"False"},
        {"name":"http_tot_Requests","rate":"True"},
        {"name":"http_tot_Responses","rate":"True"},
        {"name":"http_tot_Gets","rate":"True"},
        {"name":"http_tot_Posts","rate":"True"},
        {"name":"http_tot_Others","rate":"True"},
    ]

The following table explains the counters mentioned in this sample:

Counter name Description
cpu_use Tracks the CPU utilization percentage (CPU utilization percentage * 10).
tcp_tot_rxpkts TCP packets received.
tcp_tot_rxbytes Bytes of TCP data received.
tcp_tot_txpkts TCP packets transmitted.
tcp_tot_txbytes Bytes of TCP data transmitted.
tcp_cur_ClientConnEst Current client connections in the Established state, which indicates that data transfer can occur between the NetScaler appliance and the client.
tcp_cur_ServerConnEst Current server connections in the Established state, which indicates that data transfer can occur between the NetScaler appliance and the server.
tcp_cur_ClientConn Client connections, including connections in the Opening, Established, and Closing state. Server connections, including connections in the Opening, Established, and Closing state.
tcp_cur_ClientConnClosing Client connections in the Closing state, which indicates that the connection termination process has initiated but is not complete.
tcp_cur_ServerConn Server connections, including connections in the Opening, Established, and Closing state.
tcp_cur_ServerConnClosing Server connections in the Closing state, which indicates that the connection termination process has initiated but is not complete.
http_tot_Requests This counter tracks HTTP requests received using the GET method.
http_tot_Responses This counter tracks HTTP requests received using the POST method.
http_tot_Gets This counter tracks HTTP requests received using the GET method.
http_tot_Posts This counter tracks HTTP requests received.
http_tot_Others This counter tracks HTTP requests received using methods other than GET and POST.

Following is a sample of counters from the vserver_ssl entity.

"vserver_ssl":
[
    {"name":"ssl_ctx_tot_session_hits","rate":"True"},
    {"name":"ssl_ctx_tot_session_new","rate":"True"},
    {"name":"ssl_ctx_tot_enc_bytes","rate":"True"},
    {"name":"ssl_ctx_tot_dec_bytes","rate":"True"},
]

The following table explains the SSL counters mentioned in this sample:

Counter name Description
ssl_ctx_tot_session_hits This counter tracks the number of session hits.
ssl_ctx_tot_session_new This counter tracks the number of new sessions created.
ssl_ctx_tot_enc_bytes This counter tracks the number of encrypted bytes per SSL virtual server.
ssl_ctx_tot_dec_bytes This counter tracks the number of bytes decrypted per SSL virtual server.

Configure read-only Prometheus access for a non-super user

Perform the following steps to configure read-only Prometheus access for a non-super user.

  1. Add a new user to the NetScaler appliance.

     add system user <ns_user_name> <ns_user's_password> -externalAuth enabled -promptString user-%u-at-%T logging enaBLED

    Example :

     add system user nspaul nspaul -externalAuth enabled -promptString user-%u-at-%T logging enaBLED

  2. Create a command policy for a read only user. This command policy allows read-only access from any file under the /var/nslog/ directory.

    add system cmdPolicy read-only-prometheus ALLOW \"\(^man.\*)|\(^show\s+\(?!system)\(?!configstatus)\(?!ns ns\.conf)\(?!ns savedconfig)\(?!ns runningConfig)\(?!gslb runningConfig)\(?!audit messages)\(?!techsupport).\*)|\(^stat.\*)|\(show system file .\* -filelocation \"/var/nslog\")\"

  3. If metrics are only written to a certain file, you may even limit user access such that they can only get that specific file.

    add system cmdPolicy read-only-prometheus ALLOW \"\(^man.\*)|\(^show\s+\(!system)\(!configstatus)\(!ns ns\.conf)\(!ns savedconfig)
\(!ns runningConfig)\(!gslb   runningConfig)\(!audit messages)\(!techsupport).\*)|\(^stat.\*)
|\(show system file metrics\_prom\_<name\_of\_timeseries\_profile>.log -filelocation \"/var/nslog\")\"

    Note:

    In the show system file command, specify the name of the time series profile you have configured in place of name_of_ timeseries_profile.

  4. Bind a user with the command policy.

    bind system user <userName> \(\(<policyName> <priority>) | -partitionName <string>)

    For example :

    bind system user user1 read-only-prometheus 0

To unbind and remove a user from the command policy use the following commands:

  1. Unbind a configured user from the system command policy.

    unbind system user <userName> \(<policyName> | -partitionName <string>

    For example:

    unbind system user user1 read-only-prometheus

  2. Remove the command Policy from NetScaler.

    rm system cmdPolicy read-only-prometheus

Subscription of counters for multiple time series profiles

Now, NetScaler supports creating multiple time series profiles and specifies different set of counters for each profile. Also, you can export only the counters based on your requirements.

You must create multiple schema.json files containing the necessary counters with unique names and the .json extension to configure multiple time series profiles. A reference schema file reference_schema.json is available under the path /var/metrics_conf/. You can modify the reference schema as per your requirements and use accordingly.

The configuration of the two new time series profiles is as follows:

add analytics profile ns_analytics_timeseries_profile_1   -type timeseries -schemaFile schema1.json

set analytics profile  ns_analytics_timeseries_profile_1 -outputMode prometheus -serveMode PULL -metrics ENABLED

add analytics profile ns_analytics_timeseries_profile_2   -type timeseries -schemaFile schema2.json

set analytics profile  ns_analytics_timeseries_profile_2 -outputMode prometheus -serveMode PULL -metrics ENABLED

In this example, schema1.json and schema2.json have different sets of counters.

Prometheus configuration

Configuration of a sample prometheus.yml file is as follows:

scrape_configs:
  - job_name: 'vpx2_metrics_direct'
    metrics_path: /nitro/v1/config/systemfile
    params:
      args: ['filename:metrics_prom_ns_analytics_time_series_profile.log,filelocation:/var/nslog']
      format: ['prometheus']
    basic_auth:
     username: 'prom_user'
     password: 'user_password'
    scheme: https
    scrape_interval: 30s
    static_configs:
     - targets: ['<ADC1-ip>:<port>', '<ADC2-ip>:<port>']
<!--NeedCopy-->
Monitoring NetScaler, applications, and application security using Prometheus
August 11, 2023
Contributed by: 
C
August 11, 2023
Contributed by: 
C

In this article

Site feedback | Your privacy choices footer linkYour Privacy Choices | Privacy and legal terms | Cookie preferences | Consent settings | docs.cloud.com
© 1999- Cloud Software Group, Inc. All rights reserved.