Product Documentation
ADC
14.1 - Current Release 13.1 13.0 12.1
View PDF

Bot Detection

January 4, 2024
Contributed by: 
B S

The NetScaler bot management system uses various techniques to detect the incoming bot traffic. The techniques are used as detection rules to detect the bot type. The techniques are as follows:

Note:

Bot management supports a maximum of 32 configuration entities for block list, allow list, and rate limiting techniques.

Bot Allowed list - A customized list of IP addresses (IPv4 and IPv6), subnets (IPv4 and IPv6), and policy expressions that can be bypassed as an allowed list.

Bot Blocked list - A customized list of IP addresses (IPv4 and IPv6), subnets (IPv4 and IPv6), and policy expressions that have to be blocked from accessing your web applications.

IP reputation - This rule detects if the incoming bot traffic is from a malicious IP address.

Device fingerprint - This rule detects if the incoming bot traffic has the device fingerprint ID in the incoming request header and browser attributes of an incoming client bot traffic.

Limitation:

  1. Java Script must be enabled in the client browser.
  2. Does not work for XML responses.

Bot log expression - The detection technique enables you to capture additional information as log messages. The data can be the name of the user who requested the URL, the source IP address, and the source port from which the user sent the request or data generated from an expression.

Rate limit - This rule rate limits multiple requests coming from the same client.

Bot trap - Detects and blocks automated bots by advertising a trap URL in the client response. The URL appears invisible and not accessible if the client is a human user. The detection technique is effective in blocking attacks from automated bots.

TPS - Detects incoming traffic as bots if the maximum number of requests and percentage increase in requests exceeds the configured time interval.

CAPTCHA - This rule uses a CAPTCHA for mitigating bot attacks. A CAPTCHA is a challenge-response validation to determine if the incoming traffic is from a human user or an automated bot. The validation helps block automated bots that cause security violations to web applications. You can configure CAPTCHA as a bot action in IP reputation and device fingerprint detection techniques.

Now, let us see how you can configure each technique to detect and manage your bot traffic.

How to upgrade your appliance to NetScaler CLI-based bot management configuration

If you are upgrading your appliance from an older version (NetScaler release 13.0 build 58.32 or earlier), you must first manually convert the existing bot management configuration to the NetScaler CLI based bot management configuration only once. Complete the following steps to manually convert your bot management configuration.

  1. After upgrading to the latest version connect to the upgrade tool “upgrade_bot_config.py” by using the following command

    At the command prompt, type:

    shell "/var/python/bin/python /netscaler/upgrade_bot_config.py > /var/bot_upgrade_commands.txt"

  2. Run the configuration using the following command.

    At the command prompt, type:

    batch -f /var/bot_upgrade_commands.txt

  3. Save the upgraded configuration.

save ns config

Configure NetScaler CLI-based bot management

The bot management configuration enables you to bind one or more bot detection techniques to a specific bot profile. You must complete the following steps to configure NetScaler-based bot management:

  1. Enable bot management
  2. Import bot signature
  3. Add bot profile
  4. Bind bot profile
  5. Add bot policy
  6. Bind bot policy
  7. Configure bot settings

Note:

If you are upgrading your appliance from an older version, you must first manually convert the existing bot management configuration. For more information, see How to upgrade to NetScaler CLI-based bot management configuration section.

Enable bot management

Before you can begin, ensure that the Bot Management feature is enabled on the appliance. If you have a new NetScaler or VPX, you must enable the feature before you configure it. If you are upgrading a NetScaler appliance from an earlier version to the current version, you must need to enable the feature before you configure it. At the command prompt, type:

enable ns feature Bot

Import bot signature

You can import the default signature bot file and bind it to the bot profile. At the command prompt, type:

import bot signature [<src>] <name> [-comment <string>] [-overwrite]

Where:

src - Local path name, or URL (protocol, host, path, and file name). Maximum Length: 2047. > Note: > > The import fails if the object to be imported is on an HTTPS server that requires client certificate authentication for access.

name - Name of the bot signature file object. This is a mandatory argument. Maximum Length: 31.

comment - Description about the signature file object. Maximum Length: 255.

overwrite - Action that overwrites the existing file. > Note: > > Use the overwrite option to update the content in the signature file. Alternately, use the update bot signature <name> command to update the signature file on the NetScaler appliance.

Example

import bot signature http://www.example.com/signature.json signaturefile -comment commentsforbot –overwrite

Note:

You can use the overwrite option to update the content in the signature file. Also, you can use the update bot signature <name> command to update the signature file in the NetScaler appliance.

Add bot profile

A bot profile is a collection of profile settings to configure bot management on the appliance. You can configure the settings to perform bot detection.

At the command prompt, type:

add bot profile <name> [-signature <string>] [-errorURL <string>] [-trapURL <string>] [-whiteList ( ON | OFF )] [-blackList ( ON | OFF )] [-rateLimit ( ON | OFF )] [-deviceFingerprint ( ON | OFF )] [-deviceFingerprintAction ( none | log | drop | redirect | reset | mitigation )] [-ipReputation ( ON | OFF )] [-trap ( ON | OFF )]

Example:

add bot profile profile1 -signature signature -errorURL http://www.example.com/error.html -trapURL /trap.html -whitelist ON -blacklist ON -ratelimit ON -deviceFingerprint ON -deviceFingerprintAction drop -ipReputation ON -trap ON

Bind bot profile

After you create a bot profile, you must bind the bot detection mechanism to the profile.

At the command prompt, type:

bind bot profile <name> | (-ipReputation [-category <ipReputationCategory>] [-enabled ( ON | OFF )] [-action ( none | log | drop | redirect | reset | mitigation )] [-logMessage <string>]

Example:

The following example is for binding the IP reputation detection technique to a specific bot profile.

bind bot profile profile5 -ipReputation -category BOTNET -enabled ON -action drop -logMessage message

Add bot policy

You must add the bot policy for evaluating bot traffic.

At the command prompt, type:

add bot policy <name> -rule <expression> -profileName <string> [-undefAction <string>] [-comment <string>] [-logAction <string>]

Where,

Name- Name for the bot policy. Must begin with a letter, number, or the underscore character (_), and must contain only letters, numbers, and the hyphen (-), period (.) pound (#), space ( ), at (@), equals (=), colon (:), and underscore characters. Can be changed after the bot policy is added.

Rule- An expression that the policy uses to determine whether to apply the bot profile on the specified request. This is a mandatory argument. Maximum Length: 1499

profileName- Name of the bot profile to apply if the request matches this bot policy. This is a mandatory argument. Maximum Length: 127

undefAction- Action to perform if the result of policy evaluation is undefined (UNDEF). An UNDEF event indicates an internal error condition. Maximum Length: 127

Comment- Description about this bot policy. Maximum Length: 255

logAction - Name of the log action to use for requests that match this policy. Maximum Length: 127

Example:

add bot policy pol1 –rule "HTTP.REQ.HEADER(\"header\").CONTAINS(\"custom\")" - profileName profile1 -undefAction drop –comment commentforbotpolicy –logAction log1

Bind bot policy global

At the command prompt, type:

bind bot global -policyName <string> -priority <positive_integer> [-gotoPriorityExpression <expression>][-type ( REQ_OVERRIDE | REQ_DEFAULT )] [-invoke (-labelType ( vserver | policylabel ) -labelName <string>) ]

Example:

bind bot global –policyName pol1 –priority 100 –gotoPriorityExpression NEXT -type REQ_OVERRIDE

Bind bot policy to a virtual server

At the command prompt, type:

bind lb vserver <name>@ ((<serviceName>@ [-weight <positive_integer>] ) | <serviceGroupName>@ | (-policyName <string>@ [-priority <positive_integer>] [-gotoPriorityExpression <expression>]

Example:

bind lb vserver lb-server1 –policyName pol1 –priority 100 –gotoPriorityExpression NEXT -type REQ_OVERRIDE

Configure bot settings

You can customize the default settings if necessary. At the command prompt, type:

set bot settings [-defaultProfile <string>] [-javaScriptName <string>] [-sessionTimeout <positive_integer>] [-sessionCookieName <string>] [-dfpRequestLimit <positive_integer>] [-signatureAutoUpdate ( ON | OFF )] [-signatureUrl <URL>] [-proxyServer <ip_addr|ipv6_addr|*>] [-proxyPort <port|*>]
<!--NeedCopy-->

Where,

defaultProfile - Profile to use when a connection does not match any policy. Default setting is “ “, which sends unmatched connections back to the NetScaler without attempting to filter them further. Maximum Length: 31

javaScriptName - Name of the JavaScript that the BotNet feature uses in response. Must begin with a letter or number, and can consist of from 1 to 31 letters, numbers, and the hyphen (-) and underscore (_) symbols. The following requirement applies only to the NetScaler CLI: If the name includes one or more spaces, enclose the name in double or single quotation marks (for example, “my cookie name” or ‘my cookie name’). Maximum Length: 31

sessionTimeout - Session times out, in seconds, after which a user session is terminated.

Minimum value - 1, Maximum value: 65535

sessionCookieName - Name of the SessionCookie that the BotNet feature uses it for tracking. Must begin with a letter or number, and can consist of from 1 to 31 letters, numbers, and the hyphen (-) and underscore (_) symbols. The following requirement applies only to the NetScaler CLI: If the name includes one or more spaces, enclose the name in double or single quotation marks (for example, “my cookie name” or ‘my cookie name’). Maximum Length: 31

dfpRequestLimit - Number of requests to allow without bot session cookie if device fingerprint is enabled. Minimum value: 1, Maximum Value: 4294967295

signatureAutoUpdate - Flag used to enable/disable bot auto update signatures. Possible values: ON, OFF. Default value: OFF

signatureUrl - URL to download the bot signature mapping file from the server. Default value: https://nsbotsignatures.s3.amazonaws.com/BotSignatureMapping.json. Maximum Length: 2047

proxyServer - Proxy Server IP to get updated signatures from AWS.

proxyPort - Proxy Server Port to get updated signatures from AWS. Default value: 8080

proxyUsername - User name to authenticate to the proxy server for downloading signature updates.

proxyPassword – Password to authenticate to the proxy server for downloading signature updates.

Example:

set bot settings –defaultProfile profile1 –javaScriptName json.js –sessionTimeout 1000 –sessionCookieName session -proxyServer 10.102.30.112 -proxyPort 3128 -proxyUsername defaultuser -proxyPassword defaultPassword

Configuring bot management by using NetScaler GUI

You can configure NetScaler bot management by first enabling the feature on the appliance. Once you enable, you can create a bot policy to evaluate the incoming traffic as bot and send the traffic to the bot profile. Then, you create a bot profile and then bind the profile to a bot signature. As an alternative, you can also clone the default bot signature file and use the signature file to configure the detection techniques. After creating the signature file, you can import it into the bot profile.

Bot Management Page

  1. Enable bot management feature
  2. Configure bot management settings
  3. Clone NetScaler bot default signature
  4. Import NetScaler bot signature
  5. Configure bot signature settings
  6. Create bot profile
  7. Create bot policy

Enable bot management feature

Complete the following steps to enable bot management:

  1. On the navigation pane, expand System and then click Settings > Configure Advanced Features.
  2. On the Configure Advanced Features page, select the NetScaler Bot Management checkbox.
  3. Click OK.

Configure bot management settings for device fingerprint technique

Complete the following step to configure the device fingerprint technique:

  1. Navigate to Security > NetScaler bot Management.
  2. In the details pane, under Settings click Change NetScaler bot Management Settings.

  3. In the Configure NetScaler bot Management Settings, set the following parameters.

    1. Default Profile. Select a bot profile.
    2. JavaScript Name. Name of the JavaScript file that bot management uses in its response to the client.
    3. Session Timeout. Timeout in seconds after which the user session is terminated.
    4. Session Cookie. Name of the session cookie that the bot management system uses for tracking.
    5. Device Fingerprint Request Limit. Number of requests to allow without a bot session cookie, if device fingerprint is enabled.
    6. Proxy Server - Proxy server IP address from where the latest signatures are uploaded.
    7. Proxy Port – Port number of machine from where the latest signatures are uploaded.
    8. Proxy Username – User name for the authentication of the proxy server
    9. Proxy Password - Password for the authentication of the proxy server.

      Note:

      Proxy User name and the Proxy Password fields are enabled if the Proxy Server and the Proxy Port fields are configured.

  4. Click OK.

Clone bot signature file

Complete the following step to clone the bot signature file:

  1. Navigate to Security > NetScaler Bot Management and Signatures.
  2. In the NetScaler Bot Management Signatures page, select the default bot signatures record and click Clone.
  3. In the Clone Bot Signature page, enter a name and edit the signature data.
  4. Click Create.

Import bot signature file

If you have your own signature file, then you can import it as a file, text, or URL. Perform the following steps to import the bot signature file:

  1. Navigate to Security > NetScaler Bot Management and Signatures.
  2. On the NetScaler Bot Management Signatures page, import the file as URL, File, or text.
  3. Click Continue.
  4. On the Import NetScaler Bot Management Signature page, set the following parameters.
    1. Name - Name of the bot signature file.
    2. Comment - Brief description about the imported file.
    3. Overwrite - Select the checkbox to allow overwriting of data during file update.
    4. Signature Data - Modify signature parameters
  5. Click Done.

Configure bot allow list by using NetScaler GUI

This detection technique enables you to bypass URLs that you configure an allowed listed one. Complete the following step to configure an allow list URL:

  1. Navigate to Security > NetScaler Bot Management and Profiles.
  2. On the NetScaler Bot Management Profiles page, select a file and click Edit.
  3. On the NetScaler Bot Management Profile page, go to the Signature Settings section and click White List.
  4. In the White List section, set the following parameters:
    1. Enabled. Select the checkbox to validate the allow list URLs as part of the detection process.
    2. Configure Types. Configure an allow list URL. The URL is bypassed during bot detection. Click Add to add a URL to the bot allow list.
    3. In the Configure NetScaler Bot Management Profile Whitelist Binding page, set the following parameters:
      1. Type. URL type can be an IPv4 address, subnet IP address, or an IP address matching a policy expression.
      2. Enabled. Select the checkbox to validate the URL.
      3. Value. URL address.
      4. Log. Select the checkbox to store log entries.
      5. Log Message. Brief description of the log.
      6. Comments. Brief description about the allow list URL.
      7. Click OK.

    Configure bot allow list

  5. Click Update.
  6. Click Done.

Configure bot block list by using NetScaler GUI

This detection technique enables you to drop the URLs that you configure as block listed one. Complete the following step to configure a block list URL.

  1. Navigate to Security > NetScaler Bot Management and Profiles.
  2. On the NetScaler Bot Management Profiles page, select a signature file and click Edit.
  3. On the NetScaler Bot Management Profile page, go to the Signature Settings section and click Black List.

  4. In the Black List section, set the following parameters:

    1. Enabled. Select the checkbox to validate block list URLs as part of the detection process.
    2. Configure Types. Configure a URL to be part of the bot block list detection process. These URLs are dropped during bot detection. Click Add to add a URL to the bot block list

    3. In the Configure NetScaler Bot Management Profile Blacklist Binding page, set the following parameters.

      1. Type. URL type can be an IPv4 address, subnet IP address, or IP address.
      2. Enabled. Select the checkbox to validate the URL.
      3. Value. URL address.
      4. Log. Select the checkbox to store log entries.
      5. Log Message. Brief description of the login.
      6. Comments. Brief description about the block list URL.
      7. Click OK.

    Configuring bot block list

  5. Click Update.
  6. Click Done.

Configure IP reputation by using NetScaler GUI

The IP reputation bot technique uses Webroot’s IP reputation database and cloud service provider database to verify if a client request is a malicious IP address or a public cloud IP address. As part of the bot categories is configured and then a bot action is associated to it. Complete the following steps to configure Webroot IP reputation and cloud service provider database categories.

  1. Navigate to Security > NetScaler bot Management and Profiles.
  2. On the NetScaler bot Management Profiles page, select a profile and click Edit.
  3. On the NetScaler bot Management Profile page, go to the Profile Settings section and click IP Reputation.
  4. On the IP Reputation section, set the following parameters:
    1. Enabled. Select the checkbox to validate incoming bot traffic as part of the detection process.
    2. Configure Categories. You can use the IP reputation technique for incoming bot traffic under different categories. Based on the configured category, you can drop or redirect the bot traffic. Click Add to configure a malicious bot category.

    3. In the Configure NetScaler bot Management Profile IP Reputation Binding page, set the following parameters:

      1. Category. Select a Webroot IP reputation bot category to validate a client request as a malicious IP address.

        1. IP_BASED - This category checks whether the client IP address (IPv4 and IPv6) is malicious or not.
        2. BOTNET - This category includes Botnet C&C channels, and infected zombie machines controlled by Bot master.
        3. SPAM_SOURCES - This category includes tunneling spam messages through a proxy, anomalous SMTP activities, and forum spam activities.
        4. SCANNERS - This category includes all reconnaissance such as probes, host scan, domain scan, and password brute force attack.
        5. DOS - This category includes DOS, DDOS, anomalous sync flood, and anomalous traffic detection.
        6. REPUTATION - This category denies access from IP addresses (IPv4 and IPv6) currently known to be infected with malware. This category also includes IP addresses with average low Webroot Reputation Index score. Enabling this category prevents access from sources identified to contact malware distribution points.
        7. PHISHING - This category includes IP addresses (IPv4 and IPv6) hosting phishing sites and other kinds of fraud activities such as ad click fraud or gaming fraud.
        8. PROXY - This category includes IP addresses (IPv4 and IPv6) providing proxy services.
        9. NETWORK - IPs providing proxy and anonymization services including The Onion Router aka TOR or dark net.
        10. MOBILE_THREATS - This category checks the client IP address (IPv4 and IPv6) with the list of addresses harmful for mobile devices.

      2. Category. Select a Webroot public cloud service provider category to validate a client request is a public cloud IP address.

        1. AWS - This category checks client IP address with list of public cloud addresses from AWS.
        2. GCP - This category checks client IP address with list of public cloud addresses from the Google Cloud Platform.
        3. AZURE - This category checks client address with list of public cloud addresses from Azure.
        4. ORACLE - This category checks client IP address with list of public cloud addresses from Oracle
        5. IBM - This category checks client IP address with list of public cloud addresses from IBM.
        6. SALESFORCE - This category checks client IP address with list of public cloud addresses from Salesforce.

        Possible values for Webroot IP reputation bot category: IP, BOTNETS, SPAM_SOURCES, SCANNERS, DOS, REPUTATION, PHISHING, PROXY, NETWORK, MOBILE_THREATS.

        Possible values for Webroot public cloud service provider category: AWS, GCP, AZURE, ORACLE, IBM, SALESFORCE.

      3. Enabled. Select the checkbox to validate the IP reputation signature detection.
      4. Bot action. Based on the configured category, you can assign no action, drop, redirect, or mitigation action.
      5. Log. Select the checkbox to store log entries.
      6. Log Message. Brief description of the log.
      7. Comments. Brief description about the bot category.
  5. Click OK.
  6. Click Update.

  7. Click Done.

    Configure IP Reputation

Note

If you disable IP reputation, ensure to stop its downloads. Complete the following steps to stop the IP reputation downloads:

  1. Navigate to Security > NetScaler bot Management > Change NetScaler bot Management Settings
  2. Change the Default Nonintrusive Profile to BOT_BYPASS.

Configure bot rate limit technique

The bot rate limit technique enables you to limit bot traffic within certain timeframe based on the user’s geolocation, client IP address, session, cookie, or configured resource (URL). By configuring the bot rate limit technique, you can ensure the following:

  • Block malicious bot activity.
  • Reduce traffic strain to web servers.

Configure bot rate limit by using NetScaler CLI

At the command prompt, type:

bind bot profile <name>... -ratelimit  -type <type> Geolocation -countryCode <countryName> -rate  <positive_integer> -timeSlice <positive_integer> [-action <action> ...] [-limitType ( BURSTY | SMOOTH )] [-condition <expression>] [-enabled ( ON | OFF )]
<!--NeedCopy-->

Where,

*SOURCE_IP - Rate-limiting based on the client IP address.

*SESSION - Rate-limiting based on the configured cookie name.

*URL - Rate-limiting based on the configured URL.

*GEOLOCATION - Rate-limiting based on the configured country name.

Possible values - SESSION, SOURCE_IP, URL, GEOLOCATION

Example:

 bind bot profile geo_prof -ratelimit -type Geolocation -countryCode IN -rate 100 -timeSlice 1000 -limitType SMOOTH -condition HTTP.REQ.HEADER("User-Agent").contains("anroid") -action log,drop -enabled on
<!--NeedCopy-->

Configure bot rate limit by using NetScaler GUI

Complete the following steps to configure the bot rate limit detection technique:

  1. Navigate to Security > NetScaler Bot Management and Profiles.
  2. In the NetScaler Bot Management Profiles page, select a profile and click Edit.
  3. In the NetScaler Bot Management Profile page, go to the Profile Settings section and click Rate Limit.

  4. In the Rate Limit section, set the following parameters:

    1. Enabled. Select the checkbox to validate the incoming bot traffic as part of the detection process.
    2. Click Add to configure rate limit bindings.

  5. In the Configure NetScaler Bot Management Rate Limit page, set the following parameters.

    1. Type - Rate limit bot traffic based on the following parameters:

      1. Geolocation - Rate limit based on user’s geographic location.
      2. Source_IP - Rate limit traffic based on client IP address.
      3. Session - Rate limit bot traffic based on session or cookie name.
      4. URL - Rate limit bot traffic based on the configured URL.
    2. Country - Select a geolocation as country or region.
    3. Rate Limit Type – Limits the type of traffic based on the following types.
      • Bursty – Forward all requests that are within the set threshold and specified time period.
      • Smooth – Forward the requests evenly across the specified time period.
    4. Rate Limit Connection - Allows you to create multiple rules for a condition.
    5. Enabled - Select the checkbox to validate the incoming bot traffic.
    6. Request Threshold - Maximum number of requests allowed within certain timeframe.
    7. Period - Timeframe in milliseconds.
    8. Action - Choose a bot action for the selected category.
    9. Log - Select the checkbox to store log entries.
    10. Log Message - Brief description of the log.
    11. Comments - Brief description about the bot category.
  6. Click OK.
  7. Click Update.
  8. Click Done.

Geolocation based bot rate limit technique

Configure device fingerprint technique by using NetScaler GUI

This detection technique sends a java script challenge to the client and extracts the device information. Based on device information, the technique drops or bypasses the bot traffic. Follow the steps to configure the detection technique.

  1. Navigate to Security > NetScaler Bot Management and Profiles.
  2. On the NetScaler Bot Management Profiles page, select a signature file and click Edit.

  3. On the NetScaler Bot Management Profile page, go to the Signature Settings section and click Device Fingerprint. In the Device Fingerprint section, set the following parameters:

    1. Enabled - Select to enable the rule.
    2. Configuration - Select one of the following options:
      1. None - Allows the traffic.
      2. Drop - Drops the traffic.
      3. Redirect - Redirects the traffic to error URL.

      4. Mitigation, or CAPTCHA - Validates and allows the traffic.

        Note:

        During session replay attacks using the device fingerprint cookies, requests are dropped even if the device fingerprint configuration is set to Mitigation.

  4. Click Update.
  5. Click Done.

Configure device fingerprint

Configure device fingerprint technique for mobile (Android) applications

Device fingerprint technique detects an incoming traffic as a bot by inserting a JavaScript script in the HTML response to the client. The JavaScript script when invoked by the browser, it collects browser and client attributes and sends a request to the appliance. The attributes are examined to determine whether the traffic is a Bot or a human.

The detection technique is further extended to detect bots on a mobile (Android) platform. Unlike web applications, in mobile (Android) traffic, bot detection based on JavaScript script do not apply. To detect bots in a mobile network, the technique uses a bot mobile SDK which is integrated with mobile applications on the client-side. The SDK intercepts the mobile traffic, collects device details, and sends the data to the appliance. On the appliance side, the detection technique examines the data and determines whether the connection is from a Bot or a human.

How the device fingerprint technique for the mobile application works

The following steps explain the bot detection workflow to detect if a request from a mobile device is from a human or a bot.

  1. When a user interacts with a mobile application, the bot mobile SDK records the device’s behavior.
  2. Client sends a request to NetScaler appliance.
  3. When sending the response, the appliance inserts a bot session cookie with session details, and parameters to collect client parameters.
  4. When the mobile application receives the response, the NetScaler bot SDK which is integrated with the mobile application validates the response, retrieves the recorded device fingerprint parameters, and sends it to the appliance.
  5. The device fingerprint detection technique on the appliance side validates the device details and updates the bot session cookie if it is a suspected bot or not.
  6. When the cookie is expired or device fingerprint protection prefers to validate and collect device parameters periodically, the whole procedure or challenge is repeated.

Pre-requisite

To get started with the NetScaler device fingerprint detection technique for mobile applications, you must download and install the bot mobile SDK in your mobile application.

Configure fingerprint detection technique for mobile (Android) applications by using the CLI

At the command prompt, type:

set bot profile <profile name> -deviceFingerprintMobile ( NONE | Android )

Example:

set bot profile profile 1 –deviceFingerprintMobile Android

Configure device fingerprint detection technique for mobile (Android) applications by using the GUI

  1. Navigate to Security > NetScaler Bot Management and Profiles.
  2. On the NetScaler Bot Management Profiles page, select a file and click Edit.
  3. On the NetScaler Bot Management Profile page, click Device Fingerprint under Profile Settings.
  4. In the Configure Bot Mobile SDK section, select the mobile client type.
  5. Click Update and Done.

Bot management device fingerprint detection technique for mobile applications

Configure bot log expression

If the client is identified as a bot, the NetScaler bot management enables you to capture additional information as log messages. The data can be the name of the user who requested the URL, the source IP address, and the source port from which the user sent the request or data generated from an expression. To perform custom logging, you must configure a log expression in the bot management profile.

Bind the log expression in the bot profile by using the CLI

At the command prompt, type:

bind bot profile <name> (-logExpression -name <string> -expression <expression> [-enabled ( ON | OFF )]) -comment <string>
<!--NeedCopy-->

Example:

bind bot profile profile1 –logExpression exp1 –expression HTTP.REQ.URL –enabled ON -comment "testing log expression"

Bind log expression to bot profile by using the GUI

  1. Navigate to Security > NetScaler Bot Management > Profiles.
  2. On the NetScaler Bot Management Profiles page, select Bot Log Expressions from the Profile Settings section.
  3. In the Bot Log Expression Settings* section, click **Add.
  4. In the Configure NetScaler Bot Management Profile Bot Log Expression Binding page, set the following parameters.
    1. Log Expression Name. Name of the log expression.
    2. Expression. Enter the log expression.
    3. Enabled. Enable or disable the log expression binding.
    4. Comments. A brief description about the bot log expression binding.
  5. ClickOK and Done.

Configure bot trap technique

The NetScaler bot trap technique randomly or periodically inserts a trap URL in the server response. You can also create a trap URL list and add URLs for that The URL appears invisible and not accessible if the client is a human user. However, if the client is an automated bot, the URL is accessible and when accessed, the attacker is categorized as bot and any subsequent request from the bot is blocked. The trap technique is effective in blocking attacks from bots.

The trap URL is an alpha-numeric URL of configurable length and it is auto-generated at configurable interval. Also the technique allows you to configure a trap insertion URL for top visited websites or frequently visited websites. By doing this, you can mandate the purpose of inserting the bot trap URL for requests matching the trap insertion URL.

Note:

Although the bot trap URL is auto-generated, the NetScaler bot management still allows you to configure a customized trap URL in the bot profile. This is done to strengthen the bot detection technique and make it harder for attackers to access the trap URL.

To complete the bot trap configuration, you must complete the following steps.

  1. Enable bot trap URL
  2. Configure bot trap URL in bot profile
  3. Bind bot trap insertion URL to bot profile
  4. Configure bot trap URL length and interval in bot settings

Enable bot trap URL protection

Before you can begin, you must ensure that the Bot trap URL protection is enabled on the appliance. At the command prompt, type:

enable ns feature Bot

Configure bot trap URL in bot profile

You can configure the bot trap URL and specify a trap action in the bot profile.
At the command prompt, type:

add bot profile <name> -trapURL <string> -trap ( ON | OFF ) -trapAction <trapAction>

Where,

  • trapURL is the URL that Bot protection uses as the Trap URL. Maximum Length: 127

  • trap is to enable bot trap detection. Possible values: ON, OFF. Default value: OFF

  • trapAction is an action to be taken based on bot detection. Possible values: NONE, LOG, DROP, REDIRECT, RESET, MITIGATION. Default value: NONE

Example:

add bot profile profile1 -trapURL www.bottrap1.com trap ON -trapAction RESET

Bind bot trap insertion URL to bot profile

You can configure the bot trap insertion URL and bind it to the bot profile. At the command prompt, type:

bind bot profile <profile_name> trapInsertionURL –url <url> -enabled ON|OFF -comment <comment>

Where,

URL - The request URL regex pattern for which the bot trap URL is inserted. Maximum Length: 127

Example:

bind bot profile profile1 trapInsertionURL –url www.example.com –enabled ON –comment insert a trap URL randomly

Configure bot trap URL length and interval in bot settings

You can configure the bot trap URL length and also set the interval to auto generate the bot trap URL. At the command prompt, type:

set bot settings -trapURLAutoGenerate ( ON | OFF ) –trapURLInterval <positive_integer> -trapURLLength <positive_integer>

Where,

  • trapURLInterval is the time in seconds after which the bot trap URL is updated. Default value: 3600, Minimum value: 300, Maximum value: 86400

  • trapURLLength. Length of the auto-generated bot trap URL. Default value: 32, Minimum value: 10, Maximum value: 255

Example:

set bot settings -trapURLAutoGenerate ON –trapURLInterval 300 -trapURLLength 60

Configure bot trap URL by using the GUI

  1. Navigate to Security > NetScaler Bot Management > Profiles.
  2. In the NetScaler Bot Management Profiles page, click Edit to configure the bot trap URL technique.
  3. In the Create NetScaler Bot Management Profile page, enter the bot trap URL in the general section.
  4. In the NetScaler Bot Management Profile page, click Bot Trap from Profile Settings.

  5. In the Bot Trap section, set the following parameters.

    1. Enabled. Select the checkbox to enable bot trap detection
    2. Description. Brief description about the URL.
    3. In the Configure Action section, set the following parameters.
    1.  Action. Action to be taken for bot detected by bot trap access.
1.  Log. Enable or disable logging for bot trap binding.
  6. In the Configure Trap Insertion URLs section, click Add.

  7. In the Configure NetScaler Bot Management Profile Bot Trap Binding page, set the following parameters.

    1. Trap URL. Type the URL that you want to confirm as the bot trap insertion URL.
    2. Enabled. Enable or disable bot trap insertion URL.
    3. Comment. A brief description about the trap insertion URL.
  8. Click Update and Done.

Configure bot trap URL settings

Complete the following steps to configure bot trap URL settings:

  1. Navigate to Security > NetScaler Bot Management.
  2. In the details pane, under Settings click Change NetScaler Bot Management Settings.

  3. In the Configure NetScaler Bot Management Settings, set the following parameters.

    1. Trap URL Interval. Time in seconds after which the bot trap URL is updated.
    2. Trap URL Length. Length of the auto-generated bot trap URL.
  4. Click OK and Done.

Client IP policy expression for bot detection

The NetScaler bot management now enables you to configure an advanced policy expression to extract the client IP address from an HTTP request header, HTTP request body, HTTP request URL, or using an advanced policy expression. The extracted values are used by a bot detection mechanism (such as TPS, bot trap, or rate limit) to detect if the incoming request is a bot.

Note:

If you have not configured a client IP expression, the default or existing source client IP address is used for bot detection. If an expression is configured, then the evaluation result provides the client IP address that can be used for bot detection.

You can configure and use the client IP expression to extract the actual client IP address if the incoming request is coming through a proxy server and if the client IP address is present in the header. By adding this configuration, the appliance can use the bot detection mechanism in providing more security to software clients and servers.

Configure client IP policy expression in bot profile by using the CLI

At the command prompt, type:

add bot profile <name> [-clientIPExpression <expression>]
<!--NeedCopy-->

Example:

add bot profile profile1 –clientIPExpression 'HTTP.REQ.HEADER("X-Forwarded-For") ALT CLIENT.IP.SRC.TYPECAST_TEXT_T'

add bot profile profile1 –clientIPExpression 'HTTP.REQ.HEADER("X-Forwarded-For") ALT CLIENT.IPv6.SRC.TYPECAST_TEXT_T'

Configure client IP policy expression in bot profile by using the GUI

  1. Navigate to Security > NetScaler Bot Management > Profiles.
  2. In the details pane, click Add.
  3. In the Create NetScaler Bot Management Profile page, set the Client IP Expression.

  4. Click Create and Close.

    GUI configuration for client IP address using policy expression in bot profile

Configure CAPTCHA for IP reputation and device fingerprint detection

CAPTCHA is an acronym that stands for “Completely Automated Public Turing test to tell Computers and Humans Apart”. CAPTCHA is designed to test if an incoming traffic is from a human user or an automated bot. CAPTCHA helps to block automated bots that cause security violations to web applications. In the NetScaler, CAPTCHA uses the challenge-response module to identify if the incoming traffic is from a human user and not an automated bot.

Configure bot static signatures

This detection technique enables you to identify the user agent info from the browser details. Based on user agent information, the bot is identified as a bad or a good bot and then you assign a bot action to it.

Perform the following the steps to configure the static signature technique:

  1. On the navigation pane, expand Security > NetScaler Bot Management > Signatures.
  2. On the NetScaler Bot Management Signatures page, select a signature file and click Edit.
  3. On the NetScaler Bot Management Signature page, go to the Signature Settings section and click Bot Signatures.
  4. In the Bot Signatures section, set the following parameters:
    1. Configure Static Signatures. This section has a list of bot static signature records. You can select a record and click Edit to assign a bot action to it.
    2. Click OK.
  5. Click Update Signature.
  6. Click Done.

Bot Static Signature

Bot static signature delineation

NetScaler bot management protects your web application against bots. Bot static signatures help in identifying good and bad bots based on request parameters such as user-agent in the incoming request.
The list of signatures in the file is huge and also new rules get added and stale ones are removed periodically. As an administrator, you might want to search for a specific signature or list of signatures under a category. To filter signatures easily, the bot signature page provides an enhanced search capability. The search function enables you to find signature rules and configure its property based on one or more signature parameters like action, signature ID, developer, and signature name.

Action - Select a bot action that you prefer to configure for a specific category of signature rules. Following are the available action types:

  • Enable Selected - Enable all the selected signature rules.
  • Disable Selected - Disable all the selected signatures rules.
  • Drop Selected - Select the “Drop” action to all the selected signature rules.
  • Redirect Selected - Apply the “Redirect” action to all the selected signature rules.
  • Reset Selected - Apply the “Reset” action to all the selected signature rules.
  • Log Selected - Apply the “Log” action to all the selected signature rules.
  • Remove Drop Selected - Unset the drop action to all the selected signature rules.
  • Remove Redirect Selected - Unset the redirect action to all the selected signature rules.
  • Remove Reset Selected - Unset the reset action to all the selected signature rules.
  • Remove Log Selected - Unset the log action to all the selected signature rules.

Category - Select a category to filter signature rules accordingly. Following is the list of categories that are available for sorting signature rules.

  • Action - Sort based on bot action.
  • Category - Sort based on bot category.
  • Developer - Sort based on the host company publisher.
  • Enabled - Sort based on signature rules that are enabled.
  • Id - Sort based on signature rule ID.
  • Log - Sort based on signature rules that have logging enabled.
  • Name - Sort based on signature rule name.
  • Type - Sort based on signature type.
  • Version - Sort based on signature rule version.

Search bot static signature rules based on action and category types by using the NetScaler GUI

  1. Navigate to Security > NetScaler Bot Management > Signature.
  2. In the details page, click Add.
  3. In the NetScaler Bot Management Signatures page, click edit in the Static Signature section.
  4. In the Configure Static Signature section, select a signature action from the drop-down list.
  5. Use the search function to select a category and filter the rules accordingly.
  6. Click Update.

Edit the bot static signature rule property by using the NetScaler GUI

  1. Navigate to Security > NetScaler Bot Management > Signature.
  2. In the details page, click Add.
  3. In the NetScaler Bot Management Signatures page, click edit in the Static Signature section.
  4. In the Configure Static Signature section, select an action from the drop-down list.
  5. Use the search function to select a category and filter the rules accordingly.
  6. From the static signature list, select a signature to modify its property.
  7. Click OK to confirm.

How CAPTCHA works in NetScaler bot management

In NetScaler bot management, CAPTCHA validation is configured as a policy action to be run after bot policy is evaluated. The CAPTCHA action is available only for IP reputation and device fingerprint detection techniques. Following are the steps to understand how CAPTCHA works:

  1. If a security violation is observed during IP reputation or device fingerprint bot detection, the ADC appliance sends a CAPTCHA challenge.
  2. The client sends the CAPTCHA response.
  3. The appliance validates the CAPTCHA response and if the CAPTCHA is valid, the request is allowed and it is forwarded to the back-end server.
  4. If the CATCHA response is invalid, the appliance sends a new CAPTCHA challenge until the maximum number of attempts is reached.
  5. If the CAPTCHA response is invalid even after the maximum number of attempts, the appliance drops or redirects the request to the configured error URL.
  6. If you have configured log action, then the appliance stores the request details in the ns.log file.

Configure CAPTCHA settings by using the NetScaler GUI

The bot management CAPTCHA action is supported only for IP reputation and device fingerprint detection techniques. Complete the following steps to configure the CAPTCHA settings.

  1. Navigate to Security > NetScaler Bot Management and Profiles.
  2. On the NetScaler Bot Management Profiles page, select a profile and click Edit.
  3. On the NetScaler Bot Management Profile page, go to the Profile Settings section and click CAPTCHA.
  4. In the CAPTCHA Settings section, click Add to configure CAPTCHA settings to the profile:
  5. In the Configure NetScaler Bot Management CAPTCHA page, set the following parameters.

    1. URL. Bot URL for which the CAPTCHA action is applied during IP reputation and device fingerprint detection techniques.

    2. Enabled. Set this option to enable CAPTCHA support.
    3. Grace time. Duration until when no new CAPTCHA challenge is sent after the current valid CAPTCHA response is received.
    4. Wait time. Duration taken for the ADC appliance to wait until the client sends the CAPTCHA response.
    5. Mute Period. Duration for which the client which sent an incorrect CAPTCHA response must wait until allowed to try next. During this mute period, the ADC appliance does not allow any requests. Range: 60–900 seconds, Recommended: 300 seconds
    6. Request Length limit. Length of the request for which the CAPTCHA challenge is sent to the client. If the length is greater than the threshold value, the request is dropped. Default value is 10–3000 bytes.
    7. Retry Attempts. Number of attempts the client is allowed to retry to solve the CAPTCHA challenge. Range: 1–10, Recommended: 5.
    8. No Action/Drop/Redirect action to be taken if the client fails the CAPTCHA validation.
    9. Log. Set this option to store request information from the client when response CAPTCHA fails. The data is stored in ns.log file.
    10. Comment. A brief description about the CAPTCHA configuration.
  6. Click OK and Done.
  7. Navigate to Security > NetScaler Bot Management > Signatures.
  8. On the NetScaler Bot Management Signatures page, select a signature file and click Edit.
  9. On the NetScaler Bot Management Signature page, go to the Signature Settings section and click Bot Signatures.

  10. In the Bot Signatures section, set the following parameters:

  11. Configure Static Signatures. Select a bot static signature record and click Edit to assign a bot action to it.
  12. Click OK.
  13. Click Update Signature.
  14. Click Done.

Auto update for bot signatures

The bot static signature technique uses a signature lookup table with a list of good bots and bad bots. The bots are categorized based on user-agent string and domain names. If the user-agent string and domain name in incoming bot traffic matches a value in the lookup table, a configured bot action is applied. The bot signature updates are hosted on the AWS cloud and the signature lookup table communicates with the AWS database for signature updates. The auto signature update scheduler runs every 1-hour to check the AWS database and updates the signature table in the NetScaler appliance.

The signature auto update URL to configure is, https://nsbotsignatures.s3.amazonaws.com/BotSignatureMapping.json

Note:

You can also configure a proxy server and periodically update signatures from the AWS cloud to the appliance through the proxy. For proxy configuration, you must set the proxy IP address and port address in the bot settings.

How bot signature auto update works

The following diagram shows how the bot signatures are retrieved from the AWS cloud, updated on NetScaler, and viewed on NetScaler Console for signature update summary.

Bot signature auto update

The bot signature auto-update scheduler does the following:

  1. Retrieves the mapping file from the AWS URI.
  2. Checks the latest signatures in the mapping file with the existing signatures in the ADC appliance.
  3. Downloads the new signatures from AWS and verifies the signature integrity.
  4. Updates the existing bot signatures with the new signatures in the bot signature file.
  5. Generates an SNMP alert and sends the signature update summary to NetScaler Console.

Configure bot signature auto update

For configuring bot signature auto update, complete the following steps:

Enable bot signature auto update

You must enable the auto update option in the bot settings on the ADC appliance. At the command prompt, type:

set bot settings –signatureAutoUpdate ON

Configure proxy server settings (optional)

If you are accessing the AWS signature database through a proxy server, you must configure the proxy server and port. set bot settings –proxyserver –proxyport

Example:

set bot settings –proxy server 1.1.1.1 –proxyport 1356

Configure bot signature auto update using the NetScaler GUI

Complete the following steps to configure bot signature auto update:

  1. Navigate to Security > NetScaler Bot Management.
  2. In the details pane, under Settings click Change NetScaler Bot Management Settings.
  3. In the Configure NetScaler Bot Management Settings, select the Auto Update Signature checkbox.
  4. Click OK and Close.

Create bot management profile

A bot profile is a collection of bot management settings that are used for detecting the bot type. In a profile, you determine how the Web App Firewall applies each of its filters (or checks) to bot traffic to your websites, and responses from them.

Complete the following steps to configure the bot profile:

  1. Navigate to Security > NetScaler Bot Management > Profiles.
  2. In the details pane, click Add.

  3. In the Create NetScaler Bot Management Profile page, set the following parameters.

    1. Name. Bot profile name.
    2. Signature. Name of the bot signature file.
    3. Error URL. URL for redirects.
    4. Comment. Brief description about the profile.
  4. Click Create and Close.

Create bot policy

The bot policy controls the traffic going to the bot management system and also to control the bot logs sent to the auditlog server. Follow the procedure to configure the bot policy.

  1. Navigate to Security > NetScaler Bot Management > Bot Policies.
  2. In the details pane, click Add.
  3. In the Create NetScaler Bot Management Policy page, set the following parameters.
    1. Name. Name of the Bot policy.
    2. Expression. Type the policy expression or rule directly in the text area.
    3. Bot Profile. Bot profile to apply the bot policy.
    4. Undefined Action. Select an action that you prefer to assign.
    5. Comment. Brief description about the policy.
    6. Log Action. Audit log message action for logging bot traffic. For more information about audit log action, see Audit logging topic.
  4. Click Create and Close.

Bot Transactions Per second (TPS)

The Transactions Per Second (TPS) bot technique detects incoming traffic as a bot if the number of requests per second (RPS) and percentage increase in RPS exceeds the configured threshold value. The detection technique protects your web applications from automated bots that can cause web scraping activities, brute forcing login, and other malicious attacks.

Note:

The bot technique detects an incoming traffic as bot only if both the parameters are configured and if both values increase beyond the threshold limit. Let us consider a scenario, where the appliance receives many requests coming from a specific URL and you want the NetScaler bot management to detect if there is a bot attack. The TPS detection technique examines the number of requests (configured value) coming from the URL within 1 second and the percentage increase (configured value) in the number of requests received within 30 minutes. If the values exceed the threshold limit, the traffic is considered as bot and the appliance runs the configured action.

Configure bot transactions per second (TPS) technique

To configure TPS, you must complete the following steps:

  1. Enable bot TPS
  2. Bind TPS settings to bot management profile

Bind TPS settings to bot management profile

Once you enable the bot TPS feature, you must bind the TPS settings to the bot management profile.

At the command prompt, type:

bind bot profile <name>… (-tps [-type ( SourceIP | GeoLocation | RequestURL | Host )] [-threshold <positive_integer>] [-percentage <positive_integer>] [-action ( none | log | drop | redirect | reset | mitigation )] [-logMessage <string>])

Example:

bind bot profile profile1 -tps -type RequestURL -threshold 1 -percentage 100000 -action drop -logMessage log

Enable bot transaction per second (TPS)

Before you can begin, you must ensure that the Bot TPS feature is enabled on the appliance. At the command prompt, type:

set bot profile profile1 –enableTPS ON

Configure bot transactions per second (TPS) by using the NetScaler GUI

Complete the following steps to configure bot transactions per second:

  1. Navigate to Security > NetScaler Bot Management > Profiles.
  2. In the NetScaler Bot Management Profiles page, select a profile and click Edit.
  3. In the Create NetScaler Bot Management Profile page, click TPS under Signature Settings section.
  4. In the TPS section, enable the feature and click Add.

  5. In Configure NetScaler bot Management Profile TPS Binding page, set the following parameters.

    1. Type - Select any one of the following type:

      SOURCE_IP – TPS based on client IP address.

      GEOLOCATION – TPS based on the client’s geographic location.

      HOST - TPS based on client requests forwarded to a specific back-end server IP address.

      URL – TPS based on client requests coming from a specific URL.

    2. Fixed Threshold - Maximum number of requests allowed from a TPS input type within 1 second time interval.

    3. Percentage Threshold - Maximum percentage increase in requests from a TPS input type within 30 minute time interval.

    4. Action - Action to be taken for bot detected by TPS binding.

    5. Log - Enable or disable logging for TPS binding.

    6. Log Message. Message to log for bot detected by TPS binding. Maximum Length: 255.

    7. Comments - A brief description about the TPS configuration. Maximum Length: 255

  6. Click OK and then Close.

Bot detection based on mouse and keyboard dynamics

To detect bots and mitigate web scraping anomalies, the NetScaler bot management uses an enhanced bot detection technique based on mouse and keyboard behavior. Unlike conventional bot techniques that require direct human interaction (for example, CAPTCHA validation), the enhanced technique passively monitors the mouse and the keyboard dynamics. The NetScaler appliance then collects the real-time user data and analyses the behavioral between a human and a bot.

The passive bot detection using mouse and keyboard dynamics has the following benefits over existing bot detection mechanisms:

  • Provides continuous monitoring throughout the user session, and eliminates single checkpoint.
  • Requires no human interaction and it is transparent to users.

How bot detection using mouse and keyboard dynamics works

The bot detection technique using keyboard and mouse dynamics consists of two components, a webpage logger and bot detector. The webpage logger is a JavaScript that records keyboard and mouse movements when a user is performing a task on the webpage (for example, filling a registration form). The logger then sends the data in batches to the NetScaler appliance. The appliance then stores the data as a KM record and sends it to the bot detector on the NetScaler Console server, which analyses if the user is a human or bot.

The following steps explain how the components interact with each other:

  1. The NetScaler admin configures policy expression through the ADM StyleBook, CLI, or NITRO or any other method.
  2. The URL is set in the bot profile when the admin enables the feature on the appliance.
  3. When a client sends a request, the NetScaler appliance tracks the session and all requests in the session.
  4. The appliance inserts a JavaScript (webpage logger) in the response if the request matches the configured expression on the bot profile.
  5. The JavaScript then collects all the keyboard, mouse activity and sends the KM data in a POST URL (transient).
  6. The NetScaler appliance stores the data and sends it to the NetScaler Console server at the end of the session. Once the appliance receives the complete data of a POST request, the data is sent it to the ADM server.
  7. The NetScaler Console Service analyses the data and based on the analysis, the result is available on the NetScaler Console service GUI.

The JavaScript logger records the following mouse and keyboard movements:

  • Keyboard events – all events
  • Mouse events - mouse move, mouse up, mouse down
  • Clipboard events - paste
  • Custom events - autofill, autofill cancel
  • timestamp of each event

Configure bot detection using mouse and keyboard dynamics

The NetScaler bot management configuration includes enabling or disabling the keyboard and mouse-based detection feature, and configures the JavaScript URL in the bot profile. Complete the following steps to configure bot detection using mouse and keyboard dynamics:

  1. Enable keyboard and mouse-based detection
  2. Configure expression to decide when the JavaScript can be injected in the HTTP response

Enable keyboard mouse-based bot detection

Before you begin the configuration, ensure you have enabled the keyboard and mouse-based bot detection feature on the appliance.

At the command prompt, type:

add bot profile <name> -KMDetection ( ON | OFF )
<!--NeedCopy-->

Example:

add bot profile profile1 –KMDetection ON

Configure bot expression for JavaScript insertion

Configure bot expression to evaluate the traffic and insert JavaScript. The JavaScript is inserted only if the expression is evaluated as true.

At the command prompt, type:

bind bot profile <name> -KMDetectionExpr -name <string> -expression <expression> -enabled ( ON | OFF ) –comment <string>
<!--NeedCopy-->

Example:

bind bot profile profile1 -KMDetectionExpr -name test -expression http.req.url.startswith("/testsite") -enabled ON

Configure JavaScript file name inserted in the HTTP response for keyboard-mouse based bot detection

To collect the user action details, the appliance sends a JavaScript file name in the HTTP response. The JavaScript file collects all the data in a KM record and sends it to the appliance.

At the command prompt, type:

set bot profile profile1 – KMJavaScriptName <string>
<!--NeedCopy-->

Example:

set bot profile profile1 –KMJavaScriptName script1

Configure behavior biometrics size

You can configure the maximum size of mouse and keyboard behavior data that can be sent as KM record to the appliance and processed by the ADM server.

At the command prompt, type:

set bot profile profile1 -KMEventsPostBodyLimit <positive_integer>
<!--NeedCopy-->

Example:

set bot profile profile1 – KMEventsPostBodyLimit 25

After you have configured the NetScaler appliance to configure the JavaScript and collect keyboard and mouse behavior biometrics, the appliance sends the data to the NetScaler Console server. For more information on how the NetScaler Console server detects bots from behavior biometrics, see Bot Violations topic.

Configure keyboard and mouse bot expression settings by using the GUI

  1. Navigate to Security > NetScaler Bot Management and Profiles.
  2. On the NetScaler Bot Management Profiles page, select a profile and click Edit.
  3. On the NetScaler Bot Management Profile page, click edit icon.
  4. In the Basic Setting, under Keyboard and mouse based bot detection section, set the following parameters:
    1. Enable detection. Select the checkbox to detect the bot based keyboard and mouse dynamics behavior.
    2. Event post body limit. Size of the keyboard and mouse dynamics data sent by the browser to be processed by the NetScaler appliance.

  5. Click OK.

  6. On the NetScaler bot Management Profile page, go to the Profile Settings section and click Keyboard and Mouse Based Bot Expression Settings.
  7. In the Keyboard and Mouse Based Bot Expression Settings section, click Add.
  8. In the Configure NetScaler bot Management Profile Bot Keyboard and Mouse Expression Binding page, set the following parameters:
    1. Expression Name. Name of the bot policy expression for detection keyboard and mouse dynamics.
    2. Expression. Bot policy expression.
    3. Enabled. Select the checkbox to enable the keyboard and bot keyboard and mouse expression binding.
    4. Comments. A brief description about the bot policy expression and its binding to the bot profile.
    5. Click OK and Close.
  9. In the Keyboard and Mouse Based Bot Expression Settings section, Click Update.

Verbose logging for bot traffic

When an incoming request is identified as a bot, the NetScaler appliance logs more HTTP header details for monitoring and troubleshooting. The bot verbose logging capability is similar to the verbose logging in the Web App Firewall module.

Consider an incoming traffic from a client. If the client is identified as a bot, the NetScaler appliance uses the verbose logging functionality to log full HTTP header information such as domain address, URL, user-agent header, cookie header. The log details are then sent to the ADM server for monitoring and troubleshooting the purpose. The verbose log message is not stored in the “ns.log” file.

Configure bot verbose logging by using the CLI

To capture detailed HTTP header information as logs, you can configure the verbose logging parameter in the bot profile. At the command prompt, type:

set bot profile <name> [-verboseLogLevel ( NONE | HTTP_FULL_HEADER ) ]
<!--NeedCopy-->

Example:

set bot profile p1 –verboseLogLevel HTTP_FULL_HEADER

Configure bot verbose logging by using the NetScaler GUI

Follow the procedure to configure the verbose log level in the bot profile.

  1. On the navigation pane, navigate to Security > NetScaler Bot Management.
  2. In the NetScaler Bot Management Profiles page, click Add.
  3. In the Create NetScaler Bot Management Profile page, select verbose log level as HTTP Full Header.
  4. Click OK and Done.

Configure an action for spoofed bot requests

An attacker might attempt to impersonate a good bot and send requests to your application server. Such bots are identified as spoofed bots using the bot signature. Configure the following actions against spoofed bots to protect your application server:

  • DROP
  • NONE
  • REDIRECT
  • RESET

Configure an action for spoofed bot requests using CLI

Run the following command to configure an action for spoofed bot requests:

set bot profile <bot-profile-name> -spoofedReqAction <action> LOG
<!--NeedCopy-->

Example:

set bot profile bot_profile -spoofedReqAction DROP LOG
<!--NeedCopy-->

In this example, the requests from spoofed bots are dropped and logged in a NetScaler appliance.

Tip

To log events from the spoofed bots, specify LOG in the command.

Configure an action for spoofed bot requests using GUI

Follow the steps to configure an action for the spoofed bot requests:

  1. Navigate to Security > NetScaler Bot Management.

  2. In the NetScaler Bot Management Profiles page, click Add.

  3. Select an action from the Spoofed Request Action list.

  4. Select Log Spoofed Request.

    This action logs the events from spoofed bots.

  5. Click Create.

Request headers dropped by the NetScaler bot Management

Many of the request headers related to caching is dropped to view every request within the context of a session. Similarly, if the request includes an encoding header to allow the web server to send compressed responses, the bot management deletes this header so that the contents in the uncompressed server response is inspected by the bot management to insert the JavaScript.

The bot management drops the following request headers:

Range - Used to recover from a failed or partial file transfers.

If-Range - Allows a client to retrieve a partial object when it contains a part of that object in its cache already (conditional GET).

If-Modified-Since - If the requested object is not modified since the time specified in this field, an entity is not returned from the server. You get an HTTP 304 not modified error.

If-None-Match - Allows efficient updates of cached information with a minimum amount of overhead.

Accept-Encoding - What encoding methods are allowed for a particular object, such as gzip.

Bot Detection
January 4, 2024
Contributed by: 
B S
January 4, 2024
Contributed by: 
B S

In this article

Site feedback | Your privacy choices footer linkYour Privacy Choices | Privacy and legal terms | Cookie preferences | Consent settings | docs.cloud.com
© 1999- Cloud Software Group, Inc. All rights reserved.