Product Documentation
ADC
14.1 - Current Release 13.1 13.0 12.1
View PDF

Bot Detection

August 13, 2024
Contributed by: 
B C S

The NetScaler bot management system uses various techniques to detect the incoming bot traffic. The techniques are used as detection rules to detect the bot type. The techniques are as follows:

Note:

Bot management supports a maximum of 32 configuration entities for block list, allow list, and rate limiting techniques.

Bot Allowed list - A customized list of IP addresses (IPv4 and IPv6), subnets (IPv4 and IPv6), and policy expressions that can be bypassed as an allowed list.

Bot Blocked list - A customized list of IP addresses (IPv4 and IPv6), subnets (IPv4 and IPv6), and policy expressions that have to be blocked from accessing your web applications.

IP reputation - This rule detects if the incoming bot traffic is from a malicious IP address.

Device fingerprint - This rule detects if the incoming bot traffic has the device fingerprint ID in the incoming request header and browser attributes of an incoming client bot traffic.

Limitation:

  1. Java Script must be enabled in the client browser.
  2. Does not work for XML responses.

Bot log expression - The detection technique enables you to capture additional information as log messages. The data can be the name of the user who requested the URL, the source IP address, and the source port from which the user sent the request or data generated from an expression.

Rate limit - This rule rate limits multiple requests coming from the same client.

Bot trap - Detects and blocks automated bots by advertising a trap URL in the client response. The URL appears invisible and not accessible if the client is a human user. The detection technique is effective in blocking attacks from automated bots.

TPS - Detects incoming traffic as bots if the maximum number of requests and percentage increase in requests exceeds the configured time interval.

CAPTCHA - This rule uses a CAPTCHA for mitigating bot attacks. A CAPTCHA is a challenge-response validation to determine if the incoming traffic is from a human user or an automated bot. The validation helps block automated bots that cause security violations to web applications. You can configure CAPTCHA as a bot action in IP reputation and device fingerprint detection techniques.

Now, let us see how you can configure each technique to detect and manage your bot traffic.

How to upgrade your appliance to NetScaler CLI-based bot management configuration

If you are upgrading your appliance from an older version (NetScaler release 13.0 build 58.32 or earlier), you must first manually convert the existing bot management configuration to the NetScaler CLI based bot management configuration only once. Complete the following steps to manually convert your bot management configuration.

  1. After upgrading to the latest version connect to the upgrade tool “upgrade_bot_config.py” by using the following command

    At the command prompt, type:

    shell "/var/python/bin/python /netscaler/upgrade_bot_config.py > /var/bot_upgrade_commands.txt"

  2. Run the configuration using the following command.

    At the command prompt, type:

    batch -f /var/bot_upgrade_commands.txt

  3. Save the upgraded configuration.

save ns config

Configure NetScaler CLI-based bot management

The bot management configuration enables you to bind one or more bot detection techniques to a specific bot profile. You must complete the following steps to configure NetScaler-based bot management:

  1. Enable bot management
  2. Import bot signature
  3. Add bot profile
  4. Bind bot profile
  5. Add bot policy
  6. Bind bot policy
  7. Configure bot settings

Note:

If you are upgrading your appliance from an older version, you must first manually convert the existing bot management configuration. For more information, see How to upgrade to NetScaler CLI-based bot management configuration section.

Enable bot management

Before you can begin, ensure that the Bot Management feature is enabled on the appliance. If you have a new NetScaler or VPX, you must enable the feature before you configure it. If you are upgrading a NetScaler appliance from an earlier version to the current version, you must need to enable the feature before you configure it. At the command prompt, type:

enable ns feature Bot

Import bot signature

You can import the default signature bot file and bind it to the bot profile. At the command prompt, type:

import bot signature [<src>] <name> [-comment <string>] [-overwrite]

Where:

src - Local path name, or URL (protocol, host, path, and file name). Maximum Length: 2047. > Note: > > The import fails if the object to be imported is on an HTTPS server that requires client certificate authentication for access.

name - Name of the bot signature file object. This is a mandatory argument. Maximum Length: 31.

comment - Description about the signature file object. Maximum Length: 255.

overwrite - Action that overwrites the existing file. > Note: > > Use the overwrite option to update the content in the signature file. Alternately, use the update bot signature <name> command to update the signature file on the NetScaler appliance.

Example

import bot signature http://www.example.com/signature.json signaturefile -comment commentsforbot –overwrite

Note:

You can use the overwrite option to update the content in the signature file. Also, you can use the update bot signature <name> command to update the signature file in the NetScaler appliance.

Add bot profile

A bot profile is a collection of profile settings to configure bot management on the appliance. You can configure the settings to perform bot detection.

Note:

Starting from release 14.1 build 29.x, a new parameter addCookieFlags is added to the add bot profile and set bot profile commands.

At the command prompt, type:

add bot profile <name> [-signature <string>] [-errorURL <string>] [-trapURL <string>] [-whiteList ( ON | OFF )] [-blackList ( ON | OFF )] [-rateLimit ( ON | OFF )] [-deviceFingerprint ( ON | OFF )] [-deviceFingerprintAction ( none | log | drop | redirect | reset | mitigation )] [-ipReputation ( ON | OFF )] [-trap ( ON | OFF )] [-addCookieFlags ([**none**] | [**httpOnly**] | [**secure**] | [**all**])]

Where:

addCookieFlags - Add flags to cookies during transformation. Specify any one of the following:

  • None - Do not add flags to cookies.
  • HTTP only - Add the HttpOnly flag to all cookies. With this flag set, browsers that support the HttpOnly flag prevent scripts from accessing cookies. The default.
  • Secure - Add the Secure flag to cookies that are to be sent only over an SSL connection. Browsers that support the secure flag do not send the flagged cookies over an insecure connection.
  • All - Add the HttpOnly flag to all cookies, and the Secure flag to cookies that are to be sent only over an SSL connection.

Example:

add bot profile profile1 -signature signature -errorURL http://www.example.com/error.html -trapURL /trap.html -whitelist ON -blacklist ON -ratelimit ON -deviceFingerprint ON -deviceFingerprintAction drop -ipReputation ON -trap ON -addCookieFlags secure

Bind bot profile

After you create a bot profile, you must bind the bot detection mechanism to the profile.

At the command prompt, type:

bind bot profile <name> | (-ipReputation [-category <ipReputationCategory>] [-enabled ( ON | OFF )] [-action ( none | log | drop | redirect | reset | mitigation )] [-logMessage <string>]

Example:

The following example is for binding the IP reputation detection technique to a specific bot profile.

bind bot profile profile5 -ipReputation -category BOTNET -enabled ON -action drop -logMessage message

Add bot policy

You must add the bot policy for evaluating bot traffic.

At the command prompt, type:

add bot policy <name> -rule <expression> -profileName <string> [-undefAction <string>] [-comment <string>] [-logAction <string>]

Where,

Name- Name for the bot policy. Must begin with a letter, number, or the underscore character (_), and must contain only letters, numbers, and the hyphen (-), period (.) pound (#), space ( ), at (@), equals (=), colon (:), and underscore characters. Can be changed after the bot policy is added.

Rule- An expression that the policy uses to determine whether to apply the bot profile on the specified request. This is a mandatory argument. Maximum Length: 1499

profileName- Name of the bot profile to apply if the request matches this bot policy. This is a mandatory argument. Maximum Length: 127

undefAction- Action to perform if the result of policy evaluation is undefined (UNDEF). An UNDEF event indicates an internal error condition. Maximum Length: 127

Comment- Description about this bot policy. Maximum Length: 255

logAction - Name of the log action to use for requests that match this policy. Maximum Length: 127

Example:

add bot policy pol1 –rule "HTTP.REQ.HEADER(\"header\").CONTAINS(\"custom\")" - profileName profile1 -undefAction drop –comment commentforbotpolicy –logAction log1

Bind bot policy global

At the command prompt, type:

bind bot global -policyName <string> -priority <positive_integer> [-gotoPriorityExpression <expression>][-type ( REQ_OVERRIDE | REQ_DEFAULT )] [-invoke (-labelType ( vserver | policylabel ) -labelName <string>) ]

Example:

bind bot global –policyName pol1 –priority 100 –gotoPriorityExpression NEXT -type REQ_OVERRIDE

Bind bot policy to a virtual server

At the command prompt, type:

bind lb vserver <name>@ ((<serviceName>@ [-weight <positive_integer>] ) | <serviceGroupName>@ | (-policyName <string>@ [-priority <positive_integer>] [-gotoPriorityExpression <expression>]

Example:

bind lb vserver lb-server1 –policyName pol1 –priority 100 –gotoPriorityExpression NEXT -type REQ_OVERRIDE

Configure bot settings

You can customize the default settings if necessary. At the command prompt, type:

set bot settings [-defaultProfile <string>] [-javaScriptName <string>] [-sessionTimeout <positive_integer>] [-sessionCookieName <string>] [-dfpRequestLimit <positive_integer>] [-signatureAutoUpdate ( ON | OFF )] [-signatureUrl <URL>] [-proxyServer <ip_addr|ipv6_addr|*>] [-proxyPort <port|*>]
<!--NeedCopy-->

Where,

defaultProfile - Profile to use when a connection does not match any policy. Default setting is “ “, which sends unmatched connections back to the NetScaler without attempting to filter them further. Maximum Length: 31

javaScriptName - Name of the JavaScript that the BotNet feature uses in response. Must begin with a letter or number, and can consist of from 1 to 31 letters, numbers, and the hyphen (-) and underscore (_) symbols. The following requirement applies only to the NetScaler CLI: If the name includes one or more spaces, enclose the name in double or single quotation marks (for example, “my cookie name” or ‘my cookie name’). Maximum Length: 31

sessionTimeout - Session times out, in seconds, after which a user session is terminated.

Minimum value - 1, Maximum value: 65535

sessionCookieName - Name of the SessionCookie that the BotNet feature uses it for tracking. Must begin with a letter or number, and can consist of from 1 to 31 letters, numbers, and the hyphen (-) and underscore (_) symbols. The following requirement applies only to the NetScaler CLI: If the name includes one or more spaces, enclose the name in double or single quotation marks (for example, “my cookie name” or ‘my cookie name’). Maximum Length: 31

dfpRequestLimit - Number of requests to allow without bot session cookie if device fingerprint is enabled. Minimum value: 1, Maximum Value: 4294967295

signatureAutoUpdate - Flag used to enable/disable bot auto update signatures. Possible values: ON, OFF. Default value: OFF

signatureUrl - URL to download the bot signature mapping file from the server. Default value: https://nsbotsignatures.s3.amazonaws.com/BotSignatureMapping.json. Maximum Length: 2047

proxyServer - Proxy Server IP to get updated signatures from AWS.

proxyPort - Proxy Server Port to get updated signatures from AWS. Default value: 8080

proxyUsername - User name to authenticate to the proxy server for downloading signature updates.

proxyPassword – Password to authenticate to the proxy server for downloading signature updates.

Example:

set bot settings –defaultProfile profile1 –javaScriptName json.js –sessionTimeout 1000 –sessionCookieName session -proxyServer 10.102.30.112 -proxyPort 3128 -proxyUsername defaultuser -proxyPassword defaultPassword

Configuring bot management by using NetScaler GUI

You can configure NetScaler bot management by first enabling the feature on the appliance. Once you enable, you can create a bot policy to evaluate the incoming traffic as bot and send the traffic to the bot profile. Then, you create a bot profile and then bind the profile to a bot signature. As an alternative, you can also clone the default bot signature file and use the signature file to configure the detection techniques. After creating the signature file, you can import it into the bot profile.

Bot Management Page

  1. Enable bot management feature
  2. Configure bot management settings
  3. Clone NetScaler bot default signature
  4. Import NetScaler bot signature
  5. Configure bot profile and signature settings
  6. Create bot profile
  7. Create bot policy

Enable bot management feature

Complete the following steps to enable bot management:

  1. On the navigation pane, expand System and then click Settings > Configure Advanced Features.
  2. On the Configure Advanced Features page, select the NetScaler Bot Management checkbox.
  3. Click OK.

Configure bot management settings

Complete the following step to configure the device fingerprint technique:

  1. Navigate to Security > NetScaler bot Management.
  2. In the details pane, under Settings click Change NetScaler bot Management Settings.

  3. In the Configure NetScaler bot Management Settings, set the following parameters.

    1. Default Profile. Select a bot profile.
    2. JavaScript Name. Name of the JavaScript file that bot management uses in its response to the client.
    3. Session Timeout. Timeout in seconds after which the user session is terminated.
    4. Session Cookie. Name of the session cookie that the bot management system uses for tracking.
    5. Device Fingerprint Request Limit. Number of requests to allow without a bot session cookie, if device fingerprint is enabled.
    6. Proxy Server - Proxy server IP address from where the latest signatures are uploaded.
    7. Proxy Port – Port number of machine from where the latest signatures are uploaded.
    8. Proxy Username – User name for the authentication of the proxy server
    9. Proxy Password - Password for the authentication of the proxy server.

      Note:

      Proxy User name and the Proxy Password fields are enabled if the Proxy Server and the Proxy Port fields are configured.

  4. Click OK.

Clone bot signature file

Complete the following step to clone the bot signature file:

  1. Navigate to Security > NetScaler Bot Management and Signatures.
  2. In the NetScaler Bot Management Signatures page, select the default bot signatures record and click Clone.
  3. In the Clone Bot Signature page, enter a name and edit the signature data.
  4. Click Create.

Import bot signature file

If you have your own signature file, then you can import it as a file, text, or URL. Perform the following steps to import the bot signature file:

  1. Navigate to Security > NetScaler Bot Management and Signatures.
  2. On the NetScaler Bot Management Signatures page, import the file as URL, File, or text.
  3. Click Continue.
  4. On the Import NetScaler Bot Management Signature page, set the following parameters.
    1. Name - Name of the bot signature file.
    2. Comment - Brief description about the imported file.
    3. Overwrite - Select the checkbox to allow overwriting of data during file update.
    4. Signature Data - Modify signature parameters
  5. Click Done.

Configure IP reputation

The IP reputation bot technique uses Webroot’s IP reputation database and cloud service provider database to verify if a client request is a malicious IP address or a public cloud IP address. As part of the bot categories is configured and then a bot action is associated to it. Complete the following steps to configure Webroot IP reputation and cloud service provider database categories.

  1. Navigate to Security > NetScaler bot Management and Profiles.
  2. On the NetScaler bot Management Profiles page, select a profile and click Edit.
  3. On the NetScaler bot Management Profile page, go to the Profile Settings section and click IP Reputation.
  4. On the IP Reputation section, set the following parameters:
    1. Enabled. Select the checkbox to validate incoming bot traffic as part of the detection process.
    2. Configure Categories. You can use the IP reputation technique for incoming bot traffic under different categories. Based on the configured category, you can drop or redirect the bot traffic. Click Add to configure a malicious bot category.

    3. In the Configure NetScaler bot Management Profile IP Reputation Binding page, set the following parameters:

      1. Category. Select a Webroot IP reputation bot category to validate a client request as a malicious IP address.

        1. IP_BASED - This category checks whether the client IP address (IPv4 and IPv6) is malicious or not.
        2. BOTNET - This category includes Botnet C&C channels, and infected zombie machines controlled by Bot master.
        3. SPAM_SOURCES - This category includes tunneling spam messages through a proxy, anomalous SMTP activities, and forum spam activities.
        4. SCANNERS - This category includes all reconnaissance such as probes, host scan, domain scan, and password brute force attack.
        5. DOS - This category includes DOS, DDOS, anomalous sync flood, and anomalous traffic detection.
        6. REPUTATION - This category denies access from IP addresses (IPv4 and IPv6) currently known to be infected with malware. This category also includes IP addresses with average low Webroot Reputation Index score. Enabling this category prevents access from sources identified to contact malware distribution points.
        7. PHISHING - This category includes IP addresses (IPv4 and IPv6) hosting phishing sites and other kinds of fraud activities such as ad click fraud or gaming fraud.
        8. PROXY - This category includes IP addresses (IPv4 and IPv6) providing proxy services.
        9. NETWORK - IPs providing proxy and anonymization services including The Onion Router aka TOR or dark net.
        10. MOBILE_THREATS - This category checks the client IP address (IPv4 and IPv6) with the list of addresses harmful for mobile devices.

      2. Category. Select a Webroot public cloud service provider category to validate a client request is a public cloud IP address.

        1. AWS - This category checks client IP address with list of public cloud addresses from AWS.
        2. GCP - This category checks client IP address with list of public cloud addresses from the Google Cloud Platform.
        3. AZURE - This category checks client address with list of public cloud addresses from Azure.
        4. ORACLE - This category checks client IP address with list of public cloud addresses from Oracle
        5. IBM - This category checks client IP address with list of public cloud addresses from IBM.
        6. SALESFORCE - This category checks client IP address with list of public cloud addresses from Salesforce.

        Possible values for Webroot IP reputation bot category: IP, BOTNETS, SPAM_SOURCES, SCANNERS, DOS, REPUTATION, PHISHING, PROXY, NETWORK, MOBILE_THREATS.

        Possible values for Webroot public cloud service provider category: AWS, GCP, AZURE, ORACLE, IBM, SALESFORCE.

      3. Enabled. Select the checkbox to validate the IP reputation signature detection.
      4. Bot action. Based on the configured category, you can assign no action, drop, redirect, or mitigation action.
      5. Log. Select the checkbox to store log entries.
      6. Log Message. Brief description of the log.
      7. Comments. Brief description about the bot category.
  5. Click OK.
  6. Click Update.

  7. Click Done.

    Configure IP Reputation

Note

If you disable IP reputation, ensure to stop its downloads. Complete the following steps to stop the IP reputation downloads:

  1. Navigate to Security > NetScaler bot Management > Change NetScaler bot Management Settings
  2. Change the Default Nonintrusive Profile to BOT_BYPASS.

For more information on profile settings, see Configure bot profile setting

Configure bot static signatures

This detection technique enables you to identify the user agent info from the browser details. Based on user agent information, the bot is identified as a bad or a good bot and then you assign a bot action to it.

Perform the following the steps to configure the static signature technique:

  1. On the navigation pane, expand Security > NetScaler Bot Management > Signatures.
  2. On the NetScaler Bot Management Signatures page, select a signature file and click Edit.
  3. On the NetScaler Bot Management Signature page, go to the Signature Settings section and click Bot Signatures.
  4. In the Bot Signatures section, set the following parameters:
    1. Configure Static Signatures. This section has a list of bot static signature records. You can select a record and click Edit to assign a bot action to it.
    2. Click OK.
  5. Click Update Signature.
  6. Click Done.

Bot Static Signature

Bot static signature delineation

NetScaler bot management protects your web application against bots. Bot static signatures help in identifying good and bad bots based on request parameters such as user-agent in the incoming request.
The list of signatures in the file is huge and also new rules get added and stale ones are removed periodically. As an administrator, you might want to search for a specific signature or list of signatures under a category. To filter signatures easily, the bot signature page provides an enhanced search capability. The search function enables you to find signature rules and configure its property based on one or more signature parameters like action, signature ID, developer, and signature name.

Action - Select a bot action that you prefer to configure for a specific category of signature rules. Following are the available action types:

  • Enable Selected - Enable all the selected signature rules.
  • Disable Selected - Disable all the selected signatures rules.
  • Drop Selected - Select the “Drop” action to all the selected signature rules.
  • Redirect Selected - Apply the “Redirect” action to all the selected signature rules.
  • Reset Selected - Apply the “Reset” action to all the selected signature rules.
  • Log Selected - Apply the “Log” action to all the selected signature rules.
  • Remove Drop Selected - Unset the drop action to all the selected signature rules.
  • Remove Redirect Selected - Unset the redirect action to all the selected signature rules.
  • Remove Reset Selected - Unset the reset action to all the selected signature rules.
  • Remove Log Selected - Unset the log action to all the selected signature rules.

Category - Select a category to filter signature rules accordingly. Following is the list of categories that are available for sorting signature rules.

  • Action - Sort based on bot action.
  • Category - Sort based on bot category.
  • Developer - Sort based on the host company publisher.
  • Enabled - Sort based on signature rules that are enabled.
  • Id - Sort based on signature rule ID.
  • Log - Sort based on signature rules that have logging enabled.
  • Name - Sort based on signature rule name.
  • Type - Sort based on signature type.
  • Version - Sort based on signature rule version.

Search bot static signature rules based on action and category types by using the NetScaler GUI

  1. Navigate to Security > NetScaler Bot Management > Signature.
  2. In the details page, click Add.
  3. In the NetScaler Bot Management Signatures page, click edit in the Static Signature section.
  4. In the Configure Static Signature section, select a signature action from the drop-down list.
  5. Use the search function to select a category and filter the rules accordingly.
  6. Click Update.

Edit the bot static signature rule property by using the NetScaler GUI

  1. Navigate to Security > NetScaler Bot Management > Signature.
  2. In the details page, click Add.
  3. In the NetScaler Bot Management Signatures page, click edit in the Static Signature section.
  4. In the Configure Static Signature section, select an action from the drop-down list.
  5. Use the search function to select a category and filter the rules accordingly.
  6. From the static signature list, select a signature to modify its property.
  7. Click OK to confirm.

For more information on profile settings, see Configure bot signature setting

Create bot management profile

A bot profile is a collection of bot management settings that are used for detecting the bot type. In a profile, you determine how the Web App Firewall applies each of its filters (or checks) to bot traffic to your websites, and responses from them.

Complete the following steps to configure the bot profile:

  1. Navigate to Security > NetScaler Bot Management > Profiles.
  2. In the details pane, click Add.

  3. In the Create NetScaler Bot Management Profile page, set the following parameters.

    1. Name. Bot profile name.
    2. Signature. Name of the bot signature file.
    3. Error URL. URL for redirects.
    4. Comment. Brief description about the profile.
  4. Click Create and Close.

Create bot policy

The bot policy controls the traffic going to the bot management system and also to control the bot logs sent to the auditlog server. Follow the procedure to configure the bot policy.

  1. Navigate to Security > NetScaler Bot Management > Bot Policies.
  2. In the details pane, click Add.
  3. In the Create NetScaler Bot Management Policy page, set the following parameters.
    1. Name. Name of the Bot policy.
    2. Expression. Type the policy expression or rule directly in the text area.
    3. Bot Profile. Bot profile to apply the bot policy.
    4. Undefined Action. Select an action that you prefer to assign.
    5. Comment. Brief description about the policy.
    6. Log Action. Audit log message action for logging bot traffic. For more information about audit log action, see Audit logging topic.
  4. Click Create and Close.

Request headers dropped by the NetScaler bot Management

Many of the request headers related to caching is dropped to view every request within the context of a session. Similarly, if the request includes an encoding header to allow the web server to send compressed responses, the bot management deletes this header so that the contents in the uncompressed server response is inspected by the bot management to insert the JavaScript.

The bot management drops the following request headers:

Range - Used to recover from a failed or partial file transfers.

If-Range - Allows a client to retrieve a partial object when it contains a part of that object in its cache already (conditional GET).

If-Modified-Since - If the requested object is not modified since the time specified in this field, an entity is not returned from the server. You get an HTTP 304 not modified error.

If-None-Match - Allows efficient updates of cached information with a minimum amount of overhead.

Accept-Encoding - What encoding methods are allowed for a particular object, such as gzip.

Bot Detection
August 13, 2024
Contributed by: 
B C S
August 13, 2024
Contributed by: 
B C S

In this article

Site feedback | Your privacy choices footer linkYour Privacy Choices | Privacy and legal terms | Cookie preferences | Consent settings | docs.cloud.com
© 1999- Cloud Software Group, Inc. All rights reserved.