Configure bot management policies with the NetScaler Ingress Controller
A bot is a software application that automates manual tasks. Using Bot management policies you can allow useful bots to access your cloud native environment and block the malicious bots.
Custom Resource Definitions (CRDs) are the primary way of configuring policies in cloud native deployments. Using the Bot CRD provided by NetScaler, you can configure the bot management policies with the NetScaler Ingress Controller on the NetScaler VPX. The Bot CRD enables communication between the NetScaler Ingress Controller and NetScaler for enforcing bot management policies.
In a Kubernetes deployment, you can enforce bot management policy on the requests and responses from and to the server using the Bot CRD. For more information on security vulnerabilities, see Bot Detection.
With the Bot CRD, you can configure the bot management security policy for the following types of security vulnerabilities for the Kubernetes-native applications:
- Allow list
- Block list
- Device Fingerprint (DFP)
- Bot TPS
- Trap insertion
- IP reputation
- Rate limit
Based on the type of protections required, you can specify the metadata and use the CRD attributes in the Bot CRD .yaml file to define the bot policy.
Bot CRD definition
The Bot CRD is available in the NetScaler Ingress Controller GitHub repo at bot-crd.yaml. The Bot CRD provides attributes for the various options that are required to define the bot management policies on NetScaler.
Bot CRD attributes
The following table lists the various attributes provided in the Bot CRD:
| CRD attribute | Description |
|---|---|
security_checks |
List of security checks to be applied for incoming traffic. |
allow_list |
List of allowed IP, subnet, and policy expressions. |
block_list |
List of disallowed IP, subnet, and policy expressions. |
device_fingerprint |
Inserts javascript and collects the client browser and device parameters. |
trap |
Inserts hidden URLs in the response. |
tps |
Prevents bots which cause unusual spike in requests based on configured percentage change in transactions. |
reputation |
Prevents access to bad IPs based on configured reputation categories. |
ratelimit |
Prevents bots based on rate limit. |
redirect_url |
Redirect URL when block is enabled on protection. |
ingressclass |
Specifies the ingress class so that only the ingress controller associated with the specified ingress class processes the resource. Otherwise, all the controllers in the cluster will process this resource. |
servicenames |
Name of the services to which the bot policies are applied. |
signatures |
Location of external bot signature file. |
target |
Determines which traffic to be inspected by the bot. If you do not specify the traffic targeted, every traffic is inspected by default. |
paths |
List of HTTP URLs to be inspected. |
method |
List of HTTP methods to be inspected. |
header |
List of HTTP headers to be inspected. |
Deploy the Bot CRD
Perform the following steps to deploy the Bot CRD:
- Download the bot-crd.yaml.
- Deploy the Bot CRD using the following command:
kubectl create -f bot-crd.yaml
For example,
root@master:~# kubectl create -f bot-crd.yaml
customresourcedefinition.apiextensions.k8s.io/bots.citrix.com created
<!--NeedCopy-->
How to write a Bot configuration
After you have deployed the Bot CRD provided by NetScaler in the Kubernetes cluster, you can define the bot management policy configuration in a YAML file. In the YAML file, specify bot in the kind field. In the spec section, add the Bot CRD attributes based on your requirements for the policy configuration.
After you deploy the YAML file, the NetScaler Ingress Controller applies the bot configuration on the Ingress NetScaler device.
Following are some examples for bot policy configurations:
Block malicious traffic using known IP, subnet, or ADC policy expressions
When you want to define and employ a web bot management policy in NetScaler to enable bot for blocking malicious traffic, you can create a YAML file called botblocklist.yaml and use the appropriate CRD attributes to define the bot policy as follows:
apiVersion: citrix.com/v1
kind: bot
metadata:
name: botblocklist
spec:
servicenames:
- frontend
security_checks:
block_list: "ON"
bindings:
block_list:
- subnet:
value:
- 172.16.1.0/12
- 172.16.2.0/12
- 172.16.3.0/12
- 172.16.4.0/12
action:
- "drop"
- ip:
value: 10.102.30.40
- expression:
value: http.req.url.contains("/robots.txt")
action:
- "reset"
- "log"
<!--NeedCopy-->
Allow known traffic without bot security checks**
When you want to avoid security checks for certain traffic such as staging or trusted traffic, you can avoid such traffic from security checks. You can create a YAML file called botallowlist.yaml and use the appropriate CRD attributes to define the bot policy as follows:
apiVersion: citrix.com/v1
kind: bot
metadata:
name: botallowlist
spec:
servicenames:
- frontend
security_checks:
allow_list: "ON"
bindings:
allow_list:
- subnet:
value:
- 172.16.1.0/12
- 172.16.2.0/12
- 172.16.3.0/12
- 172.16.4.0/12
action:
- "log"
- ip:
value: 10.102.30.40
- expression:
value: http.req.url.contains("index.html")
action:
- "log"
<!--NeedCopy-->
Enable bot signatures to detect bots
NetScaler provides thousands of inbuilt signatures to detect bots based on user agents. Citrix threat intelligence team keeps on updating and releasing new bot signatures in every two weeks. The latest bot signature file is available at: Bot signatures. You can create a YAML file called botsignatures.yaml and use the appropriate CRD attributes to define the bot policy as follows:
apiVersion: citrix.com/v1
kind: bot
metadata:
name: botsignatures
spec:
servicenames:
- frontend
redirect_url: "/error_page.html"
signatures: "http://10.106.102.242/ganeshka/bot_sig.json"
<!--NeedCopy-->
Enable the bot device fingerprint and customize the action
Device fingerprinting involves inserting a JavaScript snippet in the HTML response to the client. This JavaScript snippet, when invoked by the browser on the client, collects the attributes of the browser and client. And sends a POST request to NetScaler with that information. These attributes are examined to determine whether the connection is requested from a bot or a human being. You can create a YAML file called botdfp.yaml and use the appropriate CRD attributes to define the bot policy as follows:
apiVersion: citrix.com/v1
kind: bot
metadata:
name: botdfp
spec:
servicenames:
- frontend
redirect_url: "/error_page.html"
security_checks:
device_fingerprint:
action:
- "log"
- "drop"
<!--NeedCopy-->
Enable the bot TPS and customize the action
If the bot TPS is configured, it detects incoming traffic as bots if the maximum number of requests or increase in requests exceeds the configured time interval. You can configure the TPS limits as per geolocation, host, source IP, and URL in the bindings section. You can create a YAML file called bottps.yaml and use the appropriate CRD attributes to define the bot policy as follows:
apiVersion: citrix.com/v1
kind: bot
metadata:
name: bottps
spec:
servicenames:
- frontend
redirect_url: "/error_page.html"
security_checks:
tps: "ON"
bindings:
tps:
geolocation:
threshold: 101
percentage: 100
host:
threshold: 10
percentage: 100
action:
- "log"
- "mitigation"
<!--NeedCopy-->
Enable the trap insertion protection and customize the action
Detects and blocks automated bots by advertising a trap URL in the client response. The URL is invisible and not accessible to the client, if it is human. The detection method is effective in blocking attacks from automated bots. Insertion of the trap URL in the URL responses is random. You can enforce the trap URL insertion to a particular URL response by configuring the trap bindings. You can create a YAML file called trapinsertion.yaml and use the appropriate CRD attributes to define the bot policy as follows:
apiVersion: citrix.com/v1
kind: bot
metadata:
name: trapinsertion
spec:
servicenames:
- frontend
redirect_url: "/error_page.html"
security_checks:
trap:
action:
- "log"
- "drop"
bindings:
trapinsertion:
urls:
- "/index.html"
- "/submit.php"
- "/login.html"
<!--NeedCopy-->
Enable IP reputation to reject requests of a particular category
The following is an example of a Bot CRD configuration for enabling only specific threat categories of IP reputation that are suitable for the user environment. You can create a YAML file called botiprepcategory.yaml and use the appropriate CRD attributes to define the bot policy as follows:
apiVersion: citrix.com/v1
kind: bot
metadata:
name: botiprepcategory
spec:
servicenames:
- frontend
redirect_url: "/error_page.html"
security_checks:
reputation: "ON"
bindings:
reputation:
categories:
- SPAM_SOURCES:
action:
- "log"
- "redirect"
- MOBILE_THREATS
- SPAM_SOURCES
<!--NeedCopy-->
Enable rate limit to control request rate
The following is an example of a Bot CRD configuration for enforcing the request rate limit using the parameters: URL, cookies, and IP. You can create a YAML file called botratelimit.yaml and use the appropriate CRD attributes to define the bot policy as follows:
apiVersion: citrix.com/v1
kind: bot
metadata:
name: botratelimit
spec:
servicenames:
- frontend
redirect_url: "/error_page.html"
security_checks:
ratelimit: "ON"
bindings:
ratelimit:
- url:
value: index.html
rate: 2000
timeslice: 1000
- cookie:
value: citrix_bot_id
rate: 2000
timeslice: 1000
- ip:
rate: 2000
timeslice: 1000
action:
- "log"
- "reset"
<!--NeedCopy-->