URL List
The URL List feature enables you to control access to customized URL lists (up to one million entries). The feature filters websites by applying a URL filtering policy bound to a virtual server.
As an administrator, you must import the URL List into the Citrix ADC appliance. This imported list is internally stored as a Policy data set called a URL Set. The appliance then applies a unique fast URL matching algorithm to the incoming URL requests. If the incoming URL request matches an entry in the set, the appliance applies the associated policy action to control access.
URL List Types
Each entry in a URL set can include a URL and, optionally, its metadata (URL category, category groups, or any other related data). For URLs with a metadata, the appliance uses a policy expression that evaluates the metadata. For more information, see URL Sets.
A Citrix ADC appliance supports custom URL list.
Custom URL List. You can create a customized URL set of up to 1,000,000 URL entries and import it as a text file into your appliance. The list can contain URLs with or without metadata (which could be like a URL category). TheCitrix ADC platform automatically detects whether metadata is present. It also supports storing the imported lists securely. For more information, see URL Set.
You can host the URL list and configure the Citrix ADC appliance to periodically update the list without requiring manual intervention. Once the URL list is updated, the appliance can automatically detect the metadata and the categories by using policy expressions to evaluate each incoming URL and then apply actions such as allow, block, redirect, or notify the user.
URL List Policy Expressions
The following table describes the basic expressions you can use to evaluate incoming traffic. After you import an URL List to the appliance, it is called a URL Set.
Expression | Operation |
---|---|
<URL expression>.URLSET_MATCHES_ANY(<URLSET>) |
Evaluates to TRUE if the URL exactly matches any entry in the URL set. |
<URL expression>.GET_URLSET_METADATA(<URLSET> ) |
The GET_URLSET_METADATA() expression returns the associated metadata if the URL exactly matches any pattern within the URL set. An empty string is returned if there is no match. |
<URL expression>.GET_ URLSET_METADATA(<URLSET>).EQ(<METADATA>) |
Evaluates to TRUE if the matched metadata is equal to <METADATA> . |
<URLexpression>.GET_URLSET_METADATA(<URLSET>).TYPECAST_LIST_T(‘,’).GET(0).EQ(<CATEGORY>) |
Evaluates to TRUE if the matched metadata is at the beginning of the category. This pattern can be used to encode separate fields within metadata, but match only the 1<sup>st</sup> field. |
HTTP.REQ.HOSTNAME.APPEND(HTTP.REQ.URL) | Joins the host and URL parameters, which can then be used as a <URL expression> for matching. |
URL List Policy Actions
The most common enforcement action for URLs that match a URL list is to restrict access. Create a URL list policy with a desired URL list matching expression and enforcement action. The policy group usage depends upon the incoming traffic type (HTTP or HTTPS) and the virtual server configured on the appliance. You can use a Responder policy for HTTP traffic or a Video Optimization policy for HTTPS traffic. Specify actions to apply to the URLs that match the expressions in the policies. The following table lists the available actions.
Action Type | Policy | Description |
---|---|---|
ALLOW | Responder | Allow the request to access the target URL. |
REDIRECT | Responder | Redirect the request to the URL specified as the target. |
DENY | Responder | Deny the request. |
RESET | Responder, VideoOptimization | Reset the connection. |
DROP | Responder, VideoOptimization | Drop the connection. |
Prerequisites
To configure URL List feature, make sure you have configured the following server.
DNS Server for DNS Requests
You must configure a DNS server if you import a URL Set from a hostname URL.
At the command prompt, type:
add dns nameServer ((<IP> [-local]) | <dnsVserverName>) [-state (ENABLED | DISABLED )] [-type <type>] [-dnsProfileName <string>]
<!--NeedCopy-->
Example:
add dns nameServer 10.140.50.5
<!--NeedCopy-->
Configuring a URL List
To configure a URL List, do the following:
- Import a custom URL list
- Configure a URL list for HTTP traffic.
- Add a URL List action.
- Add a URL List policy.
- Add an HTTP load balancing virtual server for HTTP traffic.
- Bind the URL List policy to the HTTP load Balancing virtual server for HTTP traffic.
- Configure a URL list for HTTPS traffic.
- Add a video optimization detection policy.
- Add an SSL-bridge load balancing virtual server for HTTPS traffic.
- Bind the video optimization detection policy to the SSL-bridge load balancing virtual server for HTTPS traffic
Importing a custom URL list
To import a URL set, see See URL Set topic.
Configuring a URL List for HTTP traffic
The Citrix ADC appliance supports HTTP and HTTPS traffic. To configure a load balancing virtual server for HTTP traffic and bind URL list policies to the server, do the following:
- Add URL List actions.
- Add URL List policies.
- Add an HTTP load balancing virtual server for HTTP traffic
- Bind the URL List policies to the HTTP load balancing virtual server for HTTP traffic
To add a URL list action
At the command prompt, type the following:
add responder action <name> <type> (<target> | <htmlpage>) [-comment <string>] [-responseStatusCode <positive_integer>] [-reasonPhrase <string>]
<!--NeedCopy-->
Example:
add responder action act_url respondwith '"HTTP/1.1 451 Unavailable For Legal ReasonsrnrnURL is NOT authorizedn"'
<!--NeedCopy-->
To add a URL list policy
At the command prompt, type the following:
add responder policy <name> <rule> <action> [<undefAction>] [-comment <string>] [-logAction <string>] [-appflowAction <string>]
<!--NeedCopy-->
Example:
add responder policy pol_http 'HTTP.REQ.HOSTNAME.APPEND(HTTP.REQ.URL).URLSET_MATCHES_ANY("set")' act_url
<!--NeedCopy-->
To add a HTTP load balancing virtual server for HTTP traffic
At the command prompt, type the following:
add lb vserver <name> [-td <positive_integer>] <serviceType> [-cltT imeout <secs>]
<!--NeedCopy-->
Example:
add lb vserver vsrv-HTTP HTTP * 80 -persistenceType NONE -cltTimeout 120
<!--NeedCopy-->
To bind URL list policy to HTTP load balancing virtual server
At the command prompt, type the following:
bind lb vserver <vServerName> -policyName <string> [-priority <positive_integer>]
<!--NeedCopy-->
Example:
bind lb vserver vsrv-HTTP -policyName pol_http -priority 10 -type REQUEST
<!--NeedCopy-->
Configuring URL List for HTTPS traffic
The Citrix ADC appliance supports HTTP and HTTPS traffic. To configure a SSL-bridge load balancing virtual server for HTTPS traffic and bind URL list policies to the server, do the following:
- Add URL List actions.
- Add URL List policies.
- Add a SSL-bridge load balancing virtual server for HTTP traffic
- Bind the URL List policies to the SSL-bridge load balancing virtual server for HTTP traffic
To add a URL List policy for HTTPS traffic
At the command prompt, type:
add videooptimization detectionpolicy <name> -rule <expression> -action <string> [-undefAction <string>] [-comment <string>] [-logAction <string>]
<!--NeedCopy-->
Example:
add videooptimization detectionpolicy pol_https -rule CLIENT.SSL.DETECTED_DOMAIN.URLSET_MATCHES_ANY("set") -action RESET
<!--NeedCopy-->
To add a SSL-bridge load balancing virtual server
At the command prompt type:
add lb vserver <name> [-td <positive_integer>] <serviceType> [-cltT imeout <secs>]
<!--NeedCopy-->
Example:
add lb vserver vsrv-HTTPS SSL_BRIDGE * 443 -persistenceType NONE -cltTimeout 180
<!--NeedCopy-->
To bind URL List policy with SSL-bridge load balancing by using the CLI
At the command prompt type:
bind lb vserver <vServerName> -policyName <string> [-priority <positive_integer>]
<!--NeedCopy-->
Example:
bind lb vserver vsrv-HTTPS -policyName pol_https -priority 20 -type REQUEST
<!--NeedCopy-->
Configuring a URL List by using the GUI
The GUI enables you to:
- Import a URL list.
- Add a URL list.
- Configure URL list actions.
- Configure URL list policies for HTTP traffic.
- Add an HTTP load balancing virtual server for HTTP traffic.
- Add an SSL-bridge load balancing virtual server for HTTPS traffic.
- Bind URL list policies to the HTTP load balancing virtual server.
- Bind a URL list policies to the SSL-bridge load balancing virtual server.
To import a URL list
- In the navigation pane, expand AppExpert > URL Sets.
- In the details pane, click Import.
- On the Configure URL Set page, set the following parameters.
- Name. Name of the URL set.
- URL. Web address of the location at which to access the URL Set.
- Overwrite. Overwrite a previously imported URL set.
- Delimiter. Character sequence that delimits a CSV file record.
- Row Separator. Row separator used in the CSV file. A single character value is permitted for example “/n”.
- Interval. Interval in seconds, rounded off to the nearest 15 minutes, at which the URL set is updated.
- Private Set. Option to prevent exporting the URL set
- Canary URL. Internal URL for testing whether the content of the URL set is to be kept confidential. The maximum length of the URL is 2047 characters
- Click Create, and then Close.
To add a URL list
- In the navigation pane, expand AppExpert > URL Sets.
- In the details pane, click Add.
- On the Create URL Set page, set the following parameters.
- Name. The name of the URL set that was given when it was imported.
- Comments. A short description about the URL set.
- Click Create.
To configure a URL list action
- Log on to the Citrix ADC appliance and navigate to Configuration tab page.
- In the menu pane, navigate to AppExpert > Responder > Actions.
- In the details pane, click Add.
- On the Create Responder Action page, set the following parameters.
- Name. Name of the URL List policy action.
- Type. Select an action type.
- Expression. Use the expression editor to create the policy expression.
- Comments. A short description about the policy action.
- Click Create and Close.
To configure a URL list policy
- In the navigation pane, expand AppExpert > Responder > Policies.
- In the details pane, click Add.
- On the Create Responder Policy page, set the following parameters.
- Name. Name of the URL List policy action.
- Action. Select the URL List action that you prefer to associate with the policy.
- Log Action. Select the log action.
- AppFlow. Select an AppFlow action.
- Expression. Use the expression editor to create the policy expression.
- Comments. A short description about the policy.
- Click Create and Close.
To add an HTTP load balancing virtual server
- Navigate to the Traffic Management > Load Balancing > Virtual Servers page.
- In the details pane, click Add.
- On the Load Balancing Virtual Server screen, set the following parameters:
- Name. Name of the load balancing virtual server.
- Protocol. Choose protocol type as HTTP.
- IP Address Type. IP addressable type.
- IP Address. IP 4 or IP6 IP address assigned to the virtual server.
- Port. Port number of the virtual server.
- Click OK to continue with the configuration of other, optional, parameters. For more information, see Creating a Virtual Server.
To bind a URL List policy to the HTTP load balancing virtual server
- Navigate to Traffic Management > Load Balancing > Virtual Servers screen.
- In the details pane, select the load balancing virtual server and click Edit.
- In the Advanced Setting section, click Policies.
- In the Policies section, click the + icon to access the Policies slider.
- In the Policies section, set the following parameters.
- Choose Policy. Select a URL categorization policy from the drop-down list.
- Choose Type. Select the policy type as Request.
- Click Continue.
- In the Policies page, select the URL List policy from the list and click Select.
- In the Policies slider, click Bind and Close.
To add URL List policy for HTTPS traffic
- Log on to the Citrix ADC appliance and navigate to Configuration > Optimization > Video Optimization > Detection.
- On the Detection page, click the Video Optimization Detection Policies link.
- On the Video Optimization Detection Policies page, click Add.
- On the Create Video Optimization Detection Policy page, set the following parameters.
- Name. Name of the optimization policy
- Expression. Configure policy using custom expressions.
- Action. Optimization action associated with the policy to handle the incoming video traffic.
- UNDEF Action. Undefined event if the incoming request does not match the optimization policy.
- Comment. A short description of the policy.
- Log Action. Select an audit log action that specifies the action to be performed for the log messages.
- Click Create and Close.
To add a SSL-bridge load balancing virtual server for HTTPS traffic
- Navigate to the Traffic Management > Load Balancing > Virtual Servers page.
- In the details pane, click Add.
- On the Load Balancing Virtual Server screen, set the following parameters:
- Name. Name of the load balancing virtual server.
- Protocol. Select protocol type as SSL-bridge.
- IP Address Type. IP address type: IPv4 or IPv6.
- IP Address. IPv4 or IP6vIP address assigned to the virtual server.
- Port. Port number of the virtual server.
- Click OK to continue with the configuration of other, optional, parameters. For more information, see “Creating a Virtual Server” topic.
To bind a URL List Policy to the SSL-bridge load balancing virtual server
- Navigate to the Traffic Management > Load Balancing > Virtual Servers screen.
- In the details pane, select the SSL-bridge load balancing virtual server and click Edit.
- In the Advanced Setting section, click Policies.
- In the Policies section, click the + icon to access the Policies slider.
- Set the following parameters.
- Choose Policy. Select video detection policy from the drop-down list.
- Choose Type. Select the policy type as Request.
- Click Continue.
- Select the video detection policy from the list and click Close.
Configuring Audit Log Messaging
Audit logging enables you to review a condition or a situation in any phase of URL List process. When a Citrix ADC appliance receives an incoming URL, if the responder policy has an URL Set advanced policy expression, the audit log feature collects URL Set information in the URL and stores the details as a log message for any target allowed by audit logging.
The log message contains the following information:
- Timestamp.
- Log message type.
- The predefined log levels (Critical, Error, Notice, Warning, Informational, Debug, Alert, and Emergency).
- Log message information, such as URL set name, policy action, URL.
To configure audit logging for URL List feature, you must complete the following tasks:
- Enable Audit Log.
- Create Audit Log message action.
- Set URL List responder policy with Audit Log message action.
For more information, see Audit Logging.
URL List Semantics
The following table lists the URL Match patterns and describes how the URLs within a URL list are matched against the incoming-request URLs. For example, the pattern www.example.com/bar
matches only with one page at www.example.com/bar. To match all the pages whose URL starts with www.example.com/bar
, you would add an asterisk (*) to the end of the URL.
Semantics | URL Pattern | Matched | Unmatched |
---|---|---|---|
Subdomain matching | domain.com | domain.com; www.domain.com ; sub.one.domain.com |
yourdomain.com; wwwdomain.com |
URL matching, exact path | domain.com/example/bar/index.html | domain.com/example/bar/index.html; www.domain.com/example/bar/index.html; s.domain.com/example/bar/index.html | wwwdomaincom/example/bar/index.html; domain.com/example/bar/index.html/one.jpg |
URL matching, exact path | domain.com/example/bar/index.html | domain.com/example/bar/index.html?key=value; www.domain.com/example/bar/index.html?; s.domain.com/example/bar/index.html | wwwdomaincom/example/bar/index.html; domain.com/example/bar/index.html/one.jpg |
URL matching, subpath matching | domain.com/example/bar/ | domain.com/example/bar/; domain.com/example/bar/index.html; www.domain.com/example/bar/index.html ; domain.com/example/bar/index.html/one.jpg |
wwwdomaincom/example/bar/index.html |
In this article
- URL List Types
- URL List Policy Expressions
- URL List Policy Actions
- Prerequisites
- DNS Server for DNS Requests
- Configuring a URL List
- Importing a custom URL list
- Configuring a URL List for HTTP traffic
- Configuring URL List for HTTPS traffic
- Configuring a URL List by using the GUI
- Configuring Audit Log Messaging
- URL List Semantics