URL list

The URL List feature enables enterprise customers to control access to specific websites and website categories. The feature filters websites by applying a responder policy bound to a URL matching algorithm. The algorithm matches the incoming URL against a URL set consisting of up to one million (1,000,000) entries. If the incoming URL request matches an entry in the set, the appliance uses the responder policy to evaluate the request (HTTP/HTTPS) and control access to it.

URL set types

Each entry in a URL set can include a URL and, optionally, its metadata (URL category, category groups, or any other related data). For URLs with metadata, the appliance uses a policy expression that evaluates the metadata. For more information, see URL Set.

Citrix SWG supports custom URL sets. You can also use pattern sets to filter URLs.

Custom URL set. You can create a customized URL set with up to 1,000,000 URL entries and import it as a text file into your appliance.

Pattern set. An SWG appliance can use pattern sets to filter URLs before granting access to websites. A pattern set is a string-matching algorithm that looks for an exact string match between an incoming URL and up to 5000 entries. For more information, see Pattern Set.

Each URL in an imported URL set can have a custom category in the form of URL metadata. Your organization can host the set and configure the SWG appliance to periodically update the set without requiring manual intervention.

After the set is updated, the Citrix ADC appliance automatically detects the metadata, and the category is available as a policy expression for evaluating the URL and applying an action such as allow, block, redirect, or notify the user.

Advanced policy expressions used with URL sets

The following table describes the basic expressions you can use to evaluate incoming traffic.

1. .URLSET_MATCHES_ANY - Evaluates to TRUE if the URL exactly matches any entry in the URL set.
2. .GET_URLSET_METADATA() - The GET_URLSET_METADATA() expression returns the associated metadata if the URL exactly matches any pattern within the URL set. An empty string is returned if there is no match.
3. .GET_ URLSET_METADATA().EQ(<METADATA) - .GET_ URLSET_METADATA().EQ(<METADATA)
4. .GET_URLSET_METADATA ().TYPECAST_LIST_T(',').GET(0).EQ() - Evaluates to TRUE if the matched metadata is at the beginning of the category. This pattern can be used to encode separate fields within metadata but match only the first field.
5. HTTP.REQ.HOSTNAME.APPEND(HTTP.REQ.URL) - Joins the host and URL parameters, which can then be used as a for matching.

Responder action types

Note: In the table, HTTP.REQ.URL is generalized as <URL expression>.

The following table describes the actions that can be applied to incoming internet traffic.

Prerequisites

You must configure a DNS server if you import a URL Set from a hostname URL. This is not required if you use an IP address.

At the command prompt, type:

add dns nameServer ((<IP> [-local]) | <dnsVserverName>) [-state (ENABLED | DISABLED )] [-type <type>] [-dnsProfileName <string>]

Example:

add dns nameServer 10.140.50.5

Configure a URL list

To configure a URL list, you can use the Citrix SWG wizard or the Citrix ADC command-line interface (CLI). On the Citrix SWG appliance, you must first configure the responder policy and then bind the policy to a URL set.

Citrix recommends that you use the Citrix SWG Wizard as the preferred option to configure a URL list. Use the wizard to bind a responder policy to a URL set. Alternatively, you can bind the policy to a pattern set.

Configure a URL list by using the Citrix SWG wizard

To configure URL List for HTTPS traffic by using the Citrix SWG GUI:

1. Log on to the Citrix SWG appliance and navigate to Secured Web Gateway page.
2. In the details pane, do one of the following:
   1. Click Secured Web Gateway Wizard to create a new SWG configuration with URL List feature.
   2. Select an existing configuration and click Edit.
3. In the URL Filtering section, click Edit.
4. Select the URL List check box to enable the feature.
5. Select a URL List policy and Click Bind.
6. Click Continue and then Done.

For more information, see How to Create a URL List Policy.

Configure a URL list by using the Citrix SWG CLI

To configure a URL list, do the following.

1. Configure a proxy virtual server for HTTP and HTTPS traffic.
2. Configure SSL interception for intercepting HTTPS traffic.
3. Configure a URL list containing a URL set for HTTP traffic.
4. Configure URL list containing URL set for HTTPS traffic.
5. Configure a private URL set.

Note

If you have already configured an SWG appliance, you can skip steps 1 and 2, and configure with step 3.

Configuring a proxy virtual server for Internet traffic

The Citrix SWG appliance supports transparent and explicit proxy virtual servers. To configure a proxy virtual server for internet traffic in explicit mode, do the following:

1. Add a proxy SSL virtual server.
2. Bind a responder policy to the proxy virtual server.

To add a proxy virtual server by using the Citrix SWG CLI:

At the command prompt, type:

add cs vserver <name> <serviceType> <IPAddress> <port>
<!--NeedCopy-->

Example:

add cs vserver starcs PROXY 10.102.107.121 80 -cltTimeout 180
<!--NeedCopy-->

To bind a responder policy to a proxy virtual server by using the Citrix SWG CLI:

bind ssl vserver <vServerName> -policyName <string> [-priority <positive_integer>]
<!--NeedCopy-->

Note

If you have already configured the SSL interceptor as part of Citrix SWG configuration, you can skip the following procedure.

Configure SSL interception for HTTPS traffic

To configure SSL interception for HTTPS traffic, do the following:

1. Bind a CA certificate-key pair to the proxy virtual server.
2. Enable the default SSL profile.
3. Create a front-end SSL profile, and bind it to the proxy virtual server and enable SSL interception in the front-end SSL profile.

To bind a CA certificate-key pair to the proxy virtual server by using the Citrix SWG CLI:

At the command prompt, type:

bind ssl vserver <vServerName> -certkeyName <certificate-KeyPairName>
<!--NeedCopy-->

To configure a front-end SSL profile by using the Citrix SWG CLI:

At the command prompt, type:

set ssl parameter -defaultProfile ENABLED

add ssl profile <name> -sslInterception ENABLED -ssliMaxSessPerServer <positive_integer>
<!--NeedCopy-->

To bind a front-end SSL profile to a proxy virtual server by using the Citrix SWG CLI

At the command prompt, type:

set ssl vserver <vServer name>  -sslProfile <name>
<!--NeedCopy-->

Configure a URL list by importing a URL set for HTTP traffic

For information about how to configure a URL Set for HTTP traffic, see URL Set.

Perform explicit subdomain match

You can now perform an explicit subdomain match for an imported URL set. To do this, a new parameter, “subdomainExactMatch” is added to the import policy URLset command.

When you enable the parameter, the URL Filtering algorithm performs an explicit subdomain match. For example, if the incoming URL is news.example.com and if the entry in the URL set is example.com, the algorithm does not match the URLs.

At the command prompt, type: import policy urlset <name> [-overwrite] [-delimiter <character>][-rowSeparator <character>] -url [-interval <secs>] [-privateSet][-subdomainExactMatch] [-canaryUrl <URL>]

Example import policy urlset test -url http://10.78.79.80/top-1k.csv -privateSet -subdomainExactMatch -interval 900

Configure a URL set for HTTPS traffic

To configure a URL Set for HTTPS traffic by using the Citrix SWG CLI

At the command prompt type:

add ssl policy <name> -rule <expression> -action <string> [-undefAction <string>] [-comment <string>]
<!--NeedCopy-->

Example:

add ssl policy pol1 -rule "client.ssl.client_hello.SNI.URLSET_MATCHES_ANY("top1m") -action INTERCEPT
<!--NeedCopy-->

To configure a URL set for HTTPS traffic by using the Citrix SWG wizard

Citrix recommends that you use the Citrix SWG wizard as the preferred option to configure a URL list. Use the wizard to import a custom URL set and bind to a responder policy.

1. Log on to the Citrix SWG appliance and navigate to Secured Web Gateway > URL Filtering > URL Lists.
2. In the details pane, click Add.
3. On the URL List Policy page, specify the policy name.
4. Select an option to import a URL set.
5. On the URL List Policy tab page, select the Import URL Set check box and specify the following URL Set parameters.
   1. URL Set Name—Name of the custom URL set.
   2. URL—Web address of the location at which to access the URL Set.
   3. Overwrite—Overwrite a previously imported URL set.
   4. Delimiter—Character sequence that delimits a CSV file record.
   5. Row Separator—Row separator used in the CSV file.
   6. Interval—Interval in seconds, rounded off to the nearest number of seconds equal to 15 minutes, at which the URL set is updated.
   7. Private Set—Option to prevent exporting the URL set.
   8. Canary URL—Internal URL for testing whether the content of the URL set is to be kept confidential. The maximum length of the URL is 2047 characters.
6. Select a responder action from the drop-down list.
7. Click Create and Close.

Configure a private URL set

If you configure a private URL set and keep its contents confidential, the network administrator might not know the blacklisted URLs in the set. For such cases, you can configure a Canary URL and add it to the URL set. Using the Canary URL, the administrator can request the private URL Set to be used for every lookup request. You can refer to the wizard section for descriptions of each parameter.

To import a URL set by using the Citrix SWG CLI:

At the command prompt, type:

import policy urlset <name> [-overwrite] [-delimiter <character>] [-rowSeparator <character>] -url <URL> [-interval <secs>] [-privateSet] [-canaryUrl <URL>]
<!--NeedCopy-->

Example:

import policy urlset test1 –url http://10.78.79.80/alytra/top-1k.csv -private -canaryUrl http://www.in.gr
<!--NeedCopy-->

Display imported URL set

You can now display imported URL sets in addition to added URL sets. To do this, a new parameter “imported” is added to the “show urlset” command. If you enable this option, the appliance displays all imported URL sets and distinguishes the imported URL sets from the added URL sets.

At the command prompt, type: show policy urlset [<name>] [-imported]

Example show policy urlset -imported

Configure audit log messaging

Audit logging enables you to review a condition or a situation in any phase of URL List process. When a Citrix ADC appliance receives an incoming URL, if the responder policy has an URL Set advanced policy expression, the audit log feature collects URL Set information in the URL and stores the details as a log message for any target allowed by audit logging.

1. The log message contains the following information:
2. Timestamp.
3. Log message type.
4. The predefined log levels (Critical, Error, Notice, Warning, Informational, Debug, Alert, and Emergency).
5. Log message information, such as URLset name, policy action, URL.

To configure audit logging for URL List feature, you must complete the following tasks:

1. Enable Audit Log.
2. Create Audit Log message action.
3. Set URL List responder policy with Audit Log message action.

For more information, see Audit Logging topic.