Product Documentation
ADC
14.1 - Current Release 13.1 13.0 12.1
View PDF

Configuring URL Set

November 7, 2023
Contributed by: 
C S

You can perform the following tasks to configure a URL set and restrict URLs on a NetScaler platform:

  1. Import a URL set (download and encrypt it). Importing a URL set in a NetScaler appliance allows you:
  • To download the URL file.
  • To add the file to the appliance.
  • To encrypt the file. Until you add the URL set to the system, it is not visible to the user.

You can download a set in the following ways:

  • Download a URL set once from a remote server and specify it as http://myserver.com/file_with_urlset.csv

  • add a file under the /var/tmp/ path inside ADC and use the command, as in the example:

> shell cat /var/tmp/test_urlset.csv
example.com
google.com
> import policy urlset top10
k -url local:test_urlset.csv -delimiter "," -rowSeparator "n" -interval 10 -privateSet -canaryUrl http://www.in.gr
Done

<!--NeedCopy-->

The imported URL set is further categorized into different categories and category groups in the database. This is valid only if categories exist in the metadata of the URL set file.

Note: There can be a chance that you might have URL patterns without metadata.

Once you have imported the file, you can update, delete, or display file properties. After the file is pushed into the appliance, you can modify the entries by more adding rows.

The imported set is then stored in an encrypted file format on the NetScaler directory. The imported list contains millions of URL entries. To the following ‘The imported list can contain up to 1 million URL entries. Otherwise, the appliance returns an error message saying that the value exceeds the limit. If the imported URL set has blacklisted entries with metadata, the metadata it is detected by the appliance when it is imported.

Once you import a URL set and add it into the appliance, the URL set is available for advanced policies to identify the correct URL set during incoming URL evaluation. HTTP.REQ.HOSTNAME.APPEND(HTTP.REQ.URL).URLSET_MATCHES_ANY(<URL set name>)  

  1. Updating a URL set on the NetScaler appliance. Once you have pushed the file into the appliance, at this interval you can manually update a URL file by using command line interface.

  2. Exporting a URL set. If you prefer a backup of the URL set, you can export the list of URL patterns and save a copy of it to a destination URL. Before you export, check whether the URL set is marked as private. If is marked private, the URL set cannot be exported. Export functionality does not work with private set. So a new url set myurl would be imported without private set defined, and then it would be exported to another file in a local path, as below:

> shell touch /var/tmp/test_urlset_export.csv
Done
> shell cat /var/tmp/test_urlset_export.csv
Done
> shell cat /var/tmp/test_urlset.csv
example.com
google.com
Done
> export urlset myurl -url local:test_urlset_export.csv

> import urlset myurl -url local:test_urlset.csv
Done
(a non-private urlset is imported)

<!--NeedCopy-->

  1. Removing a URL set. If you want to delete a URL set of blacklisted entries, you can use the remove command to delete the URL set from the NetScaler appliance.

  2. Displaying a URL set. You can display the properties of a URL set by using the show command.

Note: URLs with query part are removed during import.

Example:

show urlset
Name: top100 PatternCount: 100 Delimiter:  RowSeparator: Interval: 0
Done
<!--NeedCopy-->

Import a URL set with meta by using the command line interface

At command prompt, type:

import urlset <name> [-overwrite] [–delimiter <character>] [-rowSeparator <character>] [-url] <url> [-interval <seconds>] [-privateSet] [-canaryUrl <URL>]
<!--NeedCopy-->

Where,

Delimiter is a CSV file record with default value set as 44.

rowSeparator is a CSV file row separator with default value set as 10.

Interval is the time interval in secs, rounded to the nearest 15 minutes at which the update of the url set occurs.

CanaryURL is a URL used for testing when the contents of the url set is kept confidential.

Example

import policy urlset -url local:test_urlset.csv -delimiter "," -rowSeparator "n" -interval 10 -privateSet -canaryUrl http://www.in.gr

Perform explicit subdomain match for an imported URL set

You can now perform an explicit subdomain match for an imported URL set. A new parameter, “subdomainExactMatch” is added to the “import policy URLset” command. When you enable the parameter, the URL Filtering algorithm performs an explicit subdomain match. For example, if the incoming URL is “news.example.com” and if the entry in the URL set is “example.com,” the algorithm does not match the URLs.

At the command prompt, type: import policy urlset <name> [-overwrite] [-delimiter <character>][-rowSeparator <character>] -url <URL> [-interval <secs>] [-privateSet][-subdomainExactMatch] [-canaryUrl <URL>]

Example:

import policy urlset forth_urlset -url local:test_urlset.csv -interval 3600 -subdomainExactMatch

To show the URL set by using the command line interface

At the command prompt, type:

show urlset <name>

Example:

At the command prompt, type:

        URLset     Count
        ------     -----
1)      top1k      100
 Done

> show urlset top1k
        Count      Delimiter  Interval   RowSeparator
        -----      ---------  --------   ------------
        100             ,        0         0x0a
 Done
>

<!--NeedCopy-->

To show the URL set imported by using the command line interface

At the command prompt, type:

show urlset -imported

Example:

At the command prompt, type:

        URLset
        ------
1)      top1k
 Done
<!--NeedCopy-->

To show URL set by using the command line interface

At the command prompt, type:

show urlset <name>

To export a URL set by using the command line interface

At the command prompt, type:

export urlset <name> <url>

To add a URL set by using the command line interface

At the command prompt, type:

add urlset <urlset_name>

To update a URL set by using the command line interface

At the command prompt, type:

update urlset <name>

To remove a URL set command by using the command line interface

At the command prompt, type:

remove urlset <name>

Example:

Note:

Before you import or export a URLset, you must make sure the test_urlset_export.csv and test_urlset.csv files are created and available under the /var/tmp directory.

import policy urlset -url local:test_urlset.csv -delimiter "," -rowSeparator "n" -interval 10 -privateSet -overwrite –canaryUrl http://www.in.gr

add policy urlset top10k

update policy urlset top10k

sh policy urlset

sh policy urlset top10k

export policy urlset urlset1 -url local:test_urlset_export.csv

import policy urlset top10k -url local:test_urlset.csv –privateSet

add policy urlset top10k

update policy urlset top10k

show policy urlset top10k
<!--NeedCopy-->

Display imported URL sets

You can now display imported URL sets in addition to added URL sets. To do this, a new parameter “imported” is added to the “show url set” command. If you enable this option, the appliance displays all imported URL sets and distinguishes the imported URL sets from the added URL sets.

At the command prompt, type:

show policy urlset [<name>] [-imported]

Example:

show policy urlset -imported

To import a URL set by using the GUI

Navigate to AppExpert > URL Sets, click Import to download the URL set.

To add a URL set by using the GUI

Navigate to AppExpert > URL Sets, click Add to create a URL set file for the downloaded URL set.

To edit a URL set by using the GUI

Navigate to AppExpert > URL Sets, select a URL set and click Edit to modify.

To Update a URL set by using the GUI

Navigate to AppExpert > URL Sets, select a URL set and click Update URL Set to update the URL set with the latest modifications made to the file.

To Export a URL set by using the GUI

Navigate to AppExpert > URL Sets, select a URL set, and click Export URL Set to export the URL patterns in a set to a destination URL and save it in that location.

Configuring URL Set
November 7, 2023
Contributed by: 
C S
November 7, 2023
Contributed by: 
C S

In this article

Site feedback | Your privacy choices footer linkYour Privacy Choices | Privacy and legal terms | Cookie preferences | Consent settings | docs.cloud.com
© 1999- Cloud Software Group, Inc. All rights reserved.