AppFlow and IPFIX
AppFlow and IPFIX are flow export standards used to identify and collect application and transaction data in the network infrastructure. This data gives better visibility into application traffic utilization and performance.
The collected data, called flow records are transmitted to one or more IPv4 or IPv6 collectors. The collectors aggregate the flow records and generate real-time or historical reports.
AppFlow
AppFlow exports flow level data for HDX / ICA connections only. You can enable either the TCP only for HDX dataset template or the HDX dataset template. The TCP only for HDX dataset provides multi-hop data. The HDX dataset provides HDX insight data.
Note
HDX template is available for Citrix SD-WAN PE edition and Two-box appliances only. It should be enabled on the Data Center appliance.
AppFlow Collectors like Splunk and Citrix ADM have dashboards to interpret and present these templates.
IPFIX
IPFIX is a collector export protocol used for exporting flow level data for all connections. For any connection, you can view information such as packet count, byte count, type of service, flow direction, routing domain, application name and so on. IPFIX flows are transmitted through the management interface. Most collectors can receive IPFIX flow records, but may need to build a custom dashboard to interpret IPFIX template.
The IPFIX template defines the order in which the data stream is to be interpreted. The collector receives a template record, followed by the data records. Citrix SD-WAN uses templates 611 and 613 to export IPv4 IPFIX flow data, 615 and 616 to export IPv6 IPFIX flow data along with Options template 612.
Application Flow Info (IPFIX) exports data sets as per templates 611 for IPv4 flows, 615 for IPv6 flows and 612 options Template with Application info.
Basic Properties (IPFIX) exports data sets as per templates 613 for IPv4 flows and 616 for IPv6 flows.
The following tables provide the detailed list of flow data associated with each IPFIX template.
Application Flow Info (IPFIX) - V10 templates
Template ID - 611
| Info Element (IE) | IE name & ID | Type and len | Description |
|---|---|---|---|
| Observation point ID | observationPointId, 138 | Unsigned32, 4 | |
| Export process ID | exportingProcessId, 144 | Unsigned32, 4 | |
| Flow ID | flowId, 148 | Unsigned64, 8 | |
| Ipv4 SRC IP | sourceIPv4Address, 8 | Ipv4address, 4 | |
| Ipv4 DST IP | destinationIpv4Addres, 12 | Ipv4address, 4 | |
| Ipversion | ipVersion, 60 | Unsigned8, 1 | Set to 4. |
| IP protocol number | protocoldentifier,4 | Unsigned8, 1 | |
| Padding | N/A | Unsigned16, 2 | |
| SRC Port | sourceTransportPort, 7 | Unsigned16, 2 | |
| DST Port | destinationTransportPort,11 | Unsigned16, 2 | |
| Pkt Count | packetDeltaCount, 2 | Unsigned64, 8 | |
| Byte Count | octetDeltaCount, 1 | Unsigned64, 8 | |
| Time for first pkt in microseconds | flowStartMicroseconds, 154 | dateTimeMicroseconds, 8 | |
| Time for lastpkt in microseconds | flowEndMicroseconds, 155 | dateTimeMicroseconds, 8 | |
| IP ToS | ipClassOfService, 5 | Unsigned8, 1 | |
| Flow Flags | tcpControlBits, 6 | Unsigned8, 2 | Currently set to 0. |
| Flow Direction | flowDirection, 61 | Unsigned8, 1 | 0x00: ingress flow0x01: egress flowWAN-WAN and LAN-LAN flows are a possibility in SDWAN |
| Input Interface | ingressInterface, 10 | Unsigned32, 4 | Citrix SD-WAN load balances data flows through multiple member paths, hence a single data flow can have multiple input/output interface combinations. |
| Output Interface | egressInterface, 14 | Unsigned32, 4 | Citrix SD-WAN load balances data flows through multiple member paths, hence a single data flow can have multiple input/output interface combinations. |
| Input Vlan ID | vlanId, 58 | Unsigned16, 2 | |
| Output Vlan ID | postVlanId, 59 | Unsigned16, 2 | |
| VRF ID | ingressVRFID, 234 | Unsigned32, 4 | |
| Flow Key Indicator | flowKeyIndicator, 173 | Unsigned64, 8 | Set to 0x1E037F. |
| Application ID | applicationId, 95 | octetArray, variable | The Application ID is same as the ID of the applications classified by the DPI engine. The application IDs remain constant. The application IDs for Custom domain name based applications change with every configuration update. |
Template ID – 615 (IPv6 flows)
| Info Element (IE) | IE name & ID | Type and len | Comment | |
|---|---|---|---|---|
| Observation point ID | observationPointId, 138 | Unsigned32, 4 | ||
| Export process ID | exportingProcessId, 144 | Unsigned32, 4 | ||
| Flow ID | flowId, 148 | Unsigned64, 8 | ||
| Ipv6 SRC IP | sourceIPv6Address, 27 | Ipv6address, 16 | ||
| Ipv6 DST IP | destinationIpv6Addres, 28 | Ipv6address, 16 | ||
| Ipversion | ipVersion, 60 | Unsigned8, 1 | Set to 6 | |
| IP protocol number | protocoldentifier, 4 | Unsigned8, 1 | ||
| Padding | N/A | Unsigned16, 2 | ||
| SRC Port | sourceTransportPort, 7 | Unsigned16, 2 | ||
| DST Port | destinationTransportPort, 11 | Unsigned16, 2 | ||
| Pkt Count | packetDeltaCount, 2 | Unsigned64, 8 | ||
| Byte Count | octetDeltaCount, 1 | Unsigned64, 8 | ||
| Time for first pkt in microseconds | flowStartMicroseconds, 154 | dateTimeMicroseconds, 8 | ||
| Time for lastpkt in microseconds | flowEndMicroseconds, 155 | dateTimeMicroseconds, 8 | ||
| IP ToS | ipClassOfService, 5 | Unsigned8, 1 | ||
| Flow Flags | tcpControlBits, 6 | Unsigned8, 2 | Currently set to 0. | |
| Flow Direction | flowDirection, 61 | Unsigned8, 1 | 0x00: ingress flow0x01: egress flowWAN-WAN and LAN-LAN flows are a possibility in SDWAN | |
| Input Interface | ingressInterface, 10 | Unsigned32, 4 | Citrix SD-WAN load balances data flows through multiple member paths, hence a single data flow can have multiple input/output interface combinations. | |
| Output Interface | egressInterface, 14 | Unsigned32, 4 | Citrix SD-WAN load balances data flows through multiple member paths, hence a single data flow can have multiple input/output interface combinations. | |
| Input Vlan ID | vlanId, 58 | Unsigned16, 2 | ||
| Output Vlan ID | postVlanId, 59 | Unsigned16, 2 | ||
| VRF ID | ingressVRFID, 234 | Unsigned32, 4 | ||
| Flow Key Indicator | flowKeyIndicator, 173 | Unsigned64, 8 | Set to 0x1E037F. | |
| Application ID | applicationId, 95 | octetArray, variable | The Application ID is same as the ID of the applications classified by the DPI engine. The application IDs remain constant. The application IDs for Custom domain name based applications change with every configuration update. | |
Template 612 (Options Template)
| Info Element (IE) | IE name & ID | Type | Comment |
|---|---|---|---|
| Application ID | applicationId, 95 | octetArray | The Application ID is same as the ID of the applications classified by the DPI engine. The application IDs remain constant. The application IDs for Custom domain name based applications change with every configuration update. |
| Application Name | applicationName, 96 | string | Specifies the name of the Citrix SDWAN specific proprietary application. |
| Application Description | applicationDescription, 94 | string | Specifies the description of the application. |
Basic Properties (IPFIX) – V9 compliant template - Template 613 (IPv4 flows)
| Info Element (IE) | IE name & ID | Type and len | Comment |
|---|---|---|---|
| Ipv4 SRC IP | sourceIPv4Address, 8 | Ipv4address, 4 | |
| Ipv4 DST IP | destinationIpv4Addres, 12 | Ipv4address, 4 | |
| Ipversion | ipVersion, 60 | Unsigned8, 1 | |
| IP protocol number | protocoldentifier, 4 | Unsigned8, 1 | |
| IP ToS | ipClassOfService, 5 | Unsigned8, 1 | |
| Flow Direction | flowDirection, 61 | Unsigned8, 1 | 0x00: ingress flow0x01: egress flowWAN-WAN and LAN-LAN flows are a possibility in SDWAN |
| SRC Port | sourceTransportPort, 7 | Unsigned16, 2 | |
| DST Port | destinationTransportPort, 11 | Unsigned16, 2 | |
| Pkt Count | packetDeltaCount, 2 | Unsigned64, 8 | |
| Byte Count | octetDeltaCount, 1 | Unsigned64, 8 | |
| Input Interface | ingressInterface, 10 | Unsigned32, 4 | Citrix SD-WAN load balances data flows through multiple member paths, hence a single data flow can have multiple input/output interface combinations. |
| Output Interface | egressInterface, 14 | Unsigned32, 4 | Citrix SD-WAN load balances data flows through multiple member paths, hence a single data flow can have multiple input/output interface combinations. |
| Input Vlan ID | vlanId, 58 | Unsigned16, 2 | |
| Output Vlan ID | postVlanId, 59 | Unsigned16, 2 |
Template ID – 616 (IPv6 flows)
| Info Element (IE) | IE name & ID | Type and len | Comment | |
|---|---|---|---|---|
| Ipv6 SRC IP | sourceIPv6Address, 27 | Ipv6address, 16 | ||
| Ipv6 DST IP | destinationIpv6Addres, 28 | Ipv6address, 16 | ||
| Ipversion | ipVersion, 60 | Unsigned8, 1 | Set to 6 | |
| IP protocol number | protocoldentifier,4 | Unsigned8, 1 | ||
| IP ToS | ipClassOfService, 5 | Unsigned8, 1 | ||
| Flow Direction | flowDirection, 61 | Unsigned8, 1 | 0x00: ingress flow0x01: egress flowWAN-WAN and LAN-LAN flows are a possibility in SDWAN | |
| SRC Port | sourceTransportPort, 7 | Unsigned16, 2 | ||
| DST Port | destinationTransportPort, 11 | Unsigned16, 2 | ||
| Pkt Count | packetDeltaCount, 2 | Unsigned64, 8 | ||
| Byte Count | octetDeltaCount, 1 | Unsigned64, 8 | ||
| Input Interface | ingressInterface, 10 | Unsigned32, 4 | Citrix SD-WAN load balances data flows through multiple member paths, hence a single data flow can have multiple input/output interface combinations. | |
| Output Interface | egressInterface, 14 | Unsigned32, 4 | Citrix SD-WAN load balances data flows through multiple member paths, hence a single data flow can have multiple input/output interface combinations. | |
| Input Vlan ID | vlanId, 58 | Unsigned16, 2 | ||
| Output Vlan ID | postVlanId, 59 | Unsigned16, 2 |
Limitations
- AppFlow does not support IPv6 collector and flow records.
- The export interval for Net Flow is increased from 15 seconds to 60 seconds.
- AppFlow/IPFIX flows are transmitted over UDP, on connection loss not all data is retransmitted. If the export interval is set to X minutes, the appliance stores X minutes of data only. Which is retransmitted after X minutes of connection loss.
- In Citrix SD-WAN, release 10 version 2 the AppFlow settings are made local to every appliance, while in the previous releases it was a global setting. If the SD-WAN software release is downgraded to any of the previous releases and if AppFlow is configured on any one of the appliances, it will be applied globally to all alliances.
Configuring AppFlow/IPFIX
You can configure AppFlow / IPFIX on individual SD-WAN appliances or configure it on SD-WAN Center and push the configuration to a group of appliances.
To configure AppFlow / IPFIX on SD-WAN appliances:
- In Citrix SD-WAN SE/PE web interface, navigate to Configuration > AppFlow/IPFIX.
-
Click Enable.
-
In the Data Update Interval field, specify the time interval, in minutes, at which the flow reports are exported to AppFlow/IPFIX collector. The maximum interval is 10 minutes.
- Select the AppFlow dataset template, you can choose either one of the following dataset templates:
- TCP only for HDX (AppFlow): The AppFlow dataset template to collect and send multi-hop data of ICA connections to the AppFlow collector.
- HDX (AppFlow): The AppFlow dataset template to collect and send HDX insight data of ICA connections to AppFlow collector.
Note
HDX template is available for Citrix SD-WAN PE and Two Box appliances only.
-
You can configure up to four AppFlow / IPFIX collectors. For each collector specify the following parameters:
- IP Address: The IP Address of the external AppFlow / IPFIX collector system.
-
Port: The port number on which the external AppFlow / IPFIX collector system listens. The default value is 4739. You can change the port number depending on the collector used.
-
Application Flow Info (IPFIX): Sends flow records, as per IPFIX templates 611, 615, and 612 to IPFIX collectors.
-
Basic Properties (IPFIX): Sends flow records, as per IPFIX template 613 and 616 to IPFIX collectors.
- Citrix ADM: Select this to use Citrix ADM as the AppFlow collector.
Note
- Citrix ADM currently does not support IPFIX collection.
- Citrix ADM does not support IPv6 addresses for AppFlow and IPFIX.
-
Citrix ADM User: User name of the Citrix ADM collector
-
Password: Citrix ADM collector password.
The user name and password are used to seamlessly log in into Citrix ADM and store flow data.
- Click Apply Settings.
To configure AppFlow / IPFIX collector using Citrix SD-WAN Center:
- In Citrix SD-WAN Center management UI, navigate to Configuration > Appliance Settings.
- Navigate to the AppFlow / IPFIX section and choose Include in File.
-
Select Enable IPFIX / AppFlow Collection.
- In the Data Update Interval field, specify the time interval, in minutes, at which the AppFlow reports are exported to the AppFlow / IPFIX collector.
- Select the AppFlow dataset template, you can choose either one of the following dataset templates:
- TCP only for HDX: The AppFlow dataset template to collect and send multi-hop data of ICA connections to the AppFlow collector.
- HDX: The AppFlow dataset template to collect and send HDX insight data of ICA connections to AppFlow collector.
Note
HDX template is available for Citrix SD-WAN PE and Two Box appliances only.
-
You can configure up to four AppFlow / IPFIX collectors. For each collector specify the following parameters:
- IPFIX / AppFlow Collector: The IP Address of the external AppFlow / IPFIX collector system.
- Port: The port number on which the external AppFlow / IPFIX collector system listens. The default value is 4739. You can change the port number depending on the collector used.
- Application Flow Info: Sends flow records, as per IPFIX templates 611, 615, and 612 to IPFIX collectors.
- Basic Properties (IPFIX): Sends flow records, as per IPFIX template 613 and 616 to IPFIX collectors.
-
Citrix ADM: Select this to use Citrix ADM as the AppFlow collector.
Note
Citrix ADM currently does not support IPFIX collection.
- Citrix ADM User: User name of the Citrix ADM collector.
-
Password: Citrix ADM collector password.
The user name and password are used to seamlessly log in into Citrix ADM and store flow data.
- Save and Export the configuration to the managed appliances.
Note
If SD-WAN Center version is lower than 10.2 and SD-WAN appliances version is 10.2 and above then you can observe the following conditions.
- If local collectors are enabled on the appliances, the AppFlow / IPFIX configuration pushed from SD-WAN center does not affect the existing configuration.
- If local collectors are not enabled on the appliances, the AppFlow/IPFIX configuration pushed from SD-WAN center will be applied to the appliance.
- If the global AppFlow/IPFIX configuration is enabled in SD-WAN Center configuration, all the local collectors are enabled on the appliances.
Log files
For troubleshooting issues related to AppFlow / IPFIX export protocols, you can view and download the SDWAN_export.log files. Navigate to Configuration > Logging / Monitoring and select the SDWAN_export.log files.
