Deployment

After the sites are configured, the Deployment page allows you to change the software version, stage, and deploy the configuration across the network.

You can upgrade the SD-WAN software on all the appliances across the network, by selecting an appliance software version in the Software Version field.

A confirmation message appears. Click Proceed.

Rollback on Error

With Rollback on Error feature enabled, sites that fail to connect to Citrix SD-WAN Orchestrator service post performing network activation (as part of deployment), triggers an automatic rollback to the previous version (last staged package) to try to restore the connectivity.

Note The automatic rollback is only for the site that failed to connect to Citrix SD-WAN Orchestrator service and not for the entire network.

The rollback only triggers if the appliance loses Citrix SD-WAN Orchestrator service connectivity, not in other scenarios such as, virtual path status goes down or so on.

If at least one site in the network initiates a rollback, a warning message displays a list of sites that are trying to rollback and an option to initiate a network-wide rollback of all the online sites. You can check the progress of these sites and choose the appropriate action.

To enable the rollback on error feature, navigate to Configuration > Deployment > Settings > Rollback on Error.

You can select the Rollback on Error check box to enable automatic rollback of sites which have failed to connect to Citrix SD-WAN Orchestrator service post activation. The Rollback on Error feature must be enabled before you start the deployment to enable its functionality.

For a site to trigger automatic rollback, it must stay offline for at least 30 minutes (currently non-changeable) post activation. If in case the site can connect to Citrix SD-WAN Orchestrator service within 30 minutes, then rollback does not get triggered.

Note Rollback on sites is only performed when the site loses connectivity after activation. Rollback is not triggered in cases where site is online and activation has failed.

Click Done once you set the Rollback on Error enabled.

Use case 1: Non-hitless Upgrade

A site waits for activation to complete for a specified time with a status as Activation in progress.

Post that timeout, if the Site is still offline, Citrix SD-WAN Orchestrator service waits for another 30 mins (rollback initiation timeout) to give a chance to the site to connect back. At this stage the status shows as Activation Timeout, Waiting to Initiate Rollback (remaining time in mins).

Post the 30 minutes waiting period, the appliance triggers an automatic rollback to the previous configuration or (and) software to try and restore Citrix SD-WAN Orchestrator service connectivity. Citrix SD-WAN Orchestrator service wait for 20 mins (non-configurable setting) for the appliance to connect to Citrix SD-WAN Orchestrator service and during this period, status is shown as Rollback in progress (remaining time in mins).

If the appliance fails to connect back, in this 20 minutes, Citrix SD-WAN Orchestrator service marks the rollback operation as failed and status is shown as Device Rollback Failed.

In the network, if at least one device has initiated the automatic rollback, a banner is presented to the user as follows:

Based on the stage in Network Activation, the displayed options perform the following operations:

• Ignore Network Rollback:

  • For non-Hitless upgrade scenario: End the current deployment.
  • First step in Hitless upgrade scenario: Deployment proceeds to second step of Activation.
  • Second step in Hitless upgrade scenario: End the current deployment.

• Rollback entire Network:

  • For non-Hitless upgrade scenario: Trigger rollback on all online sites in the network.
  • First step in Hitless upgrade scenario: Trigger rollback on all online standby devices in the network.
  • Second step in Hitless upgrade scenario: Trigger rollback on all online sites (active and standby). Near-hitless software upgrade for high availability devices is not applicable in this scenario.

You can click the more Click Here hyperlink to view the list of sites for which rollback is in progress or completed and take the above actions for that page.

You can also wait until the sites that have triggered rollback to either succeed or fail before deciding on triggering the network-wide rollback.

If you select the Rollback entire Network option, the following pop-up box appears.

Note The Near-hitless software upgrade for high availability appliance is not applicable in this scenario, that is if there are any high availability sites in the network, triggering a network-wide rollback activates both the high availability appliances of that site at once which can cause some network downtime.

Click Proceed to start the network-wide rollback on all the online sites.

Use case 2: Hitless Upgrade

In the case of Hitless upgrade, the standby appliances would be activated first followed by the active and non-high availability appliances. As part of the first step if the standby appliance goes offline post activation and initiates a rollback, the following options are available:

• Ignore Network Rollback: Ignore the standby appliances which are offline and proceed with the activation of the active appliances.

• Rollback entire Network: Rollback all the online standby appliances which have completed the activation and end the ongoing deployment. No activation of active and non-high availability appliance is done in this case.

The next step of the hitless upgrade that is activation of active and non-high availability appliance, the same rollback on error workflow is followed as mentioned in the above non-hitless upgrade section. In this scenario, if you choose Rollback entire Network, the rollback triggers for all the (both active and standby) appliance.

Once the site completes rollback and connects back to Citrix SD-WAN Orchestrator service, the status for that site shows Device Rollback Successfully and the sites are online.

Limitations

Autocorrection for Rolling back or Rolled back appliances and network is not supported.

Note Automatic site rollback is only a backup mechanism to try and restore the lost connectivity to Citrix SD-WAN Orchestrator service. If the appliance still fails to connect to Citrix SD-WAN Orchestrator service, check the network configuration of this appliance.

You can export the filtered results in to a CSV or PDF file by using the Export as CSV and Export as PDF options. The CSV and PDF file name is prefixed with Deployment Site List followed by the date and time when the file is exported.

• Stage: Once the verification of configuration is successful, click Stage to distribute the configuration files to all the appliances in your network. By default Citrix SD-WAN Orchestrator service the waits for all the Control nodes (MCN, RCN, Geo MCN, Geo RCN) and the online branch appliances to get staged before allowing the user to activate.

  If the staging process fails at any site, use the Retry Staging option, under the Actions column, to reinitiate the staging process.

• Activate: Click Activate to activate the staged configuration on all the sites across the network.

• Ignore Incomplete: When selected, the Activate check box is enabled only after all the online control nodes (MCN, RCN, Geo MCN, Geo RCN) get staged. You can choose to activate even if some of the online branch appliances are not staged. The online branch appliances that fail to get staged are ignored.

• Partial Site Upgrade Setting: The Partial Site Upgrade option is added to upgrade or downgrade the selected sites with a different version. The Partial Site Upgrade feature provides the ability to test a new version before deploying to the entire network.

With the Partial Site Upgrade feature, upgrades can be staggered and thereby reducing the impact of software upgrades during business hours.

Note

Partial Site Upgrade can be performed only when all the sites in the network are running Citrix SD-WAN software version 11.2.2 or above.

Any configuration changes for the Partial Site Upgrade need a change management for the changes to take effect. The Partial Site Upgrade picks the lower version and generates the configuration for the same. Any new features cannot be tested while the network is in the Partial Site Upgrade mode.

When you are downgrading from a newer to older version using the Partial Site Upgrade, if a feature which is supported only in the newer version (with the similar configuration present both in the new and older version), audit errors occur. For example, a new platform is selected which is only supported on the newer version then this will throw audit errors.

To perform the partial site upgrade:

1. Click the Setting … icon and select the Partial Site Upgrade option.

   

2. Select the Partial Site Upgrade check box, choose the software version, and click Select Sites to add new sites.

   

3. Select the sites and click Save.

   

   

In the case of a configuration-only update, only the sites that have configuration changes are staged and activated. For the remaining sites, the timestamp is updated and processed.

If the software version is being changed, both configuration and software package are staged and activated on all the sites in the network.

The Deployment History section helps to review the previous deployment operations and results.

HA near-hitless software upgrade

During software upgrade (11.0.x and earlier versions), the staging, and activation of all the appliances in the net work are done at the same time. This includes the High Availability (HA) pair, leading to network downtime. With the HA near-hitless software upgrade feature, the Citrix SD-WAN Orchestrator service ensures that the downtime during the software upgrade (11.1.x and above) process is not more than the HA switch over time.

Note

The HA near-hitless software upgrade is applicable for the following:

• The sites that are deployed in High Availability (HA) mode. It is not applicable for Non-HA sites.
• Citrix SD-WAN Orchestrator service based deployments only and not for the networks that are managed using the SD-WAN Center or MCN.
• Software upgrade only and not configuration updates. If there is configuration change along with the software as part of the upgrade, the Citrix SD-WAN Orchestrator service does not perform HA near-hitless software upgrade and continues to upgrade in the earlier fashion (single-step upgrade).

The upgrade sequence summary:

1. Citrix SD-WAN Orchestrator service checks for the HA state of all the appliances in the network.
2. Upgrades all the secondary appliances that are in the Standby state.
3. HA switch-over is triggered and the state of the Active and Standby appliances are switched.
4. Upgrades the primary appliances that are now in Standby state.

The HA near-hitless software upgrade is a two-step upgrade process:

Step-1: During software upgrade, after the 11.1 release, the Citrix SD-WAN Orchestrator service first performs software upgrade on all the appliances that are in the Standby state across the network. The network is still up and running with the Active appliances in place.

After all the Standby appliances are upgraded to the latest software, the HA switch-over is triggered across the network. The Standby appliances (with the latest software) become Active.

Step-2: The current Standby appliances with an old software version are upgraded to the latest software and will continue to run in Standby mode.

During this software upgrade process, all other Non-HA sites will also be activated with the latest software.

For more information, see the FAQs.

You can view the upgrade status by navigating to Deployment Tracker > Current deployment.

• Stage: Click Stage to distribute the configuration files to all the appliances in your network. By default the Citrix SD-WAN Orchestrator service waits for all the Control nodes (MCN, RCN, Geo MCN, Geo RCN) and the online branch appliances to get staged before allowing the user to activate.
• Activate: Click Activate to activate the staged configuration on all the sites across the network.
• Restore previous version: Click Restore previous version to roll back to the previously activated configuration on your network. The HA near-hitless software upgrade is applicable when you restore the previous version if the previously active version is just a software version change and not a configuration change. For more information about this functionality, see Restore previous version.
• Ignore Incomplete: When selected, the Activate check box is enabled only after all the online control nodes (MCN, RCN, Geo MCN, Geo RCN) get staged. You can choose to activate even if some of the online branch appliances are not staged. The online branch appliances that fail to get staged are ignored.

In the case of a configuration-only update, only the sites that have configuration changes are staged and activated. For the remaining sites, the timestamp is updated and processed. The Not Needed column lists the number of sites that do not have any configuration change.

If the software version is being changed, both configuration and software package are staged and activated on all the sites in the network.

Restore previous version

In the restore previous version functionality, Citrix SD-WAN Orchestrator service initiates a network wide activation of the previous configuration and restores the previously activated configuration (and/or software) on your network.

When you select the Restore previous version option, the following confirmation message is displayed:

Note

The Restore previous version action can be performed when the network is not in the staged state. This option is disabled for staged networks.

Auto-correction for configuration and software upgrade

In the Citrix SD-WAN Orchestrator service, the auto-correction feature is implemented in the change management workflow.

When the staging failed for one site, and if the site that had failed staging is a control node, you need to restage after getting the staging failure message. The Activate button will not be enabled if the staging fails for the control nodes. If the site that had failed staging is a branch node, you are still allowed to move ahead with the activation. But to bring that branch in sync with the network, perform another round of change management.

Note

• The auto-correction check starts only after the Activate button has been clicked and stops once the next stage is issued from the Citrix SD-WAN Orchestrator service UI.

• The maintenance mode functionality is only applicable for the auto-correction feature. If you initiate a Staging and Activation, the appliance with the maintenance mode enabled also gets updated with the software and configuration changes.

With the auto-correction feature enhancement, when a staging failure happens, the auto-correction mechanism pushes the expected software and configuration version to the failed branch and tries to bring it up in sync with the current network. The auto-correction feature is applicable for staging failure on the branch node and activation failure on any node.

The following are the two trigger points when the auto-correction starts:

• In the Citrix SD-WAN Orchestrator service deployment tracker UI, once you get a Staging Failed or Activation Failed message, the auto-correction starts running in the background. The auto-correction check starts once the activation is completed.

• In the case of a software and configuration mismatch, where the appliance didn’t come up with the expected software and configuration version, the Citrix SD-WAN Orchestrator service starts pushing the actual required software and configuration copy down to the appliance for activation.

To troubleshoot an appliance manually, enable the maintenance mode check box under the Change Management Settings. This check box is used to control if the device needs to be checked for auto-correction or not. Once the maintenance mode check box is cleared, auto-correction brings the appliance in sync with the network software and configuration version.

Site details

The Site Details tab under the Deployment Tracker provides information about all the devices in the network. The table contains the appliance name, Citrix SD-WAN Orchestrator service connectivity, High Availability (HA) state, and currently running software version.

Verify Configuration

You can click Verify Config to validate the network configuration and check for any audit error or warning. When you click Verify Config, the Configuration results page is displayed. This page contains details of audit errors and warnings.

The configuration results display the total number of audit errors and warnings. The results are also filtered based on the audit type (error or warning) and displayed with different color codes. You can click the numbers links to view the filtered results.

The Type column displays an icon to indicate whether it is an error or a warning. The Audit Scope column specifies if the error or warning is for a site or at the network level. If the error or warning is specific to a site, then the name of the site is displayed. If the error or warning is at the global level, then Global Error or Global Warning is displayed respectively. The Audit Message column contains the error code and the error message.

You can use the search bar to search for any specific errors or warnings based on the type, error code, site name, or error message.

When you click Verify Config for the second time, the Configuration results page opens up displaying the same results when the configuration was last verified along with the date and time stamp. If necessary, you can click Verify Again to rerun the validation.