Product Documentation
ADC
Current Release 13.1 13.0 12.1
View PDF

In Service Software Upgrade support for high availability for performing zero downtime upgrade

May 2, 2023
Contributed by: 
S

During a regular upgrade process in a high availability setup, at some point, both nodes run different software builds. These two builds can have the same or different internal high availability version numbers.

If both the builds have different high availability version numbers, connection failover (even if it is enabled) for existing data connections is not supported. In other words, all existing data connections are lost, which leads to downtime.

To address this issue, in Service Software Upgrade (ISSU) can be used for high availability set-ups. ISSU introduces a migration functionality, which replaces the force failover operation step in the upgrade process. The migration functionality takes care of honoring the existing connections and includes the force failover operation.

After a migration operation is performed, the new primary node always receives traffic (request and response) related to the existing connections but steers them to the old primary node. The old primary node processes the data traffic and then sends them directly to the destination.

How the enhanced ISSU works

The regular upgrade process in a high availability setup consists of the following steps:

  1. Upgrade the secondary node. This step includes software upgrade of the secondary node and restart of the node.

  2. Force Failover. Running the force failover makes the upgraded secondary node to primary, and the primary node to secondary.

  3. Upgrade the new secondary node. This step includes software upgrade of the new secondary node and restart of the node.

During the time frame between step 1 and step 3, both nodes run different software builds. These two builds can have the same or different internal high availability versions.

If both the builds have different high availability version numbers, connection failover (even if it is enabled) for existing data connections is not supported. In other words, all existing data connections are lost, which leads to downtime.

The ISSU upgrade process in a high availability setup consists of the following steps:

  1. Upgrade the secondary node. This step includes software upgrade of the secondary node and restart of the node.

  2. ISSU migration operation. The step includes the force failover operation and takes care of the existing connections. After you perform the migration operation, the new primary node always receives traffic (request and response) related to the existing connections but steers them to the old primary node through the configured SYNC VLAN in GRE tunnel. The old primary node processes the data traffic and then sends them directly to the destination. The ISSU migration operation is completed when all the existing connections are closed.

  3. Upgrade the new secondary node. This step includes software upgrade of the new secondary node and restart of the node.

Before you begin

Before you start performing the ISSU process in a high availability setup, go through the following pre-requisites, limitations, and points to note:

  • Make sure the SYNC VLAN is configured on both the nodes of the high availability setup. For more information, see Restricting high availability synchronization traffic to a VLAN.
  • ISSU is not supported on the Microsoft Azure cloud because Microsoft Azure does not support GRE tunneling.
  • High availability config propagation and synchronization do not work during ISSU.
  • ISSU is not supported for IPv6 high availability setup.

  • ISSU is not supported for the following sessions:

    • Jumbo frames
    • IPv6 sessions
    • Large scale NAT (LSN)
  • In an HA setup in INC mode, ISSU migration operation migrates only the client side connections. The migration of server-side connections is not required because both the HA nodes have independent SNIP configurations.

Configuration steps

ISSU includes a migration feature, which replaces the force failover operation in the regular upgrade process of a high availability setup. The migration functionality takes care of honoring the existing connections and includes the force failover operation.

During the ISSU process of a high availability setup, you run the migration operation just after you upgraded the secondary node. You can perform the migration operation from either of the two nodes.

CLI Procedure

To perform the high availability migration operation by using the CLI:

At the command prompt type:

start ns migration
<!--NeedCopy-->

GUI Procedure

To perform the high availability migration operation by using the GUI:

Navigate to System, click System Information tab, click Migration tab, and then click Start Migration.

Display ISSU statistics

You can view the ISSU statistics for monitoring the current ISSU process in a high availability setup. The ISSU statistics displays the following information:

  • Current status of ISSU migration operation
  • Start time of the ISSU migration operation
  • End time of the ISSU migration operation
  • Start time of the ISSU rollback operation
  • Total number of connections that are processed as part of ISSU migration operation
  • Number of remaining connections that are being processed as part of ISSU migration operation

You can view the ISSU statistics on either of the high availability nodes by using CLI or GUI.

CLI Procedure

To display the ISSU statistics by using the CLI:

At the command prompt type:

show ns migration
<!--NeedCopy-->

GUI Procedure

To display the ISSU statistics by using the GUI:

Navigate to System, click System Information tab, click Migration tab, and then click Click to show migration details.

Display ISSU statistics - the list of existing connections that the old primary node is processing

You can display the list of existing connections that the old primary node is currently serving as part of the ISSU migration operation by using the dumpsession(Dump Session) option of the show migration operation.

The show migration operation with the dumpsession option must be run only on the new primary node during the ISSU operation.

CLI Procedure

To display the list of existing connections that the old primary node is currently processing by using the CLI:

At the command prompt type:

show ns migration –dumpsession YES
<!--NeedCopy-->

> sh migration -dumpsession yes

Index   remote-IP-port          local-IP-port           idle-time(x 10ms)

1       192.0.2.10      22      192.0.2.1       15998       703
2       198.51.100.20   7375    98.51.100.2     22          687
3       203.0.113.30    5506    203.0.113.3     22          687


<!--NeedCopy-->

GUI Procedure

To display the list of existing connections that the old primary node is currently processing by using the GUI:

Navigate to System, click System Information tab, click Migration tab, and then click Click to show migration connections.

Rollback of the ISSU process

High availability (HA) setups now support rollback of the In Service Software Upgrade (ISSU) process. The ISSU rollback feature is helpful if you observe that the HA setup during the ISSU migration operation is not stable, or is not performing at an optimum level as expected.

The ISSU rollback is applicable when the ISSU migration operation is in progress. The ISSU rollback does not work if the ISSU migration operation is already completed. In other words, you must run the ISSU rollback operation when the ISSU migration operation is in progress.

The ISSU rollback functions differently based on the state of the ISSU migration operation when the ISSU rollback operation is triggered:

  • Force failover has not yet happened during the ISSU migration operation. The ISSU rollback stops the ISSU migration operation, and removes any internal data related to the ISSU migration stored in both the nodes. The current primary node remains as primary node and continues to process data traffic related to existing and new connections.

  • Force failover has happened during ISSU migration operation. If the high availability failover has happened during the ISSU migration operation, then the new primary node (say it is N1) processes traffic related to the new connections. The old primary node (new secondary node, say it is N2) processes traffic related to the old connections (existing connections before the ISSU migration operation).

    The ISSU rollback stops the ISSU migration operation and triggers a force failover. The new primary node (N2) now starts processing traffic related to the new connections. The new primary node (N2) also continues to processes traffic related to old connections (existing connections established before the ISSU migration operation). In other words, the existing connections established before the ISSU migration operation are not lost.

    The new secondary node (N1) removes all the existing connections (new connections created during the ISSU migration operation) and does not process any traffic. In other words, any existing connections that were established after the force failover of the ISSU migration operation are lost forever.

Configuration steps

You can use the NetScaler CLI or GUI to perform the ISSU rollback operation.

CLI Procedure

To perform the ISSU rollback operation by using the CLI:

At the command prompt type:

stop ns migration
<!--NeedCopy-->

GUI Procedure

To perform the ISSU rollback operation by using the GUI:

Navigate to System, click System Information tab, click Migration tab, and then click Stop Migration.

SNMP traps for In Service Software Upgrade process

The In Service Software Upgrade (ISSU) process for a high availability setup supports the following SNMP trap messages at the start and end of the ISSU migration operation.

SNMP Trap Description
migrationStarted This SNMP trap is generated and sent to the configured SNMP trap listeners when the ISSU migration operation starts.
migrationComplete This SNMP trap is generated and sent to the configured SNMP trap listeners when the ISSU migration operation completes.

The primary node (before the start of the ISSU process) always generates these two SNMP traps and sends them to the configured SNMP trap listeners.

There are no SNMP alarms associated with the ISSU SNMP traps. In other words, these traps are generated irrespective of any SNMP alarm. You only have to configure the trap SNMP listeners.

For more information on configuring SNMP trap listeners, see SNMP traps on NetScaler.

In Service Software Upgrade support for high availability for performing zero downtime upgrade
May 2, 2023
Contributed by: 
S
May 2, 2023
Contributed by: 
S

In this article

Site feedback | Privacy and legal terms | Cookie preferences | Consent settings | docs.cloud.com
© 1999- Cloud Software Group, Inc. All rights reserved.