Hot Standby (HA)

Genesys uses the term hot standby to describe the redundancy type in which a backup server application remains initialized, clients connect to both the primary and the backup servers at startup, and the backup server data is synchronized with the primary server. Data synchronization and existing client connections to the backup guarantee higher availability of a component.

Starting with release 8.5.109, Stat Server supports Hot Standby redundancy for Stat Servers operating in high availability (HA) mode on Windows and Linux platforms.

Introduction

In HA mode, primary and backup instances of Stat Server replicate historical statistics and their aggregates to each other. Therefore, there is no single point of failure, as if one Stat Server instance fails and is then restarted, it can restore the statistics and their aggregates from another Stat Server instance.

The communication between primary and backup Stat Server instances is performed within a session. In case of a disconnect with the subsequent reconnect, the backup instance tries to restore the previous session. If it succeeds then not the whole population of the statistics but just those that were opened during the disconnect, are replicated. If a session cannot be restored, the whole population of statistics is replicated.

Overview

At startup, both Stat Server instances "read" their configuration and establish connections to each other. When the backup Stat Server instance connects to the primary, both ensure that they are configured as HA pair. Both Stat Server instances run concurrently.

After the successful initialization, two Stat Server instances start synchronization of non-replicated stats. When a new statistic is successfully opened, it is added to a non-replicated collection. Each Stat Server instance pushes statistics to another instance along with their aggregates. This is done in chunks (see the chunk-size option) by the timer (see the chunk-timeout option). If for a receiving Stat Server a statistic is new, this statistic is opened. If a statistic is not new for a receiving Stat Server, then only aggregate synchronization is performed. Aggregates need to be synchronized, except for JavaCategory, where only statistic definitions are synchronized – that requires identically absolute or relative paths to the same extension on two Stat Server instances.

If some mandatory statistical attribute (such as object, filter, time profile, or time range) is deleted in the configuration, the associated Stat Server instance deletes this statistic but still waits for the acknowledgement from another instance.

If a connection between primary and backup Stat Servers is lost, the backup instance tries to reconnect (using frequency, configured as the value of the connect-timeout option) and restore a session. If the session is restored, only new statistics (statistics opened during the disconnect) are replicated. Otherwise, all statistics are replicated.

Configuration

1. In Genesys Administrator, under Provisioning > Environment > Applications, select your primary Stat Server application.
2. In the Server Info section, in the Backup Server field, specify the backup Stat Server application.
3. In the Redundancy Type list, select Hot Standby.

   

   Configure HA port for Stat Server instances
   (Click thumbnail to enlarge.)

4. For Listening Ports, click Add. The Port Info dialog appears.

   In the Port Info dialog, configure the port for communication between the Primary and Backup Stat Server instances. The Stat Server instance configured as a backup always tries to connect to the HA Port that is set on the Stat Server instance configured as primary. You must:

   1. In the ID field, enter ha.
   2. Choose what connection protocol the Stat Server HA pair use; the following protocols are supported: default, addp, or tls.
      • For default connection protocol between the Primary and Backup Stat Servers:
        1. In the Listening Mode list, choose Unsecured,
        2. In the Connection Protocol field, enter default.
      • For ADDP connection protocol between the Primary and Backup Stat Servers:
        1. In the Listening Mode list, choose Unsecured,
        2. In the Connection Protocol field, enter addp,
        3. Configure the ADDP-specific options described in step 5.
      • For TLS connection protocol between the Primary and Backup Stat Servers:
        1. Ensure that trusted CA certificates are installed on the hosts where the HA Stat Servers are running. For more information about TLS connections and CA certificate generation, see the Secure Connections (TLS) section of the Genesys Security Deployment Guide.
        2. In the Listening Mode list, choose Secured,
        3. In the Connection Protocol field, enter default.
      • If the Connection Protocol for ha Listening Port is not specified, an ADDP connection protocol is established for HA Stat Servers.

   Changes in ha port configuration take effect after Stat Server restart.

5. On the Options tab, configure relevant Hot Standby configuration options in the [ha] section, which contains the following options:
   • addp-remote-timeout
   • addp-timeout
   • addp-trace
   • chunk-size
   • chunk-timeout
   • connect-timeout
   • session-expiration-period
   • session-expiration-timeout

Important

New resources, for example: stat types, time profiles, time ranges, filters, can be dynamically added to Stat Server options. However, statistics, using those resources, should only be opened after making identical changes to both Stat Server instances of the HA pair. Otherwise, such statistics may not replicate properly.

Consistency

After all activity in the system is stopped, if both Stat Server instances are configured identically and connected, then they have exactly the same open historical statistics (see exceptions below), and the values for most of those statistics are closed in the HA pair.

Due to the way of how the aggregate synchronization procedure works the double-counting of statistics is possible, but it has a low probability, and even if it happens, its relative contribution is very small.

Special cases of a double-counting are possible during synchronization of GroupBy statistics, such as when one Stat Server instance is started later than the other one, so it did not see the user-data on a call, and created different child for the same call, and after sync, the same call counted twice for different user data keys.

Current statistics are not synchronized and can be different in two instances for the following reasons:

• One Stat Server instance "saw" a real action start, while another instance did not.
• One Stat Server instance did not "see" user data for the call, since it is not provided in the EventRegistered, while another instance "saw" it, as it connected to T-Server prior to when the call was started.

Historical custom formulas, having current component (for example: SUM(CallInternal,duration)+@SUM(CallInternal;duration )), can be out-of-sync for the same reasons, as for the current statistics above.

Only the historical component of the aggregate is replicated for the following statistics:
Category=Formula
Subject=<Status>
Expression=<combination of current and historical component>
As a result, such statistics might have different values in different instances of the same HA pair.

The primary and backup Stat Server instances might have an inconsistent statistic with the Subject=DNAction when an Action ends, if prior to that during the synchronization of Stat Server instances (while the Action was incomplete) one of the following conditions occurred:

• The last reset point of the TimeProfile was different in the primary and backup Stat Server instances.
• The start of the Filter validation was different in the primary and backup Stat Server instances.

Diagnostic

Starting with release 8.5.109, Stat Server logging is enhanced with the new HA value for the [statserver]/debug-level option. If the HA value is set, Stat Server logs messages related to HA functionality.