Resource Manager HA (Linux)

Two options exists to achieve high availability when the Resource Manager is installed on Linux operating systems: Simple Virtual IP failover and Bonding Driver failover.

When you configure either of these two options, each host in the HA-pair has a static private IP address, however, all hosts share one public virtual IP address. The public IP address is used by external Session Initiation Protocol (SIP) endpoints to interact with the Resource Manager on each host in the cluster. If an instance of the Resource Manager fails on any host, the failover is transparent to the caller.

Furthermore, when you use the Bonding Driver failover option, and two or more network cards are installed on the same server, the bonding driver provides active standby functionality for the individual network interfaces.

Important

If a VMWare installation is used for Resource Manager in active backup HA mode, you must use Procedure: Configuring Bonding Driver Failover. Typically two NICs are recommended for bonding (to achieve maximum robustness), however, bonding can also be achieved by using one NIC only.

Precaution for Failover Scenarios

In certain failover scenarios, it is expected that both Resource Manager instances will have an active (and identical) Virtual IP address. In most environments this is not an issue, because the network layer ARP table is updated automatically to prioritize traffic to the primary Resource Manager. However, Genesys recommends that you confirm with your Network Operations team that no actions are taken at the network layer when duplicate IP addresses (specifically the Virtual IP address) have been identified.

Use the procedures in this section to configure Resource Manager HA in active standby mode on the Linux host(s).

Limitation

A limitation exists when the Resource Manager is configured in HA mode on Linux, where the NICs, that are associated with the aliases and used for the virtual IPs, are active on both Resource Manager hosts. Depending on the network topology, a situation might arise where a SIP User Agent sends a query to find the active Resource Manager, and instead of sending it to the active instance, sends it to the backup Resource Manager.

To workaround for this issue, configure the primary and secondary Resource Manager instances as described in the section, Creating Alarm Reaction Scripts, Conditions, and Reaction Applications on page 435.

Task Summaries

Task Summary: Configuring the RM in High Availability Active-Standby Mode (Linux)

Objective	Related Procedures and Actions
Configure the Resource Manager hosts for HA.	Procedure: Configuring Simple Virtual IP Failover or Procedure: Configuring Bonding Driver Failover.
Configure the member IDs for the Resource Manager Applications.	Procedure: Configuring the INIT and NLB Script Files (Linux).
Specify the NICs that require monitoring.	Procedure: Specifying the NICs for Monitoring (Linux).
Complete finals steps before executing the Resource Manager HA-pair in NLB mode.	Start Resource Manager HA-pair in NLB mode.

Task Summary: Configuring the RM in High Availability Active-Active Mode (Linux)

Objective	Related Procedures and Actions
Configure the member IDs in the Resource Manager Applications.	Procedure: Configuring the Resource Manager HA-Pair.
Configure the virtual IP in the Media Control Platform, Call Control Platform, and CTI Connector Applications.	Procedure: Integrating Application Objects with Resource Manager and Procedure: Configuring the Call Control Platform. Note: When you use these procedures to configure active active HA mode, the virtual IP is used as the Resource Manager IP.
Configure the external load balancer.	the vendor documentation for the type of load balancer you are using (for example, F5 or Radware).

Procedure: Configuring Simple Virtual IP Failover

Configure Resource Manager HA in active standby mode by using Simple Virtual IP failover.

Use this failover method if you do not have multiple NICs in the Resource Manager host.

1. Verify that:
   • The Resource Manager hosts conform to the prerequisites for Linux. See Prerequisites.
   • Management Framework is installed, and fully functional. See the Management Framework 8.5 Deployment Guide.
2. At the Linux host, log in as root.
3. Copy the contents of the /etc/sysconfig/network-scripts/ifcfg-eth0 file to the ifcfg-eth0:1 file.
4. On the active host, modify the lines in the ifcfg-eth0:1 file as follows, replacing <virtual_IP_addr> with the actual virtual IP address:

   IPADDR=<virtual_IP_addr>
   ONBOOT=no
   ONPARENT=no
   DEVICE=eth0:1
   BOOTPROTO=none

   For RHEL 5 Releases only:

5. Prepare the ifup-eth script:
   Copy /etc/sysconfig/network-scripts/ifup-eth file to <RM_Install_Dir>/bin directory.
   1. In the <RM_Install_Dir>/bin/ifup-eth file, comment out lines 266 to 269, as follows:

      # if !arping -q -c 2 -w 3 -D -I ${REALDEVICE} ${IPADDR}; then
      # echo $"Error, some other host already uses address ${IPADDR}."
      # exit 1
      # fi

   2. Enable executable permission, by typing chmod +x <RM_Install_Dir>/bin/ifup-eth, and then press Enter.
6. Prepare the ifup script:
   1. Copy the /etc/sysconfig/network-scripts/ifup script to the <RM_Install_Dir>/bin directory.
   2. In the <RM_Install_Dir>/bin/ifup file, modify lines 145 to 149, as follows:

      OTHERSCRIPT="<RM_Install_Dir>/bin/ifup-eth"
      # if [!-x ${OTHERSCRIPT}]; then
      # OTHERSCRIPT="/etc/sysconfig/network-scripts/ifup-eth"
      # fi

   3. Enable executable permission by typing chmod +x <RM_Install_Dir>/bin/ifup, and then pressing Enter.
   4. Repeat these steps on the standby host.

   For RHEL 4 Releases only:

7. Enable executable permission by typing chmod +x <RM_Install_Dir>/bin/ifup, and then pressing Enter.
8. Repeat Steps 1 to 3, and 7 on the standby host.
9. Modify the INIT and NLB script files. See Procedure: Configuring the INIT and NLB Script Files (Linux).

Procedure: Configuring Bonding Driver Failover

[+] Purpose

To configure Bonding Driver failover on the Resource Manager to achieve High Availability.

[+] Prerequisites

• The Resource Manager hosts conform to the prerequisites for Linux. See Prerequisites.

• Management Framework is installed, and fully functional. See the Framework 8.1 Deployment Guide.

[+] Steps

1. At the Linux host, log in as root.
   Tip

   Genesys recommends that root user perform the steps in this procedure. However, if a non-root user must use this procedure, see Procedure: Enabling a Non-root User to Configure and Run Resource Manager to make that possible.
2. In the /etc/modprobe.conf file, on separate lines add: alias bond0 bonding options bond0 miimon=1000 mode=1
3. In the /etc/sysconfig/network-scripts directory, copy the contents of the ifcfg-eth0 file to the ifcfg-bond0 file.
4. In the ifcfg-bond0 file:
   1. Change DEVICE=eth0 to DEVICE=bond0.
   2. Remove any line that refers to the hardware address for example, HWADDR=.
5. In the ifcfg-eth0 file:
   1. Remove any line that refers to the hardware address for example, HWADDR=.
   2. Remove any line that refers to the IP address for example, IPADDR=.
   3. On a separate line, add MASTER=bond0.
   4. On a separate line, add SLAVE=yes.
6. Repeat Step 5 in the ifcfg-eth1 file.
7. Restart the host computer.
8. After the host has restarted, modify the ifcfg-bond0:1 file, as follows, substituting <virtual_IP_addr> for the actual virtual IP address:

   IPADDR=<virtual_IP_addr>
   ONBOOT=no
   ONPARENT=no
   DEVICE=bond0:1
   BOOTPROTO=none

   For RHEL 5 Releases only:

9. Prepare the ifup-eth script:
   • *Copy /etc/sysconfig/network-scripts/ifup-eth file to <RM_Install_Dir>/bin directory.
   • *In the <RM_Install_Dir>/bin/ifup-eth file, comment out lines 266 to 269, as follows:

   # if !arping -q -c 2 -w 3 -D -I ${REALDEVICE} ${IPADDR}; then
   # echo $"Error, some other host already uses address ${IPADDR}."
   # exit 1
   # fi

   • Enable executable permission, by typing chmod +x <RM_Install_Dir>/bin/ifup-eth, and then press Enter.
10. Prepare the ifup script:
    • Copy the /etc/sysconfig/network-scripts/ifup script to the <RM_Install_Dir>/bin directory.
    • In the <RM_Install_Dir>/bin/ifup file, modify lines 145 to 149, as follows:

    OTHERSCRIPT="<RM_Install_Dir>/bin/ifup-eth"
    # if [!-x ${OTHERSCRIPT}]; then
    # OTHERSCRIPT="/etc/sysconfig/network-scripts/ifup-eth"
    # fi

    • Enable executable permission by typing chmod +x <RM_Install_Dir>/bin/ifup, and then pressing Enter.
11. Repeat Steps 1 to 10 on the standby host in the HA-pair. For RHEL 4 Releases only:
12. Enable executable permission by typing chmod +x <RM_Install_Dir>/bin/ifup, and then pressing Enter.
13. Repeat Steps 1 to 8 and 12 on the standby host in the HA-pair.

[+] Next

• Configure the member IDs and NLB script path in the Resource Manager HA Application. See Procedure: Configuring the Resource Manager HA-Pair, on page 416.

• Modify the INIT and NLB script files. See Procedure: Configuring the INIT and NLB Script Files (Linux).

Procedure: Enabling a Non-root User to Configure and Run Resource Manager

[+] Purpose

Create a user with permission to operate Resource Manager without root access. This user will access the Resource Manager install directory, and the directories /bin/ifup and /sbin/ifdown.

[+] Steps

1. Create a new user gvp-user for operating GVP without the root access. Run the following commands through the root account to give sudo permission for selected commands (<RM Install directory>/bin/ifup, /sbin/ifdown) to this user.
2. Run the following commands in the console of the RHEL machine:
   1. visudo
      This command opens the file.
   2. Beneath the line root ALL= (ALL) ALL, add this line:
      gvp-user ALL= NOPASSWD: <RM Install directory>/bin/ifup, /sbin/ifdown
   3. c. Comment the new line Defaults requiretty.
3. Execute the following commands:
   ...to grant permission to gvp-user to edit/use the necessary files and folders.
   • chown gvp-user /etc/modprobe.conf ...for RHEL 5
   • chown -R gvp_qa /etc/modprobe.d ...for RHEL 6
   • chown -R gvp-user /etc/sysconfig/network-scripts
   • chown -R gvp-user /opt/genesys ...Creates the directory /opt/genesys and grants permission to gvp-user for installing the GVP software.
4. Log on (using ssh) as gvp-user.
5. Configure Resource Manager for High Availability with the Procedure: Configuring the Resource Manager HA-Pair. Once you have completed the Resource Manager High Availability configuration...
6. Modify the following files:
   1. NLB.bat: add sudo to the beginning of lines 10 and 13...
      • line number 10 should read:
        sudo ./ifup $activeIntf
      • line number 13 should read:
        sudo /sbin/ifdown $activeIntf
   2. INIT.sh: add sudo at the beginning of line 6...
      • line number 6 should read:
        sudo /sbin/ifdown $activeIntf

[+] Next

Non-root user, gvp-user, should begin the procedure Procedure: Configuring Bonding Driver Failover at step 2.

Procedure: Configuring the INIT and NLB Script Files (Linux)

Ensure that the mymemberid parameter that is configured in the cluster section of the Resource Manager Application for mymemberid=2 is the same as the configuration in the NLB.bat file for mymemberid=2.

[+] Purpose

To configure the INIT and NLB script files so that each Resource Manager host is assigned a unique member ID in the HA-pair.

[+] Prerequisites

• HA is set up on the Resource Manager hosts. See Procedure: Configuring Simple Virtual IP Failover, or Procedure: Configuring Bonding Driver Failover.

[+] Steps

1. At the Linux host, log in as root. NLB.bat File
2. Follow the direction in the NLB.bat file to update the mymemberids:
   • On the Resource Manager host that is assigned memberID=1 in the <RM_Install_Dir>/bin directory, update the mymemberid=1.
   • On the Resource Manager host that is assigned memberID=2 in the <RM_Install_Dir>/bin directory, update mymemberid=2.
3. Save the changes. INIT.sh File
4. Follow the direction in the INIT.sh file to update the following lines:
   • If you are using Simple Virtual IP Failover activeIntf="eth0:1".
   • If you are using Bonding Driver Failover activeIntf="bond0:1".
5. Save the changes.
   Important

   2
6. Repeat Steps *to 5 on the Resource Manager host that is assigned memberID=2.

[+] Next

• Specify the NICs that you want to monitor. See Procedure: Specifying the NICs for Monitoring (Linux).

Procedure: Specifying the NICs for Monitoring (Linux)

[+] Purpose

To specify the NICs that are to be monitored by the Resource Manager.

[+] Prerequisites

• More than two NICs are configured on the same host, and they are fully functional.

• Two NICs are configured as part of an HA-pair. See Procedure: Configuring Simple Virtual IP Failover or Procedure: Configuring Bonding Driver Failover.

• Assign a unique member ID to each Resource Manager host in the HA-pair. See Procedure: Configuring the INIT and NLB Script Files (Linux), on page 432.

[+] Steps

1. Log in to Genesys Administrator.
2. On the Provisioning tab, select Environment > Applications.
3. Select the Resource Manager Application you want to configure.
4. On the Options tab, scroll to the GVP section. (Create the gvp section if it does not exist.)
5. Add the nic parameter that corresponds nic.ethX parameter.
6. If not configured by default, edit the nic.upvalue parameter as follows:
   nic.upvalue = up
7. If not configured by default, edit the nic.linkattribute parameter as follows:
   nic.linkattribute = MII Status:
8. Edit the nic.eth0 parameter as follows:
   nic.eth0 = "/proc/net/bonding/bond0"
   Tip

   Configure the nic.eth0 parameter value as shown in Step 8 even when Simple Virtual IP failover is used. If the file cannot be read, the NIC status is queried directly by default during NIC detection.
9. Add the nic.eth1 parameter as follows:
   nic.eth1 = "/proc/net/bonding/bond0"
   Important

   2
10. If more than two NICs exist, configure the nics option value to 0 1.
    Tip

    The instructions in Step 8, 9, and 10 are based on the assumption that the chosen network interfaces are numbered 0 and 1. If this configuration does not match the actual interface numbers in your system, change the values accordingly.
11. Save the changes.

[+] Next

• Execute the INIT file on each Resource Manager host. See Start Resource Manager HA-pair in NLB mode.

Creating Alarm Reaction Scripts, Conditions, and Reaction Applications

When an active Resource Manager goes down, but does not stop it's virtual IP, and then the backup Resource Manager becomes active and starts it's virtual IP, the two systems will claim the virtual IP. Therefore, when a system sends an ARP query to determine where the virtual IP can be reached, it might obtain or use the response from the system with the inactive Resource Manager.

To ensure this does not occur, complete each task in the Task Summary: Configuring the Resource Manager to Use Alarm Scripts on both the primary and backup Resource Manager instances.

Tip

Alternatively, you can use a wizard in Genesys Administrator to complete the first two tasks in the Task Summary table. See Procedure: Using the Create New Application Wizard. As you proceed through the wizard, enter the information in the required fields as outlined in the procedures in this section.

Task Summary: Configuring the Resource Manager to Use Alarm Scripts

Objective	Related Procedures and Actions
Create a new Third Party Server template.	Procedure: Creating the Third Party Server Template.
Create two Reaction Applications.	Procedure: Creating the Reaction Applications.
Create and configure two Alarm Reaction scripts.	Procedure: Creating and Configuring the Alarm Reaction Scripts.
Create two Alarm Conditions to send an alarm when either instance of the Resource Manager is stopped.	Procedure: Creating an Alarm Condition for an RM Stopped Intentionally.
Create two Alarm Conditions to send an alarm when either instance of the Resource Manager stops unexpectedly.	Procedure: Creating an Alarm Condition for an RM Stopped Unexpectedly.

Procedure: Creating the Third Party Server Template

[+] Purpose

To create a Third Party Server template to use for the Reaction Applications (created in the next procedure).

[+] Steps

1. Log in to Genesys Administrator.
2. On the Provisioning tab, select Provisioning > Environment > Application Templates.
3. Click New.
4. On the Configuration tab, populate the following fields as shown here (see also Figure 33):
   • Name: Third Party Server
   • Type: Third Party Server
   • Version: 1.0
5. Click Save & Close.

[+] Next

• Create the Reaction Applications. See Procedure: Creating the Reaction Applications.

Procedure: Creating the Reaction Applications

[+] Purpose

To create the Reaction Applications that stops the NIC (the virtual IP interface) on the Resource Manager that is down (intentionally or unintentionally).

[+] Steps

1. In Genesys Administrator, go to Provisioning > Environment > Applications.
2. Click New.
3. On the Configuration tab, in the General section, populate the following fields:
   • Name: stop_pri_VIP or stop_bac_VIP
   • Application template: Third Party Server
4. In the Server Info section, populate the following fields:
   • Host: <Primary RM host object> or <Backup RM host object>
     (Add the name of the primary or backup Resource Manager host object.)
   • Listening Ports: <Port Number>
     (Add a default unused port, typically in the 70xx range.)
   • Working Directory: <RM bin directory>
     (Add the actual Resource Manager bin directory.)
   • Command Line: ./NLB.bat
     For IPTakeover, you must use IPTakeOver.bat
     Command Line=IPTakeOver.bat}}
   • Command Line Arguments: disable 2
     (for stop_pri_VIP) or disable 1 (for stop_bac_VIP)
5. Click Save & Close.

[+] Next

• Create the Alarm Scripts. See Procedure: Creating and Configuring the Alarm Reaction Scripts.

Procedure: Creating and Configuring the Alarm Reaction Scripts

[+] Purpose

To configure the Alarm Reaction scripts that cause the Reaction Applications to be run when the Alarm Reaction script is called.

[+] Steps

1. In Genesys Administrator, go to Provisioning > Environment > Scripts.
2. Click New.
3. On the Configuration tab, in the General section, populate the following fields:
   • Name: pri_rm_not_running or bac_rm_not_running
   • Script Type: Alarm Reaction
4. Click Save & Close.

Configuring Alarm Scripts

5. In the list of Scripts, click (to highlight) the Alarm script you created in Procedure: Creating and Configuring the Alarm Reaction Scripts.
6. In the Tasks pane, click Script Wizard and enter the following in each step of the wizard:
   • Tenant and Name: Select the applicable tenant.
   • Alarm Reaction Type: Start a specified application.
   • Alarm Reaction Details: stop_pri_VIP or stop_bac_VIP (See Procedure: Creating the Reaction Applications.)
7. When the wizard is complete, Click Finish.

[+] Next

• Create the Alarm Conditions. See Procedure: Creating an Alarm Condition for an RM Stopped Intentionally.

Procedure: Creating an Alarm Condition for an RM Stopped Intentionally

[+] Purpose

To create an Alarm condition under which an Alarm script is activated when the Resource Manager is stopped intentionally.

[+] Steps

1. In Genesys Administrator, go to Provisioning > Environment > Alarm Conditions.
2. Click New.
3. On the Configuration tab, in the General section, populate the following fields as shown here:
   • Name: rm_pri_stopped or rm_bac_stoppedINSERT_TEXT
   • Description: primary RM was manually stopped or backup RM was manually stopped
   • Detect Clearance Timeout: 1 (change the default value from 86400)
   • Detect Log Event ID: 5091
   • Detect Selection Mode: Select by Application (from drop-down menu)
   • Detect Application: Enter primary or backup Resource Manager Application object (the actual Resource Managers, not the reaction objects).
4. In the Scripts section, populate the following fields with equivalent data: colspan="2"|Scripts

   Field	Sample Data
   General
   Name	rm_pri_stopped
   Description	Primary RM was manually stopped
   Category	Major (selected from a drop-down list)
   Detect Script	(select from a search field)
   Clearance Timeout	1
   Detect Log Event ID	5091
   Detect Selection mode	Select by Application (from a drop-down list of applications)
   Detect Application Type	GVP Resource Manager (selected from a drop-down list)
   Detect Application	HA_RM_2 (selected from a search field)
   Cancel Log Event ID: 0
   State: Enabled
   Name	pri_rm_not_running
   Script Type	Alarm Reaction
   State	Enabled

   • Click Add in the Reaction Scripts: field to add pri_rm_not_running or bac_rm_not_running. (See Procedure: Creating and Configuring the Alarm Reaction Scripts.)
5. Click Save & Close.

[+] Next

• Provision the Alarm Conditions. See Procedure: Creating an Alarm Condition for an RM Stopped Intentionally

Procedure: Creating an Alarm Condition for an RM Stopped Unexpectedly

[+] Purpose

To create an Alarm condition under which an Alarm script is activated when the Resource Manager is stopped unexpectedly.

[+] Steps

1. In Genesys Administrator, go to Provisioning > Environment > Alarm Conditions.
2. Click New.
3. On the Configuration tab, in the General section, populate the following fields:
   • Name: rm_pri_down or rm_bac_down
   • Description: primary RM stopped unexpectedly or "backup RM stopped unexpectedly"
   • Detect Clearance Timeout: 1 (change the default value from 86400)
   • Detect Log Event ID: 5064
   • Detect Selection Mode: Select by Application (from drop-down menu)
   • Detect Application: Enter primary or backup Resource Manage Application object (the actual Resource Manager instances, not the reaction objects).
4. In the Scripts section, populate the following fields:
   • Click Add in the Reaction Scripts: field to add pri_rm_not_running or bac_rm_not_running. (See Procedure: Creating and Configuring the Alarm Reaction Scripts.)
5. Click Save & Close.