Revision as of 17:27, February 26, 2013 by Tanyai (talk | contribs)
Jump to: navigation, search

Disaster Recovery

Disaster Recovery is intended to ensure that an enterprise continues to operate in the event of a Site Failure.

Genesys Info Mart supports the Site Failure scenario as described in this section. The steps that you need to take in the event of disaster and to restore normal operations are also covered.

Site Failure

In the event of the catastrophic failure of one of the sites--in other words, a failure in which all Genesys components on that site become unavailable, including locally paired HA servers--site redundancy is used to provide ongoing reporting on contact center activity. The recommended SIP Business Continuity architecture also enables ongoing support for all logged in agents.

GIMArchitectureBCSiteFailure.png


All contact center activity is redirected to Site 2. Interaction Concentrators at Site 2 continue to store data about Site 2 activity, which now represents the entire contact center. You must bring the standby Genesys Info Mart instance at Site 2 into service and configure it to write data into the Info Mart database at Site 2. You must stop Oracle GoldenGate components at Site 2 because the replication source database is absent. You also need to bring a standby GI2 instance at Site 2 into service to provide historical reports for all users.

As a result of the disaster, some reporting data may be lost. See Potential Data Loss.

Disaster Recovery Procedure

In the case that Site 1 fails, perform the following Disaster Recovery procedure.

  1. Stop the Oracle GoldenGate components at Site 2 to account for the fact that the replication source is no longer available.
  2. Adjust connections configuration for Genesys Info Mart Application at Site 2 as follows:
    • In the event that non-extracted data from Site 1 is available in redundant IDBs located at Site 2, keep connections to the ICON Applications at Site 2 and the Database Access Point (DAP) Applications that provide access to these redundant IDBs.
    • Check the ICON Applications that collect data from data sources at Site 2.
    • Add the ICON Applications at Site 2 to the connections if necessary.
    • Check the DAP Applications that provide access to the IDBs from which Genesys Info Mart at Site 2 can extract Site 2 data.
    • Add the DAP Application that provides access to the Info Mart database at Site 2.
    • If a dedicated DAP is configured for the Genesys Info Mart Administration Console at Site 2 to access the Info Mart database, add this DAP to connections.
    • Revise any other connections as necessary for the Site 2-only operation.
  3. Install Genesys Info Mart Administration Console at Site 2, if not already installed. For instructions, refer to the Genesys Info Mart 8.1 Deployment Guide.
  4. Start Genesys Info Mart server at Site 2.
  5. Review Genesys Info Mart logs to verify that the configuration is correct.
  6. For the ETL to quickly extract any remaining Site 1 data and redirect the extraction process to the Site 2 data exclusively, temporarily lower the value of the extract-data-stuck-threshold option to 1 or 2 hours.
  7. After the time passes that is slightly greater than the original value of the extract-data-stuck-threshold option, restore the original option value.
  8. In a deployment with GI2 or RAA, re-aggregate the data from the last 24 hours before the site failure until the current time.
    To re-aggregate the data:
    1. Run the following command from the Genesys Info Mart server root directory:
      java -jar .\agg\GIMAgg.jar -user=<Info Mart user> -pass=<Info Mart password> -jdbcurl=jdbc:oracle:thin:@<Oracle host>:1521/<Info Mart Oracle Service Name> -insertPendingAggRaw ALLTENANTS:ALLSETS:<Unix epoh time 24 hours ago>:<current Unix epoh time>
    2. Run Genesys Info Mart with RAA enabled as usual.


    Note: The 24 hours period allows to ensure that all pending notifications are accounted for, assuming that both Genesys Info Mart and RAA were operating successfully before the site failure.


Procedure to Return to Normal Operation

Once a replacement site with required infrastructure is set up after the disaster (or the original site is operational again), the contact center can return to normal, two-site operation.

As explained in Operation Modes, you have to decide which site will host the active Genesys Info Mart and how you transfer a copy of the Info Mart database to the replacement site. two options to re-establish Genesys Info Mart redundancy and replication.

  • (Option 1) With Site 3 hosting the active Genesys Info Mart, you have to set up Oracle GoldenGate at Site 3 and configure it as a replication source. You can reuse existing Oracle GoldenGate configuration at Site 2, with potential modifications (if Oracle GoldenGate replication is used to accomplish database synchronization for the new site).
  • (Option 2) With Site 2 hosting the active Genesys Info Mart, you have to set up Oracle GoldenGate at Site 3 and configure it as a replication target. You also have to reconfigure Oracle GoldenGate at Site 2 to serve as a replication source.

To establish database replication between the new active and standby sites, with either of the above options, you must transfer a copy of the Info Mart database from the surviving site to the new site. If it is unacceptable to stop Genesys Info Mart server for a long period of time that transfer of a large database may take, and if you choose Option 1, synchronization of Info Mart databases is an additional task that is required before you can start normal replication. You can use Oracle GoldenGate for database synchronization.

Genesys recommends returning Genesys Info Mart at Site 2 to standby operation and making the Info Mart database at Site 2 the target of replication once again (Option 1). The replacement site (Site 3) is set up to host the active Genesys Info Mart and the replication source database. Because requirements for Oracle and GoldenGate configuration are different for source and target databases, this strategy leverages the existing configuration at Site 2 without shutting down Oracle and is, therefore, less complex. The instructions below describe the steps for Option 1.

As part of the procedure, you must synchronize the Info Mart databases between the two sites. It is assumed that Genesys Info Mart is shut down for the time that is required for database synchronization. If the Info Mart database is very large, see Synchronizing Data During Runtime below.

The instructions below describe the steps to make Genesys Info Mart active at Site 3 (Option 1). Additional steps are required for synchronization of large Info Mart databases if you run Genesys Info Mart at Site 2 during the database transfer from Site 2 to Site 3. If you decide to use Oracle GoldenGate functionality for database synchronization, you have to run Oracle GoldenGate in reverse direction first (from Site 2 to Site 3). This means that Oracle GoldenGate at both sites needs to be configured for replication in both directions. After the Info Mart databases are synchronized, you must switch the replication direction to normal (from Site 3 to Site 2) at both sites.

Note: The steps for making Genesys Info Mart active at Site 2 (Option 2) differ slightly from the procedure described below.

To restore two-site operations:

  1. Deploy Genesys Info Mart server application and Genesys Info Mart Administration Console at Site 3.
    Notes:
    • Consider reusing the Application configuration object that was created for Genesys Info Mart at the failed site.
    • Skip the deployment of the Info Mart database because you will copy the database from Site 2 later.
  2. Deploy Interaction Concentrators to collect reporting data from data sources at Site 3.
    Notes:
    • Deploy a redundant pair of ICON server applications at Site 3 per data source.
    • Consider reusing the Application configuration objects that were created for ICONs servers and ICON DAPs at the failed site.
    • For redundant instances, deploy one of the IDBs at Site 3 and the other, at Site 2.
  3. Update the connections of the Genesys Info Mart server application that you plan to use during the site switchover to add all the ICON applications and the DAP applications that provide access to IDBs at both sites.
  4. Install and configure Oracle GoldenGate at Site 3. Follow instructions in Setting Up Replication, performing only the steps required for the replication source.
  5. Clean up the Oracle GoldenGate working folders at Site 2, such as dirchk, dirdat, dirrpt, and TRACE.
  6. Drop and re-create the Oracle GoldenGate schema at Site 2.
  7. (For deployments with a large Info Mart database) If you plan to use Oracle GoldenGate replication in reverse direction for data synchronization immediately following the database transfer, configure Oracle GoldenGate for the replication in both directions (as source and target) at each site.
  8. Once Site 3 becomes active, If it is acceptable to stop Genesys Info Mart at Site 2 for the time required for database transfer, do this now.
    7. To prevent accidental writing to the Info Mart database at Site 2, remove the Info Mart DAP object from the connections of the Genesys Info Mart Application at Site 2.
  9. Copy the Info Mart database from Site 2 to Site 3. To do so, use the Export/Import RDBMS functionality.
    Warning! To avoid data inconsistency in the exported Info Mart database, Genesys recommends that you do not run Genesys Info Mart during the Export procedure. You can run Genesys Info Mart at Site 2 after Export is completed and while the database copy is being transferred to the new site. In this case, you must synchronize the Info Mart databases at Site 2 and Site 3 before continuing with remaining steps.
  10. Configure the DAP Application to provide access to the Info Mart database at Site 3.
  11. Configure all connections for the new Genesys Info Mart Application at Site 3 as necessary.
  12. Start Replicat at Site 2 (the target database) from the new value of current_scn (retrieved from the source database) as described in Setting Up Replication.
  13. Start Extract and DataPump at Site 3 (the source database) as described in Setting Up Replication.
  14. Stop Genesys Info Mart at Site 2 if it is currently running.
  15. Make sure the Info Mart databases at Site 2 and Site 3 are completely in sync.
  16. Start Genesys Info Mart at Site 3 and run ETL jobs.
  17. Check the Genesys Info Mart server logs to verify that configuration is correct and deployment is successful.
  18. Verify that replication runs as expected at Site 3 (new source).
  19. Check the Oracle GoldenGate logs and the Info Mart database at Site 2 to verify that the data is replicated to the target database as expected.
  20. To prevent accidental writing to the Info Mart database at Site 2, remove the Info Mart DAP object from the connections of the Genesys Info Mart Application at Site 2.
  21. In a deployment with GI2 or RAA, re-aggregate the data from the last 24 hours before the site failure until the current time.
    To re-aggregate the data:
    1. Run the following command from the Genesys Info Mart server root directory:
      java -jar .\agg\GIMAgg.jar -user=<Info Mart user> -pass=<Info Mart password> -jdbcurl=jdbc:oracle:thin:@<Oracle host>:1521/<Info Mart Oracle Service Name> -insertPendingAggRaw ALLTENANTS:ALLSETS:<Unix epoh time 24 hours ago>:<current Unix epoh time>
    2. Run Genesys Info Mart with RAA enabled as usual.


    Note: The 24 hours period allows to ensure that all pending notifications are accounted for, assuming that both Genesys Info Mart and RAA were operating successfully before the site failure.

Synchronizing Data During Runtime

The Info Mart database in a large environment grows over time to a significant size. Copying such a large database can take a significant amount of time. If the Info Mart database in your environment is very large, evaluate whether it is acceptable to have Genesys Info Mart server shut down while the database Export/Import runs.

If it is unacceptable to stop Genesys Info Mart server for a long period of time, the Procedure to Return to Normal Operation differs in the following ways:

  • Do not stop Genesys Info Mart at Site 2 until you are ready to start Genesys Info Mart at Site 3.
  • Continue to extract data from all available data sources during the site switchover.
    In particular, once you deploy new ICON servers at Site 3 and initialize their respective IDBs, configure Genesys Info Mart at Site 2 to extract data from new IDBs. To do so, add the new ICON Applications and the DAP Application that provides access to the new IDBs, to the connections of Genesys Info Mart Application at Site 2.


Use the following recommendations to export the Info Mart database during Genesys Info Mart runtime.

  • Keep REDO logs for the duration of the export and use Oracle GG replication to transfer the data that has been collected during the database export.

  • Perform the Export/Import incrementally, step by step:

  1. First time, do this for the entire database.
  2. Second time, do this for the time period of the first export.
  3. Finally, use Oracle GoldenGate replication functionality to transfer the final records that were collected during the time of the second export.
Comments or questions about this documentation? Contact us for support!