Troubleshooting Genesys Info Mart Jobs

Information about errors that Genesys Info Mart jobs encounter. Some of the errors cause jobs to fail, while others result in incorrectly processed data.
Recommendations for situations in which the extract, transform, and load (ETL) cycle has not run for an extended period of time. These recommendations help you process the backlog of source data in a way that leads to the best ETL performance.
Information about resources for related information, including information about a standby Genesys Info Mart.

Job failures

The jobs that Genesys Info Mart Server launches, or that you execute or schedule, may encounter errors that cause them to fail. Some job failures are caused by an error that the job encounters directly. These are called single-job failures. Because of job interdependencies (described in Job sequencing rules), some job failures are caused by an error that some other job encounters. These are called job-interdependency job failures.

When you use Genesys Info Mart Server to launch jobs, you likely will not see job-interdependency job failures, because Genesys Info Mart Server will not launch a job until all interdependent jobs have completed successfully. If the jobs are run automatically by Scheduler, jobs are recovered automatically in cases of failures that are caused by environment situations. For example, if a database shutdown or Genesys Info Mart Server failure occurs during transformation and then the database or Genesys Info Mart Server is restarted, the next instance of Job_TransformGIM resumes processing from the point where the interrupted instance left off.

Types of errors

Several types of errors can cause a job to fail or to produce incomplete data:

Configuration errors

If Genesys Info Mart Server reports a configuration error, refer to the log messages to identify the issue so that you can correct it.

For additional information on the configuration checking process, including information on when Genesys Info Mart performs the check, what it checks for, and how it responds when it finds errors, see the section about deployment verification in the “Data Processing” chapter of the Genesys Info Mart 8.1 Deployment Guide.

In addition to the items Configuration Checker reviews, other possible configuration errors include the following:

Job_TransformGIM encountered errors in the configuration of either Outbound Contact Server (OCS) record fields in Configuration Database or Outbound Contact–related user data fields in the Info Mart database. In either case, the job does not process the incorrectly configured fields and/or user data. Job_TransformGIM depends on this configuration for Outbound Contact data transformation. Although the job does not fail when these errors are encountered, the resulting data in the Info Mart database would be missing certain information about Outbound Contact–related interactions.

You should monitor the Genesys Info Mart local log for messages that indicate configuration errors and take action to resolve them promptly to avoid incomplete data. Genesys recommends that you use the Solution Control Interface (SCI) to set up an alarm condition. Refer to the Framework 8.x Solution Control Interface Help for information about how to use the Alarm Condition Wizard to create alarm conditions.

Job_TransformGIM encountered missing configuration data during transformation of Configuration details or data from other domains. For more information about how Job_TransformGIM handles configuration data errors, see the error-handling subsection about missing configuration data in the chapter about ETL processing in the Genesys Info Mart Deployment Guide.

In the case of missing configuration data during transformation of data from other domains, you can often resolve the error by forcing Interaction Concentrator (ICON) to perform a resynchronization of the configuration data between the Configuration Database and the IDB from which Genesys Info Mart extracts the configuration history. When the resynchronized configuration data is extracted, the incomplete Genesys Info Mart data that Job_TransformGIM created about the resources is updated.

Database connection errors

The extraction job could not connect to a source database from which it extracts data because the database is not running or the network connection between the Genesys Info Mart Server and the database is down.
The Genesys Info Mart job could not connect to a target database because the database is not running or the network connection between the Genesys Info Mart Server and the database is down.
The Genesys Info Mart job or Genesys Info Mart Server could not connect to a source or target database because of a JDBC Driver class not found: exception. Ensure that the CLASSPATH environment variable has been updated to include the JDBC-specific jar files needed for the appropriate database type. Restart the Genesys Info Mart Server after the classpath is updated.
In an HA configuration, Job_ExtractICON could not connect to one of the HA IDBs because the database is not running or the network connection between the Genesys Info Mart Server and the database is down.

At the start of each extraction job, the Genesys Info Mart Server automatically checks connectivity for each database access point (DAP) to which the Genesys Info Mart Application has a configured connection. If it determines that a configured data source or IDB is not available, Genesys Info Mart logs an error and does not proceed with the extract.

You should always check the logs to obtain additional detailed information about any possible connection errors.

SQL errors

The job encountered a Structured Query Language (SQL) error that caused the failure. For example, there may be insufficient resources, such as memory or physical storage, on the database.
As a special case in deployments that use 3rd Party Media, the transformation job might encounter a unique constraint violation error if you were trying to add media types to the MEDIA_TYPE dimension table at the exact moment that the transformation job was dynamically adding an unknown media type to the MEDIA_TYPE dimension.
If the transformation job fails for this reason, no action is required. During the next ETL cycle, the transformation job will take the appropriate action with regard to the interaction and the associated media type.

For more information, see Preparing the Info Mart Database for 3rd Party Media in the Genesys Info Mart Deployment Guide.
Genesys Info Mart is not able to access a Microsoft SQL database. In order for the connection to be made, Microsoft SQL JDBC driver version 1.1 or higher must be installed.

Genesys Info Mart job error

The job encountered a critical error that caused the failure. For example, there may be insufficient operating system resources or a software defect.
Genesys Info Mart determined that data from an active data source is not available. Genesys Info Mart logs an error, and Job_ExtractICON does not proceed in this situation. For more information, see the section about determining data availability in the chapter about maintenance and other activities in the Genesys Info Mart Deployment Guide.
Job_TransformGIM has been configured to fail when errors are encountered, and the job encountered a data inconsistency for which the applicable interaction-level error policy generated an exception. For more information about the configurable error policy options and the situations that might give rise to data inconsistency errors, see the section about error handling in the chapter about ETL processing in the Genesys Info Mart Deployment Guide.
(Starting with release 8.5.010) Job_MaintainGIM encountered an exception while processing an input JSON file provided by a customer for General Data Protection Regulation (GDPR) "export" or "forget" requests. Log message 55-20172 (GIM_ETL_GDPR_ERROR) logs the failure. In most cases, the job failure is a result of errors in the JSON file formatting. Since compliance with GDPR requests is a legal requirement, Genesys recommends setting an alarm on message 55-20172 so that you can fix the issue timeously. You can use log message 55-31406 (GIM_ETL_GDPR_SUCCESS) to cancel the alarm.

Error recovery

This section provides several recommendations to consider when Genesys Info Mart jobs fail:

General recommendations

When the Genesys Central Logger or Genesys Info Mart Manager indicates that a job failed, the cause of the failure dictates the recovery steps. Messages in the Genesys Info Mart local log or Genesys Central Logger will indicate the error that caused the failure. Correct the cause of the failure before attempting to restart any job.

If a job continues to fail and it takes a long time to resolve the issue, follow the suggestions provided in Recovering from a prolonged ETL outage for the time period that the ETL is not running.

Recovering from a prolonged ETL outage

If certain circumstances (such as a failure of a particular job) prevent you from running ETL for an extended period of time, no special steps are required to process the backlog when normal processing resumes. However, carefully review settings for the options that control transaction size and limit data extraction. They might be set too large for the situation in which you are running normal ETL cycles to catch up a large backlog. In all circumstances, Genesys recommends that you never set the transaction size (extract-data-chunk-size) to more than two hours (7200 seconds).

Recovering from data-source unavailability

Data from a particular data source might become unavailable because of problems with the data source itself, the ICON monitoring it, the DAP that ICON uses to write to the IDB, the DAP that Genesys Info Mart uses to extract from the IDB (the extraction DAP), or any of the connections between these objects.

The extraction algorithm will fail the extraction job when an ICON session from a particular active data source is not available. In an HA environment, this means that the extraction algorithm will fail the extraction job when all of the IDBs from a redundant set are not available; as long as data from the active data source is available through one of the Interaction Concentrator instances in the Genesys Info Mart connections, the extraction job will not fail.

Alternatively, you might be alerted by an alarm on log message 55-20110 that data has been delayed. (For information about the conditions under which the log message is generated, see delayed-data-threshold, as well as the information about delayed data in the chapter about ETL processing in the Genesys Info Mart 8.1 Deployment Guide.)

If Job_ExtractICON fails or Job_TransformGIM is held up because of missing data:

Examine the Genesys Info Mart logs to determine the reason for the failure and identify the data source whose data is no longer available.
Evaluate whether it is worth continuing to extract data from remaining data sources for that data domain, under the risk of permanently losing data from the unavailable data source. For example, if the Genesys Info Mart connection to a non-HA IDB experiences frequent connectivity problems for a certain period of time, there might be little risk that the extraction window will advance past data from the intermittently available data source, if you continue to extract data from the remaining data sources.

If you decide that you cannot risk continuing to extract data from remaining data sources, suspend the ETL schedule by setting the run-scheduler configuration option in the [schedule] section of the Genesys Info Mart Application to false. Resume the ETL schedule when the data availability problem is resolved.
If you make the decision to proceed with the ETL process, consider temporarily excluding the problematic data source from extraction by doing one of the following:
- Temporarily disable the ICON and DAP by clearing the State Enabled check box on the General tab of the ICON and DAP Application objects. This approach is suitable if the delayed data is from a data source that is monitored by a dedicated ICON or if you identify that the problem is with the Genesys Info Mart connection to the ICON or IDB, so that there is no IDB data that would otherwise be available that Genesys Info Mart will skip. Alternatively, you can remove the applicable ICON and extraction DAP from the Genesys Info Mart Application connections.
  Disabling the ICON and DAP, like removing them from the Genesys Info Mart Application connections, does not interfere with their functioning. If you use a shared DAP for ICON and Genesys Info Mart to access the IDB, disabling the DAP does not prevent ICON from continuing to write to the IDB.
  
  Genesys recommends temporarily disabling the ICON and DAP applications, because it is easier and less error-prone than removing and then reconfiguring the connections. Remove the connections when you do not want a data source to be considered part of the deployment for the long term.
- Remove the applicable data-source application from the ICON Application connections. This approach is suitable if the associated ICON monitors more than one data source, particularly if you identify that the problem is with the data source itself, so that suspending ICON monitoring of the unavailable data source does not affect data availability from available data sources.
  Removing the connections or disabling the ICON and DAP applications means that Genesys Info Mart will no longer consider them or the applicable data source to be part of the deployment.
The configuration check will no longer identify the data source as one that is supposed to be active, so Job_ExtractICON will proceed to extract data from available data sources.

Job_TransformGIM will not delay transformation of data from other data sources in the same data domain.

Important
In earlier versions of this document, Genesys recommended disabling the data-source application, so that Genesys Info Mart would no longer consider the data source to be active and, therefore, would not wait for data from it. Genesys no longer recommends this approach, because disabling the data source can interfere with its functioning.

Tip

Extracting data from a limited number of data sources impacts the data quality in the Info Mart database and, therefore, undermines the accuracy of contact center reports. To ensure that the accuracy of reports is not compromised on a permanent basis, reverse the steps to suspend the ETL schedule or to exclude a data source from extraction as soon as the data-source availability issue is resolved.

High availability recommendations

In HA deployments, the IDB redundancy provides an additional layer of failure prevention for the extraction job. Nevertheless, you will experience a failure of Job_ExtractICON in a situation when it cannot access any IDBs in a redundant set.

You might also experience a failure of Job_ExtractICON in a situation when, because of environmental or non–Genesys Info Mart configuration conditions, the extraction job effectively times out. For example, if connection-opening timeouts are very long, the cumulative delay while Genesys Info Mart waits to identify available and unavailable data sources might amount to hours; in these situations, the Genesys Info Mart job scheduler terminates the instance of Job_ExtractICON after one hour.

In these cases, follow the guidelines in Recovering from data-source inavailability to decide whether you want the extraction job to ignore the missing data sources and, hence, compromise the completeness or accuracy of the reporting data.

Standby and disaster recovery

This section describes steps you can take to protect against downtime or data loss:

Genesys Info Mart Server Redundancy—To protect against the Genesys Info Mart Server being unavailable for an extended period of time or to enable quick substitution (for example, for server maintenance), you can deploy a second instance of the Genesys Info Mart Server to act as a standby.

If you deploy a second instance of the Genesys Info Mart Server as a standby for operation against the same Info Mart database, it is important to ensure that only one instance of Genesys Info Mart Server accesses the Info Mart database at a time. In Oracle deployments, a locking mechanism prevents a second instance from connecting to the database by mistake. In non-Oracle deployments, ensure that there is no connection to the Info Mart DAP from the standby Genesys Info Mart application.

Info Mart Database Redundancy—To protect against the Info Mart database being lost, you can also deploy a second instance of the Info Mart database. There are two types of architecture:

Active-active—Two full instances of Genesys Info Mart (Genesys Info Mart Server and Info Mart database) operate in parallel, extracting data from the same IDBs and independently populating their respective Info Mart databases.
Active-standby—Two full instances of Genesys Info Mart are deployed. With one instance of Genesys Info Mart active, you replicate data to the second instance of the Info Mart database.

The active-active topology can be used to provide both disaster-recovery protection and protection against the Info Mart database being unavailable for an extended period of time because of, for example, network connectivity problems between the other Genesys Info Mart Server and Info Mart database hosts; for disaster recovery, the second Genesys Info Mart, at a different site, protects against the Info Mart database being lost because of a site failure. The active-standby topology provides disaster-recovery protection. In Oracle deployments, you can use Oracle GoldenGate to replicate data to the second database instance. The replicated Info Mart database, in combination with a redundant Genesys Info Mart Server application and Info Mart DAP, protects against Genesys Info Mart being unavailable for an extended period of time and minimizes data loss in the event of catastrophic failure of the primary site.

More Information—For more information about configuring redundant Genesys Info Mart Applications and connections, see the section about a standby Genesys Info Mart Server in the chapter about supported topologies in the Genesys Info Mart Deployment Guide. For more information about the architecture, deployment, and operation of redundant Genesys Info Marts in the active-active and active-standby configurations, including information about GoldenGate setup and procedures for disaster recovery, see the Genesys Info Mart Business Continuity Deployment Guide.

Resources to consult for additional problem-solving information

Consult the following resources for information that will help you resolve problems:

Genesys Info Mart Deployment Guide —Contains information to help you tune performance parameters for your Info Mart database and correct errors in Genesys Info Mart configuration parameters. This guide also contains information that you can use to configure the ICON applications that populate the databases from which you want to extract data.
Genesys Info Mart Operations Guide (this document)—Contains information that you can use to correct errors in the job configuration. This guide also contains information about how to execute or schedule jobs and job interdependencies.
Genesys Central Logger—Contains events that are logged by jobs. The logs indicate configuration errors, when a job begins, when a job ends, and whether it ends successfully or unsuccessfully. When a job fails, use one of the following methods to obtain detailed information about the failure.
- Use a log file in the Genesys Info Mart Server’s local directory to view log messages.
- Use SCI to view log messages that are received by Message Server, provided that the Genesys Info Mart application has been configured with a connection to the Genesys Message Server.
Genesys Info Mart local log—Contains detailed events that are logged to the local log file on the Genesys Info Mart Server host by Genesys Info Mart Server and some of the jobs. When a job fails, view these logs to obtain detailed information about the failure.
Voice of Process—Provides administrators with information related to the processing history of jobs. You can use this information to quickly check the state of all jobs, to track the data extraction or data transformation progress of each data extraction or data transformation job cycle, to detect new and/or updated data for subsequent aggregation or other processing, and to diagnose ETL problems. See About Voice of Process for more information.
Publications for your database—Contain information for your specific RDBMS about database connections, SQL errors, configuration parameter settings that affect database performance, and the usage of operating system resources on the database server.