Data Migration from iWD 8.5.1x to iWD 9.0
There are two different options for data migration:
- Migrating GTL History Events—transfer GTL history data to iWD Manager 9.0.
- Migrating Data Mart History Events—transfer iWD task data to iWD Data Mart 9.0.
- Data migration is not a mandatory step. Depending on your needs you can choose one option, both or none.
- The migration process might be intensive and time consuming.
Please also note that depending on the selected iWD History Node configuration, IF:
- The option [iWD]/process-gtl = false, there is no point in migrating GTL History Events.
- The option [iWD]/process-dm = false, there is no point in migrating Data Mart History Events.
If you cannot avoid the GTL History migration, please consider using taskEventLimit option to reduce the amount of data to be migrated.
Migrating GTL History Events
The purpose of this migration is to transfer task history data to iWD Manager 9.0.
You can find Task History by clicking the History tab in the GTL view. It displays all the task attributes’ values and the task history events.
Limitations
Task history migration is supported for tasks that are still in the Interaction Server database.
Prerequisites
- Installing iWD History Node is done
- iWD History Node Configuration is done
- Stop all iWD 8.5 components.
- iWD Manager's Tomcat.
- iWD Runtime Node.
- iWD Web.
- Interaction Server.
Procedure
You can migrate GTL history events from the Event Log database to the History Node database by using History Node's built-in mechanism, which can be run with the following command:
On Linux
./iwd_history.sh -host CFG_SRV_HOST -port CFG_SRV_PORT -app HN_APP events-migrate <yaml-config-filename>On Windows
JavaServerStarter.exe -host CFG_SRV_HOST -port CFG_SRV_PORT -app HN_APP events-migrate <yaml-config-filename>where <yaml-config-filename> is the path to the YAML configuration file providing connection settings to the old environment.
- No History Node applications are running.
- If any History Node applications are running, they must have history events removal disabled (that is, the stop-processing option must be set to keep)
- The GTL History Events Migration mechanism only migrates events into the iWD Manager database and doesn't support migration to the Data Mart database.
Prepare YAML Configuration File
- Please find the configuration template file according to your database vendor:
- Template path
${INSTALLATION DIRECTORY}/config/template_iwd_events_migrate_<DB-VENDOR>.yaml.template #:
- Copy and rename.
- Edit the template and set the following parameters:
- solutionId: <solution-id>
- taskEventLimit: <number>
- interaction-server:
- url: <url>
- user: <user>
- password: <password>
- eventlog:
- url: <url>
- user: <user>
- password: <password>
Configuration Options logging - Defines History Node logging configuration. solutionId - The runtime id of the solution for which events will be migrated. Solution with this id must be configured in Configuration Server. taskEventLimit - Allows to specify the maximum number of events to be migrated for one iWD task. If a task consists of more events than specified, only the last of them will be migrated. It might be helpful for stability and performance reasons. All the tasks with skipped events will be displayed in the migration log. possible values: 0 - there is no limits (default) 1 to 2147483647 - the maximum number of events NOTE!: iWD Manager cannot show more than 5000 events per one interaction. If you don't have any other specific needs then 5000 is recommended value. interaction-server - Points to the Interaction Server database. eventlog - Points to the Event Log database. Working Command Line with Output Example Output Example INFO [2020-06-22 17:03:02,104] com.genesyslab.iwd.eventlog.EventLogEventsMigrator: #50011|Command description: Migrate events from Event Log db to the History Node INFO [2020-06-22 17:03:10,229] com.genesyslab.iwd.eventlog.EventLogEventsMigrator: #50012|Connecting to the Interaction Server database INFO [2020-06-22 17:03:11,979] com.genesyslab.iwd.eventlog.EventLogEventsMigrator: #50013|Events for 74480 interaction(s) will be migrated INFO [2020-06-22 17:03:11,979] com.genesyslab.iwd.eventlog.EventLogEventsMigrator: #50014|Connecting to the Event Log database WARN [2020-06-22 17:03:12,925] com.genesyslab.iwd.eventlog.EventLogEventsMigrator: #50071|Events limit exceeded, interaction: 01T6XTJXMYTMNEFR INFO [2020-06-22 17:03:54,544] com.genesyslab.iwd.eventlog.EventLogEventsMigrator: #50016|Migration progress: 7448/74480 ... INFO [2020-06-22 17:08:14,220] com.genesyslab.iwd.eventlog.EventLogEventsMigrator: #50016|Migration progress: 67032/74480 WARN [2020-06-22 17:08:44,247] com.genesyslab.iwd.eventlog.EventLogEventsMigrator: #50071|Events limit exceeded, interaction: 02RHK42XSWDA4009 INFO [2020-06-22 17:08:45,247] com.genesyslab.iwd.eventlog.EventLogEventsMigrator: #50016|Migration progress: 74480/74480 INFO [2020-06-22 17:08:45,247] com.genesyslab.iwd.eventlog.EventLogEventsMigrator: #50015|Migration finished INFO [2020-06-22 17:08:45,263] com.genesyslab.iwd.history.HistoryApplication: #50047|Shutdown hook triggered INFO [2020-06-22 17:08:45,263] com.genesyslab.iwd.history.HistoryApplication: #50048|Shutdown hook completed info
Message #50071 shows interactions with skipped events due to specified taskEventLimit option.
Migrating iWD Data Mart History Events The purpose of this migration is transfer the current tasks data to the iWD Data Mart 9.0. This way you will be able to see that data in the iWD 9.0 reporting with some limitations <limitations as link here>.
From technical standpoint the procedure consists of two steps:
Migration from EventLog DB to iWD History Node Loading the data from iWD History Node to iWD DataMart Limitations Only not finished tasks are migrated (backlog) The Fact Tables and the Aggregate Tables are not populated with already finished tasks. The whole task event history is not migrated included Agent's activity events: ASSIGN, FINISH, FINISH_RETURN, TRANSFER A task "snapshot" or the latest task state - migrated as NEW excluded All other event types not listed as included (the list of event types: TASK_EVENT_TYPE) The following metrics are not calculated Held Activity TASK_FACT.TOTAL_HELD_TIME_SEC TASK_WORK_FACT.HELD_TIME_SEC Queue Metrics
TASK_QUEUE_FACT.ENTERED_TASK_COUNT TASK_QUEUE_FACT.EXITED_TASK_COUNT
Step 1: Migration from EventLog DB to iWD History Node Prerequisites Installing iWD History Node is done iWD History Node Configuration is done Stop all iWD 8.5 components. iWD Manager's Tomcat.
iWD Runtime Node.
iWD Web.
Interaction Server.
Warning
If the Interaction Server is not stopped, all iWD tasks activity during the migration process will not be recorded.
Procedure You can migrate DataMart history events from the Event Log database to the History Node database by using History Node's built-in mechanism, which can be run with the following command:
On Linux: startup command linux ./iwd_history.sh -host CFG_SRV_HOST -port CFG_SRV_PORT -app HN_APP datamart-migrate <yaml-config-filename> On Windows: startup command windows JavaServerStarter.exe -host CFG_SRV_HOST -port CFG_SRV_PORT -app HN_APP datamart-migrate <yaml-config-filename> Where <yaml-config-filename> is the path to the YAML configuration file providing connection settings to the old environment.
Prepare YAML Configuration File Please find the configuration template file according to your database vendor: template path ${INSTALLATION DIRECTORY}/config/template_iwd_datamart_migrate_<DB-VENDOR>.yaml.template Copy and rename Please modify the following options:
solutionID: <solution-id>
fetchSize: 100
customAttributes: - <attribute1> - ... - <attribute10>
interaction-server:
<url>
<user>
<password>
eventlog:
<url>
<user>
<password>
Warning
The customAttributes section should be set properly, otherwise they won't be migrated.
Configuration Options logging - Defines History Node logging configuration. solutionId - The runtime id of the solution for which events will be migrated. Solution with this id must be configured in Configuration Server. fetchSize - Determines how many interactions are processed per batch. This option in combination with processor-threads could be useful for performance and stability tuning. possible values: 1 to 2147483647 (100 by default) customAttributes - Defines a list of the custom attributes. NOTE! : You should provide the full list of Task Attributes from iWD 8.5, the attributes not listed in this field will not be migrated. interaction-server - Points to the Interaction Server database. eventlog - Points to the Event Log database. Working Command Line with Output Example Output example INFO [2020-06-10 16:40:22,668] com.genesyslab.iwd.eventlog.EventLogData MartMigrator: #50011|Command description: Migrate events from Event Log db to the Data Mart INFO [2020-06-10 16:40:32,356] com.genesyslab.iwd.eventlog.EventLogData MartMigrator: #50012|Connecting to the Interaction Server database INFO [2020-06-10 16:40:33,496] com.genesyslab.iwd.eventlog.EventLogData MartMigrator: #50013|Events for 71320 interaction(s) will be migrated INFO [2020-06-10 16:40:33,496] com.genesyslab.iwd.eventlog.EventLogData MartMigrator: #50014|Connecting to the Event Log database INFO [2020-06-10 16:40:39,584] com.genesyslab.iwd.eventlog.EventLogData MartMigrator: #50016|Migration progress: 10/71320 ... INFO [2020-06-10 17:42:29,923] com.genesyslab.iwd.eventlog.EventLogData MartMigrator: #50016|Migration progress: 71320/71320 INFO [2020-06-10 17:42:29,923] com.genesyslab.iwd.eventlog.EventLogData MartMigrator: #50015|Migration finished Info
Message '#50016|Migration progress' shows the current migration progress.
In case of any errors they will be logged. For more detailed information you can use DEBUG or TRACE log level.
Step 2: Loading the data from iWD History Node to iWD DataMart Prerequisites Step 1: Migration from EventLog DB to iWD History Node<link to the paragraph here> has been completed. The Installing Runtime Node is done. The iWD Runtime Node Options are configured. The IWD Aggregate Plugins are configured. Procedure Configure the specific migration options
Open the iWD Data Mart Application Properties: Configuration → iWD Datamart Properties → options
Add a new option settings\etl-start-date Description: the earliest date for which iWD Data Mart initialize time data in a new deployment. Possible values: date in format "yyyy-MM-dd HH:mm", like "2015-10-26 00:00" Default value: the beginning of the current year. NOTE!: Perform the following query to the Interaction Server Database in order to get known the proper value.
SELECT min(submitted_at) as ETL_START_DATE FROM interactions Add a new option settings\aggregate-start-date
Description: the earliest date for which iWD Data Mart aggregates data in a new deployment. Possible values: date in format "yyyy-MM-dd HH:mm", like "2015-10-26 00:00" should be greater than etl-start-date Default value: today - 30 days
Warning
If the etl-start-date or aggregate-start-date options are not set (or set inappropriately), it may cause "orphaned" rows in facts and aggregates. "Orphaned" rows means that such records don't have a link to the DATE_TIME dimension and cannot be selected or filtered by date.
Add a new option schedules\manual-mode = true
Description: It switches off a scheduled mode and allows to start ETL jobs manually Start the Runtime Node 9.0 Open Data Mart Dashboard—A real-time view of the status of iWD Services. Run ETL jobs manually in the following pattern: Run Initialize job Run Load Config Run Load Intraday Job Run Aggregate Intraday Job NOTE! Aggregate Intraday Job can process only 1 DAY of data per run. In order to catch up faster you should repeat this job as many times as needed until all the data is aggregated. Run Load Historical Job Run Aggregate Historical Job Make sure that all the data was loaded The iWD History Node datamart table is empty
SELECT count(*) cnt_rows FROM <HISTORY_NODE_DB>.datamart The iWD Data Mart loaded data without fails
SELECT * FROM <DATAMART_DB>.etl_audit WHERE STATUS in ('IN PROGRESS', 'FAILED') Rollback the specific migration options
remove settings\etl-start-date remove settings\aggregate-start-date remove schedules\manual-mode