Deployment Procedures

Preparing Engage Configuration for the EX integration

1. Review Folder Filtering rules and how they are applied to the configuration objects of different types.
   • General approach to apply the folder filtering is to create one or multiple mapping folders throughout the existing configuration folder tree for a particular configuration object type (for example, Skills) and then move the configuration objects selected for the EX integration into the newly created mapping folder.
2. Select a folder name to be used for the Engage configuration object selection. Name EX_EVOLUTION is used in this section as an example.
3. Identify a routing domain to map to the EX Org
   • In this context Engage Routing Domain refers to a set of configuration objects, which create isolated or semi-isolated routing environment. Line of Business (LOB) can be an example of such environment. One LOB can be responsible for processing inbound calls landing on a set of toll free numbers of a contact center. In the Engage contact center such LOB would be configured with a set of dedicated resources: RP and VQ DNs, skills, agents, agent groups, and business attributes. Those are main configuration object, which define the routing domain of a selected LOB.
   • Usually, it is not possible to find a fully isolated LOB in the contact center. It is recommended to start with a small routing domain, observe mapping results for all three injected data types (configuration, agent states, and interactions), and then gradually increase the routing domain by adding more Engage configuration objects to it with the goal of injecting all Engage contact center configuration and activity into the EX Org.
   • Take current EXEC limitations into account while defining the routing domain. For example, consider selecting voice agents serving inbound calls because currently EXEC doesn't support digital interactions. Also, outbound campaign calls cannot be injected into the EX Org. So, if digital, blended, or voice outbound agents are selected their workload wouldn't be properly represented in the EX Org through the injected data, which may impact the quality of the WFM and Gamification services.
4. Skills:
   • Create one or multiple EX_EVOLUTION folders at the existing Skills configuration folder tree in CME and move the Skills selected for the EX integration to the newly created mapping folder(s).
5. Persons:
   • Create one or multiple EX_EVOLUTION folders at the existing Persons configuration folder tree in CME and move the Persons selected for the EX integration to the newly created mapping folder(s).
6. Extension DNs
   1. Extension DNs are not mapped to the EX Org but EXCS reads the DNs of this type and stores them into Redis. Those DNs are used by the EXCP for the call event processing.
   2. All Extension DNs used to log in the Persons selected for the EX integration at the previous step should also become a part of the EX integration context.
   3. Review the EXCP Deployment Topologies section and select a SIP Server, which processes contact center voice calls and is panned to be used as a source for the interaction event injection to the EX Org, and identify a Switch this SIP Server uses.
   4. Create one or multiple EX_EVOLUTION folders at the existing DNs configuration unit under the selected Switch and move the Extension DNs selected for the EX integration to the newly created mapping folder(s).
7. Queues:
   1. EXCS can convert four types of the Engage DNs into the ACD queues in GC EX Org: ACD Queue, Routing Queue, Virtual Queue (VQ), and Routing Point (RP). However, the only supported call distribution model from a queue to an agent is the one where call lands on RP DN, then it is placed on a VQ DN for an agent selection, and then is distributed to a selected agent. From this perspective it is recommended to map Engage queue DNs of only two types to the EX Org:
      1. RP: select all RP DNs, which belong to a routing domain being mapped
      2. VQ: select only routing VQ DNs, which are used by the URS for target selection, do not map reporting VQ DNs
   2. Select a SIP Server, which processes contact center voice calls and is panned to be used as a source for the interaction event injection to the EX Org, and identify a Switch this SIP Server uses.
   3. Create one or multiple EX_EVOLUTION folders at the existing DNs configuration unit under the selected Switch and move the Extension DNs selected for the EX integration to the newly created mapping folder(s).
   4. Review the EXCP Deployment Topologies section and select a SIP Server, which processes contact center voice calls and is panned to be used as a source for the interaction event injection to the EX Org, and identify a Switch this SIP Server uses.
   5. Create one or multiple EX_EVOLUTION folders at the existing DNs configuration unit under the selected Switch and move the RP and VQ DNs selected for the EX integration to the newly created mapping folder(s).
8. Agent Groups
   • Create a folder EX_EVOLUTION at the Agent Group configuration.
   • Identify agent groups (AG) and virtual agent groups (VAG) used for routing or reporting by the Persons selected to be mapped to the EX Org.
   • Link selected AGs and VAGs to their corresponding queue DNs by applying the following configuration to the properties of these groups:
     • Annex / hybrid / WFM_queue_number = <ROUTING_VQ_DN_NAME>
9. Supervisors
   • Engage Supervisors should be marked as shown below to get a Supervisor role assigned in the EX Org:
     • Person has 'Is Agent' checked
     • Person has Annex/htcc/roles list containing a 'Supervisor' as one of its elements
   • If customers use WWE/GWS, then their supervisors should already be configured like this. In other cases this configuration should be added.
10. Review all objects of all types moved to the EX_EVOLUTION folders for the case-sensitivity conflicts. Refer to the Case-Sensitivity of Object Names section. Resolve those conflicts if found.

Deploying Apache WebDAV Server for Recording Injection

1. Deploy WebServer that supports the WebDAV module and ensure that the WebDAV module is installed and enabled as well.
2. Create the directory to keep the recording files, and provide the required permission for the webserver user to access those recording files.
3. Create the user and password and configure it for authentication of the recording folder.
4. Open the firewall for the port used by the WebDAV server.

The sample installation and configuration for the Apache2 WebDAV server is described below:

1. Install Apache Web Server. The installation process for WebDAV Server on an Ubuntu machine can be found at How to install Apache2 page.
2. Check if the DAV module is installed and enabled in the machine.
3. Edit the /etc/apache2/apache2.conf file, and append the following to the end of the file:

   Alias /recordings /mnt/recordings
   <Directory /mnt/recordings>
      Options Indexes MultiViews FollowSymLinks
      EnableSendfile off
      AllowOverride None
      Order allow,deny
      allow from all
   </Directory>
   <Location "/recordings">
      DAV On
      AuthType Basic
      AuthName "user"
      AuthUserFile /var/www/htpasswd
      Require valid-user
   </Location>

4. Open the firewall. Ensure that the default incoming port 80 is open.
5. Create the directory to keep the recording files, and set the permission to Apache, using the following commands

   mkdir /mnt/recordings
   chown www-data:www-data /mnt/recordings

6. Create an Apache HTTP Server User for httpd, and configure the password. The following example creates a user called user:

   htpasswd -cm /var/www/htpasswd user

7. Configure the Apache HTTP server start on boot up (and start it now) using the following commands:

   sudo systemctl enable apache2

8. Test the Apache HTTP Server installation:
   • Upload a sample hello.world file to the Apache HTTP Server using the following command:

     curl -T hello.world -u user:password http://<WEBDAV_URL>/hello.world  // WEBDAV_URL: <WEBDAV_VM_ADDRESS>:<WEBDAV_PORT>/recordings

   • Using a browser, open the http://<WEBDAV_URL>/hello.world URL. The browser will request user credentials.

HTTP Load Balancing Configurations needed for Recording Injection

1. Deploy HTTP Load Balancer in a different VM from the EXEC service VMs.
2. Configure the upstream instances for EXCP and EXRP that will be used for load balancing the request.
3. Configure the port that the HTTP Load Balancer will listen for both EXCP and EXRP requests.

The sample installation and configuration for the NGNIX HTTP Load Balancer is described below:

1. Install NGNIX Proxy in a different VM which is accessible to EXEC VM. (https://nginx.org/en/docs/install.html).
2. Edit the nginx-upstreams.conf under the /etc/nginx/ directory in NGINX VM.

   upstream excp {
       server <EXCP_1_VM_HOST_PAIR_IPADDR1>:<EXCP_1_PORT> max_fails=0;
       server <EXCP_1_VM_HOST_PAIR_IPADDR2>:<EXCP_1_PORT> max_fails=0;
   }
   upstream exrp {
       server <EXRP_VM_HOST_IPADDR1>:<EXRP_PORT> max_fails=0;
       server <EXRP_VM_HOST_IPADDR2>:<EXRP_PORT> max_fails=0;
       server <EXRP_VM_HOST_IPADDR3>:<EXRP_PORT> max_fails=0;
   }

3. Edit the nginx.conf under /etc/nginx/ directory in NGINX VM.

   events {
       worker_connections 1000;
   }
     
   http {
       include nginx-upstreams.conf;
     
       # conversation-provider
       server {
           listen <HTTP_LB_EXCP_PORT>; #eg: listen 3650
     
           location / {
               proxy_pass http://excp/;
               proxy_next_upstream_tries 3;
           }
       }
     
       # recording-provider
       server {
           listen <HTTP_LB_EXRP_PORT>; #eg: listen 8080
     
           location / {
               proxy_pass http://exrp/;
               proxy_next_upstream_tries 3;
           }
       }
   }

Configuring OAuth Client in the GC Org

EXCS requires an OAuth client with grant type Client Credentials created in the EX-Org to connect to GC. Create a new OAuth client and assign the EX Integration role to the OAuth client. Refer to Create an OAuth client for more information.

Configuring hybrid_integration transaction in CME

EX Org parameters required for the hybrid_integration transaction configuration

The following configuration parameters should be fetched from EX Org to provision the hybrid_integration transaction in the Engage CME. Configuration Alias is used to refer to the value of the EX Org parameter.

Configuration Alias	EX Org Parameter Path	Parameter Description
CFG_ALIAS_EX_ORG_SHORT_NAME	Organization Setting / Organization Details / Short Name	Short name of the EX Org
CFG_ALIAS_EX_ORG_ID	Organization Setting / Organization Details / Advanced / Organization ID	EX Org Organization ID
CFG_ALIAS_EX_ORG_OAUTH_CLIENT_ID	Integrations / OAuth / <OAUTH_CLIENT_NAME> / Client Details / Client ID	OAuth client ID
CFG_ALIAS_EX_ORG_OAUTH_CLIENT_SECRET	Integrations / OAuth / <OAUTH_CLIENT_NAME> / Client Details / Client Secret	OAuth client secret

Transaction Object

Transaction object hybrid_integration contains EXEC configuration. It should be configured as:

• Path to the transaction object:
  • for the single-tenant Engage deployment the transaction object should be created in the Transactions configuration unit in the Resources structure
  • for the multi-tenant Engage deployment there must be a separate transaction object under each of the tenant. Each transaction should point at a dedicated EX Org. EX Orgs cannot be shared by multiple Engage tenants.
• Name: hybrid_integration
• Type: List

Transaction Annex Folders

Folder	Parameter Name	Value	Description
general	organization_sname	CFG_ALIAS_EX_ORG_SHORT_NAME	Short name of the EX Org.
	organization_id	CFG_ALIAS_EX_ORG_ID	EX Org Organization ID.
	base_service_url		Base service URL used for framing the REST API calls to the EX Org.
	region	GENESYS_CLOUD_REGION_URL	Service URL for the particular region, used for uploading recordings.
config_sync	client_id	CFG_ALIAS_EX_ORG_OAUTH_CLIENT_ID	OAuth client ID.
	client_secret	CFG_ALIAS_EX_ORG_OAUTH_CLIENT_SECRET	OAuth client secret.
	skill_level_max_value	ENGAGE_MAX_SKILL_LEVEL	Maximum skill level value set for Engage agents to match the skill proficiency in Genesys Cloud. See ACD Skills for more information.
	skill_level_min_value	ENGAGE_MIN_SKILL_LEVEL	Minimum skill level value set for Engage agents to match the skill proficiency in Genesys Cloud. See ACD Skills for more information.
agent_state_sync	client_id	CFG_ALIAS_EX_ORG_OAUTH_CLIENT_ID	OAuth client ID.
	client_secret	CFG_ALIAS_EX_ORG_OAUTH_CLIENT_SECRET	OAuth client secret.
conversation_provider	client_id	CFG_ALIAS_EX_ORG_OAUTH_CLIENT_ID	OAuth client ID.
	client_secret	CFG_ALIAS_EX_ORG_OAUTH_CLIENT_SECRET	OAuth client secret.
	sta_default_programid	<DEFAULT_STA_PROGRAM_ID>	Program UUID for topic detection
	sta_default_language	<DEFAULT_STA_LANGUAGE>	ISO 639-1 two letter language code + '-' + ISO 3166-1 alpha-2 two letter country code string. Default language to use for transcription
	recording_enabled	True/False	Enable Recording Support. Default Value: False.
recording-uploader.exec.transientStorage	type	WebDAV	The value must be WebDAV.
	username	WEBDAV_USERNAME	The username used for downloading recording files from WebDAV.
	password	WEBDAV_PASSWORD	Password used for downloading recording files from WebDAV.
	url	WEBDAV_URL	WebDAV URL from where the recording files are downloaded.
recording-uploader.exec.genesysCloud	client_id	CFG_ALIAS_EX_ORG_OAUTH_CLIENT_ID	OAuth client ID.
	password	CFG_ALIAS_EX_ORG_OAUTH_CLIENT_SECRET	OAuth client secret.

Configuring record IVR profile inside Voice Platform Profiles under CME

Engage deployments with third-party recording solutions use recording client recdest parameters for connecting to third-party recorders. It is expected that parameter callrec_dest is not used for the third-party recorder integration and can be utilized to direct recording metadata to EXEC.

For recording injection functionality, MCP posts the recording file into the WebDAV server and also it posts the recording metadata to the EXCP service. FAs part of this process, the recording IVR profile inside the Voice Platform Profile must be configured.

Folder	Parameter	Value	Description
gvp.general	service-type	record	IVR profile type: always set to record.
gvp.service-parameters	recordingclient.callrec_dest	http://<HTTP_LB_HOST_IPADDR>:<HTTP_LB_EXCP_PORT>/api/v1/recording	HTTP URL for posting recording metadata: pointed at the NGINX front-ending all EXCP instances.
gvp.service-parameters	recordingclient.httpauthorization2	<WEBDAV_USERNAME>:<WEBDAV_PASSWORD>	Credentials to post recordings to the intermediate WebDAV recording storage.
gvp.service-parameters	recordingclient.recdest2	http://<WEBDAV_URL>/	HTTP URL of the intermediate WebDAV recording storage.
gvp.service-parameters	recordingclient.type2	audio/opus	Recording file format. Currently only the opus format is supported.

Create a genesys user on Virtual Machines

It is mandatory to create 'genesys' user on all the Virtual Machines including Bootstrap Virtual Machine. This step ensures seamless, password-free deployment of EXEC components onto their corresponding Virtual Machines.

Installing EXEC Bootstrap Script

Important

You should always deploy EX Engage Connector using the bootstrap script.

Perform the following items before deploying EX Engage Connector:

1. Move EXOP bootstrap package downloaded from the FTP (for example exop-100.0.100.XX.tgz) to the Bootstrap VM and place it under /usr/home/genesys/exec.
2. Go to the installation directory where the bootstrap script is copied and extract the package in the same folder.
3. Under the installation directory, verify if the following files are available:
   • .env
   • bootstrap.py
   • Docker Compose files for EXCS, EXCP, EXAS, and EXRP
   • alerts.yml
   • prometheus.yml
4. Make sure the file is executable: chmod +x bootstrap.py
5. Run ./bootstrap.py --help and verify if the following screen is displayed:

Welcome to EXEC deployment bootstrap script. Following actions are supported
 
init                                      - Initialize EXEC environment
start                                     - Start EXEC services
stop                                      - Stop EXEC services
restart                                   - Restart EXEC services
update                                    - upgrade/downgrade EXEC services
rollback                                  - rollback EXEC service

Configuring .env file

The .env file contains the description of the EXEC docker environment. Configure all mandatory parameters to describe the environment where EXEC components are planned to be deployed. Bootstrap script converts data from this file into the docker compose yml configuration files.

You can specify parameters for the deployment by overriding the default values in the .env file. See the Parameters table for a full list of overridable values.

Common Parameters

Parameter Name	Description
EXEC_DOCKER_REPOSITORY *	Jfrog Artifactory Edge repository for pulling EXEC images.
EXEC_INFRA_DOCKER_REPOSITORY	Docker repo for Redis, grafana, and Prometheus containers, if empty - docker hub will be used.
EXEC_HOST_LOGS_VOLUME_PATH *	Host path which is mounted inside the container to store logs.
EXEC_ORG_ID *	EX Org ID is taken from the following path in the GC UI Admin view: Organization Settings / Organization Details / Advanced / Organization ID
EXEC_TENANT_ID *	Unique ID to identify the Engage tenant. DBID of the Tenant configuration object in the Engage Configuration DB can be used.
EXEC_REDIS_HOST *	IP Address of the Redis proxy in case of enterprise Redis and IP Address of Redis server in case of Lab environment
EXEC_REDIS_PORT *	Port of Redis server
EXEC_REDIS_SECURE	If the Redis server is password protected via the requirepass option.
EXEC_REDIS_PASSWORD	Do not configure the password in the .env file, provide it as input when you run the bootstrap script
A * indicates mandatory fields.

EXCS Parameters

Parameter Name	Description
EXCS_VM_HOST *	IP Address of VM machine where CS will be deployed
EXCS_PORT *	Config Sync Service port
EXCS_TAG *	Config Sync Service Image Version
EXCS_CONFIG_DB_HOST *	IP Address of Config Database.
EXCS_CONFIG_DB_PORT *	Port Number of Config Database
EXCS_CONFIG_DB_USER *	Config Database Username
EXCS_CONFIG_DB_SSL*	Config Database SSL connectivity
EXCS_CONFIG_DB_PASSWORD *	Do not configure the password in the .env file, provide it as input when you run the bootstrap script
EXCS_CONFIG_DB_NAME *	Config Database name
EXCS_CONFIG_DB_TYPE *	Config Database Type. Supported Values are postgresql, oracle, or mssql. Default value is postgresql.
EXCS_SYNC_CUSTOMER_OBJECTS_ONLY	When set to "true", EXCS only synchronizes objects under customer folder hierarchy to Genesys Cloud. Unless SYNC_CUSTOMER_FOLDER_NAME is specified, treat TENANT_NAME as the top-level customer folder. See Folder Filtering section for more information. Default is "false", i.e. all objects of the supported types are synchronized. Values: true/false (default)
EXCS_SYNC_CUSTOMER_FOLDER_NAME	When SYNC_CUSTOMER_OBJECTS_ONLY is set to "true", use the folder name configured here instead of the value of TENANT_NAME.
EXCS_LOG_PATH	File system path for logs. If not provided, logs will be directed to stdout.
EXCS_LOG_FILENAME *	File name for logs. If LOG_FILENAME is specified, this environment variable further describes the file name. Example: 'excs-<TENANT_ID>.%Y%m%d_%H%M%S_%L.log'
EXCS_LOG_LEVEL	Defines the EXCS log level. Values: trace, debug, info, warn, error, fatal
EXCS_GC_QUEUE_NAME_SUFFIX *	Set it to -Engage-external. This suffix is needed to distinguish Queue objects synced from Engage environment.
A * indicates mandatory fields.

EXAS Parameters

Parameter Name	Description
EXAS_HOST *	IP Address of VM machine where AS will be deployed
EXAS_PORT *	Agent Sync Service port
EXAS_TAG *	Agent Sync Service Image Version.
EXAS_STAT_SERVER_PRIMARY_HOST *	Primary Stat Server's host IP/FQDN
EXAS_STAT_SERVER_PRIMARY_PORT *	Primary Stat Server's port number.
EXAS_STAT_SERVER_BACKUP_HOST *	Backup Stat Server's host IP/FQDN
EXAS_STAT_SERVER_BACKUP_PORT *	Backup Stat Server's port number.
EXAS_LOG_PATH	File system path for logs. If not provided, logs will be directed to stdout
EXAS_LOG_FILENAME *	File name for logs. If LOG_FILENAME is specified, this environment variable further describes the file name. Example: 'exas-<TENANT_ID>.%Y%m%d_%H%M%S_%L.log'
EXAS_LOG_LEVEL	Defines the EXAS log level. Values: trace, debug, info, warn, error, fatal
A * indicates mandatory fields.

EXCP Parameters

Parameter Name	Description
EXCP_1_VM_HOST_PAIR *	VM Host IPAddress list where Conversation Provider service will be running
EXCP_1_PORT *	Conversation Provider Service port
EXCP_1_TAG *	Conversation Provider Container version
EXCP_1_VOICE_SERVER_VQ *	A list of SIP Server host:port for receiving Virtual Queue Events. EXCP should connect to the default port. Format: primary1:port/backup1:port
EXCP_1_VOICE_SERVER_AGENT *	A list of SIP Server host:port for receiving Extension DN Events . EXCP should connect to the default port. Format: primary1:port/backup1:port
EXCP_1_VOICE_SERVER_CALL *	A list of SIP Server host:port for receiving Call Monitoring Events. EXCP should connect to the default port. Format: primary1:port/backup1:port
EXCP_1_LOG_PATH	File system path for logs. If not provided, logs will be directed to stdout
EXCP_1_LOG_FILE_NAME *	File name for logs. If EXCP_1_LOG_FILE_NAME is specified, this environment variable further describes the file name. Example: 'excp-.%Y%m%d_%H%M%S_%L_${TENANT_ID}_${HOSTNAME}.log'
EXCP_1_LOG_LEVEL	Defines the EXCP log level. Values: trace, debug, info, warn, error, fatal
EXCP_RECORDING_PROCESSOR_PROXY_HOST*	HTTP Load Balancer Proxy's IP address or FQDN which proxies the request to EXRP instances.
EXCP_RECORDING_PROCESSOR_PROXY_PORT*	HTTP Load Balancer Proxy's Port which proxies the request to EXRP instances.
A * indicates mandatory fields.

EXRP Parameters

Parameter Name	Description
EXRP_VM_HOST_LIST *	Comma Separated VM IP Address list where Recording Provider service will be running.
EXRP_PORT *	Recording Provider Service Port
EXRP_TAG *	Recording Provider container version
EXRP_LOG_LEVEL	Defines the EXRP log level. Values: trace, debug, info, warn, error, fatal
EXRP_CONFIG_SERVER_USER *	Config Server Username
EXRP_CONFIG_SERVER_PASSWORD *	Config Server Password
EXRP_CONFIG_SERVER_APPLICATION *	Config Server Application
EXRP_CONFIG_SERVER_PRIMARY_HOST *	IP Address of Config Server
EXRP_CONFIG_SERVER_PRIMARY_PORT*	IP Port of Config Port
A * indicates mandatory fields.

Deploying EX Engage Connector

To set up the EX Engage Connector services, run the init command: ./bootstrap.py --action init.

The init script performs the following actions:

1. Prompt for following passwords:
   • DB password - The password for the Config database.
   • Redis password - The password for the Redis server (if the redis-secure parameter is set to true).
   • Config Server password - The password for the Config Server.
2. Create a /tmp/.env file containing the provisioning information along with password information.
3. Extract the VM address for each service and copy the docker-compose file of the respective service to their respective VMs.
4. Copy the /tmp/.env file to the list of VMs on which the connector services need to be installed and delete the /tmp/.env file.

The docker-compose files and .env files are stored under the exec directory in the home path of the Genesys user.

Run EXEC Services for First Time

Once the EXEC service environment is initialized, the EXEC services should be started in the following order:

1. Start the EXCP Service by running ./bootstrap.py --action start --service excp --pair 1.
2. Check the health of each EXCP Service and if a 200 OK response is received for health API, it signifies that that the EXCP service is operational.

   curl -v http://<EXCP_VMHOST_IPADDRESS>:3650/health
   200 OK with dependency status

3. Check if all the EXCP instances are running healthy
4. Start the EXRP Service by running ./bootstrap.py --action start --service exrp.
5. Check the health of EXRP Service and if 200 OK response is received, it signifies that that the EXRP service is operational.

   curl -v http://<EXCP_VMHOST_IPADDRESS>:3650/health
   200 OK with dependency status

6. Check if all the EXRP instances are operational.
7. Check whether NGINX properly forwards requests to both EXCP and to EXRP instances

   curl -X POST http://<HTTP_LB_HOST_IPADDR>:<HTTP_LB_EXCP_PORT>/api/v1/recording
   {"status":{"code":40402,"message":"CallUUID not found","corrId":"2fdd55fe-8b16-41ab-84c9-2c5cc42d4eee"}}
   
   curl http://<HTTP_LB_HOST_IPADDR>:<HTTP_LB_EXRP_PORT>/health
   HTTP Status Code = 200

Starting EXEC Services

The EXEC services can be started/stopped/restarted using the below commands:

• ./bootstrap.py --action start - starts all the EXEC services in their respective VMs configured.
• ./bootstrap.py --action start --service=exas - starts the EXAS service in its respective VM
• ./bootstrap.py --action start --service=excp --pair 1 - starts the EXCP service in both the VMs belonging to excp pair 1
• ./bootstrap.py --action start --service=excp --pair 1 --host 1 - starts the EXCP service in the VM host 1 belonging to excp pair 1

Stopping or Restarting EXEC Services

All services or a specific service or a specific instance of a service can be stopped using the ./bootstrap.py --action stop command. The ./bootstrap.py --action restart command can be used to restart all services or a specific service or a specific instance of a service.