Jump to: navigation, search

Configuration Server Prometheus support

Configuration Server (CS) exposes the statistics of its internal operation via HTTP interface as a Prometheus metrics endpoint.

If configured, upon startup, CS opens a specified port for the HTTP GET requests from the Prometheus and/or web browser.

Both http and https protocols are supported.

GET requests for /metrics URI path are responded with content intended for Prometheus consumption. The responses contain text in Prometheus text-based format.

Requests for paths other than /metrics are responded with http error 404 Not Found.

Exposed metrics correspond to the statistical data currently exposed via the stat.log files.

To know about feature configuration and information on configuration options, refer to [prom] Section in the Management Framework Configuration Options Reference Manual.

Sample configurations

http 

[prom]
port=80

confserv ... -prom-port 80

https - Windows 

[prom]
port=443
transport=tls=1;certificate=91a15b56b95c82214cfb9a9ecd3590a874c9f619

confserv ... -prom-port 443 -prom-transport tls=1;certificate=91a15b56b95c82214cfb9a9ecd3590a874c9f619

Exposed metrics

Metric names are prefixed with - cs_ for normal operation or with - tenant_cs_ if tenant_id is specified in devops\contactcenterid annex entry of the CS’s configuration object.

In the later case, the value tenant_id is added as an index for each metrics.

Metrics with “current” following the prefix represent values since previous scrape. These values are reset at each scrape (http request). Prometheus scrapes are distinguished from the requests from other clients (i.e. browser), so the later do not interfere with the data collected by regular Prometheus scrapes.

Metrics representing data other than countables, Boolean or enumerators are suffixed with units in which the values are given.

Counters are additionally suffixed with “total”.

CS general metrics

cs_running_time_milliseconds_total counter
cs_current_time_since_last_scrape_milliseconds gauge
cs_current_mode gauge 0 - Unknown 1 - Master Primary 2 - Master Primary Readonly 3 - Restricted Readonly 4 - Master Backup 5 - Proxy Primary 6 - Proxy Backup 7 - Offline 8 - Restricted Import
cs_main_db_connection_status gauge 0 – down 1 - up
cs_db_sync_connection_status gauge 0 – down 1 - up
cs_ha_peer_connection_status gauge 0 – down 1 - up
cs_db_sync_violation_status gauge 0 – normal operation 1 – db sync violation detected
cs_master_connection_status 0 – down 1 – up -- for proxies only
cs_persistent_mode 0 – normal operation 1 – persistent mode operation -- for proxies only
cs_average_load_percent gauge
cs_processing_time_milliseconds_total counter
cs_max_atomic_processing_time_milliseconds gauge
cs_current_load_percent gauge
cs_current_processing_time_milliseconds gauge
cs_current_max_atomic_processing_time_milliseconds gauge

Cfglib protocol metrics

cs_cfglib_number_of_clients gauge
cs_cfglib_number_of_gui_clients gauge
cs_cfglib_requests_received_total counter
cs_cfglib_notifications_sent_total counter
cs_cfglib_packets_received_total counter
cs_cfglib_packets_sent_total counter
cs_cfglib_received_bytes_total counter
cs_cfglib_sent_bytes_total counter
cs_cfglib_processing_time_milliseconds_total counter
cs_cfglib_max_atomic_processing_time_milliseconds gauge
cs_cfglib_current_requests_received gauge
cs_cfglib_current_notifications_sent gauge
cs_cfglib_current_packets_received gauge
cs_cfglib_current_packets_sent gauge
cs_cfglib_current_received_bytes gauge
cs_cfglib_current_sent_bytes gauge
cs_cfglib_current_processing_time_milliseconds gauge
cs_cfglib_current_max_atomic_processing_time_milliseconds gauge

Per cfglib client metrics

The following metrics are reported for each cfglib client connected to CS at the time of the scrape.

Metrics for each client are indexed with the following:

  • client_session_id – CS unique (per CS instance run) session ID
  • socket – socket number
  • app_type – configuration application type
  • app_name - configuration application name
  • account – user account used for login
  • host – client’s host ip address
  • connected – timestamp of the client login
cs_client_requests_received_total counter
cs_client_notifications_sent_total counter
cs_client_packets_received_total counter
cs_client_packets_sent_total counter
cs_client_received_bytes_total counter
cs_client_sent_bytes_total counter
cs_client_processing_time_milliseconds_total counter
cs_client_max_atomic_processing_time_milliseconds gauge
cs_client_current_requests_received gauge
cs_client_current_notifications_sent gauge
cs_client_current_packets_received gauge
cs_client_current_packets_sent gauge
cs_client_current_received_bytes gauge
cs_client_current_sent_bytes gauge
cs_client_current_processing_time_milliseconds gauge
cs_client_current_max_atomic_processing_time_milliseconds gauge

Client deferral metrics

cs_deferred_clients gauge

The following metrics are reported for each client deferred at the time of the scrape.

Metrics for each client are indexed the same way as in the connected clients list.

cs_client_deferred_requests gauge
cs_client_defer_counter gauge

Ldap metrics

cs_ldap_requests_queue_size gauge
cs_ldap_responses_queue_size gauge

Ldap server failures metrics

For each failing ldap server, the following metrics is reported.

Metrics are indexed with: search_base - the ldap server's search base.

Ldap server is no longer considered failing once it has recovered and responded successfully for at least one auth request.

cs_ldap_server_request_failures_total count

Sample output

# TYPE cs_running_time_milliseconds_total counter
cs_running_time_milliseconds_total 25198 1660751626724
 
# TYPE cs_current_time_since_last_scrape_milliseconds gauge
cs_current_time_since_last_scrape_milliseconds 25198 1660751626724

# HELP cs_current_mode 0 - Unknown  1 - Master Primary  2 - Master Primary Readonly  3 - Restricted Readonly  4 - Master Backup  5 - Proxy Primary  6 - Proxy Backup  7 - Offline  8 - Restricted Import
# TYPE cs_current_mode gauge
cs_current_mode 1 1660751626724
 
# TYPE cs_main_db_connection_status gauge
cs_main_db_connection_status 1 1660751626724

# TYPE cs_db_sync_connection_status gauge
cs_db_sync_connection_status 1 1660751626724

# TYPE cs_ha_peer_connection_status gauge
cs_ha_peer_connection_status 0 1660751626724

# TYPE cs_db_sync_violation_status gauge
cs_db_sync_violation_status 0 1660751626724

# TYPE cs_average_load_percent gauge
cs_average_load_percent 17 1660751626724

# TYPE cs_processing_time_milliseconds_total counter
cs_processing_time_milliseconds_total 4303 1660751626724

# TYPE cs_max_atomic_processing_time_milliseconds gauge
cs_max_atomic_processing_time_milliseconds 1503 1660751626724

# TYPE cs_current_load_percent gauge
cs_current_load_percent 17 1660751626724

# TYPE cs_current_processing_time_milliseconds gauge
cs_current_processing_time_milliseconds 4303 1660751626724

# TYPE cs_current_max_atomic_processing_time_milliseconds gauge
cs_current_max_atomic_processing_time_milliseconds 1503 1660751626724

# TYPE cs_cfglib_number_of_clients gauge
cs_cfglib_number_of_clients 1 1660751626724

# TYPE cs_cfglib_number_of_gui_clients gauge
cs_cfglib_number_of_gui_clients 1 1660751626724

# TYPE cs_cfglib_requests_received_total counter
cs_cfglib_requests_received_total 22 1660751626724

# TYPE cs_cfglib_notifications_sent_total counter
cs_cfglib_notifications_sent_total 1 1660751626724

# TYPE cs_cfglib_packets_received_total counter
cs_cfglib_packets_received_total 22 1660751626724

# TYPE cs_cfglib_packets_sent_total counter
cs_cfglib_packets_sent_total 33 1660751626724

# TYPE cs_cfglib_received_bytes_total counter
cs_cfglib_received_bytes_total 1210 1660751626724

# TYPE cs_cfglib_sent_bytes_total counter
cs_cfglib_sent_bytes_total 135722 1660751626724

# TYPE cs_cfglib_processing_time_milliseconds_total counter
cs_cfglib_processing_time_milliseconds_total 859 1660751626724

# TYPE cs_cfglib_max_atomic_processing_time_milliseconds gauge
cs_cfglib_max_atomic_processing_time_milliseconds 101 1660751626724

# TYPE cs_cfglib_current_requests_received gauge
cs_cfglib_current_requests_received 22 1660751626724

# TYPE cs_cfglib_current_notifications_sent gauge
cs_cfglib_current_notifications_sent 1 1660751626724

# TYPE cs_cfglib_current_packets_received gauge
cs_cfglib_current_packets_received 22 1660751626724

# TYPE cs_cfglib_current_packets_sent gauge
cs_cfglib_current_packets_sent 33 1660751626724

# TYPE cs_cfglib_current_received_bytes gauge
cs_cfglib_current_received_bytes 1210 1660751626724

# TYPE cs_cfglib_current_sent_bytes gauge
cs_cfglib_current_sent_bytes 135722 1660751626724

# TYPE cs_cfglib_current_processing_time_milliseconds gauge
cs_cfglib_current_processing_time_milliseconds 859 1660751626724

# TYPE cs_cfglib_current_max_atomic_processing_time_milliseconds gauge
cs_cfglib_current_max_atomic_processing_time_milliseconds 101 1660751626724

# TYPE cs_client_requests_received_total counter
cs_client_requests_received_total{client_session_id="1",socket="1956",app_type="SCE",app_name="default",account="default",host="172.27.112.1",connected="2022-08-17T12:53:32.441"} 21 1660751626724

# TYPE cs_client_notifications_sent_total counter
cs_client_notifications_sent_total{client_session_id="1",socket="1956",app_type="SCE",app_name="default",account="default",host="172.27.112.1",connected="2022-08-17T12:53:32.441"} 1 1660751626724

# TYPE cs_client_packets_received_total counter
cs_client_packets_received_total{client_session_id="1",socket="1956",app_type="SCE",app_name="default",account="default",host="172.27.112.1",connected="2022-08-17T12:53:32.441"} 21 1660751626724

# TYPE cs_client_packets_sent_total counter
cs_client_packets_sent_total{client_session_id="1",socket="1956",app_type="SCE",app_name="default",account="default",host="172.27.112.1",connected="2022-08-17T12:53:32.441"} 32 1660751626724

# TYPE cs_client_received_bytes_total counter
cs_client_received_bytes_total{client_session_id="1",socket="1956",app_type="SCE",app_name="default",account="default",host="172.27.112.1",connected="2022-08-17T12:53:32.441"} 1050 1660751626724

# TYPE cs_client_sent_bytes_total counter
cs_client_sent_bytes_total{client_session_id="1",socket="1956",app_type="SCE",app_name="default",account="default",host="172.27.112.1",connected="2022-08-17T12:53:32.441"} 135656 1660751626724

# TYPE cs_client_processing_time_milliseconds_total counter
cs_client_processing_time_milliseconds_total{client_session_id="1",socket="1956",app_type="SCE",app_name="default",account="default",host="172.27.112.1",connected="2022-08-17T12:53:32.441"} 788 1660751626724

# TYPE cs_client_max_atomic_processing_time_milliseconds gauge
cs_client_max_atomic_processing_time_milliseconds{client_session_id="1",socket="1956",app_type="SCE",app_name="default",account="default",host="172.27.112.1",connected="2022-08-17T12:53:32.441"} 101 1660751626724

# TYPE cs_client_current_requests_received gauge
cs_client_current_requests_received{client_session_id="1",socket="1956",app_type="SCE",app_name="default",account="default",host="172.27.112.1",connected="2022-08-17T12:53:32.441"} 21 1660751626724

# TYPE cs_client_current_notifications_sent gauge
cs_client_current_notifications_sent{client_session_id="1",socket="1956",app_type="SCE",app_name="default",account="default",host="172.27.112.1",connected="2022-08-17T12:53:32.441"} 1 1660751626724

# TYPE cs_client_current_packets_received gauge
cs_client_current_packets_received{client_session_id="1",socket="1956",app_type="SCE",app_name="default",account="default",host="172.27.112.1",connected="2022-08-17T12:53:32.441"} 21 1660751626724

# TYPE cs_client_current_packets_sent gauge
cs_client_current_packets_sent{client_session_id="1",socket="1956",app_type="SCE",app_name="default",account="default",host="172.27.112.1",connected="2022-08-17T12:53:32.441"} 32 1660751626724

# TYPE cs_client_current_received_bytes gauge
cs_client_current_received_bytes{client_session_id="1",socket="1956",app_type="SCE",app_name="default",account="default",host="172.27.112.1",connected="2022-08-17T12:53:32.441"} 1050 1660751626724

# TYPE cs_client_current_sent_bytes gauge
cs_client_current_sent_bytes{client_session_id="1",socket="1956",app_type="SCE",app_name="default",account="default",host="172.27.112.1",connected="2022-08-17T12:53:32.441"} 135656 1660751626724

# TYPE cs_client_current_processing_time_milliseconds gauge
cs_client_current_processing_time_milliseconds{client_session_id="1",socket="1956",app_type="SCE",app_name="default",account="default",host="172.27.112.1",connected="2022-08-17T12:53:32.441"} 788 1660751626724

# TYPE cs_client_current_max_atomic_processing_time_milliseconds gauge
cs_client_current_max_atomic_processing_time_milliseconds{client_session_id="1",socket="1956",app_type="SCE",app_name="default",account="default",host="172.27.112.1",connected="2022-08-17T12:53:32.441"} 101 1660751626724

# TYPE cs_deferred_clients gauge
cs_deferred_clients 0 1660751626724

# TYPE cs_ldap_requests_queue_size gauge
cs_ldap_requests_queue_size 0 1660751626724

# TYPE cs_ldap_responses_queue_size gauge
cs_ldap_responses_queue_size 0 1660751626724

Statistical logging

This section describes the 'CS clients Monitoring' feature in CS.

CS now optionally collects certain statistics on its operation and periodically prints it out to a dedicated log file. Statistics is collected on application, reactor (currently only CfgLib) and client levels.

For each item, the following are calculated:

  • total for the entire run
  • for the period since last printout

Statistics log file uses the following line format for statistics output: 

<Timestamp>  <Parameter>=<Value> [, <Parameter>=<Value>...]

The following data is currently collected and printed out:

Application level

  • Running time
  • Current mode
  • Load - percentage of cpu core running CS code on the main thread vs total elapsed time; this does not include low-level commonlib code, but includes requests processing (outside conn_multi_scan()), commonlib and dblib callbacks
  • Proc time - total high-level processing time - CS code on the main thread
  • Proc max - max. atomic operation processing time

Reactor level

Cfglib reactor only, soap reactor is currently not included.

  • Clients - current number of connected clients
  • GUI - current number of connected GUI clients

These are totals for all clients, including those which disconnected during the reporting period - total since CS startup and since last printout:Requests - number of received requests

  • Notificats- number of received notifications
  • Pckts rcvd - number of received "packets" (that is, calls to conn_write())
  • Pckts sent - number of sent "packets"
  • Bytes rcvd
  • Bytes sent
  • Proc time - total processing time
  • Proc max - max atomic operation processing time

Client level

Lists all clients connected at the time of printout:

  • Sock
  • App Type
  • App Name
  • User
  • Host
  • Connected - timestamp
  • (+ same stats as for reactor - for that client)

The following is sample content printed in logs every x seconds. 

2018-10-10T12:00:01.001 Running Time = 2.17:01:58.277

2018-10-10T12:00:01.001 Current Mode = Primary

2018-10-10T12:00:01.001 Load = 1%

2018-10-10T12:00:01.001 Proc time = 3279447

2018-10-10T12:00:01.001 Proc Max = 3467

....

2018-10-10T12:00:01.001 socket = 540, AppType = SCE, App Name = default, User = default, Host = 192.168.158.1, Connected = 2018-08-10T18:46:31.688,.....

2018-10-10T12:00:01.001 socket = 588, AppType = SCS, App Name = SCS, User = SYSTEM, Host = 192.168.158.1, Connected = 2018-08-10T18:46:32.250,.....
This page was last edited on June 21, 2023, at 09:30.
Comments or questions about this documentation? Contact us for support!