Revision as of 10:17, September 28, 2022 by Karlabel (talk | contribs) (Added content to the draft.)
(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)
Jump to: navigation, search

Configuration Server Prometheus support

Configuration Server exposes statistics of its internal operation via HTTP interface as a Prometheus metrics endpoint.


If configured, upon startup CS opens specified port for HTTP GET requests from Prometheus and/or web browser.

Both http and https protocols are supported.

GET requests for ‘/metrics’ URI path are responded with content intended for Prometheus consumption. The responses contain text in Prometheus text-based format.

Requests for paths other than ‘/metrics’ are responded with http error 404 Not Found.


Exposed metrics correspond to the statistical data currently exposed via stat.log files.

Feature Configuration


The feature can be configured through

- command line arguments, - [prom] section in startup cfg/conf file (for master CS), - prom section in application configuration object.


Command line


-prom-port <port> -prom-transport <transport_parameters> -prom-debug <debug_level> -lport-prom <port>


If <transport_parameters> contain whitespace characters, the string must be enclosed in quotation marks.


-lport-prom value persistently overrides Prometheus port specified elsewhere.

If it is specified in the command line, -prom-transport, if required, must also be specified on the command line.

In this case <port> and <transport_parameters> specified in startup file and/or Application object are ignored, their effective values cannot be changed at runtime.


Startup cfg/conf file - for master CS


[prom] port=<port> transport=<transport_parameters> debug=<debug_level>


Application object


[prom] port=<port> transport=<transport_parameters> debug=<debug_level>


If specified in command line and/or startup configuration file, Prometheus port is opened upon startup immediately.

If specified in application object, Prometheus port is opened once configuration is read.


The set of options in command line takes precedence over that in startup file.


Unless command line specifies -lport-prom. options specified in application object prevail over command line and startup file.

If application object's settings differ from those in command line or startup file, the port is reopened according to application object's settings once they are read.

<port>


Specifies Prometheus http/https port.


Valid values: 0 - the feature is disabled integer 1-65534 - specifies Prometheus port

Default: 0

Changes Take Effect: Immediately, if changed the port is closed and reopened.


<transport_parameters>


Optionally specify TLS parameters for https secure Prometheus connection.

Transport parameters set at Application and/or Host level do not apply to Prometheus port.


Valid values: refer to "Common Configuration Options" "Transport Parameter Options" section and "Genesys 8.5 Security Deployment Guide” “Secure Connections (TLS)" section.

Default:

none -- Prometheus port is open for unsecure http connection.

Changes Take Effect: Immediately, if changed the port is closed and reopened.


<debug _level>


Enables additional debug-level logging for the feature.


Valid values: 0 - debug logging is disabled 1 - log http message headers 2 – log complete http messages

Default: 0

Changes Take Effect: Immediately


Sample configurations


http


[prom] port=80

confserv ... -prom-port 80


https - Windows


[prom] port=443 transport=tls=1;certificate=91a15b56b95c82214cfb9a9ecd3590a874c9f619


confserv ... -prom-port 443 -prom-transport tls=1;certificate=91a15b56b95c82214cfb9a9ecd3590a874c9f619


Exposed metrics


Metric names are prefixed with - cs_ for normal operation or with - tenant_cs_ if tenant_id is specified in devops\contactcenterid annex entry of the CS’s configuration object.

In the later case tenant_id value is added as an index for each metrics.


Metrics with “current” following the prefix represent values since previous scrape. These values are reset at each scrape (http request). Prometheus scrapes are distinguished from the requests from other clients (i.e. browser), so the later do not interfere with the data collected by regular Prometheus scrapes.


Metrics representing data other than countables, Boolean or enumerators are suffixed with units in which the values are given.


Counters are additionally suffixed with “total”

cs_running_time_milliseconds_total counter
cs_current_time_since_last_scrape_milliseconds gauge
cs_current_mode gauge 0 - Unknown 1 - Master Primary 2 - Master Primary Readonly 3 - Restricted Readonly 4 - Master Backup 5 - Proxy Primary 6 - Proxy Backup 7 - Offline 8 - Restricted Import
cs_main_db_connection_status gauge 0 – down 1 - up
cs_db_sync_connection_status gauge 0 – down 1 - up
cs_ha_peer_connection_status gauge 0 – down 1 - up
cs_db_sync_violation_status gauge 0 – normal operation 1 – db sync violation detected
cs_master_connection_status 0 – down 1 – up -- for proxies only
cs_persistent_mode 0 – normal operation 1 – persistent mode operation -- for proxies only
cs_ha_peer_connection_status 0 – down 1 - up -- for proxies only
cs_average_load_percent gauge
cs_processing_time_milliseconds_total counter
cs_max_atomic_processing_time_milliseconds gauge
cs_current_load_percent gauge
cs_current_processing_time_milliseconds gauge
cs_current_max_atomic_processing_time_milliseconds gauge

Cfglib protocol metrics

cs_cfglib_number_of_clients gauge
cs_cfglib_number_of_gui_clients gauge
cs_cfglib_requests_received_total counter
cs_cfglib_notifications_sent_total counter
cs_cfglib_packets_received_total counter
cs_cfglib_packets_sent_total counter
cs_cfglib_received_bytes_total counter
cs_cfglib_sent_bytes_total counter
cs_cfglib_processing_time_milliseconds_total counter
cs_cfglib_max_atomic_processing_time_milliseconds gauge
cs_cfglib_current_requests_received gauge
cs_cfglib_current_notifications_sent gauge
cs_cfglib_current_packets_received gauge
cs_cfglib_current_packets_sent gauge
cs_cfglib_current_received_bytes gauge
cs_cfglib_current_sent_bytes gauge
cs_cfglib_current_processing_time_milliseconds gauge
cs_cfglib_current_max_atomic_processing_time_milliseconds gauge

Per cfglib client metrics


The following metrics are reported for each cfglib client connected to CS at the time of the scrape.


Metrics for each client are indexed with the following:


client_session_id – CS unique (per CS instance run) session ID,

socket – socket number,

app_type – configuration application type,

app_name - configuration application name,

account – user account used for login,

host – client’s host ip address,

connected – timestamp of the client login

cs_client_requests_received_total counter
cs_client_notifications_sent_total counter
cs_client_packets_received_total counter
cs_client_packets_sent_total counter
cs_client_received_bytes_total counter
cs_client_sent_bytes_total counter
cs_client_processing_time_milliseconds_total counter
cs_client_max_atomic_processing_time_milliseconds gauge
cs_client_current_requests_received gauge
cs_client_current_notifications_sent gauge
cs_client_current_packets_received gauge
cs_client_current_packets_sent gauge
cs_client_current_received_bytes gauge
cs_client_current_sent_bytes gauge
cs_client_current_processing_time_milliseconds gauge
cs_client_current_max_atomic_processing_time_milliseconds gauge

Client deferral metrics

cs_deferred_clients gauge

The following metrics are reported for each client deferred at the time of the scrape.

Metrics for each client are indexed the same way as in the connected clients list.


cs_client_deferred_requests gauge

cs_client_defer_counter gauge

Ldap metrics


cs_ldap_requests_queue_size gauge

cs_ldap_responses_queue_size gauge

Ldap server failures metrics For each failing ldap server the following metrics is reported.

Metrics are indexed with

search_base - the ldap server's search base.

Ldap server is no longer considered failing once it has recovered and responded successfully for at least one auth request.

ldap_server_request_failures count

Sample output


  1. TYPE cs_running_time_milliseconds_total counter

cs_running_time_milliseconds_total 25198 1660751626724

  1. TYPE cs_current_time_since_last_scrape_milliseconds gauge

cs_current_time_since_last_scrape_milliseconds 25198 1660751626724

  1. HELP cs_current_mode 0 - Unknown 1 - Master Primary 2 - Master Primary Readonly 3 - Restricted Readonly 4 - Master Backup 5 - Proxy Primary 6 - Proxy Backup 7 - Offline 8 - Restricted Import
  2. TYPE cs_current_mode gauge

cs_current_mode 1 1660751626724

  1. TYPE cs_main_db_connection_status gauge

cs_main_db_connection_status 1 1660751626724

  1. TYPE cs_db_sync_connection_status gauge

cs_db_sync_connection_status 1 1660751626724

  1. TYPE cs_ha_peer_connection_status gauge

cs_ha_peer_connection_status 0 1660751626724

  1. TYPE cs_db_sync_violation_status gauge

cs_db_sync_violation_status 0 1660751626724

  1. TYPE cs_average_load_percent gauge

cs_average_load_percent 17 1660751626724

  1. TYPE cs_processing_time_milliseconds_total counter

cs_processing_time_milliseconds_total 4303 1660751626724

  1. TYPE cs_max_atomic_processing_time_milliseconds gauge

cs_max_atomic_processing_time_milliseconds 1503 1660751626724

  1. TYPE cs_current_load_percent gauge

cs_current_load_percent 17 1660751626724

  1. TYPE cs_current_processing_time_milliseconds gauge

cs_current_processing_time_milliseconds 4303 1660751626724

  1. TYPE cs_current_max_atomic_processing_time_milliseconds gauge

cs_current_max_atomic_processing_time_milliseconds 1503 1660751626724

  1. TYPE cs_cfglib_number_of_clients gauge

cs_cfglib_number_of_clients 1 1660751626724

  1. TYPE cs_cfglib_number_of_gui_clients gauge

cs_cfglib_number_of_gui_clients 1 1660751626724

  1. TYPE cs_cfglib_requests_received_total counter

cs_cfglib_requests_received_total 22 1660751626724

  1. TYPE cs_cfglib_notifications_sent_total counter

cs_cfglib_notifications_sent_total 1 1660751626724

  1. TYPE cs_cfglib_packets_received_total counter

cs_cfglib_packets_received_total 22 1660751626724

  1. TYPE cs_cfglib_packets_sent_total counter

cs_cfglib_packets_sent_total 33 1660751626724

  1. TYPE cs_cfglib_received_bytes_total counter

cs_cfglib_received_bytes_total 1210 1660751626724

  1. TYPE cs_cfglib_sent_bytes_total counter

cs_cfglib_sent_bytes_total 135722 1660751626724

  1. TYPE cs_cfglib_processing_time_milliseconds_total counter

cs_cfglib_processing_time_milliseconds_total 859 1660751626724

  1. TYPE cs_cfglib_max_atomic_processing_time_milliseconds gauge

cs_cfglib_max_atomic_processing_time_milliseconds 101 1660751626724

  1. TYPE cs_cfglib_current_requests_received gauge

cs_cfglib_current_requests_received 22 1660751626724

  1. TYPE cs_cfglib_current_notifications_sent gauge

cs_cfglib_current_notifications_sent 1 1660751626724

  1. TYPE cs_cfglib_current_packets_received gauge

cs_cfglib_current_packets_received 22 1660751626724

  1. TYPE cs_cfglib_current_packets_sent gauge

cs_cfglib_current_packets_sent 33 1660751626724

  1. TYPE cs_cfglib_current_received_bytes gauge

cs_cfglib_current_received_bytes 1210 1660751626724

  1. TYPE cs_cfglib_current_sent_bytes gauge

cs_cfglib_current_sent_bytes 135722 1660751626724

  1. TYPE cs_cfglib_current_processing_time_milliseconds gauge

cs_cfglib_current_processing_time_milliseconds 859 1660751626724

  1. TYPE cs_cfglib_current_max_atomic_processing_time_milliseconds gauge

cs_cfglib_current_max_atomic_processing_time_milliseconds 101 1660751626724

  1. TYPE cs_client_requests_received_total counter

cs_client_requests_received_total{client_session_id="1",socket="1956",app_type="SCE",app_name="default",account="default",host="172.27.112.1",connected="2022-08-17T12:53:32.441"} 21 1660751626724

  1. TYPE cs_client_notifications_sent_total counter

cs_client_notifications_sent_total{client_session_id="1",socket="1956",app_type="SCE",app_name="default",account="default",host="172.27.112.1",connected="2022-08-17T12:53:32.441"} 1 1660751626724

  1. TYPE cs_client_packets_received_total counter

cs_client_packets_received_total{client_session_id="1",socket="1956",app_type="SCE",app_name="default",account="default",host="172.27.112.1",connected="2022-08-17T12:53:32.441"} 21 1660751626724

  1. TYPE cs_client_packets_sent_total counter

cs_client_packets_sent_total{client_session_id="1",socket="1956",app_type="SCE",app_name="default",account="default",host="172.27.112.1",connected="2022-08-17T12:53:32.441"} 32 1660751626724

  1. TYPE cs_client_received_bytes_total counter

cs_client_received_bytes_total{client_session_id="1",socket="1956",app_type="SCE",app_name="default",account="default",host="172.27.112.1",connected="2022-08-17T12:53:32.441"} 1050 1660751626724

  1. TYPE cs_client_sent_bytes_total counter

cs_client_sent_bytes_total{client_session_id="1",socket="1956",app_type="SCE",app_name="default",account="default",host="172.27.112.1",connected="2022-08-17T12:53:32.441"} 135656 1660751626724

  1. TYPE cs_client_processing_time_milliseconds_total counter

cs_client_processing_time_milliseconds_total{client_session_id="1",socket="1956",app_type="SCE",app_name="default",account="default",host="172.27.112.1",connected="2022-08-17T12:53:32.441"} 788 1660751626724

  1. TYPE cs_client_max_atomic_processing_time_milliseconds gauge

cs_client_max_atomic_processing_time_milliseconds{client_session_id="1",socket="1956",app_type="SCE",app_name="default",account="default",host="172.27.112.1",connected="2022-08-17T12:53:32.441"} 101 1660751626724

  1. TYPE cs_client_current_requests_received gauge

cs_client_current_requests_received{client_session_id="1",socket="1956",app_type="SCE",app_name="default",account="default",host="172.27.112.1",connected="2022-08-17T12:53:32.441"} 21 1660751626724

  1. TYPE cs_client_current_notifications_sent gauge

cs_client_current_notifications_sent{client_session_id="1",socket="1956",app_type="SCE",app_name="default",account="default",host="172.27.112.1",connected="2022-08-17T12:53:32.441"} 1 1660751626724

  1. TYPE cs_client_current_packets_received gauge

cs_client_current_packets_received{client_session_id="1",socket="1956",app_type="SCE",app_name="default",account="default",host="172.27.112.1",connected="2022-08-17T12:53:32.441"} 21 1660751626724

  1. TYPE cs_client_current_packets_sent gauge

cs_client_current_packets_sent{client_session_id="1",socket="1956",app_type="SCE",app_name="default",account="default",host="172.27.112.1",connected="2022-08-17T12:53:32.441"} 32 1660751626724

  1. TYPE cs_client_current_received_bytes gauge

cs_client_current_received_bytes{client_session_id="1",socket="1956",app_type="SCE",app_name="default",account="default",host="172.27.112.1",connected="2022-08-17T12:53:32.441"} 1050 1660751626724

  1. TYPE cs_client_current_sent_bytes gauge

cs_client_current_sent_bytes{client_session_id="1",socket="1956",app_type="SCE",app_name="default",account="default",host="172.27.112.1",connected="2022-08-17T12:53:32.441"} 135656 1660751626724

  1. TYPE cs_client_current_processing_time_milliseconds gauge

cs_client_current_processing_time_milliseconds{client_session_id="1",socket="1956",app_type="SCE",app_name="default",account="default",host="172.27.112.1",connected="2022-08-17T12:53:32.441"} 788 1660751626724

  1. TYPE cs_client_current_max_atomic_processing_time_milliseconds gauge

cs_client_current_max_atomic_processing_time_milliseconds{client_session_id="1",socket="1956",app_type="SCE",app_name="default",account="default",host="172.27.112.1",connected="2022-08-17T12:53:32.441"} 101 1660751626724

  1. TYPE cs_deferred_clients gauge

cs_deferred_clients 0 1660751626724

  1. TYPE cs_ldap_requests_queue_size gauge

cs_ldap_requests_queue_size 0 1660751626724

  1. TYPE cs_ldap_responses_queue_size gauge

cs_ldap_responses_queue_size 0 1660751626724

Comments or questions about this documentation? Contact us for support!