Revision as of 21:16, March 6, 2015 by Sschlich (talk | contribs) (MCP IOPS)
Jump to: navigation, search

GIR-GVP Port Capacity Tests

Genesys conducted performance tests of various Genesys Interactive Recording (GIR) and GVP Capacities. This section contains test profiles, test results, and analysis.

GVP-GIR Port Capacity Test Profiles

Software (Profiles Used in 2014 GIR-GVP Port Capacity Tests

Note: Unless explicitly noted, all MP3 recordings use stereo channels.
Software Profile 1

call recording only, MP3 codec (32kbps bit rate) without encryption

Software Profile 1a

call recording only, MP3 codec (16kbps bit rate) without encryption

Software Profile 1b'

call recording only, MP3 codec (8kbps bit rate Mono) without encryption

  • dest = S3 or http://webdav
  • type = audio/mp3
  • dest2 = NOT SET
  • type2 = NOT SET
  • encryption = disabled
  • write interval = 10s
  • call duration = 210s
  • callrec_dest = HTCC
  • dest = S3 or http://webdav
  • type = audio/mp3
  • dest2 = NOT SET
  • type2 = NOT SET
  • encryption = disabled
  • write interval = 10s
  • call duration = 210s
  • callrec_dest = HTCC
  • dest = S3 or http://webdav
  • type = audio/mp3
  • dest2 = NOT SET
  • type2 = NOT SET
  • encryption = disabled
  • write interval = 10s
  • call duration = 210s
  • callrec_dest = HTCC
  • channels = 1 (specifies mono recording)
Software Profile 2

call recording only, MP3 codec (32kbps) and WAV as dest2 without encryption

Software Profile 2a

call recording only, MP3 codec (16kbps) and WAV as dest2 without encryption

  • dest = S3 or http://webdav
  • type = audio/mp3
  • dest2 = http://webdav
  • type2 = audio/wav
  • encryption = disabled
  • write interval = 10s
  • call duration = 210s
  • callrec_dest = HTCC
  • dest = S3 or http://webdav
  • type = audio/mp3
  • dest2 = http://webdav
  • type2 = audio/wav
  • encryption = disabled
  • write interval = 10s
  • call duration = 210s
  • callrec_dest = HTCC
Software Profile 3

call recording only, MP3 codec (32kbps bit rate) with encryption

Software Profile 3a

call recording only, MP3 codec (16kbps bit rate) with encryption

  • dest = S3 or http://webdav
  • type = audio/mp3
  • dest2 = NOT SET
  • type2 = NOT SET
  • encryption = enabled
  • write interval = 10s
  • call duration = 210s
  • callrec_dest = HTCC
  • dest = S3 or http://webdav
  • type = audio/mp3
  • dest2 = NOT SET
  • type2 = NOT SET
  • encryption = enabled
  • write interval = 10s
  • call duration = 210s
  • callrec_dest = HTCC
Software Profile 4

call recording only, MP3 codec (32kbps) and WAV as dest2 with encryption

Software Profile 4a

call recording only, MP3 codec (16kbps) and WAV as dest2 with encryption

  • dest = S3 or http://webdav
  • type = audio/mp3
  • dest2 = http://webdav
  • type2 = audio/wav
  • encryption = enabled
  • write interval = 10s
  • call duration = 210s
  • callrec_dest = HTCC
  • dest = S3 or http://webdav
  • type = audio/mp3
  • dest2 = http://webdav
  • type2 = audio/wav
  • encryption = enabled
  • write interval = 10s
  • call duration = 210s
  • callrec_dest = HTCC

Hardware Profiles Used in 2014 GIR-GVP Port Capacity Tests

Hardware Profile 1 Specifications & Recommendations Comment
CPU Single Hex Core Intel Xeon X5670@ 2.93GHz
Memory 8 GB or more 4GB is minimum and 8GB is recommended
Network GigaBit Ethernet 100MBit supported
Storage 15k rpm SAS HDD disk storage with at least 72GB.
RAID 0.
15k rpm recommended for maximum performance
OS Windows Server 2008 R2 x64 Enterprise Edition SP1
Hardware Profile 2 Specification & Recommendation Comment
CPU Single Hex Core Intel Xeon X5675@ 3.06GHz
Memory 16 GB or more 4GB is minimum for each VM
Network GigaBit Ethernet 100MBit supported
Storage SSD used for MCP recording cache location.

15k rpm SAS HDD disk storage with at least 136GB used for all other operations.
RAID 0

SSD and 15k rpm SAS HDD are recommended for maximum performance
OS VM vSphere or ESXi 5.x

Windows Server 2008 R2 x64 Enterprise Edition SP1

VM vSphere 5.x as host OS

Windows 2008 Server as Guest OS on VM

Hardware Profile 3 Specification & Recommendation Comment
CPU Dual Hex Core Xeon X5675 3.06 GHz
Memory 16GB or more 8GB is minimum and recommended
Network GigaBit Ethernet 100MBit supported
Storage 15k rpm SAS HDD disk storage with at least 72GB.
RAID 0
15k rpm SAS HDD is recommended for maximum performance
OS Windows Server 2008 R2 x64 Enterprise Edition SP1
Hardware Profile 4 Specification & Recommendation Comment
CPU Dual Hex Core Xeon X5675 3.06 GHz
Memory 32GB or more 4GB is minimum for each VM
Network GigaBit Ethernet 100MBit supported
Storage SSD used for MCP recording cache location.

15k rpm SAS HDD disk storage with at least 360GB used for all other operations.
RAID 0.

SSD and 15k rpm SAS HDD are recommended for maximum performance
OS VM vSphere or ESXi 5.x

Windows Server 2008 R2 x64 Enterprise Edition SP1

VM vSphere 5.x as Host OS

Windows 2008 Server as Guest OS on VM

Hardware Profile 5 Specification & Recommendation Comment
CPU Dual Hex Core Xeon X5675 3.06 GHz
Memory 32GB or more 4GB is minimum for each VM
Network GigaBit Ethernet 100MBit supported
Storage Multiple 15k rpm SAS HDDs disk storage with at least 360GB used for all other operations.
RAID 0.
Split VMs into multiple 15k rpm SAS HDDs.
OS VM vSphere or ESXi 5.x

Windows Server 2008 R2 x64 Enterprise Edition SP1

VM vSphere 5.x as Host OS

Windows 2008 Server as Guest OS on VM

Hardware Profile 6 Specification & Recommendation Comment
CPU Single Eight Core Xeon E5-2640 2.00 GHz
Memory 64GB or more 8GB is minimum for each VM
Network GigaBit Ethernet 100MBit supported
Storage SSD used for MCP logs and recording cache location. 15k rpm SAS HDD disk storage with at least 360GB used for all other operations. RAID 0. SSD and 15k rpm SAS HDD are recommended for maximum performance.
OS VM vSphere or ESXi 5.x

Windows Server 2008 R2 x64 Enterprise Edition SP1

VM vSphere 5.x as Host OS

Windows 2008 Server as Guest OS on VM

Virtual Machine (VM) Profiles Used in 2014 GIR-GVP Port Capacity Tests

VM Profile 1 Specifications & Recommendations Comment
Host Hardware Hardware Profile 2 1x X5675@3.06GHz
16GB RAM
CPU 2 x vCPU
Memory 5 GB 4GB is minimum
Network GigaBit Ethernet 100MBit supported
Storage 10GB SSD used for MCP recording cache location.36GB 15k rpm SAS HDD disk storage used for all other operations. SSD is recommended for maximum performance
Guest OS Windows Server 2008 R2 x64 Enterprise Edition SP1
VM Profile 2 Specifications & Recommendations Comment
Host Hardware Hardware Profile 4 2x X5675@3.06GHz , 32GB RAM
CPU 4 x vCPU
Memory 8 GB 4GB is minimum.
Network GigaBit Ethernet 100MBit supported
Storage 10GB SSD used for MCP recording cache location.At least 36GB 15k rpm SAS HDD disk storage used for all other operations. SSD is recommended for maximum performance.
Guest OS Windows Server 2008 R2 x64 Enterprise Edition SP1
VM Profile 3 Specifications & Recommendations Comment
Host Hardware Hardware Profile 4 2x X5675@3.06GHz , 32GB RAM
CPU 3 x vCPU
Memory 6 GB 4GB is minimum.
Network GigaBit Ethernet 100MBit supported
Storage 10GB SSD used for MCP recording cache location.At least 36GB 15k rpm SAS HDD disk storage used for all other operations. SSD is recommended for maximum performance
Guest OS Windows Server 2008 R2 x64 Enterprise Edition SP1
VM Profile 4 Specifications & Recommendations Comment
Host Hardware Hardware Profile 4 2x X5675@3.06GHz , 32GB RAM
CPU 2 x vCPU
Memory 5 GB 4GB is minimum.
Network GigaBit Ethernet 100MBit supported
Storage 10GB SSD used for MCP recording cache location.At least 36GB 15k rpm SAS HDD disk storage used for all other operations. SSD is recommended for maximum performance
Guest OS Windows Server 2008 R2 x64 Enterprise Edition SP1
VM Profile 5 Specifications & Recommendations Comment
Host Hardware Hardware Profile 5 2x X5675@3.06GHz , 32GB RAM
CPU 2 x vCPU
Memory 5 GB 4GB is minimum.
Network GigaBit Ethernet 100MBit supported
Storage At least 36GB 15k rpm SAS HDD disk storage.
Guest OS Windows Server 2008 R2 x64 Enterprise Edition SP1


GIR-GVP Port Capacity Test Results Summary

Criteria

System Port Capacity is the maximum number of ports (Port Density or PD) or rate (Call Arrivals Per Second or CAPS) that a GIR-GVP system can handle; this number must maximize the usage of hardware resources, while maintaining all criteria within the predefined threshold.

Because CPU usage is the usual deciding factor for peak port capacity, this section presents results that correlate to CPU usage (and other criteria such jitter buffer and max delta from sample recordings) to track the quality of recording.

The following criteria are required for an installation to reach the 95th percentile of quality analysis, from a sample RTP stream:

  • Packet Loss <= 1%
  • Max Jitter Buffer <= 30ms
  • Max Delta <= 200ms

Summary of Performance Testing Results

Table 3: GIR-GVP Port Capacity on Physical Servers

Test Profiles HW profile OS Peak Ports Comment
SW Profile 1 (32 Kbps bit rate) HW Profile 1 Windows 2008 R2 x64 200 (preferred)
SW Profile 1 (32 Kbps bit rate) HW Profile 1 Windows 2008 R2 x64 220 (peak) If some of audio quality criteria can be ignored or waived.
SW Profile 1a (16 Kbps bit rate) HW Profile 1 Windows 2008 R2 x64 240 (preferred)
SW Profile 1a (16 Kbps bit rate) HW Profile 1 Windows 2008 R2 x64 270 (peak) If some of audio quality criteria can be ignored or waived.
SW Profile 1a (16 Kbps bit rate) HW Profile 6 Windows 2008 R2 x64 350 (preferred) 8 Dispatchers (= # of cores)
SW Profile 1a (16 Kbps bit rate) HW Profile 6 Windows 2008 R2 x64 450 (peak) If some of audio quality criteria can be ignored or waived. 8 Dispatchers (= # of cores)
SW Profile 1b (8 Kbps bit rate Mono) HW Profile 6 Windows 2008 R2 x64 450 (preferred) 8 Dispatchers (= # of cores)
SW Profile 1b Profile 1b (8 Kbps bit rate Mono) HW Profile 6 Windows 2008 R2 x64 600 (peak) If some of audio quality criteria can be ignored or waived. 8 Dispatchers (= # of cores)
SW Profile 3a (16 Kbps bit rate) HW Profile 1 Windows 2008 R2 x64 210 (preferred)
SW Profile 3 (16 Kbps bit rate) HW Profile 1 Windows 2008 R2 x64 270 (peak) If some of audio quality criteria can be ignored or waived.
SW Profile 1(32 Kbps bit rate) HW Profile 3 Windows 2008 R2 x64 240 (preferred)
SW Profile 1 (32 Kbps bit rate) HW Profile 3 Windows 2008 R2 x64 360 (peak) If some of audio quality criteria can be ignored or waived.
SW Profile 1a (16 Kbps bit rate) HW Profile 1 RedHat EL 6.5 x64 150 (preferred)
SW Profile 1a (16 Kbps bit rate) HW Profile 1 RedHat EL 6.5 x64 210 (peak) If some of audio quality criteria can be ignored or waived.
SW Profile 1a (16 Kbps bit rate) HW Profile 6 RedHat EL 6.6 x64 220 (preferred)
SW Profile 1a (16 Kbps bit rate) HW Profile 6 RedHat EL 6.6 x64 240 (peak) If some of audio quality criteria can be ignored or waived.
SW Profile 1a (16 Kbps bit rate) HW Profile 6 RedHat EL 6.6 x64 300 (preferred) 8 Dispatchers (= # of cores)
SW Profile 1a (16 Kbps bit rate) HW Profile 6 RedHat EL 6.6 x64 360 (peak) If some of audio quality criteria can be ignored or waived. 8 Dispatchers (= # of cores)
SW Profile 1b (8 Kbps bit rate) HW Profile 6 RedHat EL 6.6 x64 600 (preferred) 8 Dispatchers (= # of cores)
SW Profile 1a (8 Kbps bit rate) HW Profile 6 RedHat EL 6.6 x64 650 (peak) If some of audio quality criteria can be ignored or waived. 8 Dispatchers (= # of cores)
SW Profile 2a (16 Kbps bit rate) HW Profile 1 RedHat EL 6.5 x64 90 (preferred)
SW Profile 2a (16 Kbps bit rate) HW Profile 1 RedHat EL 6.5 x64 150 (peak) If some of audio quality criteria can be ignored or waived.
SW Profile 3a (16 Kbps bit rate) HW Profile 1 RedHat EL 6.5 x64 150 (preferred)
SW Profile 3a (16 Kbps bit rate) HW Profile 1 RedHat EL 6.5 x64 210 (peak) If some of audio quality criteria can be ignored or waived.

Table 4: GIR-GVP Port Capacity on Virtual Machines (VMs)

SW Profile HW profile OS Port Capacity Comment
SW Profile 1 (32 Kbps bit rate) VM Profile 1 VM vSphere 5.1
Windows 2008 R2 x64
300 (preferred) 3 VMs: each VM uses 2 vCPU & 1 MCP installed per VM.
SW Profile 1 (32 Kbps bit rate) VM Profile 1 VM vSphere 5.1
Windows 2008 R2 x64
360 (peak) 3 VMs: each VM uses 2 vCPU & 1 MCP installed per VM.
SW Profile 1 (32 Kbps bit rate) VM Profile 2 VM vSphere 5.1
Windows 2008 R2 x64
360 (preferred) 3 VMs: each VM uses 4 vCPU & 1 MCP installed per VM.
SW Profile 1 (32 Kbps bit rate) VM Profile 2 VM vSphere 5.1
Windows 2008 R2 x64
390 (peak) 3 VMs: each VM uses 4 vCPU & 1 MCP installed per VM.
SW Profile 1 (32 Kbps bit rate) VM Profile 3 VM vSphere 5.1
Windows 2008 R2 x64
520 (preferred) 4 VMs: each VM uses 3 vCPU & 1 MCP installed per VM.
SW Profile 1 (32 Kbps bit rate) VM Profile 3 VM vSphere 5.1
Windows 2008 R2 x64
600 (peak) 4 VMs: each VM uses 3 vCPU & 1 MCP installed per VM.
SW Profile 1 (32 Kbps bit rate) VM Profile 4 VM vSphere 5.1
Windows 2008 R2 x64
600 (preferred) 6 VMs: each VM uses 2 vCPU & 1 MCP installed per VM.
SW Profile 1 (32 Kbps bit rate) VM Profile 4 VM vSphere 5.1
Windows 2008 R2 x64
660 (peak) 6 VMs: each VM uses 2 vCPU & 1 MCP installed per VM.
SW Profile 1a (16 Kbps bit rate) VM Profile 4 VM vSphere 5.5
Windows 2008 R2 x64
720 (peak) 6 VMs: each VM uses 2 vCPU & 1 MCP installed per VM.
SW Profile 1a (32 Kbps bit rate) VM Profile 4 VM vSphere 5.5
Windows 2008 R2 x64
840 (peak) 6 VMs: each VM uses 2 vCPU & 1 MCP installed per VM.
SW Profile 2 (32 Kbps bit rate MP3 + WAV) VM Profile 4 VM vSphere 5.1
Windows 2008 R2 x64
360 (preferred) 6 VMs: each VM uses 2 vCPU & 1 MCP installed per VM.
SW Profile 2 (32 Kbps bit rate MP3 + WAV) VM Profile 4 VM vSphere 5.1
Windows 2008 R2 x64
540 (peak) 6 VMs: each VM uses 2 vCPU & 1 MCP installed per VM.
SW Profile 3a (16 Kbps bit rate) VM Profile 4 VM vSphere 5.5/ Windows 2008 R2 x64 480 (preferred) 6 VMs: each VM uses 2 vCPU & 1 MCP installed per VM.
SW Profile 3a (16 Kbps bit rate) VM Profile 4 VM vSphere 5.5/ Windows 2008 R2 x64 840 (peak) 6 VMs: each VM uses 2 vCPU & 1 MCP installed per VM.
SW Profile 1a (16 Kbps bit rate) VM Profile 4 VM vSphere 5.5/ RH EL 6.5 x64 540 (preferred) 6 VMs: each VM uses 2 vCPU & 1 MCP installed per VM.
SW Profile 1a (16 Kbps bit rate) VM Profile 4 VM vSphere 5.5/ RH EL 6.5 x64 660 (peak) 6 VMs: each VM uses 2 vCPU & 1 MCP installed per VM.
SW Profile 2a (16 Kbps bit rate) VM Profile 4 VM vSphere 5.5/ RH EL 6.5 x64 480 (preferred) 6 VMs: each VM uses 2 vCPU & 1 MCP installed per VM.
SW Profile 2a (16 Kbps bit rate) VM Profile 4 VM vSphere 5.5/ RH EL 6.5 x64 600 (peak) 6 VMs: each VM uses 2 vCPU & 1 MCP installed per VM.
SW Profile 3a (16 Kbps bit rate) VM Profile 4 VM vSphere 5.5/ RH EL 6.5 x64 540 (preferred) 6 VMs: each VM uses 2 vCPU & 1 MCP installed per VM.
SW Profile 3a (16 Kbps bit rate) VM Profile 4 VM vSphere 5.5/ RH EL 6.5 x64 660 (peak) 6 VMs: each VM uses 2 vCPU & 1 MCP installed per VM.
SW Profile 4a (16 Kbps bit rate) VM Profile 4 VM vSphere 5.5/ RH EL 6.5 x64 480 (preferred) 6 VMs: each VM uses 2 vCPU & 1 MCP installed per VM.
SW Profile 4a (16 Kbps bit rate) VM Profile 4 VM vSphere 5.5/ RH EL 6.5 x64 600 (peak) 6 VMs: each VM uses 2 vCPU & 1 MCP installed per VM.

Parameter Adjustments

These adjustments achieve higher port capacity:

Parameter Value Adjustments Yielding Higher Port Capacity

Parameter Default Value Adjusted Value
mpc.recordnumparallelpost 30 300
mpc.recordpostretrybackoff 120000 15000
mpc.recordpostretrycount 3 1
mpc.mediamgr.recordwritetimeinterval 1000 10000
fm.http_proxy <not empty> <empty> (squid bypassed)


Detailed Studies of GVP Media Server Behavior on Windows


Performance Comparison of Physical Server and Virtual Machines

Single Hex Core

With a single hex core CPU, Genesys recommends 200 ports as a reasonable peak port capacity on a physical server with a single X5670, assuming that all criteria have been met. 300 ports can be achieved with a three-VMs configuration of the same hardware, with a single X5675 (performance is slightly better than X5670). The graph below compares overall CPU usage:

Figure 1: Comparison of System Usage between Physical Server and VM from Single Hex Core


Memory usage for MCP scales linearly against port capacity:

Figure 2: Comparison of MCP Memory Usage between Physical Server and VM from Single Hex Core


The two graphs below compare the 95th percentile value of Max Jitter Buffer and Max Delta, tracking audio quality from a sample RTP stream:

Figure 3: Comparison of Max Jitter between Physical Server and VM from Single Hex Core
Figure 4: Comparison of Max Delta between Physical Server and VM from Single Hex Core


A strong correlation exists between Max Jitter Buffer and Max Delta, regarding audio quality. A physical server can meet all criteria when its port capacity is 200 or below. Port capacity that is between 200 and 220 may impact audio quality, since both Max Jitter buffer and Max Delta are just slightly beyond the passing criteria. You can consider 220 as peak performance, if audio quality is not strictly required and can be waived. However, when port capacity reaches 230 or beyond, the two values become so big that there is apparent audio quality impact.

For VM configuration: Preferred/Recommended = 300 ports; Peak Port Capacity = 360 ports. With 390 ports, overall system CPU usage is 97%, close enough to 100% that it also observed audio quality impact.

Below are two tables of IOPS for the above two configurations:

Table 1: Disk IOPS of system level from a physical server with a single hex core

Ports Disk IOPS Physical Server
Total Reads Writes
60 11.13 0.001 11.13
120 21.82 0.001 21.82
180 32.03 0.001 32.03
200 34.95 0.001 34.95
210 36.53 0.001 36.53
220 37.76 0.001 37.76
230 39.48 0.001 39.48
240 43.33 0.002 43.33

Table 2: Disk IOPS of sum of all VMs of single hex core

Ports Disk IOPS VMs Overall
Total Reads Writes
120 20.68 0.101 20.58
240 36.29 0.070 36.22
270 41.39 0.087 41.30
300 45.57 0.065 45.50
330 48.85 0.000 48.85
360 51.69 0.000 51.69
390 57.82 0.002 57.82

Disk IOPS in Disk IOPS of sum of all VMs of single hex core table is the sum of Disk IOPS from all VMs. Also, IOPS is measured from each VM and then totaled, to determine overall IOPS. The same method is applied to all Disk IO calculations for VM environments in this series of tests.

Also in the above two tables, the IOPS Reads value is quite small because most of the operations are Writes.

The graph below compares the two IOPS tables above:

Figure 5: Comparison of System Disk IOPS Physical Server and VM from Single Hex Core
  • System level disk IOPS is scaling linearly against port capacity for both physical server and virtual machines.
  • SSD is only used on VM env as cache folder of MCP recording while SAS HDD drive is used to installed OS and MCP.

Dual Hex Cores

With a host of dual hex core CPUs (2x X5675@3.06GHz) with 32 GB RAM, we also compare the results from physical server and VM env. In VM env, on same hardware spec, 3 VMs are configured with 4 vCPU and 8 GB RAM assigned to each VM. Only one MCP installed on each VM and a SSD partition is used as cache folder for MCP recording.

The graph below depicts the overall system CPU usage:

Figure 6: Comparison of System Usage between Physical Server and VM from Dual Hex Cores


The next two graphs show 95 percentile values of Max Jitter and Max Delta from sample RTP stream quality analysis:

Figure 7: Comparison of Max Jitter between Physical Server and VM from Dual Hex Cores
Figure 8: Comparison of Max Delta between Physical Server and VM from Dual Hex Cores


The two tables below show:

  • Disk IOPS at system level on a physical server.
    and
  • Disk IOPS at system level on a VM environment.

Table 3: Disk IOPS at system level from physical server of dual hex cores

Ports Disk IOPS Physical Server
Total Reads Writes
50 9.069 0.000 9.07
100 18.587 0.000 18.59
150 28.598 0.001 28.60
200 37.460 0.001 37.46
240 41.290 0.003 41.29
280 49.031 0.020 49.01
330 53.373 0.001 53.37
350 53.150 0.001 53.15
380 61.456 0.001 61.46
400 64.123 0.001 64.12


Table 4: Disk IOPS of sum of all 3 VMs of dual hex cores

Ports Overall Disk IOPS
Total Reads Writes
120 22.38 0.024 22.35
240 38.99 0.012 38.97
300 48.60 0.017 48.59
360 56.05 0.047 56.00
390 60.24 0.002 60.24
420 64.59 0.028 64.57

The graph below compares the above two tables above:

Figure 9: Comparison of System Disk IOPS Physical Server and VM from Dual Hex Cores


Performance Comparison of Different Virtual Machines Configurations

Overall CPU usage on a physical server beyond peak port capacity is actually higher than overall CPU usage on virtual machines, while audio quality actually shows a quick downfall on a physical server. So the splitting the load into multiple MCPs in a VM environment will definitely take advantage of hardware resources and will achieve high port capacity with fewer audio quality concerns. There are three different VM configurations on the same hardware spec (counting the dual hex cores, 12 vCPUs in total) that are used for this purpose:

  • 3 VMs in total, 4 vCPU are assigned to each VM, only one MCP installed on one VM.
  • 4 VMs in total, 3 vCPU are assigned to each VM, only one MCP installed on one VM.
  • 6 VMs in total, 2 vCPU are assigned to each VM, only one MCP installed on one VM.

The graph below compares overall system CPU usage.

Figure 14: Comparison of System CPU Usage among different VMs configurations

Overall CPU usage scales linearly against port capacity, regardless of how many VMs are configured.


The two graphs below compare RTP stream quality related Max Jitter and Max Delta on these three different VM configurations:

Figure 15: Comparison of Max Jitter among different VM configurations
Figure 16: Comparison of Max Delta among different VM configurations


To achieve higher port capacity, configure more VMs and assign less vCPU to each VM. With audio quality criteria considered, Genesys recommends 600 ports as peak for six VM configurations. Six VMs with two vCPUs for each VM is the optimal configuration.

Below is the table of IOPS for 6 VM configurations:

Table: Disk IOPS of sum of all 6 VMs of dual hex cores, MP3 only

Ports Overall Disk IOPS (kbps)
Total Reads Writes
120 25.18 0.028 25.15
240 42.75 0.052 42.70
300 51.16 0.004 51.15
360 59.61 0.000 59.61
420 67.04 0.000 67.04
480 74.82 0.000 74.82
540 86.30 0.000 86.30
600 94.11 0.000 94.11
660 102.05 0.000 102.04
720 111.30 0.000 111.29


The graph below compares the two tables of IOPS (Table: Disk IOPS of sum of all 3 VMs of dual hex cores for 3 VMs and Table: Disk IOPS of sum of all 6 VMs of dual hex cores, MP3 only for 6 VMs), on the same hardware spec of dual hex cores:

Figure 18: Comparison of System Disk IOPS among different VMs.
  • System Disk IOPS scales linearly against port capacity, but not related for VM configurations.
  • We ran an additional test with only 1 vCPU assigned to each VM, on a single hex core server Hardware profile 2, with a 6-VMs in total on the one server. We could barely run beyond 150 ports—the single CPU cannot be linearly scaled—which compares with a 3-VMs configuration:
Figure 19: Comparison of System Usage for one vCPU vs two vCPUs VMs configuration


The two graphs below show that both Max Jitter and Max Delta jump significantly beyond 150 ports:

Figure 20: Comparison of Max Jitter for one vCPU vs two vCPUs VMs configuration
Figure 21: Comparison of Max Delta for one vCPU vs two vCPUs VMs configuration

The comparison indicates that MCP doesn’t perform well on a single vCPU VM.


Performance Comparison of MP3 only and MP3 + WAV

The graph below compares two test profiles (Profile 1 of MP3 only and Profile 2 of MP3 + WAV as dest2) on the same hardware spec with same 6 VM configurations of 2 vCPU per VM. Below is the CPU usage:

Figure 22: Comparison of System Usage for different test profiles

Overall CPU usage for Software Profile 2 (MP3 + WAV) is slightly higher than for Software Profile 1 (MP3 only).


The two graphs below compare audio quality criteria:

Figure 24: Comparison of Max Jitter among different test profiles
Figure 25: Comparison of Max Delta among different test profiles
  • For this test, applying Profile 2 to a 6 VMs configuration: Preferred/Recommended = 360 ports; Peak Port Capacity = 530 ports, if you can ignore some potential impact to audio quality.

The table below shows the IOPS of the sum of all 6 VMs for a test profile of MP3 + wav:

Table: IOPS of sum of all 6 VMs of dual hex cores, MP3 + wav

Ports Overall Disk IOPS (kbps)
Total Reads Writes
120 42.64 0.01 42.63
240 77.69 0.00 77.69
300 95.99 0.00 95.99
360 114.28 0.00 114.28
420 130.45 0.00 130.45
480 149.58 0.00 149.58
540 172.49 0.00 172.49
600 194.55 0.00 194.55
660 177.80 0.00 177.80

The graph below compares Table: IOPS of sum of all 6 VMs of dual hex cores, MP3 + wav with Table: Disk IOPS of sum of all 6 VMs of dual hex cores, MP3 only:

Figure 26: Comparison of System Disk IOPS for different test profiles on VMs


As we have cache folder on a different SSD drive, we can break down disk IOPS for each drive as below:

Table: Disk IOPS Break Down per Drive, Test Profile 1, MP3 only

Ports Overall Disk IOPS (kbps) SSD Drive E Disk IOPS (kbps) HDD Drive C Disk IOPS (kbps)
Total Reads Writes Total Reads Writes Total Reads Writes
120 25.18 0.03 25.15 20.88 0.00 20.88 4.30 0.03 4.28
240 42.75 0.05 42.70 36.96 0.00 36.96 5.79 0.05 5.74
300 51.16 0.00 51.15 44.63 0.00 44.63 6.53 0.00 6.53
360 59.61 0.00 59.61 52.80 0.00 52.80 6.81 0.00 6.81
420 67.04 0.00 67.04 60.31 0.00 60.31 6.74 0.00 6.74
480 74.82 0.00 74.82 67.85 0.00 67.85 6.97 0.00 6.97
540 86.30 0.00 86.30 79.31 0.00 79.31 6.99 0.00 6.99
600 94.11 0.00 94.11 87.31 0.00 87.31 6.80 0.00 6.80
660 102.05 0.00 102.04 95.12 0.00 95.12 6.92 0.00 6.92
720 111.30 0.00 111.29 104.30 0.00 104.30 6.99 0.00 6.99


Table: Disk IOPS Break Down per Drive, Test Profile 2, MP3 + wav

Ports Overall Disk IOPS (kbps) SSD Drive E Disk IOPS (kbps) HDD Drive C Disk IOPS (kbps)
Total Reads Writes Total Reads Writes Total Reads Writes
120 42.64 0.01 42.63 38.38 0.00 38.38 4.26 0.01 4.26
240 77.69 0.00 77.69 72.07 0.00 72.07 5.62 0.00 5.62
300 95.99 0.00 95.99 89.04 0.00 89.04 6.95 0.00 6.95
360 114.28 0.00 114.28 107.50 0.00 107.50 6.78 0.00 6.78
420 130.45 0.00 130.45 123.56 0.00 123.56 6.89 0.00 6.89
480 149.58 0.00 149.58 142.65 0.00 142.65 6.92 0.00 6.92
540 172.49 0.00 172.49 165.61 0.00 165.61 6.88 0.00 6.88
600 194.55 0.00 194.55 187.53 0.00 187.53 7.02 0.00 7.02

The two graphs below compare corresponding drives:

Figure 29: Comparison of cache folder of SSD Drive IOPS for different profiles

This SSD drive is used exclusively as the cache folder for MCP recording. The IOPS for Profile 2 (two dest2, MP3 + wav) is as double as Profile 1 (one dest. MP3 only).

Figure 30: Comparison of HDD Drive IOPS for different profiles


This HDD drive is used for all operations except the cache folder for MCP recording. IOPS is nearly constant at a regular load and below peak. Thus, the IOPS estimating formula can be:

IOPS1 = C + k * P (dest only)
IOPS2 = C + 2k * P (both dest + dest2)
Where P = ports, C = 7, k = 0.15


Performance Comparison between SAS HDDs and SSD

These tests compare performance between SAS HDDs and SSD for recording, using 6 VMs from the same hardware spec, and these four different HDD and SSD combinations:

  • 1 HDD: all 6 VMs on one 15 krpm SAS HDD drive.
  • 2 HDD: split 6 VMs on two 15 krpm SAS HDD drives, 3 VMs per drive.
  • 3 HDD: split 6 VMs on three 15 krpm SAS HDD drives, 2 VMs per drive.
  • SSD: all 6 VMs on one 15 krpm SAS HDD while a separate SSD drive used as cache folder only.

The testing was executed with Profile 1, MP3 only. Below is the overall system CPU usage:

Figure: Comparison of System Usage among different HDD/SSD drive combinations

The overall system CPU usage exhibits no significant different between HDD and SDD.

IOPS is almost the same for these 4 combinations, so these tests use the numbers in Table: Disk IOPS of sum of all 6 VMs of dual hex cores, MP3 only.


The graphs below compare max jitter and max delta for HDD/SSD drive combinations:

Figure 32: Comparison of Max Jitter among different HDD/SSD drive combinations
Figure 33: Comparison of Max Delta among different HDD/SSD drive combinations


This graph illustrates the average disk write queue for one drive:

Figure 34: Comparison of Avg Disk Write Queue among different HDD/SSD drive combinations
  • The queue starts to increase non linearly around 360 ports, which makes that number close to maximum port capacity of the hard drive.
  • In the three graphs above: with only one HDD drive, both max jitter and max delta started to increase dramatically from 330 ports and higher. Thus: Preferable/Recommended = 330 ports; Peak Port Capacity = 360 ports. In Table: Disk IOPS of sum of all 6 VMs of dual hex cores, MP3 only, IOPS is 51 for 330 ports; while IOPS is around 60 for 360 ports. Thus: Preferable/Recommended IOPS = 51; maximum IOPS for one 15 krpm SAS HDD = 60.
  • With multiple HDDs (2 or 3) to split the load, peak port capacity is nearly the same as SSD—660 ports since the load per drive would be 330 (for 2 HDD drives) and 220 (for 3 HDD drives). Max jitter does not exhibit big differences for these three configurations. But max delta shows a higher delay for 3 HDDs compared to SSD, and 2 HDDs compared to 3 HDDs. Thus: with strict audio quality required in these scenarios, fast media such as SSD will help improve latency and minimize any potential audio quality issues.


Data Throughput

These two formulas estimate data throughputs:

Formula 1 (for MP3 only):
MP3 bitrate * Ports / 8 = KB/sec
Or 32kbps * Ports / 8 =KB/sec if MP3 is 32kbps

Formula 2 (for MP3 + wav):
(MP3 bitrate + WAV bitrate) * Ports / 8 = KB/sec
Or (32 kbps + 128 kpbs) * Ports / 8 = 160 kbps / 8 = KB/sec if 32kpbs MP3 + wav

Six VM configurations, with SSD as the cache folder for MCP recording, produced the following measurements from testing for test SW Profile 1 (MP3 32 kbps only):

Table: Data Throughputs for MP3 32 kbps only

Ports Overall Disk (kbps) SSD Drive Disk (kbps)
Total Reads Writes Total Reads Writes
120 554.10 0.12 553.98 536.98 0.00 536.98
240 1075.70 0.50 1075.19 1053.19 0.00 1053.19
300 1332.61 0.06 1332.55 1308.69 0.00 1308.69
360 1601.09 0.00 1601.09 1575.02 0.00 1575.02
420 1847.91 0.00 1847.91 1822.30 0.00 1822.30
480 2109.57 0.00 2109.57 2082.49 0.00 2082.49
540 2461.25 0.00 2461.25 2434.04 0.00 2434.04
600 2728.83 0.00 2728.83 2702.57 0.00 2702.57
660 3010.07 0.00 3010.07 2982.84 0.00 2982.84
720 3310.64 0.00 3310.64 3280.45 0.00 3280.45

Apply Formula 1 to the 120-port and 600-port samples from the table above to achieve these results:

32 kpbs * 120 / 8 = 480 kb close to 534 in the table (in SSD)
32 kpbs * 600 / 8 = 2400 kb close to 2703 in the table (in SSD)

The measurements from real testing are slightly higher than calculations. Because other files such as metadata and JSON files are saved in the same cache folder, the formula might need adjusting.


Below is the table from testing measurement for SW Profile 2 (MP3 + wav) on the same 6 VM configuration with SSD as cache folder of MCP recording:

Table: Data Throughputs for MP3 32 kbps + wav

Ports Overall Disk (kbps) SSD Drive Disk (kbps)
Total Reads Writes Total Reads Writes
120 2444.255 0.25 2444.01 2427.025 0.000 2427.025
240 4881.163 0.01 4881.15 4859.951 0.000 4859.951
300 6083.649 0.00 6083.64 6058.294 0.000 6058.294
360 7380.491 0.00 7380.49 7354.970 0.000 7354.970
420 8547.663 0.00 8547.66 8522.034 0.000 8522.034
480 9828.785 0.00 9828.79 9802.991 0.000 9802.991
540 11093.931 0.00 11093.93 11067.838 0.000 11067.838
600 12335.993 0.01 12335.99 12309.182 0.000 12309.182

Apply Formula 2 to the 120-port and 600-port samples in the table above to achieve these results:

(32 kbps + 128 kbps) * 120 / 8 = 2400 kb close to 2427 in the table
(32 kbps + 128 kbps) * 600 / 8 = 12000 kb close to 12309 in the table


MCP IOPS

Important
IOPS is measured as it is related to MCP only, and includes IOPS for MP3, WAV and Log files (Interaction level verbosity).

A single HDD local hard drive was used for testing because the HDD itself could become the bottleneck. These tests focus on disk IOPS measurement and calculation, and certain real deployment scenarios require that a local drive not be used. Thus, the measurement for MCP IOPS is useful to calculate overall IO requirement. The three tables below offer three typical MCP IOPS configurations:

Table: MCP IOPS on physical server of single hex core, MP3 only

Ports Physical MCP IOPS (kbps)
Total Read Write
60 21.88 14.93 6.95
120 43.25 29.64 13.60
180 63.28 43.37 19.91
200 71.47 49.35 22.12
210 74.95 51.58 23.37
220 78.40 53.93 24.47
230 82.51 56.83 25.68
240 85.28 58.37 26.90


Table: MCP IOPS on 6 VMs of dual hex core, MP3 only

MP3 only
Ports
Overall 6 VM MCP IOPS (kbps)
Total Read Write
120 43.672 29.679 13.993
240 87.477 59.280 28.197
300 109.292 73.848 35.445
360 130.965 88.917 42.047
420 151.706 103.095 48.611
480 171.290 116.053 55.237
540 194.772 132.867 61.905
600 215.101 146.882 68.219
660 236.654 161.990 74.664
720 259.100 177.279 81.820


Table: MCP IOPS on 6 VMs of dual hex core, MP3 + wav

MP3 + wav
Ports
Overall 6 VM MCP IOPS (kbps)
Total Read Write
120 173.607 146.092 27.515
240 348.369 292.884 55.486
300 434.511 364.898 69.613
360 522.447 439.412 83.035
420 600.880 504.682 96.198
480 693.861 583.965 109.896
540 780.187 656.958 123.229
600 859.359 722.907 136.453
660 790.737 664.024 126.713


The graph below compares the two tables MCP IOPS on physical server of single hex core, MP3 only and MCP IOPS on 6 VMs of dual hex core, MP3 only:

Figure 40: MCP IOPS, Physical VS VMs


The graph below compares the two tables MCP IOPS on 6 VMs of dual hex core, MP3 only and MCP IOPS on 6 VMs of dual hex core, MP3 + wav:

Figure 41: MCP IOPS, MP3 only VS MP3 + WAV

The MCP IOPS is related to the test profile and ports, but on the same physical server and VMs.


MP3 16KBPS Bit Rate Compression

Support for MP3 16 kbps bit rate recording compression began with The GVP 8.5.1 release in December 2014. We tested performance on physical server and Virtual Machine (VM) environments, using Windows 2008 R2 x64.

Physical Server on Single Hex Core

Testing was performed on Hardware Profile 1: a physical server on a single hex core of Dell R410. The three graphs below compare system CPU usage and audio quality related metrics, max jitter and max delta.

Figure 42: Comparison of System CPU Usage, MP3 16kbps vs 32kbps on Physical Server
Figure 43: Comparison of Max Jitter, MP3 16kbps vs 32kbps on Physical Server
Figure 44: Comparison of Max Delta MP3 16kbps vs 32kbps on a Physical Server

MP3 16kbps consumes less CPU memory, which means higher port capacity. The two graphs above that compare Max Jitter with Max Delta also indicate the higher port capacity of MP3 16kbps. Recommended port capacity for MP3 16kbps: 240 ports (20% higher than the 200 recommended port capacity for MP3 32kbps). Peak port capacity: 270 ports (22.7% higher than the 220 peak port capacity for MP3 32kbps).

The table below lists the system disk IOPS:

Figure 45: System Disk IOPS on Physical Server, MP3 only 16 Kbps

Ports Physical Server Disk IOPS (kbps)
Total Reads Writes
60 14.66 0.036 14.62
120 24.00 0.041 23.95
180 33.42 0.029 33.39
210 37.65 0.030 37.62
240 42.21 0.029 42.18
270 47.18 0.036 47.14
300 51.44 0.011 51.43
330 55.81 0.006 55.81
360 60.99 0.002 60.99
390 67.12 0.003 67.11

The graph below compares Table: System Disk IOPS on Physical Server, MP3 only 16 Kbps with Table: Disk IOPS of system level from a physical server with a single hex core, both on a single hex core server:

Figure 46: Comparison of System Disk IOPS on Single Hex Core Physical Server, MP3 16 Kbps vs 32 Kbps

The system disk IOPS for MP3 16kbps and 32kbps are nearly identical to each other; reasonable since the disk IO operations should be the same, and at the same port capacity, no matter which MP3 bit rate is chosen.

MCP IOPS is listed here:

Figure 47: MCP IOPS on physical server of single hex core, MP3 only, 16 Kbps

Ports Physical Server MCP IOPS (kbps)
Total Reads Writes
60 14.56 7.53 7.04
120 28.64 14.92 13.72
180 42.54 22.29 20.25
210 49.42 25.93 23.48
240 56.41 29.64 26.76
270 63.38 33.34 30.04
300 70.36 36.92 33.44
330 77.53 40.79 36.74
360 85.52 44.46 41.06
390 94.68 48.14 46.54

The graph below compares Table: MCP IOPS on physical server of single hex core, MP3 only, 16 Kbps and Table: MCP IOPS on physical server of single hex core, MP3 only:

Figure 48: MCP IOPS on Single Hex Core Physical Server, MP3 16Kbps vs 32Kbps

MP3 16kbps uses less IOPS at the process level, probably be due to fewer network operations for MP3 16kbps.

VMs on Dual Hex Cores Server

The testing for MP3 16kbps was conducted on VM Profile 4 (based on Hardware Profile 4, which is a dual hex cores server). 6 VMs were configured, while only one MCP was installed on each Windows VM. The three graphs below compare overall CPU usage, audio quality related max jitter and max delta for MP3 16kbps vs 32kbps:

Figure 49: Comparison System CPU Usage of MP3 16kbps vs 32kbps on VM env
Figure 50: Comparison of Max Jitter, MP3 16kbps vs 32kbps on VM env
Figure 51: Comparison of Max Delta, MP3 16kbps vs 32kbps on VM environment

MP3 16kbps consumes less CPU memory, which matches test results on a physical server in Figure: Comparison of System CPU Usage, MP3 16kbps vs 32kbps on Physical Server. Both Max Jitter and Max Delta also show a higher port capacity for MP3 16kbps compression, which also matches test results on a physical server from Figure 4: Comparison of Max Jitter, MP3 16kbps vs 32kbps on Physical Server & Figure: Comparison of Max Delta MP3 16kbps vs 32kbps on a Physical Server. Preferred/Recommended port capacity for MP3 16 kbps: 720 ports (20% higher 600 ports for than MP3 32kbps). It’s the same increase as observed from a physical server. Peak port capacity for MP3 16kpbs can be as high as 840 ports (27.3% higher than 660 peak port capacity for MP3 32kbps.

The table below illustrates system disk IOPS:

Figure 52: Overall Disk IOPS on all 6 VMs of dual hex cores, MP3 only, 16 Kbps

Ports Overall 6 VMs Disk IOPS (kbps)
Total Reads Writes
120 26.57 0.13 26.44
360 63.47 0.13 63.34
480 80.66 0.15 80.51
600 93.73 0.04 93.69
660 109.53 0.14 109.39
720 118.76 0.13 118.62
780 126.15 0.07 126.08
840 134.12 0.04 134.09
900 142.21 0.09 142.12

The graph below compares overall disk IOPS of all 6 VMs for MP3 16kpbs against 32kbps in Table: Disk IOPS of sum of all 6 VMs of dual hex cores, MP3 only':

Figure 53: Comparison of Overall 6 VMs Disk IOPS MP3 16 kbps vs 32 kbps

The IOPS from both MP3 16kbps and 32kbps are inline with each other, as in the physical server tests.

Data throughput for MP3 16kbps is listed in following table:

Figure 54: Data Throughputs for MP3 only, 16 kbps

Ports Overall Disk (kbps) SSD Drive Disk (kbps)
Total Reads Writes Total Reads Writes
120 318.17 0.68 317.49 296.313 0.001 296.312
360 892.94 0.52 892.42 856.077 0.001 856.076
480 1175.63 0.79 1174.84 1132.997 0.001 1132.996
600 1537.43 0.19 1537.23 1510.543 0.000 1510.543
660 1729.45 0.58 1728.87 1680.374 0.003 1680.371
720 1890.48 0.58 1889.90 1837.492 0.000 1837.492
780 2045.34 0.35 2045.00 1995.239 0.004 1995.235
840 2191.98 0.15 2191.83 2142.373 0.002 2142.371
900 2349.18 0.75 2348.44 2298.426 0.004 2298.422

Using this formula:
MP3 bitrate * Ports / 8 = kbps
...where MP3 bitrate=16kbps and Ports = 120 and 720 from the table above,

The results...
16 kpbs * 120 / 8 = 240 kbps (compared to 296 in the table -- in SSD)
and
16 kpbs * 720 / 8 = 14400 kbps (compared to 1837 in the table -- in SSD)

...from real testing for MP3 16kbps are slightly higher than calculations predict, due to other files such as metadata and JSON files being saved in the same cache folder. So the formula still stands.

The following table lists MCP IOPS:

Figure 55: Overall MCP IOPS from 6 VMs of dual hex core, MP3 only, 16kbps

Ports Overall 6 VMs MCP IOPS (kbps)
Total Reads Writes
120 28.931 14.915 14.016
360 86.517 44.456 42.061
480 114.574 59.153 55.421
600 142.112 73.730 68.382
660 156.495 81.359 75.136
720 170.237 88.660 81.577
780 184.173 96.048 88.125
840 197.767 103.263 94.504
900 211.644 110.545 101.099

The graph below compares Overall MCP IOPS with MP3 32k MCP IOPS, and shows the same trend of physical server results that appeared in Figure: MCP IOPS on Single Hex Core Physical Server, MP3 16Kbps vs 32Kbps:

Figure 56: MCP IOPS on 6 VMs of Dual Hex Cores, MP3 16Kbps vs 32Kbps


MP3 16KBPS Bit Rate Compression with Encryption

We tested the MP3 16 kbps bit rate with encryption, using the dest2 physical server and Vitual Machine (VM) environments, which compares with results of non-encryption from MP3 16 kbps Bit Rate without Encryption. The OS remained Windows 2008 R2 x64.

Physical Server on Single Hex Core

These tests were performed on Hardware Profile 1: a physical server on a single hex core of Dell R410. The three graphs below compare system CPU usage and audio quality-related metrics, max jitter and max delta.

Figure 57: Comparison of Physical Server System CPU Usage of MP3 16kbps encryption vs non-encryption
Figure 58: Comparison of Physical Server Max Jitter of MP3 16kbps encryption vs non-encryption
Figure 59: Comparison of Physical Server Max Delta of MP3 16kbps encryption vs non-encryption

In the graphs above, encryption consumes slightly higher system CPU than does non-encryption. Max Jitter and Max Delta consume much more CPU with encryption, than without. If a slightly higher delay due to latency introduced by encryption is acceptable, then recommended and preferred port capacity would be 210 ports—only a 12.5% reduction from the peak capacity of 240 ports offered by non-encryption. If the audio quality strictly applies, then the recommended port capacity can be as low as 120 ports. Peak port capacity could be the same 270 ports as non-encryption, if the delay is acceptable.

The table below lists system disk IOPS:

Figure 60: IOPS on physical server of single hex core, MP3 only, 16 Kbps, encryption

Ports Physical Server Disk IOPS
Total Reads Writes
60 14.66 0.036 14.62
120 24.00 0.041 23.95
180 33.42 0.029 33.39
210 37.65 0.030 37.62
240 42.21 0.029 42.18
270 47.18 0.036 47.14
300 51.44 0.011 51.43
330 55.81 0.006 55.81
360 60.99 0.002 60.99
390 67.12 0.003 67.11

The graph below compares system disk IOPS on a physical server IOPS with non-encryption:

Figure 61: Comparison of System Disk IOPS on Single Hex Core Physical Server, MP3 16kbps encryption vs non-encryption

System disk IOPS is nearly the same for encryption and non-encryption; both increase slightly at a higher port capacity. Some of that can be attributed by other disk IO operations, such as encryption key files.

The table below lists MCP IOPS:

Figure 62: MCP IOPS on physical server of single hex core, MP3 only, 16 Kbps, encryption

Ports Physical Server MCP IOPS
Total Reads Writes
60 16.53 8.88 7.65
120 32.59 17.69 14.91
150 40.40 21.96 18.44
180 48.46 26.46 22.01
210 56.35 30.83 25.52
240 64.32 35.24 29.08
270 72.28 39.64 32.64
300 80.06 43.95 36.11
330 88.61 48.53 40.07
360 100.48 52.91 47.57

The graph below compares total MCP IOPS between encryption and non-encryption:

Figure 63: MCP IOPS on Single Hex Core Physical Server, MP3 16Kbps encryption vs non-encryption

MCP IOPS for encryption increases when port capacity increases. As seen in Figure: Comparison of System Disk IOPS on Single Hex Core Physical Server, MP3 16kbps encryption vs non-encryption, increase for disk IOPS is much smaller for encryption, so here the increase should be attributed to network IOs.

VMs on Dual Hex Cores Server

The testing for MP3 16kbps with encryption was conducted on the VM Profile 4 based on Hardware Profile 4 of a dual hex cores server, same as non-encryption in the 16knps tests VMs on Dual Hex Cores Server. Six VMs were configured while only one MCP was installed on each Windows VM. Below are three graphs comparing overall CPU usage, audio quality related max jitter and max delta for MP3 16kbps encryption vs non-encryption:

Figure 64: Comparison of Overall VMs CPU Usage of MP3 16kbps encryption vs non-encryption
Figure 65: Comparison of Overall VMs Max Jitter of MP3 16kbps encryption vs non-encryption
Figure 66: Comparison of Overall VMs Max Delta of MP3 16kbps encryption vs non-encryption

The VM environment exhibits a similar trend: slightly overall CPU usage for the encryption profile, and much higher for max jitter and max delta. Applying the same criteria from the physical server results, if a slightly higher delay (due to latency introduced by encryption) is acceptable, then the recommended and preferred port capacity could be 600 ports—only a 16.7% reduction of the peak 720 ports with non-encryption. If audio quality strictly applies, the recommended ports can be as low as 480 ports. And if some delay is acceptable, then the peak port capacity can be the same 840 ports as non-encryption.

The overall system disk IOPS for all 6 VMs is listed below:

Figure 67: Overall Disk IOPS on all 6 VMs of dual hex cores, MP3 only, 16 Kbps, encryption

Ports Overall 6 VMs Disk IOPS SSD Drive Disk IOPS
Total Reads Writes Total Reads Writes
120 28.70 0.004 28.69 21.881 0.000 21.881
360 67.46 0.004 67.46 56.238 0.000 56.238
480 87.56 0.026 87.54 74.903 0.000 74.903
600 108.01 0.015 107.99 93.647 0.000 93.647
660 119.49 0.005 119.48 104.304 0.000 104.304
720 128.76 0.020 128.74 114.441 0.000 114.441
780 137.68 0.015 137.66 123.210 0.002 123.209
840 146.99 0.009 146.98 132.646 0.002 132.644
900 154.68 0.025 154.66 140.145 0.002 140.143

The graph below compares system disk IOPS with encryption and with non-encryption, on the same VM environment:

Figure 68: Comparison of Overall 6 VMs Disk IOPS MP3 16 kbps encryption vs non-encryption

As with the physical server tests, encryption increases as port capacity increases. Also as with the physical server tests, some of that can be attributed to extra disk IO operations.

The table below lists Data throughputs for encryption:

Figure 69: Data Throughputs for MP3 only, 16 kbps, encryption

Ports Overall Disk KB/sec SSD Drive Disk KB/sec
Total Reads Writes Total Reads Writes
120 387.99 0.02 387.97 304.229 0.000 304.229
360 1096.82 22.54 1074.28 876.599 0.000 876.599
480 1344.60 107.95 1236.65 1191.403 0.006 1191.397
600 2187.50 348.40 1839.09 1532.171 0.000 1532.171
660 2024.16 35.09 1989.07 1652.232 0.000 1652.232
720 1955.33 99.81 1855.51 1803.207 0.006 1803.201
780 2572.79 205.15 2367.64 1982.733 0.024 1982.709
840 2534.97 28.65 2506.32 2097.871 0.043 2097.829
900 2851.85 119.47 2732.38 2297.264 0.007 2297.257

Using Formula 1...

MP3 bitrate * Ports / 8 = KB/sec
Or 16kbps * Ports / 8 =KB/sec if MP3 is 16kbps

...take two samples (120 & 720) from the above table above, and apply them to Formula 1:

16 kpbs * 120 / 8 = 240 kb close to 304 in the table (in SSD)
16 kpbs * 720 / 8 = 14400 kb close to 1803 in the table (in SSD)

The measurements from real testing for MP3 16kbps encryption are slightly higher than these calculations predict, due to other file, such as metadata and JSON files, being saved on the same cache folder.

The graph below compares overall data throughputs with no encryption:

Figure 70: Comparison of Overall 6 VMs Data Throughputs MP3 16 kbps encryption vs non-encryption

The data throughputs for encryption increase slightly when port capacity increases, matching a similar trend with system disk IOPS.

The table below lists overall MCP IOPS from all 6 VMs:

Figure 71: Overall MCP IOPS from 6 VMs of dual hex core, MP3 only, 16kbps, encryption

Ports Overall 6 VMs MCP IOPS
Total Reads Writes
120 34.874 17.638 17.236
360 102.624 52.900 49.724
480 130.285 70.377 59.909
600 168.849 87.766 81.083
660 186.175 96.882 89.293
720 193.248 105.171 88.077
780 219.395 114.398 104.997
840 235.730 123.009 112.720
900 252.198 131.682 120.516

The graph below compares performance of the same configuration, except with non-encryption:

Figure 72: MCP IOPS from 6 VMs of dual hex core, MP3 only, 16kbps, encryption vs non-encryption

MCP IOPS performance is affected slightly by encryption, similar to the trend expressed in the physical server results.


Detailed Studies of GVP Media Server Behavior on Linux

Because MP3 16kbps produces better results than 32kbps on Windows, Linux tests focused on MP3 16kpbs testing profiles. Also based on previous results on Windows, we chose two typical Hardware Profiles for Linux testing: Hardware Profile 1 for physical server testing, and Hardware Profile 4 with Virtual Machine Profile 4 for virtual machine environment testing.

Parameter Adjustments

These adjustments achieve higher port capacity:

Parameter Default Value Adjusted Value
mpc.recordnumparallelpost 30 300
mpc.recordpostretrybackoff 120000 15000
mpc.recordpostretrycount 3 1
mpc.mediamgr.recordwritetimeinterval 1000 10000
fm.http_proxy (empty) (squid bypassed)

Comparisons with Windows

Physical Server on Single Hex Core

These tests use Software Profile 1a on Hardware Profile 1 for a physical server. Here are three graphs illustrating overall system CPU usage, MCP CPU usage and memory usage:

Figure 73: Comparison of System CPU Usage on a physical server, MP3 16kbps without encryption, RH EL 6.5 vs. Windows 2008 R2


Figure 74: Comparison of MCP CPU Usage on a physical server, MP3 16kbps without encryption, RH EL 6.5 vs. Windows 2008 R2


Figure 75: Comparison of MCP Memory Usage on a physical server, MP3 16kbps without encryption, RH EL 6.5 vs. Windows 2008 R2

Linux uses more resources (CPU, memory etc) than Windows, and so lower capacity is achieved on Linux with a 37.5% reduction (150 vs. 240) for preferred ports and a 22.2% reduction (210 vs. 270) for peak ports.

The two graphs below compare audio quality in terms of max jitter and max delta:

Figure 76: Comparison of Max Jitter on A physical server, MP3 16kbps without encryption, RH EL 6.5 vs. Windows 2008 R2


Figure 77: Comparison of Max Delta on A physical server, MP3 16kbps without encryption, RH EL 6.5 vs. Windows 2008 R2

Note that Max Jitter is somewhat close between Windows and Linux. But Linux has a lower value at lower ports and a slightly higher value on higher ports. The Max Delta shows that Linux has the higher value even though it is nearly the same for both Windows and Linux at lower ports.

System disk IOPS is illustrated in this table for Linux EL 6.5:

Figure 78: System Disk IOPS on a physical server of single hex core on EL 6.5, MP3 only 16 kbps
Ports Disk IOPS Physical Server
Total Reads Writes
60 12.75 0.000 12.754
120 23.12 0.000 23.117
150 27.65 0.000 27.645
180 32.15 0.000 32.150
210 36.73 0.000 36.729
240 41.57 0.000 41.568

The graph below compares System Disk IOPS performance on Linux and Windows physical servers:

Figure 79: Comparison of System Disk IOPS on a Physical Server, MP3 16kbps without encryption, RH EL 6.5 vs. Windows 2008 R2

Note that IOPS on both Windows and Linux are similar; and so Disk IOPS is related to the test profile, and irrelevant to a particular OS. So the IOPS numbers from the previous Windows testing can be used generally for both Windows and Linux.

VMs on Dual Hex Cores Server

These tests use SW Profile 1a on HW Profile 1 with VM Profile 4 for virtual machine environment testing. Below are three graphs illustrating overall system CPU usage, MCP CPU usage and memory usage:


Figure 80: Comparison of System CPU Usage on VM env, MP3 16kbps without encryption, RH EL 6.5 vs. Windows 2008 R2

You can observe the same trend as with the physical server results in the previous section Linux consumes more CPU resources. Below are two graphs of audio quality-related metrics that show the same thing.


Figure 81: Comparison of Max Jitter on VM env, MP3 16kbps without encryption, RH EL 6.5 vs. Win 2008 R2
Figure 82: Comparison of Max Delta on VM env, MP3 16kbps without encryption, RH EL 6.5 vs. Windows 2008 R2

As observed on the above graphs, 540 ports are recommended and preferred. This is a 25% reduction compared with Windows 2008 R2 (540 vs. 720). Peak capacity would be 660 ports, which is a 21.4% reduction compared to Windows 2008 R2 (660 vs. 840). Similar reductions were also observed on physical server tests in the previous section.

The disk IOPS is displayed here:

Figure 83: Disk IOPS from overall 6 VMs of dual hex core, MP3 only 16 kbps, on EL 6.5
Ports Overall 6 VMs Disk IOPS SSD Drive Disk IOPS
Total Reads Writes Total Reads Writes
120 28.17 0.00 28.17 24.011 0.000 24.011
240 49.78 0.00 49.78 44.590 0.000 44.590
360 71.11 0.00 71.11 65.747 0.000 65.747
420 81.59 0.00 81.59 76.058 0.000 76.058
480 92.37 0.00 92.37 86.767 0.000 86.767
540 102.96 0.00 102.96 97.305 0.000 97.305
600 112.33 0.00 112.33 106.727 0.000 106.727
660 122.06 0.00 122.06 116.440 0.000 116.440
720 130.82 0.00 130.82 125.121 0.000 125.121

The graph below compares the above table with the corresponding table for Windows, for the same MP3-only 16 kbps profile:

Figure 84: Comparison of System Disk IOPS on VM env, MP3 16kbps without encryption, RH EL 6.5 vs. Windows 2008 R2

Note that disk IOPS results for Linux and Windows are very close, and corresponds to the results on a physical server in the previous section.

The data throughput for this MP3-only profile on EL 6.5 is illustrated below:

Figure 85: Data throughputs from overall 6 VMs of dual hex core, MP3 only 16 kbps, on EL 6.5
Ports Overall Disk KB/sec SSD Drive Disk KB/sec
Total KB/sec Read KB/sec Write KB/sec Total KB/sec Read KB/sec Write KB/sec
120 417.70 0.00 417.70 389.474 0.000 389.474
240 788.58 0.00 788.58 751.418 0.000 751.418
360 1145.77 0.00 1145.77 1104.237 0.000 1104.237
420 1317.38 0.00 1317.38 1274.484 0.000 1274.484
480 1496.20 0.00 1496.20 1451.114 0.000 1451.114
540 1677.83 0.00 1677.83 1627.798 0.000 1627.798
600 1843.65 0.00 1843.65 1795.706 0.000 1795.706
660 2023.36 0.00 2023.36 1974.070 0.000 1974.070
720 2193.62 0.00 2193.62 2142.769 0.000 2142.769

Comparing MP3 only and MP3 + WAV

Physical Server on Single Hex Core

This test uses SW Profile 2a (MP3 16 kbps + wav without encryption) on HW Profile 1 for a physical server, compared which SW Profile 1a (MP3 only 16 kbps without encryption) is used as a baseline for comparison. Below are three graphs illustrating overall system CPU usage, MCP CPU usage and memory usage:

Figure 86: Comparison of System CPU Usage on a Physical Server, MP3 + wav vs. MP3 only, on RH EL 6.5


Figure 87: Comparison of MCP CPU Usage on a Physical Server, MP3 + wav vs. MP3 only, on RH EL 6.5


Figure 88: Comparison of MCP Memory Usage on a Physical Server, MP3 + wav vs. MP3 only, on RH EL 6.5

The comparison shows apparent higher MCP usage and overall system CPU usage for the MP3 + wav profile. However, the MCP memory usage is not significantly higher.

The audio quality metric also shows some differences, below:

Figure 89: Comparison of Max Jitter on a Physical Server, MP3 + wav vs. MP3 only, on RH EL 6.5
Figure 90: Comparison of Max Delta on a Physical Server, MP3 + wav vs. MP3 only, on RH EL 6.5

Note that lower capacities would be achieved for the MP3 + WAV profile. 90 ports would be recommended and preferred, a 40% reduction (90 vs. 150) compared with the MP3-only profile, while 150 ports would be peak capacity a 28.6% reduction (150 vs. 210).

System disk IOPS is listed in the following table:

Figure 91: System Disk IOPS on a physical server of single hex core on EL 6.5, MP3 16 kbps + wav
Ports Physical Server Disk IOPS
Total Reads Writes
30 15.18 0.008 15.17
60 26.70 0.000 26.70
90 35.53 0.003 35.53
120 46.04 0.002 46.04
150 55.44 0.000 55.44
180 65.50 1.520 63.98

The graph below compares disk IOPS with the MP3-only profile:

Figure 92: Comparison of System Disk IOPS on a Physical Server, MP3 + wav vs. MP3 only, on RH EL 6.5

The MP3-only profile is almost double the disk IOPS for MP3 + wav profile, as observed in the Windows testing.

VMs on Dual Hex Cores Server

A similar trend of overall CPU usage occurs in the Virtual Machine environment.

Figure 93: Comparison of System CPU Usage on VM env, MP3 + wav vs. MP3 only, on RH EL 6.5

The audio quality metrics shows similar trends as on a physical server.

Figure 94: Comparison of Max Jitter on VM env, MP3 + wav vs. MP3 only, on RH EL 6.5


Figure 95: Comparison of Max Delta on VM env, MP3 + wav vs. MP3 only, on RH EL 6.5

480 ports are recommended and preferred for this MP3 + wav profile, an 11.1% reduction (480 vs. 540 for MP3 only); peak would be 660, a 9.1% reduction (600 vs. 660 for MP3 only).

Below is a table to illustrate overall 6 VMs disk IOPS:

Figure 96: System Disk IOPS on a VM environment of dual hex cores on EL 6.5, MP3 16 kbps + wav
Ports Overall 6 VMs Disk IOPS SSD Drive Disk IOPS
Total Reads Writes Total Reads Writes
120 52.99 0.00 52.99 48.728 0.000 48.728
240 100.50 0.00 100.50 95.174 0.000 95.174
360 144.34 0.00 144.34 138.864 0.000 138.864
420 164.65 0.00 164.65 158.979 0.000 158.979
480 183.45 0.00 183.45 177.711 0.000 177.711
540 207.27 0.00 207.27 201.564 0.000 201.564
600 224.97 0.00 224.97 219.197 0.000 219.197
660 275.49 0.00 275.49 269.584 0.000 269.584
720 187.34 0.00 187.33 179.984 0.001 179.983

Compared with the MP3-only profile, overall 6 VM disk IOPS for MP3 + wav profile shows almost double IOPS, as in the previous physical server section.


Figure 97: Comparison of System Disk IOPS on VM env, MP3 + wav vs. MP3 only, on RH EL 6.5

The table below illustrats overall data throughput for this MP3 + wav profile on VMs of RH EL 6.5 environment.

Figure 98: Data throughputs from overall 6 VMs of dual hex core, MP3 16 kbps + wav, on EL 6.5
Ports Overall Disk KB/sec SSD Drive Disk KB/sec
Total KB/sec Read KB/sec Write KB/sec Total KB/sec Read KB/sec Write KB/sec
120 2376.30 0.00 2376.30 2347.222 0.000 2347.222
240 4684.79 0.00 4684.79 4646.371 0.000 4646.371
360 6975.83 0.00 6975.83 6933.441 0.000 6933.441
420 8100.79 0.00 8100.79 8056.843 0.001 8056.842
480 9242.32 0.00 9242.32 9195.871 0.001 9195.871
540 10391.78 0.00 10391.78 10344.249 0.001 10344.249
600 11512.54 0.00 11512.54 11462.150 0.001 11462.149
660 12804.19 0.01 12804.18 12752.305 0.001 12752.304
720 9380.58 0.00 9380.58 9336.194 0.003 9336.191

Encryption

MP3 16 kbps Only on a Physical Server of Single Hex Core

This is SW Profile 3a (MP3 16 kbps only with encryption) on HW Profile 1 for a physical server which SW Profile 1a (MP3 only 16 kbps without encryption) is used as baseline to compare with. Here are the three graphs illustrating overall system CPU usage, MCP CPU usage and memory usage:


Figure 99: Comparison of System CPU Usage on a Physical Server, MP3 only 16 kbps encryption vs. non-encryption, on RH EL 6.5


Figure 100: Comparison of MCP CPU Usage on a Physical Server, MP3 only 16 kbps encryption vs. non-encryption, on RH EL 6.5
Figure 101: Comparison of MCP Memory Usage on a Physical Server, MP3 only 16 kbps encryption vs. non-encryption, on RH EL 6.5

It can be observed that both system CPU and MCP CPU are quite inline to each other between encryption and non-encryption profiles while MCP memory for encryption is slightly higher than non-encryption.

Let us look at audio quality metrics further:

Figure 102: Comparison of Max Jitter on a Physical Server, MP3 only, Encryption vs. Non-encryption on EL 6.5


Figure 103: Comparison of Max Delta on a Physical Server, MP3 only, Encryption vs. Non-encryption on EL 6.5

Max Jitter is similar for both encryption and non-encryption scenarios, as are the Max Delta metrics. Thus, the preferred ports (540) and peak ports (660) for encryption are the same as for non-encryption.

System disk IOPS is illustrated below:

Figure 104: System Disk IOPS on a Physical Server on EL 6.5, MP3 16 kbps only, Encryption
Ports Physical Server Disk IOPS
Total Reads Writes
30 8.12 0.000 8.122
60 14.22 0.000 14.220
90 19.98 0.000 19.975
120 25.12 0.000 25.122
150 30.62 0.000 30.621
180 35.07 0.000 35.074
210 39.83 0.000 39.828
240 44.74 0.000 44.739

The graph below compares encryption with non-encryption:

Figure 105: Comparison of System Disk IOPS on a Physical Server, MP3 only, on EL 6.5, Encryption vs. Non-encryption

Slightly higher system disk IOPS occurs in the encryption scenario, likely caused by the extra key/pem files required for encryption.

MP3 16 kbps Only on VMs of Dual Hex Cores

This test uses SW Profile 3a (MP3 16 kbps only with encryption) on VM Profile 4 configured as HW Profile 0 for a VM environment, compared with SW Profile 1a (MP3 only 16 kbps without encryption) on the same hardware specification. Below are graphs illustrating overall system CPU usage and memory usage:


Figure 106: Comparison of System CPU Usage on VMs, MP3 only 16 kbps encryption vs. non-encryption, on RH EL 6.5


Figure 107: Comparison of MCP Memory Usage on VMs, MP3 only 16 kbps encryption vs. non-encryption, on RH EL 6.5

As observed in previous physical server graphs, CPU usage is almost the same for both encryption and non-encryption, while MCP memory usage is slightly higher for encryption.

Consider audio quality metrics:


Figure 108: Comparison of Max Jitter on VMs, MP3 only, Encryption vs. Non-encryption on EL 6.5


Figure 109: Comparison of Max Delta on VMs, MP3 only, Encryption vs. Non-encryption on EL 6.5

Similar trends can be observed in the previous physical server section that both encryption and non-encryption achieved similar value for both Max Jitter and Max Delta. So the preferred ports (540) and peak ports (660) for encryption would be the same as non-encryption.

Figure 110: Overall System Disk IOPS on VMs of EL 6.5, MP3 16 kbps only, Encryption
Ports Overall 6 VMs Disk IOPS SSD Drive Disk IOPS
Total Reads Writes Total Reads Writes
120 30.44 0.00 30.44 25.997 0.000 25.997
240 53.41 0.00 53.41 47.939 0.000 47.939
360 75.57 0.00 75.57 70.011 0.000 70.011
420 86.37 0.00 86.37 80.600 0.000 80.600
480 97.32 0.00 97.32 91.564 0.000 91.564
540 108.20 0.00 108.20 102.393 0.000 102.393
600 117.95 0.00 117.95 112.132 0.000 112.132
660 127.85 0.00 127.85 121.911 0.000 121.911
720 136.85 0.00 136.85 130.951 0.000 130.951

The graph below compares encryption with non-encryption:


Figure 111: Comparison of System Disk IOPS on VM env, MP3 only, on EL 6.5, Encryption vs. Non-encryption

As in the previous physical server section, system disk IOPS for encryption is slightly higher than non-encryption.

Data throughput is illustrated in this table:

Figure 112: Data throughputs from overall 6 VMs of dual hex core, MP3 16 kbps only, encryption, on EL 6.5
Ports Overall Disk KB/sec SSD Drive Disk KB/sec
Total KB/sec Read KB/sec Write KB/sec Total KB/sec Read KB/sec Write KB/sec
120 435.55 0.00 435.55 403.192 0.000 403.192
240 822.06 0.00 822.06 780.379 0.000 780.379
360 1186.43 0.00 1186.43 1140.874 0.000 1140.874
420 1359.14 0.00 1359.14 1311.668 0.000 1311.668
480 1549.49 0.00 1549.49 1500.982 0.000 1500.982
540 1719.89 0.00 1719.89 1669.506 0.000 1669.506
600 1905.09 0.00 1905.09 1853.208 0.000 1853.208
660 2081.23 0.00 2081.23 2027.495 0.000 2027.495
720 2269.56 0.00 2269.56 2214.658 0.000 2214.658

MP3 16 kbps + wav on VMs of Dual Hex Cores

This test uses SW Profile 4a (MP3 16 kbps + wav with encryption) on VM Profile 4 configured as HW Profile 1 for a VM environment to compare with SW Profile 2a (MP3 16 kbps + wav without encryption) on the same HW spec. Below are two graphs illustrating overall system CPU usage and memory usage:

Figure 113: Comparison of System CPU Usage on VMs, MP3 16 kbps + wav encryption vs. non-encryption, on RH EL 6.5
Figure 114: Comparison of MCP Memory Usage on VMs, MP3 16 kbps + wav encryption vs. non-encryption, on RH EL 6.5

System CPU usage is quite close to each other for both encryption and non-encryption, while MCP memory usage for encryption is slightly higher than for non-encryption, similar to the previous MP3 only test scenarios.

The audio quality metrics of Max Jitter and Max Delta also show similar trends.

Figure 115: Comparison of Max Jitter on VMs, MP3 + wav, Encryption vs. Non-encryption on EL 6.5


Figure 116: Comparison of Max Delta on VMs, MP3 + wav, Encryption vs. Non-encryption on EL 6.5

The recommended and preferred ports for encryption of MP3 + wav would be 480 the same as non-encryption of MP3 + wav, as is 600 for peak ports.

The table below shows overall system disk IOPS, for reference:

Figure 117: Overall System Disk IOPS on VMs of EL 6.5, MP3 16 kbps + wav, Encryption
Ports Overall 6 VMs Disk IOPS SSD Drive Disk IOPS
Total Reads Writes Total Reads Writes
120 53.97 0.00 53.97 49.506 0.000 49.506
240 102.98 0.00 102.98 97.468 0.000 97.468
360 149.87 0.00 149.87 144.235 0.000 144.235
420 171.89 0.00 171.89 166.144 0.000 166.144
480 196.97 0.00 196.97 191.140 0.000 191.140
540 223.52 0.01 223.51 217.663 0.000 217.663
600 246.26 0.03 246.22 240.216 0.000 240.216
660 296.60 0.00 296.60 290.582 0.000 290.582

The graph below compares encryption with non-encryption, and shows the same trend as observed previously in this section:

Figure 118: Comparison of System Disk IOPS on VM env, MP3 + wav, on EL 6.5, Encryption vs. Non-encryption

Data throughput is also listed below as reference:

Figure 119: Data throughputs from overall 6 VMs of dual hex core, MP3 16 kbps + wav, encryption, on EL 6.5
Ports Overall Disk KB/sec SSD Drive Disk KB/sec
Total KB/sec Read KB/sec Write KB/sec Total KB/sec Read KB/sec Write KB/sec
120 2421.76 0.00 2421.76 2373.612 0.000 2373.612
240 4756.37 0.00 4756.37 4699.737 0.000 4699.737
360 7065.62 0.00 7065.62 7004.491 0.000 7004.491
420 8179.23 0.00 8179.23 8116.591 0.000 8116.591
480 9366.53 0.00 9366.53 9301.426 0.000 9301.426
540 10489.26 0.14 10489.12 10423.230 0.000 10423.230
600 11647.29 0.78 11646.51 11574.973 0.000 11574.973
660 12976.30 0.06 12976.24 12905.764 0.001 12905.763
Comments or questions about this documentation? Contact us for support!