BahmniLite Performance Baselining (AWS + Gatling)
This would be a living document to capture various baselining snapshots while :-
- Troubleshooting and applying a patch
- Changing Software or Network configurations
- Adding new scenarios and changing load share
Source Code: https://github.com/Bahmni/performance-test
⭕️ Automation Technology Stack
⭕️ Current Snapshots
40 Concurrent Users - ConfigurationHardwarePerformance environment was running on AWS EKS Custer with single node
Node (EC2: t3-large)
- RAM 8GB
- 2 vCPU
- 100GB Secondary storage
- AWS LINUX x86_64
Database (AWS RDS service: db.t3.xlarge)
- RAM 16GB,
- 4 vCPU (2 core, 2.5 GHz Intel Scalable Processor)
- 100GB Secondary storage
- MySQL, max_connections = 1304
Software
OpenMRS Tomcat - Server
OpenMRS - Heap
- Initial Heap: 1024 MB
- Max Heap: 1536 MB
-Xms1024m -Xmx1536m -XX:NewSize=512m -XX:MaxNewSize=512m -XX:MetaspaceSize=256m -XX:MaxMetaspaceSize=1024m -XX:InitialCodeCacheSize=64m -XX:ReservedCodeCacheSize=96m -XX:SurvivorRatio=4 -XX:TargetSurvivorRatio=40 -XX:+UseParNewGC -XX:ParallelGCThreads=2 -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:+CMSCompactWhenClearAllSoftRefs -XX:CMSInitiatingOccupancyFraction=85 -XX:+CMSScavengeBeforeRemark -XX:+UseGCOverheadLimit -XX:+UseStringDeduplication
Openmrs Connection Pooling
📗 Current Results - 40 Concurrent Users
- Network: 60 Mbps
- Duration: 24 hours
- Ramp Up: 5 mins
- Database pre-state: 75000 patient records
Report Observations:
| Simulations | Scenario | Load share | Patient Count | Min Time (ms) | 95th Percentile (ms) | 99th Percentile (ms) | Max Time (ms) |
|---|---|---|---|---|---|---|---|
| Frontdesk50% Traffic | New Patient Registration → Start OPD Visit | 40% | 5760 | 152 | 484 | 648 | 1389 |
| Existing Patient Search using ID → Start OPD Visit | 30% | 4320 | 54 | 472 | 676 | 1977 | |
| Existing Patient Search using Name → Start OPD Visit | 20% | 4320 | 119 | 352 | 507 | 1492 | |
| Upload Patient Document | 10% | 1440 | 142 | 482 | 581 | 1135 | |
| Doctor50% Traffic | Doctor Consultation8 Observations2 Lab Orders3 Medication | 100% | 5760 | 1364 | 4056 | 4531 | 7291 |
70 Concurrent Users - ConfigurationHardwarePerformance environment was running on AWS EKS Custer with single node
Node (EC2: m5-xlarge)
- RAM 16GB
- 4 vCPU
- 100GB Secondary storage
- AWS LINUX x86_64
Database (AWS RDS service: db.t3.xlarge)
- RAM 16GB,
- 4 vCPU (2 core, 2.5 GHz Intel Scalable Processor)
- 100GB Secondary storage
- MySQL, max_connections = 1304
Software
OpenMRS Tomcat - Server
OpenMRS - Heap
- Initial Heap: 1024 MB
- Max Heap: 2536 MB
-Xms1024m -Xmx2536m -XX:NewSize=512m -XX:MaxNewSize=512m -XX:MetaspaceSize=256m -XX:MaxMetaspaceSize=1024m -XX:InitialCodeCacheSize=64m -XX:ReservedCodeCacheSize=96m -XX:SurvivorRatio=4 -XX:TargetSurvivorRatio=40 -XX:+UseParNewGC -XX:ParallelGCThreads=2 -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:+CMSCompactWhenClearAllSoftRefs -XX:CMSInitiatingOccupancyFraction=85 -XX:+CMSScavengeBeforeRemark -XX:+UseGCOverheadLimit -XX:+UseStringDeduplication
Openmrs Connection Pooling
📗 Current Results - 70 Concurrent Users
- Network: 60 Mbps
- Duration: 24 hours
- Ramp Up: 5 mins
- Database pre-state: 90500 patient records
Report Observations:
| Simulations | Scenario | Load share | Patient Count | Min Time (ms) | 95th Percentile (ms) | 99th Percentile (ms) | Max Time (ms) |
|---|---|---|---|---|---|---|---|
| Frontdesk50% Traffic | New Patient Registration → Start OPD Visit | 40% | 10080 | 130 | 252 | 309 | 674 |
| Existing Patient Search using ID → Start OPD Visit | 30% | 7200 | 49 | 305 | 464 | 2660 | |
| Existing Patient Search using Name → Start OPD Visit | 20% | 7200 | 135 | 278 | 348 | 573 | |
| Upload Patient Document | 10% | 2160 | 111 | 206 | 253 | 459 | |
| Doctor50% Traffic | Doctor Consultation8 Observations2 Lab Orders3 Medication | 100% | 10080 | 998 | 2331 | 2608 | 4134 |
🔰 Observations:
- Doctor Consultation - The maximum response times for this particular activity was pretty high under both 40 and 70 concurrent tests. The services responsible for this activity is under analysis and will be prioritised for performance improvement.
Note: The tests performed here are of most demand to users using Bahmni lite compared to the real world clinic activities. So it is assumed safe to implement for the suggested concurrent users under a cluster even when the maximum time response numbers for some activities are not optimal at the moment under test.
⭕️ Base Configuration
The below details are historical data observed during each tests with different configurations at the start of performance analysis for Bahmni lite. Kindly view them for data purposes and not as references.
DetailsHardwarePerformance environment was running on AWS EKS Custer with single node
Node (EC2: m5-xlarge)
- RAM 16GB
- 4 vCPU
- 100GB Secondary storage
- AWS LINUX x86_64
Database (AWS RDS service: db.t3.xlarge)
- RAM 16GB,
- 4 vCPU (2 core, 2.5 GHz Intel Scalable Processor)
- 100GB Secondary storage
- MySQL, max_connections = 1304
Software
OpenMRS Tomcat - Server
OpenMRS - Heap
- Initial Heap: 256 MB
- Max Heap: 768 MB
-Xms256m -Xmx768m -XX:PermSize=256m -XX:MaxPermSize=512m
Openmrs Connection Pooling
📙 10 Concurrent UsersDetailsnoteNetwork: 30 MBPS
Duration: 30 mins
Ramp Up: 3 mins
Database pre-state: 2104 Active Patients with 9288 Observations
Report Link: https://bahmni.github.io/performance-test/baseline_report-20220911073147512_10users_30mins_768mb/index.html
Report Observations:
| Simulations | Scenario | Load share | Patient Count | Min Time (ms) | Max Time (ms) |
|---|---|---|---|---|---|
| Frontdesk50% Traffic | New Patient Registration → Start OPD Visit | 40% | 174 | 100 | 1077 |
| Existing Patient Search using ID → Start OPD Visit | 30% | 106 | 243 | 1550 | |
| Existing Patient Search using Name → Start OPD Visit | 20% | 107 | 243 | 1437 | |
| Upload Patient Document | 10% | 27 | 228 | 2169 | |
| Doctor50% Traffic | Doctor Consultation8 Observations2 Lab Orders3 Medication | 100% | 187 | 185 | 2092 |
📙 25 Concurrent Users
Report Link: https://bahmni.github.io/performance-test/baseline_report-20220912043950582_25users_15mins_2gb/index.html
🔰 Observations:
| Simulations | Scenario | Load share | Patient Count | Min Time (ms) | Max Time (ms) |
|---|---|---|---|---|---|
| Frontdesk50% Traffic | New Patient Registration → Start OPD Visit | 40% | 86 | 161 | 1193 |
| Existing Patient Search using ID → Start OPD Visit | 30% | 74 | 320 | 1113 | |
| Existing Patient Search using Name → Start OPD Visit | 20% | 64 | 310 | 1144 | |
| Upload Patient Document | 10% | 21 | 213 | 843 | |
| Doctor50% Traffic | Doctor Consultation8 Observations2 Lab Orders3 Medication | 100% | 107 | 259 | 846 |
📙 40 Concurrent Users - Standard Traffic condition
Report Link: https://bahmni.github.io/performance-test/baseline_report-20220912053731908_40users_15mins_2gb/index.html
🔰 Observations:
- OpenMRS Pod was consuming 4.15GB Memory and 2.59 CPU
- Overall Cluster was at 5.2GB RAM and 2.9 CPU
Report Observations:
| Simulations | Scenario | Load share | Patient Count | Min Time (ms) | Max Time (ms) |
|---|---|---|---|---|---|
| Frontdesk50% Traffic | New Patient Registration → Start OPD Visit | 40% | 130 | 162 | 1523 |
| Existing Patient Search using ID → Start OPD Visit | 30% | 109 | 356 | 2921 | |
| Existing Patient Search using Name → Start OPD Visit | 20% | 124 | 337 | 4125 | |
| Upload Patient Document | 10% | 36 | 238 | 579 | |
| Doctor50% Traffic | Doctor Consultation8 Observations2 Lab Orders3 Medication | 100% | 142 | 252 | 1289 |
📙 70 Concurrent Users - High Traffic condition
Report Link: https://bahmni.github.io/performance-test/baseline_report-20220912071131867_70users_30mins_2gb/index.html
🔰 Observations:
- OpenMRS Cluster Memory and CPU consumption was similar to 90 concurrent user run (Peak traffic)
- The API failure rates were very similar to peak traffic
Report Observations:
| Simulations | Scenario | Load share | Patient Count | Min Time (ms) | Max Time (ms) | Notes |
|---|---|---|---|---|---|---|
| Frontdesk50% Traffic | New Patient Registration → Start OPD Visit | 40% | 251 | 198 | 52939 | |
| Existing Patient Search using ID → Start OPD Visit | 30% | 175 | 1200 | 54692 | ||
| Existing Patient Search using Name → Start OPD Visit | 20% | 162 | 979 | 54510 | ||
| Upload Patient Document | 10% | 62 | 409 | 7848 | ||
| Doctor50% Traffic | Doctor Consultation8 Observations2 Lab Orders3 Medication | 100% | 193 | 558 | 44887 | Get Patient Lab Orders: 95% KOSave Consultation: 52% KOGet Visits: 18% KO |
📙 90 Concurrent Users - Peak Traffic condition
Report Link: https://bahmni.github.io/performance-test/baseline_report-20220912061644613_90users_30mins_8gb/index.html
🔰 Observations:
- OpenMRS Pod was consuming 11.5GB Memory and 3.22 CPU
- Overall Cluster was at 12.5GB RAM and 3.6 CPU
- More than 50% of the consultation and 95% all of Patient lab order fetch api calls suffered timeout.
Report Observations:
| Simulations | Scenario | Load share | Patient Count | Min Time (ms) | Max Time (ms) | Notes |
|---|---|---|---|---|---|---|
| Frontdesk50% Traffic | New Patient Registration → Start OPD Visit | 40% | 254 | 188 | 55802 | |
| Existing Patient Search using ID → Start OPD Visit | 30% | 192 | 1031 | 56250 | ||
| Existing Patient Search using Name → Start OPD Visit | 20% | 177 | 651 | 53063 | ||
| Upload Patient Document | 10% | 62 | 545 | 59070 | ||
| Doctor50% Traffic | Doctor Consultation8 Observations2 Lab Orders3 Medication | 100% | 200 | 626 | 53271 | Get Patient Lab Orders : 95% KOSave Consultation: with 56% KOGet Visits: 15% KO |
⭕️ Pace Based Framework
The framework is based on iterations per time type volumes i.e. a dedicated pace is set for each persona simulation.
Default JVM Configuration
📙 40 Concurrent Users - Standard Traffic condition
Report Link: https://bahmni.github.io/performance-test/baseline_report-20220923121606971_40users_30mins_defaultJVM/index.html
🔰 Observations:
- OpenMRS Pod was consuming GB Memory and CPU
- Overall Cluster was at GB RAM and CPU
Report Observations:
| Simulations | Scenario | Load share | Patient Count | Min Time (ms) | 95th Percentile (ms) | 99th Percentile (ms) | Max Time (ms) |
|---|---|---|---|---|---|---|---|
| Frontdesk50% Traffic | New Patient Registration → Start OPD Visit | 40% | 108 | 88 | 191 | 328 | 378 |
| Existing Patient Search using ID → Start OPD Visit | 30% | 96 | 243 | 481 | 557 | 571 | |
| Existing Patient Search using Name → Start OPD Visit | 20% | 87 | 243 | 579 | 694 | 749 | |
| Upload Patient Document | 10% | 27 | 164 | 209 | 280 | 304 | |
| Doctor50% Traffic | Doctor Consultation8 Observations2 Lab Orders3 Medication | 100% | 120 | 332 | 531 | 610 | 1153 |
📙 70 Concurrent Users - High Traffic condition
Report Link: https://bahmni.github.io/performance-test/baseline_report-20220923133122220_70users_30mins_defaultJVM/index.html
🔰 Observations:
- OpenMRS Pod was consuming GB Memory and CPU
- Overall Cluster was at GB RAM and CPU
Report Observations:
| Simulations | Scenario | Load share | Patient Count | Min Time (ms) | 95th Percentile (ms) | 99th Percentile (ms) | Max Time (ms) |
|---|---|---|---|---|---|---|---|
| Frontdesk50% Traffic | New Patient Registration → Start OPD Visit | 40% | 202 | 85 | 968 | 1058 | 1120 |
| Existing Patient Search using ID → Start OPD Visit | 30% | 163 | 232 | 1254 | 1700 | 1860 | |
| Existing Patient Search using Name → Start OPD Visit | 20% | 143 | 241 | 1221 | 1630 | 1866 | |
| Upload Patient Document | 10% | 40 | 144 | 984 | 1281 | 1338 | |
| Doctor50% Traffic | Doctor Consultation8 Observations2 Lab Orders3 Medication | 100% | 210 | 329 | 1736 | 2538 | 2654 |
📙 90 Concurrent Users - Peak Traffic condition
Report Link: https://bahmni.github.io/performance-test/baseline_report-20220923141108441_90users_30mins_defaultJVM/index.html
🔰 Observations:
- OpenMRS Pod was consuming GB Memory and CPU
- Overall Cluster was at GB RAM and CPU
Report Observations:
| Simulations | Scenario | Load share | Patient Count | Min Time (ms) | 95th Percentile (ms) | 99th Percentile (ms) | Max Time (ms) |
|---|---|---|---|---|---|---|---|
| Frontdesk50% Traffic | New Patient Registration → Start OPD Visit | 40% | 274 | 81 | 995 | 1064 | 1843 |
| Existing Patient Search using ID → Start OPD Visit | 30% | 192 | 250 | 1820 | 2193 | 2410 | |
| Existing Patient Search using Name → Start OPD Visit | 20% | 188 | 239 | 2109 | 2223 | 2465 | |
| Upload Patient Document | 10% | 57 | 167 | 972 | 1366 | 1416 | |
| Doctor50% Traffic | Doctor Consultation8 Observations2 Lab Orders3 Medication | 100% | 270 | 339 | 2656 | 3752 | 4149 |
Tuned JVM Configuration
The JVM was tuned for better efficiency from the default configuration
Configuration
📙 40 Concurrent Users - Standard Traffic condition
Report Link: https://bahmni.github.io/performance-test/baseline_report-20220927073846406_40users_30mins_tunedJVM/index.html
🔰 Observations:
- OpenMRS Pod was consuming 2.75 GB Memory and 0.411 CP
Report Observations:
| Simulations | Scenario | Load share | Patient Count | Min Time (ms) | 95th Percentile (ms) | 99th Percentile (ms) | Max Time (ms) |
|---|---|---|---|---|---|---|---|
| Frontdesk50% Traffic | New Patient Registration → Start OPD Visit | 40% | 119 | 154 | 294 | 361 | 544 |
| Existing Patient Search using ID → Start OPD Visit | 30% | 75 | 242 | 408 | 453 | 463 | |
| Existing Patient Search using Name → Start OPD Visit | 20% | 96 | 270 | 612 | 765 | 766 | |
| Upload Patient Document | 10% | 35 | 138 | 348 | 474 | 504 | |
| Doctor50% Traffic | Doctor Consultation8 Observations2 Lab Orders3 Medication | 100% | 120 | 490 | 687 | 769 | 778 |
📙 70 Concurrent Users - High Traffic condition
Report Link: https://bahmni.github.io/performance-test/baseline_report-20220927100508288_70users_30mins_tunedJVM/index.html
🔰 Observations:
- OpenMRS Pod was consuming 2.77 GB Memory and 1.22 CPU
Report Observations:
| Simulations | Scenario | Load share | Patient Count | Min Time (ms) | 95th Percentile (ms) | 99th Percentile (ms) | Max Time (ms) |
|---|---|---|---|---|---|---|---|
| Frontdesk50% Traffic | New Patient Registration → Start OPD Visit | 40% | 206 | 164 | 366 | 537 | 677 |
| Existing Patient Search using ID → Start OPD Visit | 30% | 146 | 249 | 661 | 806 | 894 | |
| Existing Patient Search using Name → Start OPD Visit | 20% | 157 | 238 | 654 | 743 | 767 | |
| Upload Patient Document | 10% | 60 | 132 | 600 | 710 | 710 | |
| Doctor50% Traffic | Doctor Consultation8 Observations2 Lab Orders3 Medication | 100% | 210 | 497 | 978 | 1075 | 1236 |
📙 90 Concurrent Users - Peak Traffic condition
Report Link: https://bahmni.github.io/performance-test/baseline_report-20220927092902911_90users_30mins_tunedJVM/index.html
🔰 Observations:
- OpenMRS Pod was consuming 2.76 GB Memory and 1.42 CPU
Report Observations:
| Simulations | Scenario | Load share | Patient Count | Min Time (ms) | 95th Percentile (ms) | 99th Percentile (ms) | Max Time (ms) |
|---|---|---|---|---|---|---|---|
| Frontdesk50% Traffic | New Patient Registration → Start OPD Visit | 40% | 260 | 153 | 362 | 423 | 526 |
| Existing Patient Search using ID → Start OPD Visit | 30% | 199 | 268 | 786 | 960 | 1163 | |
| Existing Patient Search using Name → Start OPD Visit | 20% | 197 | 276 | 844 | 952 | 1120 | |
| Upload Patient Document | 10% | 62 | 152 | 537 | 741 | 860 | |
| Doctor50% Traffic | Doctor Consultation8 Observations2 Lab Orders3 Medication | 100% | 270 | 511 | 1037 | 1321 | 1551 |
⭕️ Scenario based pace frameworkThe scenario based pace framework enables various pace distribution among different scenarios individually.
Configuration
📙 40 Concurrent Users - Standard Traffic condition
Report Link: https://bahmni.github.io/performance-test/baseline_report-20221012061917030_standard_30mins_tunedJVM_v2/index.html
📙 70 Concurrent Users - High Traffic condition
Report Link: https://bahmni.github.io/performance-test/baseline_report-20221012065541720_high_30mins_tunedJVM_v2/index.html
📙 90 Concurrent Users - Peak Traffic condition
Report Link: https://bahmni.github.io/performance-test/baseline_report-20221012095812872_peak_30mins_tunedJVM_v2/index.html
Bahmni Wiki · CC BY-SA 4.0