https://github.com/harness/harness-performance-tool
https://github.com/harness/harness-performance-tool
Last synced: 8 months ago
JSON representation
- Host: GitHub
- URL: https://github.com/harness/harness-performance-tool
- Owner: harness
- License: other
- Created: 2023-08-16T07:43:25.000Z (over 2 years ago)
- Default Branch: main
- Last Pushed: 2025-04-25T23:27:27.000Z (9 months ago)
- Last Synced: 2025-05-07T04:44:34.758Z (8 months ago)
- Language: Python
- Size: 1.77 MB
- Stars: 4
- Watchers: 50
- Forks: 1
- Open Issues: 1
-
Metadata Files:
- Readme: README.md
- License: LICENSE.md
Awesome Lists containing this project
README
## Harness Performance Testing Framework
This document outlines all the necessary details required for:
1. Setting up a performance test tool (Locust - https://locust.io/) to generate load on Harness system
2. Python scripts that emulate actual user scenarios via Harness apis https://apidocs.harness.io/
### Performance Test Results
[Test Report - 06-Oct-2023](./reports/06-Oct'23.md)
[Test Report - 27-Oct-2023](./reports/27-Oct'23.md)
[Test Report - 28-Dec-2023](./reports/28-Dec'23.md)
[Test Report - 31-Jan-2024](./reports/31-Jan'24.md)
[Test Report - 29-Feb-2024](./reports/29-Feb'24.md)
[Test Report - 01-Apr-2024](./reports/01-Apr'24.md)
[Test Report - 30-Apr-2024](./reports/30-Apr'24.md)
[Test Report - 05-June-2024](./reports/05-June'24.md)
[Test Report - 08-July-2024](./reports/08-July'24.md)
[Test Report - 28-Aug-2024](./reports/28-Aug'24.md)
[Test Report - 08-Jan-2025](./reports/08-Jan'25.md)
### Quick overview
[locust_setup_run.mov](https://drive.google.com/file/d/1oU9r0_IBOs908D0YmpRUrCzW9EqmR_hV/view)
### [Run Existing Pipeline](#)
If there is CI/CD pipeline already available and you would like to just run it 'n' times repeatedly.
Perform following:
- Add Harness user credentials in [credentials.csv](./data/on-prem/credentials.csv)
- [Setup Locust](#locust-installation)
- [Execute Test](#test-execution) with [Test Scenario](#test-scripts-scenarios) [TRIGGER_PIPELINE]
### [Test data setup to generate pipelines during test run](#)
#### Add Harness users for test runs
Add new users into Harness account and list them in [credentials.csv](./data/on-prem/credentials.csv)
These credentials will be used during test runs for performing authentication
> eg: auto_perf_1597@mailinator.com:Test@123
#### Generate Harness users in bulk
`#1` Run mongo query to add user (admin role) into harness User group user.
Replace the username with harness user of type : account admin.
This user will be used to provision the new users in step2.
`db.harnessUserGroups.insertOne({'name':'readOnly','memberIds':db.users.distinct('_id',{email:""})})`
`#2` Execute script [user_provision.sh](./data/scripts/user_provision.sh)
#### CI / CD pipelines (test) would need github repo details to perform git clone and pull manifest
- repo userIds
- repo tokens
- repo url
- repo branch name
- kubernetes namespace (to host build | deployment pods)
Input data and execute [testdata.sh](./data/scripts/testdata.sh)
#### Populate Harness data (optional)
Input count of entities required and execute [platform_data.sh](./data/scripts/platform_data.sh)
eg: Add 'x' organisations and 'y' projects in each organisation in Harness account
#### Install delegate in Harness account with tag : perf-delegate
### [Test execution via webhook triggers](#)
##### In order to support higher pipeline concurrency run via webhook
CI - [CI_PIPELINE_WEBHOOK_RUN](./locust_tasks/ci_pipeline_webhook_run.py)
- update webhook payload on [ci_webhook_payload.json](./resources/NG/pipeline/ci/ci_webhook_payload.json)
CD - [CD_PIPELINE_WEBHOOK_RUN](./locust_tasks/cd_pipeline_webhook_run.py)
- update webhook payload on [cd_webhook_payload.json](./resources/NG/pipeline/cd/cd_webhook_payload.json)
**Note:** Make sure to include {"zen": "$zen"} inside webhook payload which is required to match the Harness pipeline triggers to execute
### [Locust installation](#)
#### Set up Locust on local machine
1. Install python3 ```brew install python3 ```
2. Install Locust > ```pip3 install locust```
3. Git clone
4. Update locust master ip to '0.0.0.0' in [variables.sh](./variables.sh)
5. Execute > ```locust -f locust_tasks/tasks```
**Note:** If run into SSL/TLS verification error append certificates to python cacert.pem file.
Locate cacert.pem file via terminal
`python -c "import certifi; print(certifi.where())"`
#### Set up Locust on GCP cluster
1. Git clone
2. Connect to GCP cluster
3. Procure static IP address
4. Update GCP project, cluster namespace and locust master ip (static ip) in [variables.sh](./variables.sh)
5. Execute [install.sh](./install.sh) under cloned directory `./install.sh`
**Note:** If run into SSL/TLS verification error add certificates to [smp_certificates.pem](./smp_certifcates.pem) and run [install.sh](./install.sh) again
### [Test Execution](#)
#### Execute test scripts via locust webUI
1. Navigate to http://0.0.0.0:8089/ (on local setup) or http://Locust_MASTER_IP:8089
2. Parameters
```
Number of users : Total concurrent users [eg: 1]
Spawn rate : no. of users to start per second
Host : [eg: http://]
Run time : Test duration [eg: 5m]
note: Click on Advanced options to view this field
Rps : request per second per user
eg: Generate 40 requests per 60s with 1 user [ (40 req / 60 sec)/(1 user) = 0.6 ]
eg: Generate 40 requests per 60s with 5 user [ (40 req / 60 sec)/(5 user) = 0.13 ]
Note: This is applicable only for test scenarios *_RUN and TRIGGER_PIPELINE
Test scenario : Test class to execute
eg: CI_CREATE_PIPELINE,CI_EXECUTE_PIPELINE (single or comma separated class names)
Pipeline url (optional) : Pipeline url (without payload) or Custom Webhook Url to execute ‘n’ times
eg: https:///ng/account/EuVfUT4wTfqIYugqfpssQw/all/orgs/default/projects/ScaleTestPOC/pipelines/Test/pipeline-studio?storeType=INLINE
To pick pipeline url : Navigate to project > Pipelines > Select pipeline from list to view pipeline-studio > Copy pipeline url
Webhook payload : JSON payload of webhook request
eg: {"key": "inputSetId"}
Blank if there is no payload to be sent with webhook
Note: This is applicable only for test scenario TRIGGER_PIPELINE
Pipeline execution count (optional) : no. of times pipeline or webhook should run [eg: 50]
Auth mechanism :
1. Harness : If username/password authentication is confiugured on account (default)
2. Local-Login : If SSO authentication is configured on account, Local-Login will be used for making requests in tests
```
#### Testdata setup
There might be a slight delay of approximately 60 seconds before the test execution begins, as pre-requisite data is being generated
STATUS = TESTDATA SETUP is set on locust web UI during this phase
**Note:** Any error during this phase is not visible on locust WebUI and had to be checked in locust master logs
#### Execute test scripts via Curl
> curl --location ':8089/swarm' \
--header 'Content-Type: application/x-www-form-urlencoded; charset=UTF-8' \
--data 'user_count=2&spawn_rate=1&host=&run_time=2m&test_scenario=CI_CREATE_PIPELINE,CI_EXECUTE_PIPELINE&pipeline_url=&pipeline_execution_count=1'
### [View Results](#)
#### Statistics
> capture no. of requests, failures, 99%ile, avg response time, etc.
Navigate to locust server URL while tests are running to view real time metrics
#### Locust Reports
> capture requests, failures, etc.
### [Test Scripts (scenarios)](#)
| Scenario | Pre-requisite data | Locust params | Comments |
|:---------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------|
| TRIGGER_PIPELINE
Trigger pipeline or webhook 'n' times | Find pipeline link - Open Harness left nav -> Builds -> Select project ->
Click on pipelines -> Click on pipeline name -> copy pipeline url
eg: http:///ng/account/DbSRs-u5QgukRP_ODtvGkw/ci/orgs/default
/projects/ScaleTestPOC/pipelines/CITest/pipeline-studio/?storeType=INLINE | Test scenario : TRIGGER_PIPELINE
Pipeline url:
Pipeline execution count : 10 | |
| CI_PIPELINE_RUN
Execute CI pipeline 'n' times | Create Organization, project, connectors,
step template and pipeline | Test scenario : CI_PIPELINE_RUN
Pipeline url: blank
Pipeline execution count : 10 | |
| CD_PIPELINE_RUN
Execute CD pipeline 'n' times | Create Organization, project, K8s service, env, infra, connectors, pipeline | Test scenario : CD_PIPELINE_RUN
Pipeline url: blank
Pipeline execution count : n | |
| CI_PIPELINE_REMOTE_RUN
Execute CI pipeline 'n' times (gitX) | Create Organization, project, 15 github code repo connectors, 15 gitX
(step template, stage template, pipelines, pipeline input set, pipeline triggers)
Manual steps :
1. Add Harness pipeline trigger
2. Get webhook payload under github repo
3. Add one property {"zen": "$zen"} in webhook payload
4. Replace webhook payload in resources/NG/pipeline/ci/ci_webhook_payload.json file | Test scenario : CI_PIPELINE_REMOTE_RUN
Pipeline url: blank
Pipeline execution count : 10 | Pipeline is triggered via webhook and each pipeline_execution_count triggers 15 pipelines
To execute 1500 pipelines - set pipeline_execution_count = 100 |
| CI_PIPELINE_SAVE
SAVE - GET - UPDATE CI pipeline | Create Organization, project, 15 github connectors, pipeline template | Test scenario : CI_PIPELINE_SAVE
Pipeline url: blank
Pipeline execution count : 0 | ‘n’ users will Save - Get - Update CI pipeline concurrently and repeat until duration |
| CI_PIPELINE_REMOTE_SAVE
SAVE - GET - UPDATE CI pipeline (gitX) | Create Organization, project, 15 github connectors, 15 gitX step template,
15 gitX stage templates | Test scenario : CI_PIPELINE_REMOTE_SAVE
Pipeline url: blank
Pipeline execution count : 0 | ‘n’ users will Save - Get - Update gitX CI pipeline concurrently and repeat until duration |
| User behaviour under steady load | Create Organizations, projects, connectors, 200 pipeline and execute them | Test scenario : CI_CREATE_PIPELINE,CI_EXECUTE_PIPELINE,
CI_UPDATE_PIPELINE,CI_VIEW_EXECUTION
Pipeline url: blank
Pipeline execution count : 0 | |
### [Project structure](#)
- data : contains harness username list
- kubernetes-config : locust master and worker deployment yaml
- locust_tasks : test scripts and api helpers
- resources : request payload files
- locust_tasks > tasks.py : execution starting point and all the test scripts have to be imported here