{"id":19149814,"url":"https://github.com/harness/harness-performance-tool","last_synced_at":"2025-05-07T04:44:41.806Z","repository":{"id":188666214,"uuid":"679146888","full_name":"harness/harness-performance-tool","owner":"harness","description":null,"archived":false,"fork":false,"pushed_at":"2025-04-25T23:27:27.000Z","size":1852,"stargazers_count":4,"open_issues_count":1,"forks_count":1,"subscribers_count":50,"default_branch":"main","last_synced_at":"2025-05-07T04:44:34.758Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/harness.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2023-08-16T07:43:25.000Z","updated_at":"2025-02-17T22:25:07.000Z","dependencies_parsed_at":null,"dependency_job_id":"7474eb81-9259-4e01-b010-d9a9ea6b7d16","html_url":"https://github.com/harness/harness-performance-tool","commit_stats":null,"previous_names":["harness/harness-performance-tool"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/harness%2Fharness-performance-tool","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/harness%2Fharness-performance-tool/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/harness%2Fharness-performance-tool/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/harness%2Fharness-performance-tool/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/harness","download_url":"https://codeload.github.com/harness/harness-performance-tool/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":252816519,"owners_count":21808702,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-09T08:09:45.896Z","updated_at":"2025-05-07T04:44:41.784Z","avatar_url":"https://github.com/harness.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\n## Harness Performance Testing Framework\n\nThis document outlines all the necessary details required for:\n1. Setting up a performance test tool (Locust - https://locust.io/) to generate load on Harness system\n2. Python scripts that emulate actual user scenarios via Harness apis https://apidocs.harness.io/\n\n### Performance Test Results\n[Test Report - 06-Oct-2023](./reports/06-Oct'23.md)  \n[Test Report - 27-Oct-2023](./reports/27-Oct'23.md)  \n[Test Report - 28-Dec-2023](./reports/28-Dec'23.md)  \n[Test Report - 31-Jan-2024](./reports/31-Jan'24.md)  \n[Test Report - 29-Feb-2024](./reports/29-Feb'24.md)  \n[Test Report - 01-Apr-2024](./reports/01-Apr'24.md)  \n[Test Report - 30-Apr-2024](./reports/30-Apr'24.md)  \n[Test Report - 05-June-2024](./reports/05-June'24.md)  \n[Test Report - 08-July-2024](./reports/08-July'24.md)  \n[Test Report - 28-Aug-2024](./reports/28-Aug'24.md)  \n[Test Report - 08-Jan-2025](./reports/08-Jan'25.md)  \n\n### Quick overview\n[locust_setup_run.mov](https://drive.google.com/file/d/1oU9r0_IBOs908D0YmpRUrCzW9EqmR_hV/view)\n\n### [Run Existing Pipeline](#)\n\nIf there is CI/CD pipeline already available and you would like to just run it 'n' times repeatedly.  \nPerform following:\n- Add Harness user credentials in [credentials.csv](./data/on-prem/credentials.csv)\n- [Setup Locust](#locust-installation)\n- [Execute Test](#test-execution) with [Test Scenario](#test-scripts-scenarios) [TRIGGER_PIPELINE]\n\n\n### [Test data setup to generate pipelines during test run](#)\n\n#### Add Harness users for test runs\nAdd new users into Harness account and list them in [credentials.csv](./data/on-prem/credentials.csv)  \nThese credentials will be used during test runs for performing authentication\n\n\u003e eg: auto_perf_1597@mailinator.com:Test@123\n\n#### Generate Harness users in bulk\n\n`#1` Run mongo query to add user (admin role) into harness User group user.  \nReplace the username with harness user of type : account admin.  \nThis user will be used to provision the new users in step2.\n\n`db.harnessUserGroups.insertOne({'name':'readOnly','memberIds':db.users.distinct('_id',{email:\"\u003cusername\u003e\"})})`\n\n`#2` Execute script [user_provision.sh](./data/scripts/user_provision.sh)\n\n#### CI / CD pipelines (test) would need github repo details to perform git clone and pull manifest\n- repo userIds\n- repo tokens\n- repo url\n- repo branch name\n- kubernetes namespace (to host build | deployment pods)  \nInput data and execute [testdata.sh](./data/scripts/testdata.sh)\n\n#### Populate Harness data (optional)\nInput count of entities required and execute [platform_data.sh](./data/scripts/platform_data.sh)  \neg: Add 'x' organisations and 'y' projects in each organisation in Harness account\n\n#### Install delegate in Harness account with tag : perf-delegate\n\n### [Test execution via webhook triggers](#)\n##### In order to support higher pipeline concurrency run via webhook\nCI - [CI_PIPELINE_WEBHOOK_RUN](./locust_tasks/ci_pipeline_webhook_run.py)  \n- update webhook payload on [ci_webhook_payload.json](./resources/NG/pipeline/ci/ci_webhook_payload.json)  \n\nCD - [CD_PIPELINE_WEBHOOK_RUN](./locust_tasks/cd_pipeline_webhook_run.py)  \n- update webhook payload on [cd_webhook_payload.json](./resources/NG/pipeline/cd/cd_webhook_payload.json)  \n\n**Note:** Make sure to include {\"zen\": \"$zen\"} inside webhook payload which is required to match the Harness pipeline triggers to execute\n\n\n### [Locust installation](#)\n\n#### Set up Locust on local machine\n1. Install python3 ```brew install python3 ```  \n2. Install Locust \u003e ```pip3 install locust```\n3. Git clone\n4. Update locust master ip to '0.0.0.0' in [variables.sh](./variables.sh)\n5. Execute \u003e ```locust -f locust_tasks/tasks```  \n\n**Note:** If run into SSL/TLS verification error append certificates to python cacert.pem file.  \nLocate cacert.pem file via terminal  \n`python -c \"import certifi; print(certifi.where())\"`   \n\n#### Set up Locust on GCP cluster  \n1. Git clone\n2. Connect to GCP cluster\n3. Procure static IP address\n4. Update GCP project, cluster namespace and locust master ip (static ip) in [variables.sh](./variables.sh)\n5. Execute [install.sh](./install.sh) under cloned directory `./install.sh`  \n\n**Note:** If run into SSL/TLS verification error add certificates to [smp_certificates.pem](./smp_certifcates.pem) and run [install.sh](./install.sh) again\n\n### [Test Execution](#)\n\n#### Execute test scripts via locust webUI\n\n1. Navigate to http://0.0.0.0:8089/ (on local setup) or http://Locust_MASTER_IP:8089     \n\n  \n2. Parameters\n``` \nNumber of users : Total concurrent users [eg: 1]\n\nSpawn rate : no. of users to start per second\n\nHost : \u003cHarness URL\u003e [eg: http://\u003cip_address\u003e]\n\nRun time : Test duration [eg: 5m]\nnote: Click on Advanced options to view this field\n\nRps : request per second per user\neg: Generate 40 requests per 60s with 1 user [ (40 req / 60 sec)/(1 user) = 0.6 ]\neg: Generate 40 requests per 60s with 5 user [ (40 req / 60 sec)/(5 user) =  0.13 ]\nNote: This is applicable only for test scenarios *_RUN and TRIGGER_PIPELINE\n\nTest scenario : Test class to execute \neg: CI_CREATE_PIPELINE,CI_EXECUTE_PIPELINE (single or comma separated class names)\n\nPipeline url (optional) : Pipeline url (without payload) or Custom Webhook Url to execute ‘n’ times \neg: https://\u003cip_address\u003e/ng/account/EuVfUT4wTfqIYugqfpssQw/all/orgs/default/projects/ScaleTestPOC/pipelines/Test/pipeline-studio?storeType=INLINE  \n\nTo pick pipeline url : Navigate to project \u003e Pipelines \u003e Select pipeline from list to view pipeline-studio \u003e Copy pipeline url\n\nWebhook payload : JSON payload of webhook request\neg: {\"key\": \"inputSetId\"}\nBlank if there is no payload to be sent with webhook\nNote: This is applicable only for test scenario TRIGGER_PIPELINE\n\nPipeline execution count (optional) : no. of times pipeline or webhook should run [eg: 50]\n\nAuth mechanism : \n1. Harness : If username/password authentication is confiugured on account (default)\n2. Local-Login : If SSO authentication is configured on account, Local-Login will be used for making requests in tests\n\n```\n![](./docs/img/locust_params.png)\n\n#### Testdata setup \n\nThere might be a slight delay of approximately 60 seconds before the test execution begins, as pre-requisite data is being generated  \nSTATUS = TESTDATA SETUP is set on locust web UI during this phase  \n**Note:** Any error during this phase is not visible on locust WebUI and had to be checked in locust master logs\n\n![](./docs/img/testdata_setup.png)\n\n#### Execute test scripts via Curl\n\n\u003e curl --location '\u003cLOCUST_MASTER_IP\u003e:8089/swarm' \\\n--header 'Content-Type: application/x-www-form-urlencoded; charset=UTF-8' \\\n--data 'user_count=2\u0026spawn_rate=1\u0026host=\u003cHARNESS_SMP_URL\u003e\u0026run_time=2m\u0026test_scenario=CI_CREATE_PIPELINE,CI_EXECUTE_PIPELINE\u0026pipeline_url=\u0026pipeline_execution_count=1'  \n  \n\n### [View Results](#)\n\n#### Statistics \n\u003e capture no. of requests, failures, 99%ile, avg response time, etc.  \n\nNavigate to locust server URL while tests are running to view real time metrics\n\n![](./docs/img/statistics.png)\n\n\n#### Locust Reports\n\u003e capture requests, failures, etc.\n\n![](./docs/img/reports.png)\n\n\n### [Test Scripts (scenarios)](#)\n\n  \n  \n\n| Scenario                               | Pre-requisite data                                                                                                                                                                                                                                                                                                                                                                                                             | Locust params                                                                                                                                         | Comments                                                                                                                                                       |\n|:---------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------|\n| TRIGGER_PIPELINE\u003cbr/\u003e Trigger pipeline or webhook 'n' times  | Find pipeline link - Open Harness left nav -\u003e Builds -\u003e Select project -\u003e \u003cbr/\u003eClick on pipelines -\u003e Click on pipeline name -\u003e copy pipeline url\u003cbr/\u003e\u003cbr/\u003e eg: http://\u003cip_address\u003e/ng/account/DbSRs-u5QgukRP_ODtvGkw/ci/orgs/default\u003cbr/\u003e/projects/ScaleTestPOC/pipelines/CITest/pipeline-studio/?storeType=INLINE                                                                                                             | Test scenario : TRIGGER_PIPELINE\u003cbr/\u003e Pipeline url: \u003cURL\u003e\u003cbr/\u003ePipeline execution count : 10                                                           |                                                                                                                                                                |\n| CI_PIPELINE_RUN\u003cbr/\u003e Execute CI pipeline 'n' times          | Create Organization, project, connectors, \u003cbr/\u003estep template and pipeline                                                                                                                                                                                                                                                                                                                                                      | Test scenario : CI_PIPELINE_RUN\u003cbr/\u003e Pipeline url: blank\u003cbr/\u003ePipeline execution count : 10                                                            |                                                                                                                                                                |\n| CD_PIPELINE_RUN\u003cbr/\u003e Execute CD pipeline 'n' times          | Create Organization, project, K8s service, env, infra, connectors, pipeline                                                                                                                                                                                                                                                                                                                                                    | Test scenario : CD_PIPELINE_RUN\u003cbr/\u003e Pipeline url: blank\u003cbr/\u003ePipeline execution count : n                                                            |                                                                                                                                                                |\n| CI_PIPELINE_REMOTE_RUN\u003cbr/\u003e Execute CI pipeline 'n' times (gitX)   | Create Organization, project, 15 github code repo connectors, 15 gitX \u003cbr/\u003e(step template, stage template, pipelines, pipeline input set, pipeline triggers) \u003cbr/\u003e \u003cbr/\u003eManual steps :\u003cbr/\u003e 1. Add Harness pipeline trigger \u003cbr/\u003e2. Get webhook payload under github repo \u003cbr/\u003e3. Add one property {\"zen\": \"$zen\"} in webhook payload \u003cbr/\u003e4. Replace webhook payload in resources/NG/pipeline/ci/ci_webhook_payload.json file | Test scenario : CI_PIPELINE_REMOTE_RUN\u003cbr/\u003e Pipeline url: blank\u003cbr/\u003ePipeline execution count : 10                                                     | Pipeline is triggered via webhook and each pipeline_execution_count triggers 15 pipelines \u003cbr/\u003e \u003cbr/\u003eTo execute 1500 pipelines - set pipeline_execution_count = 100 |\n| CI_PIPELINE_SAVE\u003cbr/\u003e SAVE - GET - UPDATE CI pipeline        | Create Organization, project, 15 github connectors, pipeline template                                                                                                                                                                                                                                                                                                                                                          | Test scenario : CI_PIPELINE_SAVE\u003cbr/\u003e Pipeline url: blank\u003cbr/\u003ePipeline execution count : 0                                                            |                                                            ‘n’ users will Save - Get - Update CI pipeline concurrently and repeat until duration                                                                                                    | \n| CI_PIPELINE_REMOTE_SAVE\u003cbr/\u003e SAVE - GET - UPDATE CI pipeline (gitX) | Create Organization, project, 15 github connectors, 15 gitX step template, \u003cbr/\u003e15 gitX stage templates                                                                                                                                                                                                                                                                                                                        | Test scenario : CI_PIPELINE_REMOTE_SAVE\u003cbr/\u003e Pipeline url: blank\u003cbr/\u003ePipeline execution count : 0                                                     |                                                 ‘n’ users will Save - Get - Update gitX CI pipeline concurrently and repeat until duration                                                                                                               |\n| User behaviour under steady load       | Create Organizations, projects, connectors, 200 pipeline and execute them                                                                                                                                                                                                                                                                                                                                                      | Test scenario : CI_CREATE_PIPELINE,CI_EXECUTE_PIPELINE,\u003cbr/\u003eCI_UPDATE_PIPELINE,CI_VIEW_EXECUTION\u003cbr/\u003e Pipeline url: blank\u003cbr/\u003ePipeline execution count : 0 |                                                                                                                                                                |\n \n\n### [Project structure](#)\n- data : contains harness username list\n- kubernetes-config : locust master and worker deployment yaml\n- locust_tasks : test scripts and api helpers\n- resources : request payload files\n- locust_tasks \u003e tasks.py : execution starting point and all the test scripts have to be imported here\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fharness%2Fharness-performance-tool","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fharness%2Fharness-performance-tool","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fharness%2Fharness-performance-tool/lists"}