{"id":13583685,"url":"https://github.com/serputko/performance-testing-framework","last_synced_at":"2025-04-04T21:07:44.763Z","repository":{"id":28470754,"uuid":"113171357","full_name":"serputko/performance-testing-framework","owner":"serputko","description":"Framework allows to perform load testing with Apache Jmeter, view application/server metrics in real-time with Grafana, analyze errors cause with detailed traces for failed requests, compare different test runs in scripted dashboard and perform frontend performance testing with sitespeed.io+webpagetest","archived":false,"fork":false,"pushed_at":"2023-12-19T18:27:41.000Z","size":35335,"stargazers_count":410,"open_issues_count":16,"forks_count":212,"subscribers_count":43,"default_branch":"master","last_synced_at":"2025-03-28T20:07:47.051Z","etag":null,"topics":["backend-load-testing","docker","frontend-load-testing","grafana","influxdb","jenkins","jmeter","load-testing","load-tests","performance","performance-measurement","performance-testing","real-time-monitoring","sitespeed","sitespeedio","telegraf","webpagetest","webpagetest-agent","webpagetest-server"],"latest_commit_sha":null,"homepage":"","language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/serputko.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null}},"created_at":"2017-12-05T11:05:01.000Z","updated_at":"2025-03-25T09:16:40.000Z","dependencies_parsed_at":"2024-04-15T07:12:56.697Z","dependency_job_id":"71550f87-f57d-4efa-850a-5d4b682c70c0","html_url":"https://github.com/serputko/performance-testing-framework","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/serputko%2Fperformance-testing-framework","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/serputko%2Fperformance-testing-framework/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/serputko%2Fperformance-testing-framework/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/serputko%2Fperformance-testing-framework/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/serputko","download_url":"https://codeload.github.com/serputko/performance-testing-framework/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247249524,"owners_count":20908212,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["backend-load-testing","docker","frontend-load-testing","grafana","influxdb","jenkins","jmeter","load-testing","load-tests","performance","performance-measurement","performance-testing","real-time-monitoring","sitespeed","sitespeedio","telegraf","webpagetest","webpagetest-agent","webpagetest-server"],"created_at":"2024-08-01T15:03:41.974Z","updated_at":"2025-04-04T21:07:44.745Z","avatar_url":"https://github.com/serputko.png","language":"JavaScript","readme":"\u003c!--- \nCopyright © 2017-2018 Anton Serputko. Contacts: serputko.a@gmail.com\n\nLicensed under the Apache License, Version 2.0 (the \"License\");\nyou may not use this file except in compliance with the License.\nYou may obtain a copy of the License at\n\nhttp://www.apache.org/licenses/LICENSE-2.0\n\nUnless required by applicable law or agreed to in writing, software\ndistributed under the License is distributed on an \"AS IS\" BASIS,\nWITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\nSee the License for the specific language governing permissions and\nlimitations under the License.\n--\u003e\n\n# Performance testing framework\n\nThis framework could be used both for backend load testing with Apache Jmeter and frontend load test with sitespeed.io + webpagetest private instance.\n\n### Custom Grafana dashboards features:\n* **view** application/server-side metrics in real time while test is running\n* **analyze** errors cause with detailed traces for failed requests\n* **compare** different test runs in scripted dashboard\n\n#### Dashboard demo\n![](https://github.com/serputko/performance-testing-framework/blob/master/docs/img/grafana_dashboard_demo.gif)\n\n#### Tests comparison\n![](https://github.com/serputko/performance-testing-framework/blob/master/docs/img/grafana_compare_tests_demo.png)\n\n\n\n## Getting Started\n\nFramework consists of next services:\n- **Grafana**: data visualization \u0026 monitoring\n- **Influxdb**: time series DB platform for metrics \u0026 events(Time Series Data)\n- **Telegraf**: server agent for collecting \u0026 reporting metrics\n- **Sitespeed.io**: set of tools for frontend load testing\n- **Graphite**: time series DB platform for metrics\n- **Jenkins**: continuous integration server for tests execution\n- **Portainer**: service for managing docker environment\n- **Webpagetest**: private instance of webpagetest server for frontend tests execution\n- **Apache Jmeter**: tool for backend load testing\n\n## Framework architecture:\n![](https://github.com/serputko/performance-testing-framework/blob/master/docs/img/Framework%20structure.png)\n\t\n## Prerequisites\n\nTo run framework install docker: https://docs.docker.com/engine/installation/.\n\nYou should be able to run ```docker run hello-world``` with no errors.\n\n**For Windows**:\n- share C:\\ D:\\ drives with docker: https://blogs.msdn.microsoft.com/stevelasker/2016/06/14/configuring-docker-for-windows-volumes/\n- starting from 18.0.3 docker version you need to set up windows user variable to handle linux VM path in docker-compose:\nstart cmd as administrator and execute ```setx COMPOSE_CONVERT_WINDOWS_PATHS \"1\" /M``` or simply install older version of docker https://download.docker.com/win/stable/15139/Docker%20for%20Windows%20Installer.exe\n\n## Installing\n\n1. Clone this repository\n```git clone https://github.com/serputko/performance-testing-framework.git``` \n2. open performance-testing-framework dir\n\n**For Backend testing:**\n\n3. (optional) if you want to update existing services\n```\ndocker-compose pull\ndocker-compose build \ndocker-compose down\n```\n4. ```docker-compose up -d```\n\n**For Frontend + Backend testing**\n\n3. (optional) if you want to update existing services\n```\ndocker-compose -f docker-compose-with-frontend.yml pull\ndocker-compose -f docker-compose-with-frontend.yml build\ndocker-compose -f docker-compose-with-frontend.yml down\n```\n4. ```docker-compose -f docker-compose-with-frontend.yml up -d```\n\nAll containers should be up and running\n\n### Services endpoints\n- **jenkins** localhost:8181\n- **grafana** localhost:8857\n- **portainer** localhost:9000\n- **webpagetest server** localhost:80\n- **influxdb** localhost:8653\n\n## Jenkins\n\nLogin to Jenkins with admin/admin(could be changed in docker-compose file)\n![](https://github.com/serputko/performance-testing-framework/blob/master/docs/img/jenkins_dashboard.png)\nBy default jenkins consists of 2 jobs:\n- **BackendJob**: run Jmeter scenarios\n- **FrontendJob**: run tests with sitespeed.io and webpagetest private instance\n\n## Running demo jmeter scenario with BackendJob\n\n![](https://github.com/serputko/performance-testing-framework/blob/master/docs/img/jenkins_backendjob_start_test.gif)\n\nTo run jmeter demo script: **Open BackendJob -\u003e Build with Parameters -\u003e Set build parameters -\u003e Select scenario -\u003e Build**\n\nThis job will start jmeter docker container and execute demo_scenario.jmx jmeter scenario\n![](https://github.com/serputko/performance-testing-framework/blob/master/docs/img/jmeter_demo_scenario.png)\n\nTo open demo_scenario in local instance of Jmeter please install Plugin Manager https://jmeter-plugins.org/wiki/PluginsManager/. It will automatically install all required plugins.\n\n## Making your jmeter script compatible with framework\n\nCreate jmeter test scenario \n\nModify your jmeter scenario to work with framework, add:\n- **JSR223 Listener from demo scenario**\n\tContains groovy script that:\n    * Generates detailed trace about request/response info for failed samplers\n    * Posts generated trace to jmeter log(could be viewed in jenkins job)\n    * Set trace to sampler response message. \n    If failed samplers are part of Transaction Controller(generate parent sample enabled) all traces will be set to Transaction Controller response message\n- **Backend Listener from demo scenario**\n\tSends samplers data to influxdb database. Fields:\n    * **application** - groovy script set field value in next format: \n    ```\n    .jmx file title @ jenkins build number @ test start time (f.e. demo_scenario.jmx @ 18 @ Sat Jul 28 09:44:26 UTC 2018)\n    ```\n    * **testTitle** - groovy script generates parameterized title like:\n    ```\n    Jenkins build #18 http://127.0.0.1:8181/job/BackendJob/18/ with demo_scenario.jmx scenario with 10 users, 180 sec rampup and 600 sec duration was started\n    ```\n    \n    * **eventTags** - info from testTitle field in tag format\n\n    Data from these fields will be displayed in grafana dashboard annotations- Green/Red vertical lines that stands for test Start/End.\n    ![](https://github.com/serputko/performance-testing-framework/blob/master/docs/img/grafana_annotations_demo.gif)\n- (optional) **jp@gc - Console Status Logger**\n\tsends minimalistic stats to jenkins console output about test execution:\n    ```\n    #43\tThreads: 3/10\tSamples: 16\tLatency: 23\tResp.Time: 197\tErrors: 2\n    #44\tThreads: 3/10\tSamples: 16\tLatency: 32\tResp.Time: 203\tErrors: 1\n    ```\n\n## Grafana\n\nStart demo_scenario.jmx test with jenkins Backend job\n\nReal time results should be available in grafana\nOpen 'Load test monitoring' dashboard. Dashboard contains visualizations based on data from influxdb.\n\n### Available metrics\n\nDashboard has multiple rows with different metrics\n![](https://github.com/serputko/performance-testing-framework/blob/master/docs/img/grafana_dashboard_available_stats.png)\n\n### Timerange\nAll values in visualizations are calculated according to selected time range. Default timerange is last 15 min with 10 sec refresh. Timerange could be set in timepicker or selected on any graph.\n![](https://github.com/serputko/performance-testing-framework/blob/master/docs/img/grafana_set_time_range.gif)\n\n### Templating\nGraphs and series on dashboard are displayed dynamically according to variables selected \n* **group_time** - aggregation time \n* **metric** - which series to show on Response Times Over Time graph(default All)\n* **scenario** - results of which scenario to display\n* **transactions** - which transactions to show on Response Times Over Time row(default All). transaction variable depends on scenario variable\n* **server_measurements** - which server side metrics to show\n* **host** - for which servers show monitoring\n\n\n### Failed requests details\nIf JSR223 listener was added to scenario than detailed traces for failed requests can be viewed in 'Error details' table. \nExample of failed request: \n```\nNumber of samples in transaction : 1, number of failing samples : 1\nLogin; Response message: Unauthorized;\nStatus code: 401;\nNumber of failed assertions: 1\n\nSample Failed: Login\nStarted at: Mon Jul 30 11:54:59 UTC 2018\nFinished at: Mon Jul 30 11:54:59 UTC 2018\nRequest:\n{\n    \"code\": \"1234567\"\n}\nREQUEST DATA\nURL: http://demo-server.com/login\nRequest headers:\nContent-Type: application/json\n\nResponse: \nUnauthorized\nResponse code:401\n\nResponse data:\n{\n   \"Error message\": \"Incorrect password or confirmation code entered. Please try again.\"\n}\n\nAssertion results:\nNumber of failed assertions: 1\nResponse Assertion Failed; \nFailure Message: Test failed: text expected to contain /success/;\n```\n\n### Test runs comparison\nTests comparison is done using scripted js dashboard. It could be accessed at http://127.0.0.1:8857/dashboard/script/compare_tests.js\n\n![](https://github.com/serputko/performance-testing-framework/blob/master/docs/img/grafana_compare_tests_demo.gif)\n\n\n## FrontendJob\nTo run frontend test: **Open FrontendJob -\u003e Build with Parameters -\u003e Set build parameters -\u003e Build**\n![](https://github.com/serputko/performance-testing-framework/blob/master/docs/img/jenkins_frontendjob_run.png)\n\nThis job will start sitespeed.io docker container and run test with parameters using WebPageTest private instance \n\nFrontend test deliverables:\n- **sitespeed.io HTML report**\n![](https://github.com/serputko/performance-testing-framework/blob/master/docs/img/jenkins_frontendjob_sitespeed_html_report.png)\n- **webpagetest HTML report**\n![](https://github.com/serputko/performance-testing-framework/blob/master/docs/img/jenkins_frontendjob_webpagetest_html_report.png)\n","funding_links":[],"categories":["JavaScript","performance","Automation"],"sub_categories":["Frameworks"],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fserputko%2Fperformance-testing-framework","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fserputko%2Fperformance-testing-framework","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fserputko%2Fperformance-testing-framework/lists"}