{"id":21603446,"url":"https://github.com/anayrat/pgcheetah","last_synced_at":"2025-04-11T02:35:49.388Z","repository":{"id":65196908,"uuid":"180830542","full_name":"anayrat/pgcheetah","owner":"anayrat","description":"Quick and dirty project to inject workload to Postgres","archived":false,"fork":false,"pushed_at":"2023-02-25T01:23:46.000Z","size":5578,"stargazers_count":9,"open_issues_count":1,"forks_count":1,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-03-25T00:11:06.334Z","etag":null,"topics":["benchmark","golang","postgresql","replay"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/anayrat.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-04-11T16:16:00.000Z","updated_at":"2023-02-20T08:02:01.000Z","dependencies_parsed_at":"2024-06-19T06:18:44.789Z","dependency_job_id":"de47aea8-4af5-403c-b34d-13d7eb85c588","html_url":"https://github.com/anayrat/pgcheetah","commit_stats":{"total_commits":17,"total_committers":1,"mean_commits":17.0,"dds":0.0,"last_synced_commit":"40254c6be64e04bd81bf04d3e28b6c24129edbe7"},"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/anayrat%2Fpgcheetah","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/anayrat%2Fpgcheetah/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/anayrat%2Fpgcheetah/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/anayrat%2Fpgcheetah/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/anayrat","download_url":"https://codeload.github.com/anayrat/pgcheetah/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248330122,"owners_count":21085659,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["benchmark","golang","postgresql","replay"],"created_at":"2024-11-24T19:16:18.813Z","updated_at":"2025-04-11T02:35:49.367Z","avatar_url":"https://github.com/anayrat.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"[![Go Report Card](https://goreportcard.com/badge/github.com/anayrat/pgcheetah)](https://goreportcard.com/report/github.com/anayrat/pgcheetah) [![License](https://img.shields.io/badge/License-PostgreSQL-blue.svg)](https://github.com/anayrat/pgcheetah/blob/master/LICENSE)\n\n# pgcheetah\n\npgcheetah is a mix between pgbench and a tool to replay statements. It takes\na sample dataset to replay it *randomly*.\n\nHis goal is to reproduce a *real* workload from a sample of statements.\nIt also collects wait events statistics (the reason why it has been developped).\n\nThanks to Go and goroutines, pgcheetah is able to handle several hundred of clients and a quite high throughput in TPS.\nIt has been successfully tested up to thousand clients and 200K TPS with a sample collected by [pg_sampletolog\n](https://github.com/anayrat/pg_sampletolog). Note that it may necessary to clean collected sample to generate a clean dataset.\nFor example, by logging to CSV, import in a postgres database, clean \"noise\" queries and use a window function to reorder\ntransactions.\n\nYou must provide a sample dataset with ordered transactions. pgcheetah will \"parse\" the dataset to idenfify transactions.\nThen, it will start clients which choose a random transaction and replay it in the right order.\n\nThere are many options to control how to replay the workload:\n\n  * Number of clients\n  * Expected *transaction per seconds* or delay betweend each transaction\n  * Lenght of the test\n  * User think time\n\nUser think time is useful to reproduce *idle in transaction* sessions. Actually you must provide a min and a max\nin milliseconds and each client will draw a random number in this range (uniform distribution).\n\nEven if nothing forbid to replay write query, in real life there is few chance it will work. We could be facing\nproblems such as unique and foreign key constraints.\n\nIf you have an error during parsing phase, you can enable debug option. This will display each parsed line and can be helpful to\nclean the dataset.\n\n## Options\n\nUsage of ./pgcheetah:\n\n  * clients:\n    \tnumber of client (default 100)\n  * constr:\n    \tpg connstring (default \"user=postgres dbname=postgres\")\n  * datasetfraction:\n    Fraction of dataset to use between 0 - 1 (default 1)\n  * debug:\n    \tdebug mode\n  * delaystart:\n    \tspread clients start among seconds\n  * delayxact:\n    \tmillisecond between each transaction (default 5)\n  * duration:\n    \tTest duration in seconds\n  * interval:\n    \tInterval stats report (default 1 second)\n  * netpprof:\n    \tenable internal pprof web server\n  * queryfile:\n    \tpath to file containing queries to play\n  * slowstartfactor:\n    \tFactor to control how fast the delay between transaction will be changed (default 1.6)\n  * thinktimemax:\n    \tmillisecond thinktime (default 5)\n  * thinktimemin:\n    \tmillisecond thinktime (default 5)\n  * tps:\n    \texpected tps\n  * weinterval:\n        Wait Event collection interval in ms (default 500)\n\n\n## Example\n\npgcheetah will reports metrics on stdout output (each *interval* seconds) and collects [wait events](https://www.postgresql.org/docs/current/monitoring-stats.html#MONITORING-STATS-SETUP) statistics every 500ms:\n\n```\n./pgcheetah -clients 1000 -tps 200000 -constr 'user=user1 dbname=db1 host=pg.local' -thinktimemin 0 -thinktimemax 0 -delaystart 30 -delayxact 30 -queryfile play-20k.sql -duration 40 -interval 5 -slowstartfactor 2\n2019/04/26 15:37:20 Start parsing\n2019/04/26 15:37:20 Parsing done, start workers. Transactions processed: 19960\n2019/04/26 15:37:50 All workers launched\n2019/04/26 15:37:51 TPS: 302450 QPS: 338850 Xact: 30245 Queries: 33885 Delay: 30ms Remaining: 39s\n2019/04/26 15:37:56 TPS: 61580 QPS: 69070 Xact: 264135 Queries: 296984 Delay: 15.675ms Remaining: 34s\n2019/04/26 15:38:01 TPS: 96660 QPS: 108750 Xact: 660486 Queries: 742695 Delay: 9.703ms Remaining: 29s\n2019/04/26 15:38:06 TPS: 131960 QPS: 147740 Xact: 1239210 Queries: 1393970 Delay: 6.404ms Remaining: 24s\n2019/04/26 15:38:11 TPS: 183880 QPS: 206850 Xact: 2047366 Queries: 2304641 Delay: 4.377ms Remaining: 19s\n2019/04/26 15:38:16 TPS: 202420 QPS: 228370 Xact: 3040840 Queries: 3422127 Delay: 3.816ms Remaining: 14s\n2019/04/26 15:38:21 TPS: 208660 QPS: 243780 Xact: 4043817 Queries: 4550454 Delay: 3.943ms Remaining: 9s\n2019/04/26 15:38:26 TPS: 200270 QPS: 224500 Xact: 5047839 Queries: 5680047 Delay: 3.891ms Remaining: 4s\n2019/04/26 15:38:30 Test finished, stop clients\n2019/04/26 15:38:30 End test - Clients: 1000 - Elapsed: 40.090753907s - Average TPS: 145918 - Average QPS: 164192\n2019/04/26 15:38:30 Wait_event count:\nLWLockTranche-lock_manager      - 22\n```\n\nHere is a test with 1000 clients and 200KTPS expected.\n\nFirst step is parsing, then clients are started among *delaystart* seconds to avoid a spike when starting.\n\nWhen all clients are started, rate limiting begin and it changes the delay between each transaction to reach expected tps.\nFirst line is generally wrong because clients had already performed activity before rate limiter start. It is possible\nto control reduce clients activity before limiter by increasing delayxact.\n*slowstartfactor* allow to control the aggressiveness of how pgcheetah try to reach expected tps.\nHere, it took ~15s to reach 200KTPS.\n\nBy the end of the test (*duration* setting) or if you hit ctrl-c, all the clients will be stopped and wait event are reported.\n\nIn this example Average TPS is less than expected TPS, it is due to short test and slowstart.\n\n## Notice\n\nPlease note, it is a quick and dirty tool. For example, instead of using a real parser, it only identify few statements type\n(Begin, commit etc) by using regexp. It may not works with your queries and require few changes.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fanayrat%2Fpgcheetah","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fanayrat%2Fpgcheetah","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fanayrat%2Fpgcheetah/lists"}