{"id":20260437,"url":"https://github.com/influxdata/qprof","last_synced_at":"2025-04-11T01:33:38.689Z","repository":{"id":57581802,"uuid":"129422485","full_name":"influxdata/qprof","owner":"influxdata","description":"A tool for profiling the performance of InfluxQL queries","archived":false,"fork":false,"pushed_at":"2023-10-05T18:18:54.000Z","size":85,"stargazers_count":6,"open_issues_count":1,"forks_count":1,"subscribers_count":8,"default_branch":"master","last_synced_at":"2025-03-24T22:42:10.929Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/influxdata.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-04-13T15:41:34.000Z","updated_at":"2023-10-05T18:18:52.000Z","dependencies_parsed_at":"2024-06-20T09:20:35.222Z","dependency_job_id":"f1beed14-6e92-40e5-9641-dcdae4cff973","html_url":"https://github.com/influxdata/qprof","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/influxdata%2Fqprof","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/influxdata%2Fqprof/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/influxdata%2Fqprof/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/influxdata%2Fqprof/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/influxdata","download_url":"https://codeload.github.com/influxdata/qprof/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248325441,"owners_count":21084926,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-14T11:19:37.304Z","updated_at":"2025-04-11T01:33:38.662Z","avatar_url":"https://github.com/influxdata.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# qprof\n\nA tool for profiling the performance of InfluxQL queries.\n\n`qprof` works by executing a provided query one or more times against an InfluxDB \nserver and profiling the server before, during, and after execution. Profiles are\nbundled into an archive for easy dissemination.\n\nBecause `qprof` gathers profiles before and after all queries have executed, \ndelta profiles become possible, which can help to identify the cost directly \nassociated with running the queries.\n\nThe following queries are captured:\n\n  - CPU profile (before, 15s into qprof execution and after);\n  - Heap profile (before and after);\n  - goroutine profile (before, 15s into qprof execution and after);\n  - blocking profile (before and after); and\n  - mutex profile (before and after).\n\nFinally, `qprof` stores all emitted data from the program execution, as well as\nthe flags that `qprof` was initialised with, in a file called `info.txt`.\n\n## Usage\n\nIdeally the server under test should have as little traffic as possible, so as to\nensure accurate profiles. Increasing the concurrency of queries can help to bring \nthe query load to the forefront of the profiles.\n\nThe tool can either run a query `n` number of times, or it can keep running the \nquery for a fixed duration of time (using `-d`).\n\n**Note**: for the _during_ profiles to be of use, it's important that `qprof` run for a reasonable\namount of time. It should run for at least one minute. If the query portion of the\nprogram does not last for at least a minute, the following warning will be emitted:\n\n```\n***** NOTICE - QUERY EXECUTION 42.637249ms *****\nThis tool works most effectively if queries are executed for at least one minute\nwhen capturing CPU profiles. Consider increasing `-n` or setting `-t 1m`.\n```\n\n**Short queries** can be difficult to profile if they're only run serially one at\na time because they often don't involve much CPU work. If the queries are short, \ne.g., under one seconds, or they're not coming through in profiles, consider \nincreasing the concurrency setting (`-c`) to provide more query load.\n\n\n### Options\n\n```\n⇒  qprof -help\nUsage of qprof:\n  -cpu\n    \tInclude CPU profile (will take at least 30s) (default true)\n  -db string\n    \tDatabase to query (required)\n  -host string\n    \tscheme://host:port of server/cluster/load balancer. (default: http://localhost:8086) (default \"http://localhost:8086\")\n  -k\tSkip SSL certificate validation\n  -n int\n    \tRepeat query n times (default 1 if -d not specified) (default 1)\n  -out string\n    \tOutput directory (default \".\")\n  -pass string\n    \tPassword if using authentication (optional)\n  -t duration\n    \tRepeat query for this period of time (optional and overrides -n)\n  -user string\n    \tUsername if using authentication (optional)\n\n```\n\n### Examples\n\nHere are some example uses:\n\n```\n// Run this query for 10m.\n$ qprof -db mydb -host http://example.com:8086 -t 10m \"SELECT count(*) FROM m9 WHERE time \u003e now() - 4m\"\n```\n\n```\n// Run this query for 5 minutes using 20 clients.\n$ qprof -db mydb -host http://example.com:8086 -c 20 -t 5m \"SELECT count(*) FROM m9 WHERE time \u003e now() - 4m\"\n```\n\n```\n// Connect to a TLS (SSL) enabled server with a self-signed certificate.\n$ qprof -db mydb -k -host https://example.com:8086 -t 10m \"SELECT count(*) FROM m9 WHERE time \u003e now() - 4m\"\n```\n\n```\n// Save the output archive to /tmp directory\n$ qprof -db mydb -host http://example.com:8086 -out /tmp -t 10m \"SELECT count(*) FROM m9\"\n```\n\n```\n// Use authentication and run query 5 times only.\n$ qprof -db mydb -host https://secret.com:8086 -user admin -pass qwertyui -n 5 \"SELECT count(*) FROM m9 WHERE time \u003e now() - 4m\"\n```\n\n```\n// Don't get a CPU profile\n$ qprof -db mydb -host http://example.com:8086 -cpu=false -n 5 \"SELECT count(*) FROM m9 WHERE time \u003e now() - 4m\"\n```\n\nThe output will look something like the following. In this case there is only one\nclient worker. Expect more `[Worker x]` prefixes when `-c` is higher than `1`.\n\n```\n⇒  qprof -db db -t 1m -out /tmp \"SELECT sum(v0) from m0\"\n2018/04/16 13:19:04 [Worker 0] Host http://localhost:8086 responded to a ping in 6.317739ms\n2018/04/16 13:19:04 [Worker 0] Capturing CPU profile. This will take 30s...\n2018/04/16 13:19:34 [Worker 0] \"profile\" profile captured...\n2018/04/16 13:19:34 [Worker 0] \"block\" profile captured...\n2018/04/16 13:19:34 [Worker 0] \"goroutine\" profile captured...\n2018/04/16 13:19:34 [Worker 0] \"heap\" profile captured...\n2018/04/16 13:19:34 [Worker 0] \"mutex\" profile captured...\n2018/04/16 13:19:34 [Worker 0] Begin query execution...\n2018/04/16 13:19:34 [Worker 0] Waiting 15 seconds before taking concurrent profiles.\n2018/04/16 13:19:39 [Worker 0] Query \"SELECT sum(v0) from m0\" took 5.484053033s to execute.\n2018/04/16 13:19:46 [Worker 0] Query \"SELECT sum(v0) from m0\" took 6.989616778s to execute.\n2018/04/16 13:19:49 [Worker 0] Capturing CPU profile. This will take 30s...\n2018/04/16 13:19:56 [Worker 0] Query \"SELECT sum(v0) from m0\" took 9.835411294s to execute.\n2018/04/16 13:20:03 [Worker 0] Query \"SELECT sum(v0) from m0\" took 7.251047777s to execute.\n2018/04/16 13:20:11 [Worker 0] Query \"SELECT sum(v0) from m0\" took 7.595724158s to execute.\n2018/04/16 13:20:17 [Worker 0] Query \"SELECT sum(v0) from m0\" took 6.16515614s to execute.\n2018/04/16 13:20:19 [Worker 0] \"profile\" profile captured...\n2018/04/16 13:20:20 [Worker 0] \"goroutine\" profile captured...\n2018/04/16 13:20:25 [Worker 0] Query \"SELECT sum(v0) from m0\" took 7.584774771s to execute.\n2018/04/16 13:20:31 [Worker 0] Query \"SELECT sum(v0) from m0\" took 5.982338931s to execute.\n2018/04/16 13:20:36 [Worker 0] Query \"SELECT sum(v0) from m0\" took 5.886193173s to execute.\n2018/04/16 13:20:36 [Worker 0] Queries executed for at least 1m0s\n2018/04/16 13:20:36 [Worker 0] Taking final profiles...\n2018/04/16 13:20:36 [Worker 0] Capturing CPU profile. This will take 30s...\n2018/04/16 13:21:06 [Worker 0] \"profile\" profile captured...\n2018/04/16 13:21:06 [Worker 0] \"block\" profile captured...\n2018/04/16 13:21:06 [Worker 0] \"goroutine\" profile captured...\n2018/04/16 13:21:07 [Worker 0] \"heap\" profile captured...\n2018/04/16 13:21:07 [Worker 0] \"mutex\" profile captured...\n2018/04/16 13:21:07 [Worker 0] All profiles gathered and saved at /tmp/profiles.tar.gz. Total query executions: 9.\n```\n\n## Analysis\n\nWhere appropriate, profiles are captured in the `debug=1` Go `pprof` format, which\nmakes them both human readable, and supported by the `pprof` tool. If the profile\nfile ends in a `.txt`, then they're in this format.\n\n#### How much memory was allocated during the executions of these queries?\n\nYou can answer this by running generating a delta heap profile using the `-alloc_space` flag:\n\n```\nedd@Alameda:~|⇒  go tool pprof -alloc_space -base /tmp/profiles/base-heap.pb.gz $GOPATH/bin/influxd /tmp/profiles/heap.pb.gz\nFile: influxd\nType: alloc_space\nEntering interactive mode (type \"help\" for commands, \"o\" for options)\n(pprof) top\nShowing nodes accounting for 17921.69MB, 71.16% of 25184.02MB total\nDropped 118 nodes (cum \u003c= 125.92MB)\nShowing top 10 nodes out of 86\n      flat  flat%   sum%        cum   cum%\n 3916.29MB 15.55% 15.55% 14998.68MB 59.56%  github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Engine).createVarRefSeriesIterator\n 2947.41MB 11.70% 27.25%  2947.41MB 11.70%  github.com/influxdata/influxdb/query.newFloatReduceFloatIterator\n 2348.64MB  9.33% 36.58%  2351.15MB  9.34%  runtime.mapassign_faststr\n 1590.29MB  6.31% 42.89%  1590.29MB  6.31%  github.com/influxdata/influxdb/models.parseTags\n 1509.09MB  5.99% 48.89%  1515.20MB  6.02%  github.com/influxdata/influxdb/tsdb/index/inmem.(*measurement).TagSets\n 1462.17MB  5.81% 54.69%  2644.22MB 10.50%  github.com/influxdata/influxdb/tsdb/engine/tsm1.newKeyCursor\n 1432.11MB  5.69% 60.38%  2956.24MB 11.74%  github.com/influxdata/influxdb/query.(*floatReduceFloatIterator).reduce\n 1067.07MB  4.24% 64.62%  1336.57MB  5.31%  github.com/influxdata/influxdb/query.encodeTags\n  850.54MB  3.38% 67.99%  1123.05MB  4.46%  github.com/influxdata/influxdb/tsdb/engine/tsm1.(*FileStore).locations\n  798.06MB  3.17% 71.16%   798.06MB  3.17%  github.com/influxdata/influxdb/query.newSumIterator.func1\n(pprof)\n```\n\n#### How many objects were allocated during the executions of these queries?\n\nSometimes it's useful to know if all that allocated memory was done via many tiny allocations,\nor a few larger ones. Lots of small allocations make garbage collection slower due\nto all the extra scanning and checking of small objects on the heap. The `-alloc_objects` will\ntell you how many objects were allocated during the query executions:\n\n```\n edd@Alameda:~|⇒  go tool pprof -alloc_objects -base /tmp/profiles/base-heap.pb.gz $GOPATH/bin/influxd /tmp/profiles/heap.pb.gz\nFile: influxd\nType: alloc_objects\nEntering interactive mode (type \"help\" for commands, \"o\" for options)\n(pprof) top\nShowing nodes accounting for 182098616, 61.49% of 296136057 total\nDropped 123 nodes (cum \u003c= 1480680)\nShowing top 10 nodes out of 81\n      flat  flat%   sum%        cum   cum%\n  32637426 11.02% 11.02%   49654035 16.77%  github.com/influxdata/influxdb/models.Tags.Map\n  26223442  8.86% 19.88%   35152994 11.87%  github.com/influxdata/influxdb/tsdb/engine/tsm1.(*FileStore).locations\n  23721094  8.01% 27.89%   46621988 15.74%  github.com/influxdata/influxdb/query.(*floatReduceFloatIterator).reduce\n  17503214  5.91% 33.80%   60389515 20.39%  github.com/influxdata/influxdb/tsdb/engine/tsm1.newKeyCursor\n  17488255  5.91% 39.70%   26319500  8.89%  github.com/influxdata/influxdb/query.encodeTags\n  16048840  5.42% 45.12%   16048840  5.42%  github.com/influxdata/influxdb/query.newFloatReduceFloatIterator\n  15597912  5.27% 50.39%   15597912  5.27%  github.com/influxdata/influxdb/query.newFloatMergeIterator\n  14970312  5.06% 55.44%   14970312  5.06%  github.com/influxdata/influxdb/query.newSumIterator.func1\n   8978569  3.03% 58.48%    8978569  3.03%  github.com/influxdata/influxdb/tsdb/engine/tsm1.DecodeFloatBlock\n   8929552  3.02% 61.49%    8929552  3.02%  github.com/influxdata/influxdb/tsdb/engine/tsm1.readEntries\n(pprof)\n```\n\n#### How much time was spent blocking during the query execution?\n\nBlocking profiles tell you about areas where they may be contention over locks or \nother areas of contention.\n\n```\nedd@Alameda:~|⇒  go tool pprof -base /tmp/profiles/base-block.txt $GOPATH/bin/ossinfluxd-v1.5.0 /tmp/profiles/block.txt\nFile: ossinfluxd-v1.5.0\nType: delay\nEntering interactive mode (type \"help\" for commands, \"o\" for options)\n(pprof) top\nShowing nodes accounting for 985.13s, 100% of 985.13s total\nDropped 12 nodes (cum \u003c= 4.93s)\nShowing top 10 nodes out of 57\n      flat  flat%   sum%        cum   cum%\n   888.29s 90.17% 90.17%    888.29s 90.17%  runtime.selectgo\n    67.22s  6.82% 96.99%     67.22s  6.82%  runtime.chanrecv2\n    29.63s  3.01%   100%     29.63s  3.01%  sync.(*WaitGroup).Wait\n         0     0%   100%     63.94s  6.49%  github.com/bmizerany/pat.(*PatternServeMux).ServeHTTP\n         0     0%   100%     29.63s  3.01%  github.com/influxdata/influxdb/coordinator.(*LocalShardMapping).CreateIterator\n         0     0%   100%     32.91s  3.34%  github.com/influxdata/influxdb/coordinator.(*StatementExecutor).ExecuteStatement\n         0     0%   100%     29.63s  3.01%  github.com/influxdata/influxdb/coordinator.(*StatementExecutor).createIterators\n         0     0%   100%     32.91s  3.34%  github.com/influxdata/influxdb/coordinator.(*StatementExecutor).executeSelectStatement\n         0     0%   100%     88.42s  8.98%  github.com/influxdata/influxdb/monitor.(*Monitor).storeStatistics\n         0     0%   100%     32.91s  3.34%  github.com/influxdata/influxdb/query.(*QueryExecutor).executeQuery\n(pprof)\n```\n\nIn this case there isn't much of interest.\n\n#### What was the CPU doing _during_ the query execution?\n\n`qprof` captures a CPU profile _during_ the queries' executions. This is useful for \nunderstanding what the server was doing. This profile is prefixed with `concurrent`.\n\n```\nedd@Alameda:~|⇒  go tool pprof $GOPATH/bin/ossinfluxd-v1.5.0 /tmp/profiles/concurrent-cpu.pb.gz\nFile: ossinfluxd-v1.5.0\nType: cpu\nTime: Apr 16, 2018 at 1:31pm (BST)\nDuration: 30.15s, Total samples = 1.20mins (239.10%)\nEntering interactive mode (type \"help\" for commands, \"o\" for options)\n(pprof) top\nShowing nodes accounting for 39.09s, 54.22% of 72.09s total\nDropped 215 nodes (cum \u003c= 0.36s)\nShowing top 10 nodes out of 194\n      flat  flat%   sum%        cum   cum%\n     8.62s 11.96% 11.96%     18.04s 25.02%  runtime.scanobject\n     7.89s 10.94% 22.90%      7.89s 10.94%  runtime.usleep\n     6.45s  8.95% 31.85%      6.45s  8.95%  runtime.heapBitsForObject\n     3.73s  5.17% 37.02%      3.73s  5.17%  runtime.greyobject\n     3.33s  4.62% 41.64%      3.33s  4.62%  runtime.mach_semaphore_signal\n     2.96s  4.11% 45.75%     22.52s 31.24%  runtime.mallocgc\n     1.88s  2.61% 48.36%      1.88s  2.61%  runtime.(*mspan).refillAllocCache\n     1.47s  2.04% 50.40%      1.47s  2.04%  runtime.duffcopy\n     1.40s  1.94% 52.34%      1.40s  1.94%  runtime.heapBitsSetType\n     1.36s  1.89% 54.22%      4.04s  5.60%  runtime.mapassign_faststr\n(pprof)\n```\n\nIn this example, most of the time is being spent doing GC. Ignoring the `runtime` package\nprovides a bit more insight.\n\n```\n(pprof) top -runtime\nActive filters:\n   ignore=runtime\nShowing nodes accounting for 6.58s, 9.13% of 72.09s total\nDropped 62 nodes (cum \u003c= 0.36s)\nShowing top 10 nodes out of 116\n      flat  flat%   sum%        cum   cum%\n     1.15s  1.60%  1.60%      1.15s  1.60%  github.com/influxdata/influxdb/tsdb/engine/tsm1.(*indirectIndex).search.func1\n     0.96s  1.33%  2.93%      2.11s  2.93%  github.com/influxdata/influxdb/pkg/bytesutil.SearchBytesFixed\n     0.78s  1.08%  4.01%      9.60s 13.32%  github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Engine).createVarRefSeriesIterator\n     0.72s     1%  5.01%      0.72s     1%  github.com/influxdata/influxdb/models.parseTags.func1\n     0.59s  0.82%  5.83%     10.19s 14.14%  github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Engine).createTagSetGroupIterators\n     0.59s  0.82%  6.64%      0.59s  0.82%  sync.(*RWMutex).RLock\n     0.55s  0.76%  7.41%      0.67s  0.93%  github.com/influxdata/influxdb/query.(*floatMergeHeap).Less\n     0.44s  0.61%  8.02%      0.44s  0.61%  sync.(*RWMutex).RUnlock\n     0.43s   0.6%  8.61%      0.86s  1.19%  github.com/influxdata/influxdb/query.encodeTags\n     0.37s  0.51%  9.13%      0.37s  0.51%  github.com/influxdata/influxdb/query.(*TagSet).Less\n(pprof)\n```","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Finfluxdata%2Fqprof","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Finfluxdata%2Fqprof","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Finfluxdata%2Fqprof/lists"}