{"id":27088953,"url":"https://github.com/devops-works/dw-query-digest","last_synced_at":"2025-04-06T06:36:45.703Z","repository":{"id":36380201,"uuid":"164306437","full_name":"devops-works/dw-query-digest","owner":"devops-works","description":"MySQL slow log analyzer. Alternative to pt-query-digest.","archived":false,"fork":false,"pushed_at":"2023-02-25T00:56:53.000Z","size":169,"stargazers_count":43,"open_issues_count":5,"forks_count":6,"subscribers_count":6,"default_branch":"master","last_synced_at":"2024-06-20T00:35:42.176Z","etag":null,"topics":["golang","log-analysis","mysql"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/devops-works.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2019-01-06T12:48:12.000Z","updated_at":"2024-05-18T16:58:16.000Z","dependencies_parsed_at":"2022-07-29T00:38:51.921Z","dependency_job_id":"e3620f55-310d-4f7b-a310-b587dc05c773","html_url":"https://github.com/devops-works/dw-query-digest","commit_stats":null,"previous_names":[],"tags_count":15,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/devops-works%2Fdw-query-digest","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/devops-works%2Fdw-query-digest/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/devops-works%2Fdw-query-digest/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/devops-works%2Fdw-query-digest/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/devops-works","download_url":"https://codeload.github.com/devops-works/dw-query-digest/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247445686,"owners_count":20939953,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["golang","log-analysis","mysql"],"created_at":"2025-04-06T06:36:45.263Z","updated_at":"2025-04-06T06:36:45.686Z","avatar_url":"https://github.com/devops-works.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# dw-query-digest\n\nAlternative to `pt-query-digest`.\n\n[![Go Report Card](https://goreportcard.com/badge/github.com/devops-works/dw-query-digest)](https://goreportcard.com/report/github.com/devops-works/dw-query-digest)\n[![MIT Licence](https://badges.frapsoft.com/os/mit/mit.svg?v=103)](https://opensource.org/licenses/mit-license.php)\n\n## Purpose\n\n`dw-query-digest` reads slow query logs files and extracts a number of\nstatistics. It can list top queries sorted by specific metrics.\n\nIt is reasonably fast and can process ~450k slow-log lines per second.\n\nA 470MB slow query log containing 10M lines took approx 150s with `pt-query-digest`\nand approx 23s with `dw-query-digest` (6x faster).\n\n`dw-query-digest` normalizes SQL queries so identical queries can be aggregated\nin a report.\n\nSee also https://github.com/devops-works/slowql for an alternative\nimplementation conatianing a log replayer.\n\n## Usage\n\n### Binary\n\nGrab binary releases on [Github](https://github.com/devops-works/tools/dw-query-digest/releases).\n\n`bin/dw-query-digest [options] \u003cfile\u003e`\n\n### Source\n\nRequires Go 1.12+.\n\n```bash\nmake\nbin/dw-query-digest [options] \u003cfile\u003e\n```\n\n### Docker\n\nTo run using [Docker](https://hub.docker.com/r/devopsworks/dw-query-digest):\n\n```bash\ncd where_your_slow_query_log_is\ndocker run -v $(pwd):/data devopsworks/dw-query-digest /data/slow-query.log\n```\n\n### Options\n\n- `--debug`: show debugging information; this is very verbose, and meant for debugging\n- `--progress`: display a progress bar (disabled when reading from STDIN or using `--follow`)\n- `--output \u003cfmt\u003e`: produce report using _fmt_ output plugin (default: `terminal`; see \"Outputs\" below)\n- `--list-outputs`: list possible output plugins\n- `--quiet`: display only the report (no log)\n- `--reverse`: reverse sort (i.e. lowest first)\n- `--follow`: follow log file (`tail -F` style)\n- `--sort \u003cstring\u003e`: Sort key\n  - `time` (default): sort by cumulative execution time\n  - `count`: sort by query count\n  - `bytes`: sort by query bytes size\n  - `lock[time]`: sort by lock time (`lock` \u0026 `locktime` are synonyms)\n  - `[rows]sent`: sort by rows sent (`sent` and `rowssent` are synonyms)\n  - `[rows]examined`: sort by rows examined\n  - `[rows]affected`: sort by rows affected\n- `--top \u003cint\u003e`: Top queries to display (default 20)\n- `--nocache`: Disables cache (writing \u0026 reading)\n- `--version`: Show version \u0026 exit\n\n## Outputs\n\nThe default output is \"terminal\".\n\n### `terminal`\n\nSimple terminal output, designed to be read by humans.\n\n### `greppable`\n\nCSV format output designed to be used in combination with `grep`.\nIn this format, the two first lines are:\n\n- meta information (server version, duration, etc...)\n- columns header\n\nBoth lines start with `#`, so if you only want queries, you can filter them\nusing `dw-query-digest ... | grep -v '^#'`.\n\nColumn headers are prefixed by a number. You can directly use this number with\n`cut` if you only need a specific column. For instance, if I just want the max\ntime for a query, and since the column header is `16_Max(s)`, I can use:\n\n```bash\n$ dw-query-digest ... | grep -v '^#' | cut -f16 -d';'\n2.378111\n0.243685\n0.335063\n0.715469\n```\n\n### `json`\n\nPretty-printed JSON output containing general information in the `meta` key and\nqueries informations in the `stats` key. Note that the keys layout are subject\nto change.\n\n### `null`\n\nDoes not output anything, everything goes to `/dev/null`. This can be used for\nbenchmarking, to prime cache, and whatnot.\n\n## Cache\n\nWhen run against a file, `dw-query-digest` will try to find a cache file having\nthe same name and ending with `.cache`. For instance, if you invoke:\n\n```bash\ndw-query-digest -top 1 dbfoo-slow.log\n```\n\nthen `dw-query-digest` will try to find `dbfoo-slow.log.cache`. If the cache\nfile is found, the original file is not touched and results are read from the\ncache. This lets you rerun the command if needed without having to re-analyze\nthe whole original file.\n\nSince all results are cached, you can use different paramaters. For instance,\n`--top`, `--sort` or `output` can be different and will (hopefully) work.\n\nIf you don't want to read or write from/to the cache at all, you can use the\n``--nocache` option. You can also remove the file anytime.\n\nIf the analyzed file is newer than it's cache, the cache will not be used.\n\nCache format is not guaranteed to work between different versions.\n\n## Continuous reading\n\nThere is an *alpha* support for ever growing files when `--follow` is set. It\nshould support file rotation \u0026 truncation too.\n\n### Testing continuous reading\n\nAssuming you have docker and `dw-query-digest` in your PATH, you can quickly\ntest continuous reading like so:\n\n```bash\n# Prepare log dir \u0026 database\nmkdir test\nchmod 777 test\ndocker run --name mysql-test \\\n    -p 3307:3306 \\\n    -v $(pwd)/test/:/log \\\n    -e MYSQL_ALLOW_EMPTY_PASSWORD=true \\\n    -d mysql:5.7.19 \\\n    --slow-query-log=ON \\\n    --slow-query-log-file=/log/slow.log \\\n    --long-query-time=0\n\ndocker exec -ti mysql-test mysqladmin create test || echo 'Unable to create database; mysql not ready ?'\ndocker exec -ti mysql-test mysql -e 'create user sbtest;' || echo 'Unable to create user; mysql not ready ?'\ndocker exec -ti mysql-test mysql -e 'grant all privileges on *.* to sbtest@`%`;'|| echo 'Unable to grant user; mysql not ready ?'\ndocker exec -ti mysql-test mysql -e 'grant all on *.* to root@`%`;' || echo 'Unable to grant root; mysql not ready ?'\ndocker exec -ti mysql-test mysql -e 'flush privileges;'\ndocker exec -ti mysql-test mysql -e 'set global slow_query_log=\"ON\";'\n\n# Run this in another terminal\ndw-query-digest -top 10 -refresh 1000 --follow test/slow.log\n\n# Then back to previous terminal\ndocker run \\\n--rm=true \\\n--name=sb-prepare \\\n--link mysql-test:mysql \\\nseveralnines/sysbench \\\nsysbench \\\n--db-driver=mysql --table-size=1000000 --mysql-db=test \\\n--mysql-user=sbtest --mysql-port=3306 --mysql-host=mysql \\\noltp_read_write prepare\n\ndocker run \\\n--rm=true \\\n--name=sb-run \\\n--link mysql-test:mysql \\\nseveralnines/sysbench \\\nsysbench \\\n--db-driver=mysql --table-size=1000000 --mysql-db=test \\\n--mysql-user=sbtest --mysql-port=3306 --mysql-host=mysql \\\n--max-requests=0 --threads=8 --time=60 \\\noltp_read_write run\n\n# Clean all the things when done\ndocker run \\\n--rm=true \\\n--name=sb-run \\\n--link mysql-test:mysql \\\nseveralnines/sysbench \\\nsysbench \\\n--db-driver=mysql --table-size=1000000 --mysql-db=test \\\n--mysql-user=sbtest --mysql-port=3306 --mysql-host=mysql \\\noltp_read_write cleanup\n\ndocker stop mysql-test \u0026\u0026 docker rm mysql-test\n```\n\n## Caveats\n\nSome corners have been cut regarding query normalization. So YMMV regarding\naggregations.\n\nAlso, `dw-query-digest` does not support reading a header in the middle of a\nfile (and will crash with a panic). This can happen if you `FLUSH LOGS` during\na capture.\n\n## Contributing\n\nIf you spot something missing, or have a slow query log that is not parsed\ncorrectly, please open an issue and attach the log file.\n\nComments, criticisms, issues \u0026 pull requests welcome.\n\n## Roadmap\n\n- [x] cache\n- [x] `tail -f` reading (disable linecount !) with periodic reporting (in a TUI ?)\n- [x] internal statistics (logs lines/s, queries/s, ...)\n- [ ] `web` live output\n- [ ] `pt-query-digest` output ?\n- [ ] UDP json streamed output (no stats) for filebeat/logstash/graylog ?\n\n## Licence\n\nMIT\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdevops-works%2Fdw-query-digest","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdevops-works%2Fdw-query-digest","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdevops-works%2Fdw-query-digest/lists"}