{"id":15187225,"url":"https://gitlab.com/postgres-ai/postgres-checkup","last_synced_at":"2025-04-12T20:44:04.141Z","repository":{"id":51867168,"uuid":"9951336","full_name":"postgres-ai/postgres-checkup","owner":"postgres-ai","description":"[Beta] postgres-checkup: PostgreSQL Health Check and SQL Performance Analysis","archived":false,"fork":false,"pushed_at":null,"size":null,"stargazers_count":289,"open_issues_count":166,"forks_count":26,"subscribers_count":null,"default_branch":"master","last_synced_at":"2025-04-04T00:11:16.586Z","etag":null,"topics":["databases","dbms","healthchecks","pgsql","postgres","postgres.ai","postgresql","psql","sql"],"latest_commit_sha":null,"homepage":null,"language":null,"has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"agpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://gitlab.com/uploads/-/system/project/avatar/9951336/Screen_Shot_2019-03-14_at_16.34.04.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2018-12-16T18:43:35.122Z","updated_at":"2025-04-02T20:05:35.161Z","dependencies_parsed_at":"2022-09-19T18:52:12.406Z","dependency_job_id":null,"html_url":"https://gitlab.com/postgres-ai/postgres-checkup","commit_stats":null,"previous_names":[],"tags_count":9,"template":null,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/gitlab.com/repositories/postgres-ai%2Fpostgres-checkup","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/gitlab.com/repositories/postgres-ai%2Fpostgres-checkup/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/gitlab.com/repositories/postgres-ai%2Fpostgres-checkup/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/gitlab.com/repositories/postgres-ai%2Fpostgres-checkup/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/gitlab.com/owners/postgres-ai","download_url":"https://gitlab.com/postgres-ai/postgres-checkup/-/archive/master/postgres-checkup-master.zip","host":{"name":"gitlab.com","url":"https://gitlab.com","kind":"gitlab","repositories_count":4518646,"owners_count":6930,"icon_url":"https://github.com/gitlab.png","version":null,"created_at":"2022-05-30T11:31:42.605Z","updated_at":"2024-07-18T11:24:13.055Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/gitlab.com","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/gitlab.com/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/gitlab.com/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/gitlab.com/owners"}},"keywords":["databases","dbms","healthchecks","pgsql","postgres","postgres.ai","postgresql","psql","sql"],"created_at":"2024-09-27T18:04:49.923Z","updated_at":"2025-04-12T20:44:04.119Z","avatar_url":"https://gitlab.com/uploads/-/system/project/avatar/9951336/Screen_Shot_2019-03-14_at_16.34.04.png","language":null,"readme":"Please support the project giving a GitLab star (it's on [the main page](https://gitlab.com/postgres-ai/postgres-checkup/),\nat the upper right corner):\n\n![Add a star](./assets/star.gif)\n\n### Demo: [an example of postgres-checkup report](https://gitlab.com/postgres-ai/postgres-checkup-tests/tree/master/1.3.0) (based on CI, multi node).\n\n***Disclaimer: Conclusions, Recommendations – work in progress.**\nTo treat the data correctly, you need deep Postgres knowledge. Each report\nconsists of 3 sections: Observations, Conclusions, and Recommendations.\nObservations are filled automatically. As for Conclusions and Recommendations\nsections, not all reports are auto-generated.*\n\n\n# About\n\nPostgres Checkup ([postgres-checkup](https://gitlab.com/postgres-ai-team/postgres-checkup)) is a new kind of diagnostics tool for a deep analysis of a Postgres database health. It detects current and potential issues with database performance, scalability and security. It also produces recommendations on how to resolve or prevent them.\n\nA monitoring system will only show current, urgent problems. And postgres-checkup will show sneaking up, deeper problems, that may hit you in the future. It helps to solve many known database administration problems and common pitfalls. It aims to detect issues at a very early stage and to suggest the best ways to prevent them. \nWe recommend to run these on a regular basis — weekly, monthly, and quarterly. And also to run these right before and after applying any major change to a database server. Whether it’s a schema or configuration parameter or cluster settings change.\n\n\nWhy do you need postgres-checkup and why it's safe and easy to use:\n\n- *It is unobtrusive*: its impact on the observing system is\nclose to zero. It does not use any heavy queries, keeping resource usage\nvery low, and avoiding having the [“observer effect”](https://en.wikipedia.org/wiki/Observer_effect_(information_technology)).\npostgres-checkup reports were successfully tested on real-world databases\ncontaining 500,000+ tables and 1,000,000+ indexes.\n\n- *Zero install* (on observed machines): it is able to analyze any Linux\nmachine (including virtual machines), as well as cloud Postgres instances\n(such as Amazon RDs or Google Cloud SQL), not requiring any additional setup\nor any changes. It does, however, require a privileged access that a DBA usually\nhas anyway.\n\n- *Complex analysis*: unlike most monitoring tools, which provide just raw data,\npostgres-checkup combines data from various parts of the system (e.g.,\ninternal Postgres stats are combined with knowledge about system resources\nin autovacuum setting and behavior analysis) joining the data into well-formatted\nreports aimed to solve particular DBA problems. Also, it analyzes the master\ndatabase server together with all its replicas, which is neccessary in such\ncases as index analysis or search for settings deviations.\n\n# Reports Structure\n\nPostgres-checkup produces two kinds of reports for every check:\n\n- JSON reports (*.json) — can be consumed by any program or service, or\nstores in some database.\n\n- Markdown reports (*.md) — the main format for humans, may contain lists,\ntables, pictures. Being of native format for GitLab and GitHub, such reports\nare ready to be used, for instance, in their issue trackers, simplifying\nworkflow. Markdown reports are derived from JSON reports.\n\nMarkdown reports can be converted to different formats such as HTML or PDF.\n\nEach report consists of three sections:\n\n1. \"Observations\": automatically collected data. This is to be consumed by\nan expert DBA.\n1. \"Conclusions\": what we conclude from the Observations, stated in plain English\nin the form that is convenient for engineers who are not DBA experts.\n1. \"Recommendations\": action items, what to do to fix the discovered issues.\n\nBoth \"Conclusions\" and \"Recommendations\" are to be consumed by engineers who\nwill make decisions what, how and when to optimize.\n\n# Installation and Usage\n\n## Requirements\n\nFor the operator machine (from where the tool will be executed), the following\nOS are supported:\n\n* Linux (modern RHEL/CentOS or Debian/Ubuntu; others should work as well, but\nare not yet tested);\n* MacOS.\n\nThere are known cases when postgres-checkup was successfully used on Windows,\nalthough with some limitations.\n\nThe following programs must be installed on the operator machine:\n\n* bash\n* psql\n* coreutils\n* jq \u003e= 1.5\n* go \u003e= 1.17 (no binaries are shipped at the moment)\n* awk\n* sed\n* pandoc *\n* wkhtmltopdf \u003e= 0.12.4 *\n\npandoc and wkhtmltopdf are optional, they are needed for generating HTML and \nPDF versions of report (options `--html`, `--pdf`).\n\nNothing special has to be installed on the observed machines. However, they must\nrun Linux (again: modern RHEL/CentOS or Debian/Ubuntu; others should work as\nwell, but are not yet tested).\n\n:warning: Only Postgres version 9.6 and higher are currently officially supported.\n\n## How to Install\n\n#### 1. Install required programs\n\nUbuntu/Debian:\n```bash\nsudo apt-get update -y\nsudo apt-get install -y git postgresql coreutils jq golang\n\n# Optional (to generate PDF/HTML reports)\nsudo apt-get install -y pandoc\nwget https://github.com/wkhtmltopdf/wkhtmltopdf/releases/download/0.12.4/wkhtmltox-0.12.4_linux-generic-amd64.tar.xz\ntar xvf wkhtmltox-0.12.4_linux-generic-amd64.tar.xz\nsudo mv wkhtmltox/bin/wkhtmlto* /usr/local/bin\nsudo apt-get install -y openssl libssl-dev libxrender-dev libx11-dev libxext-dev libfontconfig1-dev libfreetype6-dev fontconfig\n```\n\nCentOS/RHEL:\n```bash\nsudo yum install -y git postgresql coreutils jq golang\n\n# Optional (to generate PDF/HTML reports)\nsudo yum install -y pandoc\nwget https://github.com/wkhtmltopdf/wkhtmltopdf/releases/download/0.12.4/wkhtmltox-0.12.4_linux-generic-amd64.tar.xz\ntar xvf wkhtmltox-0.12.4_linux-generic-amd64.tar.xz\nsudo mv wkhtmltox/bin/wkhtmlto* /usr/local/bin\nsudo yum install -y libpng libjpeg openssl icu libX11 libXext libXrender xorg-x11-fonts-Type1 xorg-x11-fonts-75dpi\n```\n\nMacOS (assuming that [Homebrew](https://brew.sh/) is installed):\n```bash\nbrew install postgresql coreutils jq golang git\n\n# Optional (to generate PDF/HTML reports)\nbrew install pandoc Caskroom/cask/wkhtmltopdf\n```\n\n#### 2. Clone this repo\n\n```bash\ngit clone https://gitlab.com/postgres-ai/postgres-checkup.git\n# Use --branch to use specific release version. For example, to use version 1.1:\n#   git clone --branch 1.1 https://gitlab.com/postgres-ai/postgres-checkup.git\ncd postgres-checkup\n```\n\n#### 3. Build pghrep\n\n```bash\ncd ./pghrep\nmake main\ncd ..\n```\n\n## Example of Use\n\nLet's make a report for a project named `prod1`. Assume that we have two servers,\n`db1.vpn.local` and `db2.vpn.local`.\n\nPostgres-checkup automatically detects which one is the master:\n\n```bash\n./checkup -h db1.vpn.local -p 5432 --username postgres --dbname postgres --project prod1 -e 1\n```\n\n```bash\n./checkup -h db2.vpn.local -p 5432 --username postgres --dbname postgres --project prod1 -e 1\n```\n\nWhich literally means: connect to the server with given credentials, save data into `prod1`\nproject directory, as epoch of check `1`. Epoch is a numerical (**integer**) sign of current iteration.\nFor example: in half a year we can switch to \"epoch number `2`\".\n\n`-h db2.vpn.local` means: try to connect to host via SSH and then use remote `psql` command to perform checks.\nIf SSH is not available the local 'psql' will be used (non-psql reports will be skipped) to establish\nPostgres connection. If you want to avoid \"guessing\", use `-ssh-hostname` or `--pg-hostname`.\n\nAlso, you can define a specific way to connect: SSH or `psql`:\n\n`--ssh-hostname db2.vpn.local` - SSH will be used for the connection. SSH port can be defined as well\nwith option `--ssh-port`.\n\n`--pg-hostname db2.vpn.local` - `psql` will be used for the connection. The port where PostgreSQL\naccepts connections can be defined with the option `--pg-port`.\n\nIn case when `--pg-port` or `--ssh-port` are not defined but `--port` is defined, value of `--port` option\nwill be used instead of `--pg-port` or `--ssh-port` depending on the current connection type.\n\nFor comprehensive analysis, it is recommended to run the tool on the master and\nall its replicas – postgres-checkup is able to combine all the information from\nmultiple nodes to a single report.\n\nSome reports (such as K003) require two snapshots, to calculate \"deltas\" of\nmetrics. So, for better results, use the following example, executing it during peak working\nhours, with `$DISTANCE` values from 10 min to a few hours:\n\n```bash\n$DISTANCE=\"1800\" # 30 minutes\n\n# Assuming that db2 is the master, db3 and db4 are its replicas\nfor host in db2.vpn.local db3.vpn.local db4.vpn.local; do\n  ./checkup \\\n    -h \"$host\" \\\n    -p 5432 \\\n    --username postgres \\\n    --dbname postgres \\\n    --project prod1 \\\n    -e 1 \\\n    --file resources/checks/K000_query_analysis.sh # the first snapshot is needed only for reports K***\ndone\n\nsleep \"$DISTANCE\"\n\nfor host in db2.vpn.local db3.vpn.local db4.vpn.local; do\n  ./checkup \\\n    -h \"$host\" \\\n    -p 5432 \\\n    --username postgres \\\n    --dbname postgres \\\n    --project prod1 \\\n    -e 1\ndone\n```\n\nAs a result of execution, two directories containing .json and .md files will\nbe created:\n\n```bash\n./artifacts/prod1/json_reports/1_2018_12_06T14_12_36_+0300/\n./artifacts/prod1/md_reports/1_2018_12_06T14_12_36_+0300/\n```\n\nEach of generated files contains information about \"what we check\" and collected data for\nall instances of the postgres cluster `prod1`.\n\nA human-readable report can be found at:\n\n```bash\n./artifacts/prod1/md_reports/1_2018_12_06T14_12_36_+0300/Full_report.md\n```\n\nOpen it with your favorite Markdown files viewer or just upload to a service such as gist.github.com.\n\nYou can collect and process data separately by specifying working mode name in CLI option `--mode %mode%` or using it as a \"command\" (`checkup %mode%`).  \nAvailable working modes:  \n    `collect` - collect data;\n    `process` - generate MD (and, optionally, HTML, PDF) reports with conclusions and recommendations;\n    `upload` - upload generated reports to Postgres.ai platform;\n    `run` - collect and process data at once. This is the default mode, it is used when no other mode is specified. Note, that upload is not included.\n\n## Docker 🐳\n\nIt's possible to use the `postgres-checkup` from a docker container.\nThe container will run, execute all checks and stop itself.\nThe check result can be found inside the `artifacts` folder in current directory (pwd).\n\n### Usage with `docker run`\n\nThere is an option to run postgres-checkup in a Docker container:\n\n```bash\ndocker run --rm \\\n  --name postgres-checkup \\\n  --env PGPASSWORD=\"postgres\" \\\n  --volume `pwd`/artifacts:/checkup/artifacts \\\n  postgresai/postgres-checkup:latest \\\n    ./checkup \\\n      --hostname hostname \\\n      --port 5432 \\\n      --username postgres \\\n      --dbname postgres \\\n      --project c \\\n      --epoch \"$(date +'%Y%m%d')001\"\n```\n\nIn this case some checks (those requiring SSH connection) will be skipped.\n\nIf you want to have all supported checks, you have to use SSH access to the\ntarget machine with Postgres database.\n\nIf SSH connection to the Postgres server is available, it is possible to pass\nSSH keys to the docker container, so postgres-checkup will switch to working via\nremote SSH calls, generating all reports (this approach is known to have issues\non Windows, but should work well on Linux and MacOS machines):\n\n```bash\ndocker run --rm \\\n  --name postgres-checkup \\\n  --volume \"$(pwd)/artifacts:/checkup/artifacts\" \\\n  --volume \"$(echo ~)/.ssh/id_rsa:/root/.ssh/id_rsa:ro\" \\\n  postgresai/postgres-checkup:latest \\\n  ./checkup \\\n    --hostname sshusername@hostname \\\n    --username my_postgres_user \\\n    --dbname my_postgres_database \\\n    --project docker_test_with_ssh \\\n    --epoch \"$(date +'%Y%m%d')001\"\n```\n\nIf you try to check the local instance of postgres on your host from a container,\nyou cannot use `localhost` in `-h` parameter. You have to use a bridge between\nhost OS and Docker Engine. By default, host IP is `172.17.0.1` in `docker0`\nnetwork, but it vary depending on configuration. More information [here](https://nickjanetakis.com/blog/docker-tip-65-get-your-docker-hosts-ip-address-from-in-a-container).\n\nIf you use SSH connection and `sudo` on the remote server requires a password,\nyou can provide this password using the `SSHSUDOPASSWORD` environment variable.\n\n## Credits\n\nSome reports are based on or inspired by useful queries created and improved by\nvarious developers, including but not limited to:\n * Jehan-Guillaume (ioguix) de Rorthais https://github.com/ioguix/pgsql-bloat-estimation\n * Alexey Lesovsky, Alexey Ermakov, Maxim Boguk, Ilya Kosmodemiansky et al. from Data Egret (aka PostgreSQL-Consulting) https://github.com/dataegret/pg-utils\n * Josh Berkus, Quinn Weaver et al. from PostgreSQL Experts, Inc. https://github.com/pgexperts/pgx_scripts\n\nDocker support implemented by [Ivan Muratov](https://gitlab.com/binakot).\n\n# The Full List of Reports\n\n## А. General  / Infrastructural\n\n- [x] A001 System information #6 , #56 , #57 , #86\n- [x] A002 Version information #68, #21, #86\n- [x] A003 Postgres settings  #15, #167, #86\n- [x] A004 Cluster information  #7, #58, #59, #86, #162\n- [x] A005 Extensions #8, #60, #61, #86, #167\n- [x] A006 Postgres setting deviations #9, #62, #63, #86\n- [x] A007 Altered settings #18, #86\n- [x] A008 Disk usage and file system type #19, #20\n- [ ] A010 Data checksums, wal_log_hints #22\n- [ ] A011 Connection pooling. pgbouncer #23\n- [ ] A012 Anti-crash checks #177\n\n## B. Backups and DR\n\n- [ ] B001 SLO/SLA, RPO, RTO  #24\n- [ ] B002 File system, mount flags #25\n- [ ] B003 Full backups / incremental  #26\n- [ ] B004 WAL archiving (GB/day?) - #27\n- [ ] B005 Restore checks, monitoring, alerting  #28\n\n## C. Replication and HA\n\n- [ ] C001 SLO/SLA  #29\n- [ ] C002 Sync/async, Streaming / wal transfer; logical decoding #30\n- [ ] C003 SPOFs; “-1 datacenter”, standby with traffic #31\n- [ ] C004 Failover #32\n- [ ] C005 Switchover #33\n- [ ] C006 Delayed replica (replay of 1 day of WALs) - #34\n- [ ] C007 Replication slots. Lags. Standby feedbacks\n\n## D. Monitoring / Troubleshooting\n\n- [ ] D001 Logging (syslog?), log_*** #35\n- [x] D002 Useful Linux tools  #36\n- [ ] D003 List of monitoring metrics #37\n- [x] D004 pg_stat_statements and pg_stat_kcache settings #38\n- [ ] D005 track_io_timing, …, auto_explain  #39\n- [ ] D006 Recommended DBA toolsets: postgres_dba, pgCenter, pgHeroother  #40\n- [ ] D007 Postgres-specific tools for troubleshooting  #137\n\n## E. WAL, Checkpoints\n\n- [ ] E001 WAL/checkpoint settings, IO  #41\n- [ ] E002 Checkpoints, bgwriter, IO  #42\n\n## F. Autovacuum, Bloat\n\n- [x] F001 \u003c F003 Autovacuum: current settings  #108, #164\n- [x] F002 \u003c F007 Autovacuum: transaction ID wraparound check  #16, #171\n- [x] F003 \u003c F006 Autovacuum: dead tuples  #164\n- [x] F004 \u003c F001 Autovacuum: heap bloat (estimated) #87, #122\n- [x] F005 \u003c F002 Autovacuum: index bloat (estimated) #88\n- [ ] F006 \u003c F004 Precise heap bloat analysis\n- [ ] F007 \u003c F005 Precise index bloat analysis\n- [x] F008 \u003c F008 Autovacuum: resource usage #44\n\n## G. Performance / Connections / Memory-related Settings\n\n- [x] G001 Memory-related settings #45, #190\n- [x] G002 Connections and current activity #46\n- [x] G003 Timeouts, locks, deadlocks #47\n- [ ] G004 Query planner (diff) #48\n- [ ] G005 I/O settings #49\n- [ ] G006 Default_statistics_target (plus per table?) #50\n\n## H. Index Analysis\n\n- [x] H001 Invalid indexes #192, #51\n- [x] H002 Unused indexes #51, #180, #170, #168, #322\n- [x] H003 Non-indexed foreign keys #52, #142, #173\n- [x] H004 Redundant indexes\n\n## J.  Capacity Planning\n\n- [ ] J001 Capacity planning - #54\n\n## K. SQL query Analysis\n\n- [x] K001 Globally aggregated query metrics #158, #178, #182, #184\n- [x] K002 Workload Type (\"The First Word\" Analysis) #159, #178, #179, #182, #184\n- [x] K003 Top-50 queries by total_time #160, #172, #174, #178, #179, #182, #184, #193\n\n## L. DB Schema Analysis\n- [x] L001 (was: H003) Table sizes #163\n- [ ] L002 (was: H004) Data types being used #53\n- [x] L003 Integer (int2, int4) out-of-range risks in PKs // calculate capacity remained; optional: predict when capacity will be fully used) https://gitlab.com/postgres-ai-team/postgres-checkup/issues/237\n- [ ] L004 Tables without PK/UK\n","funding_links":[],"categories":["Compiled list","Utilities","Monitoring/Statistics/Perfomance"],"sub_categories":["plv8:","Samples"],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/gitlab.com%2Fpostgres-ai%2Fpostgres-checkup","html_url":"https://awesome.ecosyste.ms/projects/gitlab.com%2Fpostgres-ai%2Fpostgres-checkup","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/gitlab.com%2Fpostgres-ai%2Fpostgres-checkup/lists"}