{"id":22484474,"url":"https://github.com/jobinau/pg_gather","last_synced_at":"2025-08-02T17:32:57.592Z","repository":{"id":50727027,"uuid":"330908378","full_name":"jobinau/pg_gather","owner":"jobinau","description":"Scan PostgreSQL Instance for potential problems. pg_gather is a SQL-only script leveraging the built-in features of psql. ","archived":false,"fork":false,"pushed_at":"2024-12-04T17:11:42.000Z","size":1005,"stargazers_count":147,"open_issues_count":22,"forks_count":19,"subscribers_count":10,"default_branch":"main","last_synced_at":"2024-12-04T18:22:04.067Z","etag":null,"topics":["database","performance-analysis","postgres","postgresql","scannner"],"latest_commit_sha":null,"homepage":"","language":"HTML","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/jobinau.png","metadata":{"files":{"readme":"README.md","changelog":"history_schema.sql","contributing":"docs/CONTRIBUTING","funding":null,"license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":"docs/security.md","support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-01-19T08:12:07.000Z","updated_at":"2024-12-04T17:11:46.000Z","dependencies_parsed_at":"2023-10-15T11:48:26.042Z","dependency_job_id":"13c44a45-ba4b-45cb-852c-708e14232efb","html_url":"https://github.com/jobinau/pg_gather","commit_stats":null,"previous_names":[],"tags_count":26,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jobinau%2Fpg_gather","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jobinau%2Fpg_gather/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jobinau%2Fpg_gather/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jobinau%2Fpg_gather/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/jobinau","download_url":"https://codeload.github.com/jobinau/pg_gather/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":228499909,"owners_count":17929978,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["database","performance-analysis","postgres","postgresql","scannner"],"created_at":"2024-12-06T17:11:03.304Z","updated_at":"2025-08-02T17:32:57.579Z","avatar_url":"https://github.com/jobinau.png","language":"HTML","funding_links":[],"categories":["HTML"],"sub_categories":[],"readme":"# pg_gather aka pgGather\n![pgGather logo](./docs/pgGather.svg)  \nScan and collect the minimal amount of data needed to identify potential problems in your PostgreSQL database, and then generate an analysis report using that data. This project provides two SQL scripts for users:  \n  \n* `gather.sql`: Gathers performance and configuration data from PostgreSQL databases.\n* `gather_report.sql`: Analyzes the collected data and generates detailed HTML reports.\n  \nEverything is SQL-only, leveraging the built-in features of psql, the command-line utility of PostgreSQL\n\n**Supported PostgreSQL Versions** :  10, 11, 12, 13, 14, 15, 16 \u0026 17\n**Older versions** : For PostgeSQL 9.6 and older, Please refer the [documentation page](docs/oldversions.md)\n\n# Highlights\n1. **Secure by Open :** Simple, Transperent, Fully auditable code.  \n   To ensure full transparency of what is collected, transmitted, and analyzed, we use an SQL-only data collection script and avoid programs with any control structures, thus improving the readability and auditability of the data collection. This is one reason for separating data collection and analysis.\n2. **No Executables :** No executables need to be deployed on the database host  \n   Using executables in secured environments poses unacceptable risks in many highly secure environments. `pg_gather` requires only the standard PostgreSQL command line utility, `psql`, and no other libraries or executables.\n3. **Authentication agnostic**\n   Any authentication mechanism supported by PostgreSQL works for data gathering in `pg_gather`, because it uses the standard `psql` command-line utility.\n4. **Any Operating System**  \n   Linux (32/64-bit), Sun Solaris, Apple macOS, and Microsoft Windows: pg_gather works wherever `psql` is available, ensuring maximum portability. \n   (Windows users, please see the [Notes section](#notes) below)\n5. **Architecture agnostic**\n   x86-64 bit, ARM, Sparc, Power, and other architectures. It works anywhere `psql` is available.\n6. **Auditable and optionally maskable data** :  \n   `pg_gather` collects data in Tab Separated Values (TSV) format, making it easy to review and audit the information before sharing it for analysis. Additional masking or trimming is also possible with [simple steps](docs/security.md).\n7. **Any cloud/container/k8s** :   \n   Works with AWS RDS, Azure, Google Cloud SQL, on-premises databases, and more.  \n   (Please see Heroku, AWS Aurora, Docker and K8s specific notes in the [Notes section](#notes) below)\n8. **Zero failure design** :   \n   `pg_gather` can generate a report from available information even if data collection is partial or fails due to permission issues, unavailable tables/views, or other reasons.\n9.  **Low overhead for data collection** :  \n   By design, data collection is separate from data analysis. This allows the collected data to be analyzed on an independent system, so that analysis queries do not adversely impact critical systems. In most cases, the overhead of data collection is negligible.\n10. **Small, single file data dump** :  \n   To generate the smallest possible file, which can be further compressed by `gzip` for the easy transmission and storage, `pg_gather` avoids redundancy in the collected data as much as possible.\n  \n\n# How to Use\n\n# 1. Data Gathering.\nTo gather configuration and performance information, run the `gather.sql` script against the database using `psql`:\n```\npsql \u003cconnection_parameters_if_any\u003e -X -f gather.sql \u003e out.tsv\n```\nOR ALTERNATIVELY pipe to a compression utilty to get a compressed output as follows:\n```\npsql \u003cconnection_parameters_if_any\u003e -X -f gather.sql | gzip \u003e out.tsv.gz\n```\nThis script may take over 20 seconds to run because it contains sleeps/delays. We recommend running the script as a privileged user (such as `superuser` or `rds_superuser`) or as an account with the `pg_monitor` privilege. The output file contains performance and configuration data for analysis.\n\u003ca name=\"notes\"\u003e\n## Notes:\u003c/a\u003e \n   1. **Heroku** and similar DaaS hostings impose very high restrictions on collecting performance data. Queries on views like `pg_statistics` may produce errors during data collection, but these errors can be ignored.\n   2. **MS Windows** users!, \n   Client tools like [pgAdmin](https://www.pgadmin.org/) include `psql`, which can be used to run `pg_gather` against local or remote databases.\n   For example:  \n   ```\n     \"C:\\Program Files\\pgAdmin 4\\v4\\runtime\\psql.exe\" -h pghost -U postgres -f gather.sql \u003e out.tsv\n   ```\n   3. **AWS Aurora** offers a \"PostgreSQL-compatible\" database. However, it is not a true PostgreSQL database, even though it looks like one. Therefore, you should do the following to the `gather.sql` script to replace any unapplicable lines with \"NULL\".\n   ```\n     sed -i -e 's/^CASE WHEN pg_is_in_recovery().*/NULL/' gather.sql\n   ```\n   4. **Docker** containers of PostgreSQL may not include the `curl` or `wget` utilities necessary to download `gather.sql`. Therefore, it is recommended to pipe the contents of the SQL file to `psql` instead.\n   ```\n     cat gather.sql | docker exec -i \u003ccontainer\u003e psql -X -f - \u003e out.tsv\n   ```\n   5. **Kubernetes**  environments also have similar restrictions as those mentioned for Docker. Therefore, a similar approach is suggested.\n   ```\n     cat gather.sql | kubectl exec -i \u003cPGpod\u003e -- psql -X -f - \u003e out.tsv\n   ```\n\n## Gathering data continuosly\nThere could be requirements for collecting data continuously and repatedly. `pg_gather` has a special lightweight mode for continuous data gathering, which is automatically enabled when it connects to the \"template1\" database. Please refer to detailed [documentation specific to continuous and repated data collection](docs/continuous_collection.md)\n\n# 2. Data Analysis\n## 2.1 Importing collected data\nThe collected data can be imported to a PostgreSQL Instance. This creates required schema objects in the `public` schema of the database. \n**CAUTION :** Avoid importing the data into critical environments/databases. A temporary PostgreSQL instance is preferable.\n```\n psql -f gather_schema.sql -f out.tsv\n```\nDeprecated usage of `sed` : sed -e '/^Pager/d; /^Tuples/d; /^Output/d; /^SELECT pg_sleep/d; /^PREPARE/d; /^\\s*$/d' out.tsv | psql -f gather_schema.sql -\n## 2.2 Generating Report\nAn analysis report in HTML format can be generated from the imported data as follows.\n```\npsql -X -f gather_report.sql \u003e GatherReport.html\n```\nYou may use your favourite web browser to read the report.\n\nNOTE: PostgreSQL version 13 or above is required to generate the analysis report.\n\n\n\n# ANNEXTURE 1 : Using PostgreSQL container and wrapper script\nThe steps for data analysis mentioned above seem simple (single command), but they require a PostgreSQL instance to import the data into. An alternative is to use the `generate_report.sh` script, which can spin up a PostgreSQL Docker container and automate the entire process. To use this script, you must place it in a directory containing the `gather_schema.sql` and `gather_report.sql` files.\n\nThe script will spin up a Docker container, import the output of `gather.sql` (out.tsv) and then it generates an HTML report. This script expects at least a single argument: path to the `out.tsv`. \n\nThere are two more additional positional arguments: \n* Desired report name with path. \n* A flag to specify whether to keep the docker container. This flag allows to usage of the container and data for further analysis.\n\nExample 1: Import data and generate an HTML file\n```\n$ ./generate_report.sh /tmp/out.tsv\n...\nContainer 61fbc6d15c626b484bdf70352e94bbdb821971de1e00c6de774ca5cd460e8db3 deleted\nFinished generating report in /tmp/out.txt.html\n```\nExample 2: Import data, keep the container intact and generate the report in the specified location\n```\n$ ./generate_report.sh /tmp/out.tsv /tmp/custom-name.html y\n...\nContainer df7b228a5a6a49586e5424e5fe7a2065d8be78e0ae3aa5cddd8658ee27f4790c left around\nFinished generating report in /tmp/custom-name.html\n```\n# Advanced configurations\n## Timezone \nBy default, the `pg_gather` report uses the same timezone of the server from which the data is collected, because it considers the `log_timezone` parameter for generating the report. This default timezone setting helps to compare the PostgreSQL log entries with the `pg_gather` report.\nHowever, this may not be the right timezone for few users, especially when cloud hostings are used. The `pg_gather` allows the user to have a custom timezone by setting the environment variable `PG_GATHER_TIMEZONE` to override the default. For example,\n```\nexport PG_GATHER_TIMEZONE='UTC'\n```\nPlease use the timezone name or abbreviation available from `pg_timezone_names`\n# Demo\n## Data collection\n[![IMAGE ALT TEXT HERE](https://img.youtube.com/vi/4EK7BoV6oOg/0.jpg)](https://youtu.be/4EK7BoV6oOg)\n## Simple Report Generation (1min): \n[![Import data and Generate report using PG](https://img.youtube.com/vi/Y8gq1dwfzQU/0.jpg)](https://youtu.be/Y8gq1dwfzQU)\n## Report generation using postgresql docker container made easy (3min): \n[![Import data and Generate report using PG docker container: made simple](https://img.youtube.com/vi/amPQRzz5D8Y/0.jpg)](https://youtu.be/amPQRzz5D8Y)","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjobinau%2Fpg_gather","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjobinau%2Fpg_gather","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjobinau%2Fpg_gather/lists"}