{"id":21224658,"url":"https://github.com/vitabaks/pgbackrest_auto","last_synced_at":"2025-07-10T14:31:50.298Z","repository":{"id":36243694,"uuid":"192348167","full_name":"vitabaks/pgbackrest_auto","owner":"vitabaks","description":"pgbackrest_auto:  Automatic Restore and Validate for physical and logical database corruption (PostgreSQL)","archived":false,"fork":false,"pushed_at":"2023-11-30T11:19:00.000Z","size":275,"stargazers_count":53,"open_issues_count":0,"forks_count":15,"subscribers_count":4,"default_branch":"master","last_synced_at":"2023-11-30T13:43:27.500Z","etag":null,"topics":["automate","pgbackrest","postgresql","restore","validate"],"latest_commit_sha":null,"homepage":"","language":"Shell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/vitabaks.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2019-06-17T13:02:26.000Z","updated_at":"2023-11-30T13:43:27.501Z","dependencies_parsed_at":"2023-11-29T13:51:13.197Z","dependency_job_id":null,"html_url":"https://github.com/vitabaks/pgbackrest_auto","commit_stats":null,"previous_names":[],"tags_count":7,"template":null,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vitabaks%2Fpgbackrest_auto","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vitabaks%2Fpgbackrest_auto/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vitabaks%2Fpgbackrest_auto/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vitabaks%2Fpgbackrest_auto/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/vitabaks","download_url":"https://codeload.github.com/vitabaks/pgbackrest_auto/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":225642427,"owners_count":17501312,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["automate","pgbackrest","postgresql","restore","validate"],"created_at":"2024-11-20T22:59:29.699Z","updated_at":"2024-11-20T22:59:30.203Z","avatar_url":"https://github.com/vitabaks.png","language":"Shell","readme":"# pgbackrest_auto\n\nThis is a simple script to automate the process of restore PostgreSQL databases from backup using [pgbackrest](https://github.com/pgbackrest/pgbackrest). With the possibility **validate for physical and logical database corruption**, and (optional) sending the final report to DBA an e-mail address.\n\n#### Data Validation\n\n\u003ePostgreSQL has supported page-level checksums since 9.3. **If page checksums are enabled** pgbackrest will [validate](https://pgbackrest.org/configuration.html#section-backup/option-checksum-page) the checksums for every file that is copied during a backup. All page checksums are validated during a full backup and checksums in files that have changed are validated during differential and incremental backups.\nThis feature allows page-level corruption to be detected early.\n\n`pgbackrest_auto` runs **additional advanced checks on your data** after the recovery stage. In case of PostgreSQL data restore and recovery, the stage of physical data validation is performed (all data blocks are sequentially read with pg_dump);\nDatabases that successfully passed the stage of physical validation additionally pass the stage of logical validation using the [amcheck](https://github.com/petergeoghegan/amcheck) extension. Tthe logical integrity of the structure of the B-Tree indexes and tables related to the target index relation is checked (used `bt_index_parent_check` with `heapallindexed` argument is `true`).\n\n![pgbackrest_auto_scheme](./scheme.png)\n\n#### Example\n```\npgbackrest_auto --from=epgdb --to=/bkpdata/rst/epgdb --checkdb\n```\n###### result:\n```\n2022-06-15 14:13:13 INFO: [STEP 1]: Starting\n2022-06-15 14:13:13 INFO: Starting. Restore Type: Full PostgreSQL Restore FROM Stanza: epgdb --\u003e TO Directory: /bkpdata/rst/epgdb\n2022-06-15 14:13:14 INFO: Starting. Restore Settings: immediate\n2022-06-15 14:13:14 INFO: Starting. Run settings: Backup host: localhost\n2022-06-15 14:13:14 INFO: Starting. Run settings: Log: /var/log/pgbackrest/pgbackrest_auto_epgdb.log\n2022-06-15 14:13:14 INFO: Starting. Run settings: Lock run: /tmp/pgbackrest_auto_epgdb.lock\n2022-06-15 14:13:14 INFO: Starting. PostgreSQL instance: epgdb\n2022-06-15 14:13:14 INFO: Starting. PostgreSQL version: 11\n2022-06-15 14:13:14 INFO: Starting. PostgreSQL port: 5432\n2022-06-15 14:13:14 INFO: Starting. PostgreSQL Database Validation: yes\n2022-06-15 14:13:14 WARN: Restoring to /bkpdata/rst/epgdb Waiting 30 seconds. The directory will be overwritten. If mistake, press ^C\n2022-06-15 14:13:44 INFO: [STEP 2]: Stopping PostgreSQL\n2022-06-15 14:13:44 INFO: attempt: 1/3600\n2022-06-15 14:13:44 INFO: PostgreSQL check status\n2022-06-15 14:13:44 INFO: PostgreSQL instance epgdb not running\n2022-06-15 14:13:44 INFO: [STEP 3]: Restoring from backup\n2022-06-15 14:13:44 INFO: Restore from backup started. Type: Full PostgreSQL Restore\n2022-06-15 14:13:44 INFO: See detailed log in the file /var/log/pgbackrest/epgdb-restore.log\npgbackrest --config=/tmp/pgbackrest_auto.conf --repo1-host=localhost --repo1-host-user=postgres --stanza=epgdb --pg1-path=/bkpdata/rst/epgdb  --type=immediate --repo1-path=/bkpdata/pgbackrest --delta restore --process-max=4 --log-level-console=error --log-level-file=detail --recovery-option=recovery_target_action=promote --tablespace-map-all=/bkpdata/rst/epgdb/remapped_tablespaces\n2022-06-15 14:13:52 INFO: Restore from backup done\n2022-06-15 14:13:52 INFO: [STEP 4]: PostgreSQL Starting for recovery\n2022-06-15 14:13:52 INFO: PostgreSQL start\n2022-06-15 14:14:02 INFO: attempt: 1/3600\n2022-06-15 14:14:03 INFO: PostgreSQL instance epgdb started and accepting connections\n2022-06-15 14:14:03 INFO: [STEP 5]: PostgreSQL Recovery Checking\n2022-06-15 14:14:03 INFO: Checking if restoring from archive is done\n2022-06-15 14:14:03 INFO: Replayed:\n2022-06-15 14:14:03 INFO: Restoring from archive is done\n2022-06-15 14:14:03 INFO: Restore done\n2022-06-15 14:14:03 INFO: [STEP 6]: Validate for physical database corruption\n2022-06-15 14:14:04 INFO: Start data validation for database postgres\n2022-06-15 14:14:04 INFO: ... starting pg_dump -p 5432 -d postgres \u003e\u003e /dev/null\n2022-06-15 14:14:06 INFO: Data validation in the database postgres - Successful\n2022-06-15 14:14:06 INFO: Start data validation for database epg\n2022-06-15 14:14:06 INFO: ... starting pg_dump -p 5432 -d epg \u003e\u003e /dev/null\n2022-06-15 14:14:12 INFO: Data validation in the database epg - Successful\n2022-06-15 14:14:12 INFO: Start data validation for database ota\n2022-06-15 14:14:12 INFO: ... starting pg_dump -p 5432 -d ota \u003e\u003e /dev/null\n2022-06-15 14:14:13 INFO: Data validation in the database ota - Successful\n2022-06-15 14:14:13 INFO: [STEP 7]: Validate for logical database corruption\n2022-06-15 14:14:13 INFO: pg_checksums: starting data checksums validation\n2022-06-15 14:14:21 INFO: pg_checksums: data checksums validation - Successful\n2022-06-15 14:14:22 INFO: amcheck: verify the logical consistency of the structure of indexes and heap relations in the database postgres\n2022-06-15 14:14:22 INFO: amcheck: verify the logical consistency of the structure of indexes and heap relations in the database epg\n2022-06-15 14:14:59 INFO: amcheck: verify the logical consistency of the structure of indexes and heap relations in the database ota\n2022-06-15 14:15:23 INFO: Finish\n```\n\n\n#### pgbackrest_auto --help\n```\nAutomatic Restore and Validate for physical and logical database corruption\n\nSupport three types of restore:\n        1) Restore last backup  (recovery to earliest consistent point) [default]\n        2) Restore latest       (recovery to the end of the archive stream)\n        3) Restore to the point (recovery to restore point)\n\nImportant: Run on the nodes on which you want to restore the backup\n\nUsage: /usr/bin/pgbackrest_auto --from=STANZANAME --to=DATA_DIRECTORY [ --datname=DATABASE [...] ] [ --recovery-type=( default | immediate | time ) ] [ --recovery-target=TIMELINE  [ --backup-set=SET ] [ --pgver= ] [ --checkdb ] [ --clear ] [ --report ] ]\n\n--from=STANZANAME\n        Stanza from which you need to restore from a backup\n\n--to=DATA_DIRECTORY\n        PostgreSQL Data directory Path to restore from a backup\n        a PostgreSQL database cluster (PGDATA) will be automatically created if it does not exist\n        Example: /bkpdata/rst/app-db\n\n--datname=DATABASE [...]\n        Database name to be restored (After this you MUST drop other databases)\n        Note that built-in databases (template0, template1, and postgres) are always restored.\n        To be restore more than one database specify them in brackets separated by spaces.\n        Example: --datname=\"db1 db2\"\n\n--recovery-type=TYPE\n        immediate - recover only until the database becomes consistent           (Type 1. Restore last backup)  [default]\n        default   - recover to the end of the archive stream                     (Type 2. Restore latest)\n        time      - recover to the time specified in --recovery-target           (Type 3. Restore to the point)\n\n--recovery-target=TIMELINE\n        time - recovery point time. The time stamp up to which recovery will proceed.\n        if --recovery-type=time\n        Example: \"2022-06-14 09:00:00\"\n\n--backup-set=SET\n        If you need to restore not the most recent backup. Example few days ago.\n        Get info of backup. Login to pgbackrest server. User postgres\n        pgbackrest --stanza=[STANZA NAME] info\n        And get it. Example:\n                    incr backup: 20220611-000004F_20220614-000003D\n        This is the name of SET: 20220611-000004F_20220614-000003D\n\n--pgver=VERSION\n        PostgreSQL cluster (instance) version [ optional ]\n        by default, the PostgreSQL version will be determined from the pgbackrest info\n\n--dummy-dump\n        Verify that data can be read out. Check with pg_dump\n\n--checksums\n        Check data checksums\n\n--amcheck\n        Validate Indexes (verify the logical consistency of the structure of indexes and heap relations)\n\n--checkdb\n        Validate for Physical and Logical Database Corruption (includes: dummy-dump, checksums, amcheck)\n\n--clear\n        Clear PostgreSQL Data directory after Restore (the path was specified in the \"--to\" parameter ) [ optional ]\n\n--report\n        Send report to mail address\n\n--norestore\n        Do not restore a stanza but use an already existing cluster\n\n--config=/path/to/pgbackrest.conf\n        The path to the custom pgbackrest configuration file [ optional ]\n\n--custom-options=\n        Costom options for pgBackRest [ optional ]\n        This includes all the options that may also be configured in pgbackrest.conf\n        Example: --option1=value --option2=value --option3=value\n        See all available options: https://pgbackrest.org/configuration.html\n\n--process-max=\n        Max processes to use for restore and validate (default 1).\n\nEXAMPLES:\n( example stanza \"app-db\" , backup host \"localhost\" (default value) )\n\n| Restore last backup:\n\n    /usr/bin/pgbackrest_auto --from=app-db --to=/bkpdata/rst/app-db\n\n| Restore backup made a few days ago:\n\n    /usr/bin/pgbackrest_auto --from=app-db --to=/bkpdata/rst/app-db --backup-set=20220611-000004F_20220614-000003D\n\n| Restore backup made a few days ago and pick time:\n\n    /usr/bin/pgbackrest_auto --from=app-db --to=/bkpdata/rst/app-db --backup-set=20220611-000004F_20220614-000003D --recovery-type=time --recovery-target=\"2022-06-14 09:00:00\"\n\n| Restore backup made a few days ago and pick time. And we have restore only one database with the name \"app_db\":\n\n    /usr/bin/pgbackrest_auto --from=app-db --to=/bkpdata/rst/app-db --backup-set=20220611-000004F_20220614-000003D --recovery-type=time --recovery-target=\"2022-06-14 09:00:00\" --datname=app_db\n\n| Restore and Validate of databases:\n\n    /usr/bin/pgbackrest_auto --from=app-db --to=/bkpdata/rst/app-db --checkdb\n\n```\n\n---\n\n :bulb: You can use this script to daily automatically check your backups, immediately after the completion of the backup process.\n\n###### Example of Cron jobs:\n\n```\n#=== pgbackrest - Backup PostgreSQL ====================\n\n01 00 * * 6 if pgbackrest --stanza=app-db --type=full backup; then pgbackrest_auto --from=app-db --to=/bkpdata/rst/app-db --checkdb --clear --report; fi\n01 00 * * 0-5 if pgbackrest --stanza=app-db --type=diff backup; then pgbackrest_auto --from=app-db --to=/bkpdata/rst/app-db --checkdb --clear --report; fi\n\n30 00 * * 6 if pgbackrest --stanza=apdb-cluster --type=full backup; then pgbackrest_auto --from=apdb-cluster --to=/bkpdata/rst/apdb-cluster --checkdb --clear --report; fi\n30 00 * * 0-5 if pgbackrest --stanza=apdb-cluster --type=diff backup; then pgbackrest_auto --from=apdb-cluster --to=/bkpdata/rst/apdb-cluster --checkdb --clear --report; fi\n\n00 01 * * 6 if pgbackrest --stanza=dbs-eu --type=full backup; then pgbackrest_auto --from=dbs-eu--to=/bkpdata/rst/dbs-eu --checkdb --clear --report; fi\n00 01 * * 0-5 if pgbackrest --stanza=dbs-eu --type=diff backup; then pgbackrest_auto --from=dbs-eu--to=/bkpdata/rst/dbs-eu --checkdb --clear --report; fi\n\n#=======================================================\n```\n\n## Compatibility\nRedHat and Debian based distros\n\n###### PostgreSQL versions:\nall supported PostgreSQL versions\n\n## Requirements\n`pgbackrest` and `jq` packages.\n\nfor `--checksums` (and `--checkdb`):\n  - `postgresql-\u003cversion\u003e-pg-checksums` package (if PostgreSQL version \u003c= 11)\n\nfor `--amcheck` (and `--checkdb`):\n  - `postgresql-\u003cversion\u003e-amcheck` package (if PostgreSQL version \u003c= 10)\n    (_the amcheck extension will be automatically installed to the restored databases_)\n\nfor `--report`:\n  - `sendemail` package\n  - specify smtp parameters `smtp_server`, `mail_from`, `mail_to` in the pgbackrest_auto script file.\n\nRun as user: `postgres`\n\nIf your PostgreSQL is installed somewhere other than the default installation path, please specify the `PG_BIN_DIR` variable in the script file.\n\n## Installation\n1. Download and copy the `pgbackrest_auto` script to `/usr/bin/` directory\n2. Grant execute rights on the scripts\n\nExample:\n```\nwget https://raw.githubusercontent.com/vitabaks/pgbackrest_auto/master/pgbackrest_auto\nsudo mv pgbackrest_auto /usr/bin/\nsudo chown postgres:postgres /usr/bin/pgbackrest_auto\nsudo chmod 750 /usr/bin/pgbackrest_auto\n```\n\n\n## Logging\nLog file: `/var/log/pgbackrest/pgbackrest_auto_\u003cSTANZANAME\u003e.log`\n\nIn addition, the script execution is written in syslog. Get the pgbackrest_auto log:\n```\nsudo grep pgbackrest_auto /var/log/syslog\n```\n\n## License\nLicensed under the MIT License. See the [LICENSE](./LICENSE) file for details.\n\n## Author\nVitaliy Kukharik (PostgreSQL DBA) vitabaks@gmail.com\n\n## Feedback, bug-reports, requests, ...\nAre [welcome](https://github.com/vitabaks/pgbackrest_auto/issues)!\n#### Help Wanted! If you noticed a bug or a missing feature or just have an idea of how this project could be enhanced, please feel free to file an issue.\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvitabaks%2Fpgbackrest_auto","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fvitabaks%2Fpgbackrest_auto","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvitabaks%2Fpgbackrest_auto/lists"}