{"id":21873334,"url":"https://github.com/jacksonkasi0/postgres-r2-backup","last_synced_at":"2025-10-08T17:30:40.585Z","repository":{"id":265009777,"uuid":"892955259","full_name":"jacksonkasi0/postgres-r2-backup","owner":"jacksonkasi0","description":"An automated PostgreSQL backup and restore system with 30-day retention, Cloudflare R2 integration, and Docker support.","archived":false,"fork":false,"pushed_at":"2024-12-14T18:09:20.000Z","size":34,"stargazers_count":10,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-01-17T19:18:18.479Z","etag":null,"topics":["automation","backup","cloudflare-r2","cron","database-backup","database-management","deno","docker","postgresql","restore"],"latest_commit_sha":null,"homepage":"","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/jacksonkasi0.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-11-23T06:17:42.000Z","updated_at":"2025-01-08T18:56:47.000Z","dependencies_parsed_at":"2024-12-14T18:36:32.512Z","dependency_job_id":null,"html_url":"https://github.com/jacksonkasi0/postgres-r2-backup","commit_stats":null,"previous_names":["jacksonkasi0/postgres-r2-backup"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jacksonkasi0%2Fpostgres-r2-backup","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jacksonkasi0%2Fpostgres-r2-backup/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jacksonkasi0%2Fpostgres-r2-backup/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jacksonkasi0%2Fpostgres-r2-backup/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/jacksonkasi0","download_url":"https://codeload.github.com/jacksonkasi0/postgres-r2-backup/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":235736813,"owners_count":19037536,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["automation","backup","cloudflare-r2","cron","database-backup","database-management","deno","docker","postgresql","restore"],"created_at":"2024-11-28T07:07:33.733Z","updated_at":"2025-10-08T17:30:35.260Z","avatar_url":"https://github.com/jacksonkasi0.png","language":"TypeScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# PostgreSQL DB Backup \u0026 Restore via Docker \u0026 Deno\n\nThis project provides a system to automate the backup and restore of a PostgreSQL database using **Deno**, **Docker**, and **Cloudflare R2**. It supports local backups, restoring from local files, and restoring directly from Cloudflare R2.\n\n---\n\n## **Technologies Used**\n\n- **Deno**: Runtime for the scripts and automation.\n- **Docker**: Containerization for portable backup/restore workflows.\n- **PostgreSQL**: Database used for backups and restores.\n- **PostgreSQL 16 Client**: Installed in the container for backup and restore operations. *(Note: Update the `postgresql\u003cx\u003e-client` version in the Dockerfile if your database uses a different version.)*\n- **Cloudflare R2**: Object storage for storing backup files.\n- **Winston**: Logging library for structured and color-coded logs.\n- **pg_restore**: PostgreSQL's native tool for restoring database dumps.\n- **Cronie**: Lightweight cron daemon for scheduling backup and restore tasks in the container.\n\n## **Getting Started**\n\n### **Requirements**\n\n1. **Deno** installed ([Install Guide](https://deno.land/manual/getting_started/installation)).\n2. **Docker** installed ([Install Guide](https://docs.docker.com/get-docker/)).\n3. A `.env` file in the root directory containing:\n\n   ```env\n   AUTOMATIC_BACKUP=If set to `true`, it will trigger a cron job to perform a backup.\n   BACKUP_DB_URL=your_postgresql_connection_url\n   RESTORE_DB_URL=your_postgresql_connection_url\n   ACCESS_KEY_ID=your_r2_access_key_id\n   SECRET_ACCESS_KEY=your_r2_secret_access_key\n   BACKUP_DIR=your_r2_bucket_db_backup_dir\n   ACCOUNT_ID=your_r2_account_id\n   BUCKET_NAME=your_r2_bucket_name\n   ```\n\n4. **Cloudflare R2** storage setup with appropriate access credentials.\n\n---\n\n## **Available Tasks**\n\n### **Backup Task**\n\nCreates a backup of the database and uploads it to Cloudflare R2.\n\n**Command:**\n\n```sh\ndeno task run:backup\n```\n\n**Process:**\n\n1. Generates a backup file in the local `db_backups/\u003cfolder\u003e` directory.\n2. Uploads the backup to the specified Cloudflare R2 bucket.\n\n---\n\n### **Restore Task**\n\nRestores the database from a local backup file.\n\n**Command:**\n\n```sh\ndeno task run:restore\n```\n\n**Process:**\n\n1. Restores the database from a specified local backup file.\n2. Ensure the local backup file exists in the `db_backups/\u003cfolder\u003e` directory.\n\n---\n\n### **Restore From R2 Task**\n\nRestores the database by downloading a backup file from Cloudflare R2.\n\n**Command:**\n\n```sh\ndeno task run:restore-r2 --key=backups/\u003cfolder\u003e/backup_1732266xxxxxx.tar\n```\n\n**Process:**\n\n1. Downloads the specified backup file from Cloudflare R2 to a temporary location.\n2. Restores the database using the downloaded file.\n3. Deletes the temporary file after restoring.\n\n---\n\n### **Automate Task**\n\nAutomates the backup process, including uploading to R2 and cleanup.\n\n**Command:**\n\n```sh\ndeno task automate\n```\n\n**Process:**\n\n1. Creates a backup of the database.\n2. Uploads the backup file to Cloudflare R2.\n3. Cleans up local backup files older than the retention period.\n4. Deletes old R2 backups based on the retention policy.\n\n---\n\n### **Restore From R2 Task (Script)**\n\nRuns the restore process from R2 using Docker for containerized execution.\n\n**Setup:**\n\n1. Set `RESTORE_DB_URL` in the `.env` file.\n2. Modify the `run_restore.sh` script to include the desired R2 object key.\n\n**Command:**\n\n```sh\nchmod +x run_restore.sh\n./run_restore.sh\n```\n\n**Process:**\n\n1. Builds and runs a Docker container using `Dockerfile.restore`.\n2. Passes environment variables and R2 object key to the container.\n3. Restores the database and removes the temporary container after completion.\n\n---\n\n## **Folder Structure**\n\n```plaintext\n.\n├── .dockerignore             # Specifies files and directories to ignore in Docker builds\n├── .env                      # Environment variables file\n├── .env.example              # Example environment variables file\n├── .vscode/\n│   └── settings.json         # VS Code-specific configuration\n├── deno.json                 # Deno configuration and tasks\n├── deno.lock                 # Dependency lock file for Deno\n├── Dockerfile.backup         # Dockerfile for the backup container\n├── Dockerfile.restore        # Dockerfile for the restore container\n├── railway.toml              # Railway platform configuration for deployment\n├── README.md                 # Documentation\n├── run_automate.sh           # Script to trigger automate task\n├── run_restore.sh            # Script to trigger restore from R2 task via Docker\n├── start.sh                  # Startup script for cron-based backups\n├── src/\n│   ├── backup/               # Backup-related functionality\n│   │   ├── cleanCloudBackups.ts  # Cleans up old backups from R2\n│   │   ├── cleanLocalBackups.ts  # Cleans up old backups locally\n│   │   └── createBackup.ts       # Logic to create a PostgreSQL backup\n│   ├── restore/              # Restore-related functionality\n│   │   └── restoreBackup.ts      # Logic to restore a PostgreSQL backup\n│   ├── utils/                # Utility functions and helpers\n│   │   ├── r2/\n│   │   │   ├── cloudflareR2.ts   # Cloudflare R2 integration logic\n│   │   │   ├── index.ts          # R2 helper functions\n│   │   │   └── multipartUploadToR2.ts  # Logic for multipart uploads to R2\n│   │   ├── execCommand.ts     # Helper for executing shell commands\n│   │   └── logger.ts          # Custom logger with color-coded log levels\n│   ├── config.ts              # Configuration file for environment variables\n│   ├── index.ts               # Entry point for backup/restore tasks\n│   └── automate.ts            # Automation script for scheduled backups\n```\n\n---\n\n## **Environment Variables**\n\nEnsure the following environment variables are defined in the `.env` file:\n\n```env\n# PostgreSQL Connection URLs\nBACKUP_DB_URL=your_postgresql_backup_url\nRESTORE_DB_URL=your_postgresql_restore_url\n\n# Cloudflare R2 Configuration\nACCESS_KEY_ID=your_r2_access_key_id\nSECRET_ACCESS_KEY=your_r2_secret_access_key\nACCOUNT_ID=your_r2_account_id\nBUCKET_NAME=your_r2_bucket_name\n```\n\n---\n\n## **Cron Integration**\n\nTo automate backups with cron:\n\n1. Add the `deno task automate` command to your crontab file.\n2. Example entry for a daily backup at midnight:\n\n   ```cron\n   0 0 * * * deno task automate \u003e\u003e /var/log/db_backup.log 2\u003e\u00261\n   ```\n\n---\n\n## **Additional Notes**\n\n- Automatic backup is configured in `Dockerfile.backup`. It is designed for deployment on the **Railway.app** platform via the `railway.toml` file to enable automatic PostgreSQL backups. If you are using a different platform, update the configuration to run `Dockerfile.backup` by default.\n- Ensure your R2 bucket permissions allow read and write operations.\n- Use the `run_restore.sh` script for containerized restore processes.\n- For schema-specific restores, customize the `restoreBackup.ts` function as needed.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjacksonkasi0%2Fpostgres-r2-backup","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjacksonkasi0%2Fpostgres-r2-backup","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjacksonkasi0%2Fpostgres-r2-backup/lists"}