{"id":28241852,"url":"https://github.com/significa/fly-pg-dump-to-s3","last_synced_at":"2026-03-17T19:38:12.600Z","repository":{"id":59890488,"uuid":"539882373","full_name":"significa/fly-pg-dump-to-s3","owner":"significa","description":"Fly pg_dump to AWS S3","archived":false,"fork":false,"pushed_at":"2025-09-04T11:33:01.000Z","size":103,"stargazers_count":9,"open_issues_count":0,"forks_count":3,"subscribers_count":3,"default_branch":"main","last_synced_at":"2025-09-04T12:24:54.167Z","etag":null,"topics":["backup","backup-script","fly","flyio","flyio-tools","postgres"],"latest_commit_sha":null,"homepage":"","language":"Shell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/significa.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2022-09-22T08:43:20.000Z","updated_at":"2025-09-04T11:33:04.000Z","dependencies_parsed_at":"2025-01-08T11:25:48.429Z","dependency_job_id":"6321df3b-f072-42e0-8787-2a18c873fe4e","html_url":"https://github.com/significa/fly-pg-dump-to-s3","commit_stats":null,"previous_names":[],"tags_count":12,"template":false,"template_full_name":null,"purl":"pkg:github/significa/fly-pg-dump-to-s3","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/significa%2Ffly-pg-dump-to-s3","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/significa%2Ffly-pg-dump-to-s3/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/significa%2Ffly-pg-dump-to-s3/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/significa%2Ffly-pg-dump-to-s3/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/significa","download_url":"https://codeload.github.com/significa/fly-pg-dump-to-s3/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/significa%2Ffly-pg-dump-to-s3/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":30629664,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-03-17T17:32:55.572Z","status":"ssl_error","status_checked_at":"2026-03-17T17:32:38.732Z","response_time":56,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["backup","backup-script","fly","flyio","flyio-tools","postgres"],"created_at":"2025-05-19T05:08:55.835Z","updated_at":"2026-03-17T19:38:12.568Z","avatar_url":"https://github.com/significa.png","language":"Shell","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Fly pg_dump to AWS S3\n\nUtilities to backup a Fly.io Postgres database to AWS S3 buckets.\n\nThis repository contains two backup strategies:\n\n1. The simple method: a github action that connects to your Fly.io application and dumps it to AWS\n   S3. Quite useful for small projects (where the database is not that big).\n\n   Advantages: simplicity, maintenance free and easy to understand.\n\n2. A more complex setup, useful for bigger databases, which triggers a backup workers via github\n   actions and performs the database dump, and backup upload directly within the Fly infrastructure.\n   From my experience the latency and bandwidth from Fly to AWS is extremely good, meaning it we\n   can create medium sized backups rather quickly.\n   It uses a dedicated app for the _backup worker_ that is _woken up_ to start the dump.\n   When it finishes it is _scaled_ back to 0, meaning **it is not billable when idle**,\n   you only pay for the backup time (it is close to free, and supper affordable even with\n   high end machines).\n   It leverages Fly machines to dynamically deploy volumes and servers on demand.\n   \n   Advantages: handles bigger databases, performs backups quickly with performant Fly.io machines\n   (instead of slow github actions), data goes directly from Fly to your bucket without going\n   though GitHub (security, compliance and obviously performance).\n\n## Why this?\n\nIndeed Fly's pg images support `wal-g` config to S3 via env vars.\nBut I wanted a way to create simple archives periodically with `pg_dump`,\nmaking it easy for developers to replicate databases, and have a simple daily snapshot that can be\nrestored with `pg_restore`.\n\n\n## Setup\n\nCreate your resources, credentials and permissions following the\n[create resources utils documentation](./create-resources-utils).\n\n### Method 1: Simple github actions backup\n\nCreate a `.github/workflows/backup-database.yaml` in your project:\n\n```yaml\nname: Backup database\n\non:\n  workflow_dispatch:\n  schedule:\n    # Every day at 6:22am UTC\n    - cron: \"22 6 * * *\"\n\njobs:\n  backup-db:\n    name: Backup db\n    uses: significa/fly-pg-dump-to-s3/.github/workflows/backup-fly-db.yaml\n    with:\n      fly-db-name: your-fly-db-name\n    secrets:\n      FLY_API_TOKEN: ${{ secrets.DB_BACKUP_FLY_API_TOKEN }}\n      DATABASE_URL: ${{ secrets.DB_BACKUP_DATABASE_URL }}\n      S3_DESTINATION_URL: ${{ secrets.DB_BACKUP_S3_DESTINATION_URL }}\n      AWS_ACCESS_KEY_ID: ${{ secrets.DB_BACKUP_AWS_ACCESS_KEY_ID }}\n      AWS_SECRET_ACCESS_KEY: ${{ secrets.DB_BACKUP_AWS_SECRET_ACCESS_KEY }}\n```\n\nThat's it, trigger the backup at any time with the `workflow_dispatch` event and adapt the \n`schedule` to your preference.\n\n### Method 2: Worker installation\n\n1. Launch your database backup worker with `fly apps create`\n\n2. Set the required fly secrets (env vars). Example:\n\n   ```env\n   AWS_ACCESS_KEY_ID=XXXX\n   AWS_SECRET_ACCESS_KEY=XXXX\n   DATABASE_URL=postgresql://username:password@my-fly-db-instance.internal:5432/my_database\n   S3_DESTINATION=s3://your-s3-bucket/backup.tar.gz\n   ```\n\n3. Automate the Call the reusable GitHub Actions workflow found in\n   `.github/workflows/trigger-backup.yaml`. Example workflow definition:\n\n   ```yaml\n   name: Backup databases\n   on:\n     workflow_dispatch:\n     schedule:\n       # Runs Every day at 5:00am UTC\n       - cron: \"00 5 * * *\"\n\n   jobs:\n     backup-databases:\n       name: Backup databases\n       uses: significa/fly-pg-dump-to-s3/.github/workflows/trigger-backup.yaml@v3\n       with:\n         fly-app: my-db-backup-worker\n         volume-size: 3\n         machine-size: shared-cpu-4x\n         region: ewr\n       secrets:\n         FLY_API_TOKEN: ${{ secrets.FLY_API_TOKEN }}\n   ```\n   \nYou can also trigger a manual backup without GitHub actions with `./trigger-backup.sh`:\n\n   - `FLY_APP`: (Required) Your fly application.\n   - `FLY_API_TOKEN`: (Required) Fly token (PAT or Deploy token).\n   - `FLY_REGION`: the region of the volume and consequently the region where the worker will run.\n     Choose one close to the db and the AWS bucket region. Defaults to `cdg`.\n   - `FLY_MACHINE_SIZE`: the fly machine size, list available in\n     [Fly's pricing page](https://fly.io/docs/about/pricing/#machines). Defaults to `shared-cpu-4x`\n   - `FLY_VOLUME_SIZE`: the size of the temporary disk where the ephemeral files live during the\n     backup, set it accordingly to the size of the db. Defaults to `3`.\n   - `DOCKER_IMAGE`:\n     Option to override the default docker image `ghcr.io/significa/fly-pg-dump-to-s3:3`\n   - `ERROR_ON_DANGLING_VOLUMES`: After the backup completes, checks if there are any volumes still\n     available, and crashes if so. This might be useful to alert that there are dangling volumes\n     (that you might want to be paying for). Defaults to `true`.\n   - `DELETE_ALL_VOLUMES`: True to delete all volumes in the backup worker instead of the one used\n      in the machine. Fly has been very inconsistent with what volume does the machine start.\n      This solves the problem but prevents having multiple backup workers running in the same app.\n      Default to `true`.\n\n\n## Backup history\n\nThe best way to keep a backup history is to setup versioning in your S3, this allows you to\nleverage retention policies.\nAlternatively, if you wish to have dedicated keys per backup you can play with `S3_DESTINATION`\n(by changing the docker CMD).\n\n\n## Backup multiple databases/backups in one go\n\nUse `BACKUP_CONFIGURATION_NAMES` to define multiple configurations (env var prefix) and backup\nmultiple connection strings (to their dedicated s3 destination):\n\n```env\nBACKUP_CONFIGURATION_NAMES=ENV1,STAGING_ENVIRONMENT,test\n\nENV1_DATABASE_URL=postgresql://username:password@env1/my_database\nENV1_S3_DESTINATION=s3://sample-bucket/sample.tar.gz\n\nSTAGING_ENVIRONMENT_DATABASE_URL=postgresql://username:password@sample/staging\nSTAGING_ENVIRONMENT_S3_DESTINATION=s3://sample-db-backups/staging_backup.tar.gz\n\nTEST_DATABASE_URL=postgresql://username:password@sample/test\nTEST_S3_DESTINATION=s3://sample-db-backups/test_backup.tar.gz\n```\n\nIt will backup all the databases to the desired s3 destination. AWS and fly tokens are reused.\n\n\n## Plain backups without compression (raw SQL backups)\n\nJust tweak `PG_DUMP_ARGS` to your liking.\nTar compression will only kick in if the resulting backup is a directory.\nFor example set `PG_DUMP_ARGS=--format=plain` and\n`S3_DESTINATION=s3://sample-db-backups/my_backup.sql` for a raw sql backup.\n\n\n## Environment variables reference (backup worker)\n\n- `DATABASE_URL`: Postgres database URL.\n  For example: `postgresql://username:password@test:5432/my_database`\n- `S3_DESTINATION`: AWS S3 fill file destination Postgres database URL.\n- `BACKUP_CONFIGURATION_NAMES`: Optional: Configuration names/prefixes for `DATABASE_URL` and\n  `S3_DESTINATION`.\n- `BACKUPS_TEMP_DIR`: Optional: Where the temp files should go. Defaults to: `/tmp/db-backups`\n- `THREAD_COUNT`: Optional: The number of threads to use for backup and compression.\n  Defaults to `4`.\n- `PG_DUMP_ARGS`: Optional: Override the default `pg_dump` args:\n  `--no-owner --clean --no-privileges --jobs=4 --format=directory --compress=0`.\n  The `--jobs` parameter defaults to `$THREAD_COUNT`.\n- `COMPRESSION_THREAD_COUNT`: Optional: The number of threads to use for compression.\n  Defaults to `$THREAD_COUNT`.\n\n\n## Will this work outside fly?\n\nYes, everything that is part of the backup worker (docker image) and creation scripts will work\noutside Fly.\nThe script `./trigger-backup.sh` and the GitHub workflow is obviously targeted to fly apps.\n\n\n## Migrating to v3 (Fly machines - apps v2)\n\nFrom version 3 _fly-pg-dump-to-s3_ uses Fly machines API (also known as _Fly apps v2_).\nMigrating an existing backup worker should be quite simple:\n\n1. Remove the `FLY_API_TOKEN` environment variable: `fly secrets unset FLY_API_TOKEN -a YOUR_APP`\n2. Migrate your app to v2: `fly migrate-to-v2 -a YOUR_APP`\n3. Start making use of `trigger-backup.sh` or the action `.github/workflows/trigger-backup.yaml`\n   to trigger the backup.\n\n\n## Manual resource creation\n\nTo create the resources without the scripts in [create-resources-utils](./create-resources-utils),\none could do it with:\n\nPostgres user setup:\n\n```sql\nCREATE USER db_backup_worker WITH PASSWORD '\u003cpassword\u003e';\nGRANT CONNECT ON DATABASE \u003cdb_name\u003e TO db_backup_worker;\nGRANT pg_read_all_data TO db_backup_worker;\n```\n\n\u003e **Note**: For Postgres \u003e= 14, `pg_read_all_data` is used for simplicity\n\n\n\u003cdetails\u003e\n\u003csummary\u003eFor older Postgres versions (\u003c 14)\u003c/summary\u003e\n\n```sql\n-- Grant these permissions for each schema (ex: public):\nGRANT USAGE ON SCHEMA public TO db_backup_worker;\nGRANT SELECT ON ALL TABLES IN SCHEMA public TO db_backup_worker;\nGRANT SELECT ON ALL SEQUENCES IN SCHEMA public TO db_backup_worker;\nALTER DEFAULT PRIVILEGES FOR USER db_backup_worker IN SCHEMA public\nGRANT SELECT ON TABLES TO db_backup_worker;\nALTER DEFAULT PRIVILEGES FOR USER db_backup_worker IN SCHEMA public\nGRANT SELECT ON SEQUENCES TO db_backup_worker;\n```\n\u003c/details\u003e\n\nCreate an AWS S3 bucket and an access token with write permissions to it, attaching the following\nIAM policy:\n\n```json\n{\n  \"Version\": \"2012-10-17\",\n  \"Statement\": [\n    {\n      \"Sid\": \"WriteDatabaseBackups\",\n      \"Effect\": \"Allow\",\n      \"Action\": [\n        \"s3:PutObject\",\n        \"s3:AbortMultipartUpload\",\n        \"s3:ListMultipartUploadParts\"\n      ],\n      \"Resource\": [\"arn:aws:s3:::your-s3-bucket/backup.tar.gz\"]\n    }\n  ]\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsignifica%2Ffly-pg-dump-to-s3","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsignifica%2Ffly-pg-dump-to-s3","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsignifica%2Ffly-pg-dump-to-s3/lists"}