{"id":30172887,"url":"https://github.com/nicobistolfi/go-postgres-s3-backup","last_synced_at":"2025-08-11T23:18:08.253Z","repository":{"id":308436634,"uuid":"1031086873","full_name":"nicobistolfi/go-postgres-s3-backup","owner":"nicobistolfi","description":"A serverless backup solution for PostgreSQL databases using AWS Lambda, with automatic daily, monthly, and yearly backup rotation to S3.","archived":false,"fork":false,"pushed_at":"2025-08-05T22:04:32.000Z","size":25,"stargazers_count":2,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-08-06T00:08:33.899Z","etag":null,"topics":["aws","backup","backup-utility","database-backup","database-backup-scripts","golang","lambda","postgres","postgresql","postgresql-database","supabase","supabase-db"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/nicobistolfi.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-08-03T01:27:29.000Z","updated_at":"2025-08-05T22:04:33.000Z","dependencies_parsed_at":"2025-08-06T00:08:38.922Z","dependency_job_id":"13a7a259-d719-4ab1-a8ef-32335930d3b6","html_url":"https://github.com/nicobistolfi/go-postgres-s3-backup","commit_stats":null,"previous_names":["nicobistolfi/go-postgres-s3-backup"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/nicobistolfi/go-postgres-s3-backup","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nicobistolfi%2Fgo-postgres-s3-backup","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nicobistolfi%2Fgo-postgres-s3-backup/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nicobistolfi%2Fgo-postgres-s3-backup/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nicobistolfi%2Fgo-postgres-s3-backup/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/nicobistolfi","download_url":"https://codeload.github.com/nicobistolfi/go-postgres-s3-backup/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nicobistolfi%2Fgo-postgres-s3-backup/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":269970135,"owners_count":24505476,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-11T02:00:10.019Z","response_time":75,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["aws","backup","backup-utility","database-backup","database-backup-scripts","golang","lambda","postgres","postgresql","postgresql-database","supabase","supabase-db"],"created_at":"2025-08-11T23:18:03.839Z","updated_at":"2025-08-11T23:18:08.245Z","avatar_url":"https://github.com/nicobistolfi.png","language":"Go","funding_links":[],"categories":["Database"],"sub_categories":["Database Tools"],"readme":"# Go PostgreSQL S3 Backup\n\n![Backup Banner](https://nico.bistol.fi/static/images/shutdown.jpeg)\n\nA serverless backup solution for PostgreSQL databases using AWS Lambda, with automatic daily, monthly, and yearly backup rotation to S3.\n\n## Features\n\n- ✅ Automated daily backups of your PostgreSQL database\n- ✅ Intelligent backup rotation (daily, monthly, yearly)\n- ✅ Daily backups retained for 7 days\n- ✅ Monthly backups automatically transitioned to Glacier storage\n- ✅ Yearly backups moved to Deep Archive for long-term retention\n- ✅ Serverless architecture with AWS Lambda\n- ✅ S3 bucket encryption and versioning enabled\n- ✅ Uses pgx/v5 for efficient PostgreSQL connectivity\n\n## Project Structure\n\n```\n/\n├── cmd/\n│   └── lambda/\n│       └── main.go           # Lambda function entry point\n├── serverless.yml            # Serverless Framework configuration\n├── Taskfile.yml              # Task runner configuration\n├── .env                      # Environment variables (not in repo)\n├── .gitignore                # Git ignore file\n├── go.mod                    # Go module file\n└── README.md                 # This file\n```\n\n## Prerequisites\n\n- Go 1.21+\n- [Task](https://taskfile.dev)\n- Docker (for building PostgreSQL layer)\n- AWS CLI configured\n- Node.js + npm (for Serverless Framework)\n\n## Quick Start\n\n1. **Clone and setup**\n```bash\ngit clone https://github.com/nicobistolfi/go-postgres-s3-backup.git\ncd go-postgres-s3-backup\nnpm install -g serverless\n```\n\n2. **Configure database**\n```bash\necho \"DATABASE_URL=postgresql://user:pass@host:5432/dbname\" \u003e .env\n```\n\n3. **Deploy**\n```bash\ntask deploy\n```\n\nThat's it! Your PostgreSQL database will be backed up daily at 2 AM UTC.\n\n## How It Works\n\n1. **Daily Execution**: The Lambda function runs daily at 2 AM UTC via EventBridge\n2. **Database Backup**: Connects to your PostgreSQL database and creates a SQL dump\n3. **Daily Backup**: Saves the backup to S3 under `daily/YYYY-MM-DD-backup.sql`\n4. **Monthly Backup**: If no backup exists for the current month, copies the daily backup to `monthly/YYYY-MM-backup.sql`\n5. **Yearly Backup**: If no backup exists for the current year, copies the daily backup to `yearly/YYYY-backup.sql`\n6. **Cleanup**: Removes daily backups older than 7 days\n7. **Lifecycle Management**: \n   - Monthly backups transition to Glacier after 30 days\n   - Yearly backups transition to Deep Archive after 90 days\n\n## Manual Operations\n\n**Trigger a backup:**\n```bash\ntask invoke\n```\n\n**View logs:**\n```bash\ntask logs\n```\n\n**Remove deployment:**\n```bash\ntask sls:remove\n```\n\n## Monitoring\n\n### View recent backups\n\n```bash\naws s3 ls s3://go-postgres-s3-backup-[stage]-backups/daily/\naws s3 ls s3://go-postgres-s3-backup-[stage]-backups/monthly/\naws s3 ls s3://go-postgres-s3-backup-[stage]-backups/yearly/\n```\n\n### Download a backup\n\n```bash\naws s3 cp s3://go-postgres-s3-backup-[stage]-backups/daily/2025-08-01-backup.sql ./\n```\n\n## Testing Backups Locally\n\n**Start local PostgreSQL:**\n```bash\ndocker run --name my-postgres -e POSTGRES_PASSWORD=postgres -d -p 5432:5432 postgres\n```\n\n**Restore backup to local instance:**\n```bash\ndocker exec -i my-postgres psql -U postgres -d postgres -W \u003c [backup-file].sql\n```\n\n**Connect and query:**\n```bash\ndocker exec -it my-postgres psql -U postgres\n```\n\n## Environment Variables\n\n| Variable | Description | Required |\n|----------|-------------|----------|\n| `DATABASE_URL` | PostgreSQL connection string | Yes |\n| `BACKUP_BUCKET` | S3 bucket name (auto-configured by Serverless) | Auto |\n\n## Security\n\n- Database credentials are stored as Lambda environment variables\n- S3 bucket has encryption enabled (AES256)\n- Public access to the S3 bucket is blocked\n- IAM role follows least privilege principle\n- Versioning is enabled on the S3 bucket\n\n## Cost Optimization\n\n- Lambda runs only once per day (minimal compute costs)\n- Daily backups are automatically deleted after 7 days\n- Monthly backups move to cheaper Glacier storage\n- Yearly backups move to Deep Archive for maximum cost savings\n\n## Troubleshooting\n\n### Lambda timeout issues\n\nIf your database is large and backups are timing out:\n1. Increase the timeout in `serverless.yml` (currently set to 300 seconds)\n2. Consider increasing the Lambda memory allocation\n\n### Connection issues\n\nEnsure your PostgreSQL database allows connections from AWS Lambda:\n1. Check your PostgreSQL connection pooling settings\n2. Verify the DATABASE_URL is correct\n3. Ensure your database is not hitting connection limits\n\n### Missing backups\n\nCheck the Lambda logs for errors:\n```bash\ntask logs\n```\n\n\n## License\n\nThis project is licensed under the MIT License - see the LICENSE file for details.","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnicobistolfi%2Fgo-postgres-s3-backup","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fnicobistolfi%2Fgo-postgres-s3-backup","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnicobistolfi%2Fgo-postgres-s3-backup/lists"}