{"id":26364960,"url":"https://github.com/kaalpanikh/automated-backups","last_synced_at":"2025-08-10T19:27:36.344Z","repository":{"id":277166496,"uuid":"931498997","full_name":"kaalpanikh/automated-backups","owner":"kaalpanikh","description":"Automated MongoDB backup system with AWS S3 storage, compression, and retention management via cron jobs","archived":false,"fork":false,"pushed_at":"2025-02-12T14:34:55.000Z","size":22,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-03-16T19:30:27.119Z","etag":null,"topics":["automation","aws-cli","aws-s3","backup","backup-restore","bash","cron","database-backup","devops","infrastructure","linux","mongodb","mongodump","mongorestore","shell-script"],"latest_commit_sha":null,"homepage":"","language":"Shell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/kaalpanikh.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2025-02-12T11:34:38.000Z","updated_at":"2025-02-12T14:38:36.000Z","dependencies_parsed_at":"2025-02-12T14:29:55.755Z","dependency_job_id":"2250c99d-00d0-4a32-94df-927019235f6f","html_url":"https://github.com/kaalpanikh/automated-backups","commit_stats":null,"previous_names":["kaalpanikh/automated-backups"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/kaalpanikh/automated-backups","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kaalpanikh%2Fautomated-backups","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kaalpanikh%2Fautomated-backups/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kaalpanikh%2Fautomated-backups/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kaalpanikh%2Fautomated-backups/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/kaalpanikh","download_url":"https://codeload.github.com/kaalpanikh/automated-backups/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kaalpanikh%2Fautomated-backups/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":269777456,"owners_count":24474379,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-10T02:00:08.965Z","response_time":71,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["automation","aws-cli","aws-s3","backup","backup-restore","bash","cron","database-backup","devops","infrastructure","linux","mongodb","mongodump","mongorestore","shell-script"],"created_at":"2025-03-16T19:28:49.687Z","updated_at":"2025-08-10T19:27:36.270Z","avatar_url":"https://github.com/kaalpanikh.png","language":"Shell","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Automated MongoDB Backups\n\nAutomated backup system for MongoDB databases with AWS S3 storage, implementing the [roadmap.sh Automated DB Backups project](https://roadmap.sh/projects/automated-backups).\n\n## Project Overview\n\nThis project implements an automated backup system for MongoDB databases that:\n- Creates compressed backups every 12 hours using `mongodump`\n- Stores backups in AWS S3 (instead of Cloudflare R2)\n- Maintains a 7-day backup retention policy\n- Provides easy restore functionality\n- Includes comprehensive logging and error handling\n\n### Key Features\n\n- **Automated Backups**: Runs every 12 hours via cron job\n- **Compression**: Uses gzip to minimize storage space\n- **Cloud Storage**: AWS S3 integration with configurable bucket and prefix\n- **Retention Policy**: Automatically removes backups older than 7 days\n- **Restore Capability**: Simple script to restore from the latest backup\n- **Error Handling**: Comprehensive checks and error reporting\n- **Logging**: Detailed logs for monitoring and troubleshooting\n\n### Implementation Differences\n\nWhile the roadmap.sh project suggests using Cloudflare R2, we chose AWS S3 because:\n1. Better integration with existing AWS infrastructure\n2. More mature tooling and documentation\n3. Wider industry adoption\n4. Compatible with other S3-like services (including R2) if needed\n\n## Project Structure\n\n```\nautomated-backups/\n├── scripts/\n│   ├── backup.sh        # Creates and uploads backups\n│   └── restore.sh       # Downloads and restores backups\n├── .env.example         # Template for environment variables\n└── README.md           # Documentation\n```\n\n## Implementation Details\n\n### 1. Backup Script\n- Uses `mongodump` with gzip compression\n- Uploads to S3 using AWS CLI\n- Implements backup rotation\n- Includes error handling and logging\n\n### 2. Restore Script\n- Downloads latest backup from S3\n- Restores using `mongorestore`\n- Validates backup integrity\n- Handles errors gracefully\n\n### 3. Cron Job\n- Runs every 12 hours\n- Logs output to `/var/log/mongodb-backup.log`\n- Uses absolute paths for reliability\n\n## Proof of Implementation\n\n### 1. Successful Backup\n```bash\n$ ssh ubuntu@44.203.38.191 'cd /opt/mongodb-backup \u0026\u0026 ./backup.sh'\nStarting MongoDB backup at Wed Feb 12 14:17:37 UTC 2025\nBackup file: mongodb-backup_20250212_141737.gz\nCreating backup...\ndone dumping todos.todos (1 document)\nUploading to S3...\nupload: ./mongodb-backup_20250212_141737.gz to s3://roadmapsh-bucket/mongodb-backup/mongodb-backup_20250212_141737.gz\nCleaning up local backup...\nCleaning up old backups...\nBackup completed successfully at Wed Feb 12 14:17:39 UTC 2025\n```\n\n### 2. List Backups in S3\n```bash\n$ ssh ubuntu@44.203.38.191 'aws s3 ls s3://roadmapsh-bucket/mongodb-backup/'\n2025-02-12 14:15:19        469 mongodb-backup_20250212_141517.gz\n2025-02-12 14:17:39        469 mongodb-backup_20250212_141737.gz\n```\n\n### 3. Successful Restore\n```bash\n$ ssh ubuntu@44.203.38.191 'cd /opt/mongodb-backup \u0026\u0026 ./restore.sh'\nFinding latest backup...\nLatest backup: mongodb-backup_20250212_141737.gz\nDownloading backup from S3...\nBackup downloaded successfully\nRestoring database...\ndone restoring todos.todos (1 document)\nDatabase restored successfully\nRestore process completed successfully\n```\n\n### 4. Verify Database Content\n```bash\n$ ssh ubuntu@44.203.38.191 'docker exec multi-container-service-mongodb-1 mongosh --eval \"db.getSiblingDB('\\''todos'\\'').todos.find()\"'\n[\n  {\n    _id: ObjectId('67aade6bacff6a08472ed100'),\n    title: 'Test Todo',\n    description: 'Testing our deployment',\n    completed: false,\n    createdAt: ISODate('2025-02-11T05:21:47.910Z'),\n    updatedAt: ISODate('2025-02-11T05:21:47.910Z'),\n    __v: 0\n  }\n]\n```\n\n## Setup Instructions\n\n### Prerequisites\n- Linux server with cron support\n- MongoDB database\n- AWS account with S3 access\n- Python 3 and pip (for AWS CLI)\n\n### Configuration\n1. Clone this repository to `/opt/mongodb-backup`\n2. Copy `.env.example` to `.env` and configure:\n   ```env\n   MONGODB_URI=mongodb://172.18.0.2:27017/todos\n   AWS_ACCESS_KEY_ID=your_aws_access_key\n   AWS_SECRET_ACCESS_KEY=your_aws_secret_key\n   AWS_DEFAULT_REGION=us-east-1\n   S3_BUCKET_NAME=your_bucket_name\n   BACKUP_RETENTION_DAYS=7\n   BACKUP_PREFIX=mongodb-backup\n   ```\n3. Install dependencies:\n   ```bash\n   sudo apt-get update\n   sudo apt-get install -y mongodb-database-tools python3-pip\n   sudo pip3 install --break-system-packages awscli\n   ```\n4. Set up cron job:\n   ```bash\n   (crontab -l 2\u003e/dev/null; echo \"0 */12 * * * cd /opt/mongodb-backup \u0026\u0026 ./backup.sh \u003e\u003e /var/log/mongodb-backup.log 2\u003e\u00261\") | crontab -\n   ```\n\n## Usage\n\n### Manual Backup\n```bash\ncd /opt/mongodb-backup \u0026\u0026 ./backup.sh\n```\n\n### Manual Restore\n```bash\ncd /opt/mongodb-backup \u0026\u0026 ./restore.sh\n```\n\n### Monitor Backups\n```bash\n# View backup logs\ntail -f /var/log/mongodb-backup.log\n\n# List backups in S3\naws s3 ls s3://your-bucket-name/mongodb-backup/\n```\n\n## Future Improvements\n\n1. **Multi-Database Support**: Add ability to backup multiple databases\n2. **Compression Options**: Allow configurable compression levels\n3. **Notification System**: Add email/Slack notifications for backup status\n4. **Backup Verification**: Add integrity checks for backups\n5. **Web Interface**: Create a simple dashboard for backup management\n\n## Contributing\n\nFeel free to open issues or submit pull requests. All contributions are welcome!\n\n## License\n\nMIT\n\n## Project Status\n\n✅ Active and Maintained","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkaalpanikh%2Fautomated-backups","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fkaalpanikh%2Fautomated-backups","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkaalpanikh%2Fautomated-backups/lists"}