{"id":44695816,"url":"https://github.com/one2nc/mongodb-backup-scripts","last_synced_at":"2026-02-15T08:13:22.726Z","repository":{"id":292840859,"uuid":"978544151","full_name":"one2nc/mongodb-backup-scripts","owner":"one2nc","description":null,"archived":false,"fork":false,"pushed_at":"2025-05-12T11:56:32.000Z","size":9,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-05-12T12:43:47.275Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/one2nc.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-05-06T06:37:12.000Z","updated_at":"2025-05-12T11:56:27.000Z","dependencies_parsed_at":"2025-05-12T12:43:49.585Z","dependency_job_id":"de686781-eabf-4330-b898-4371565be941","html_url":"https://github.com/one2nc/mongodb-backup-scripts","commit_stats":null,"previous_names":["one2nc/mongodb-backup-scripts"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/one2nc/mongodb-backup-scripts","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/one2nc%2Fmongodb-backup-scripts","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/one2nc%2Fmongodb-backup-scripts/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/one2nc%2Fmongodb-backup-scripts/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/one2nc%2Fmongodb-backup-scripts/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/one2nc","download_url":"https://codeload.github.com/one2nc/mongodb-backup-scripts/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/one2nc%2Fmongodb-backup-scripts/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":29473598,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-02-15T06:58:05.414Z","status":"ssl_error","status_checked_at":"2026-02-15T06:58:05.085Z","response_time":118,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-02-15T08:13:20.651Z","updated_at":"2026-02-15T08:13:22.719Z","avatar_url":"https://github.com/one2nc.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# MongoDB Backup, Restore \u0026 Validation Toolkit\n\nThis repository contains a comprehensive set of scripts to manage MongoDB **backups**, **restorations**, and **post-backup validations**. It supports both **full database dumps** and **oplog backups** for point-in-time recovery (PITR), along with a script to inspect and validate database metadata.\n\nThe backup and restore operations are containerized and can be automated using **Kubernetes CronJobs**, making them production-ready for scheduled operations in cloud-native environments. All backup data is stored in **Google Cloud Storage (GCS)**.\n\n## Table of Contents\n\n1. [MongoDB Database \u0026 Collection Metadata Script](#mongodb-database--collection-metadata-script)\n2. [MongoDB Full Load Backup](#mongodb-full-load-backup)\n3. [MongoDB Oplog Backup](#mongodb-oplog-backup)\n4. [MongoDB Restore Script](#mongodb-restore-script)\n\n\n\n## MongoDB Database \u0026 Collection Metadata Script\n### Prerequisites \n- `mongosh` Shell \n- Credentials with atleast **read acess** to all databases.\n\n### Description\nThis script iterates over all non-system databases in a MongoDB instance and generates a detailed JSON summary of their collections.  \nIt is especially useful for **backup validation** and general metadata inspection. \nIt captures following key information:\n- Number of documents in each collection\n- Size of each collection (in bytes)\n- Minimum and Maximum `_id` values\n\n### Usage \n```bash\nmongosh \u003cCONNECTION_STRING\u003e db_stats.js\n```\nThe output will be printed as formatted JSON to the terminal.\n\n## MongoDB Full Load Backup\n\nWe have automated MongoDB Full Load Backups to GCS Buckets using a python Script. This script is scheduled to run as a Kubernetes CronJob.\n\n### Prerequisites\n\n- GCS Bucket to store backups.\n- Necessary permission to access GCS Bucket.\n\n### Setup Details\n\n- Full Backup script will simply take MongoDB backup using `mongodump` (version 100.11.0) utility and copy it to GCS Bucket.\n- Backup File Format with timestamp : `/\u003cAPP_NANE\u003e/\u003cENV\u003e/mongodb_backup_\u003cyyyymmdd_HHMMSS\u003e.gz`\n- **Script Details** :\n  - _Script Path_ : `full-backup/full-backup.py`\n  - _Environment Variables_ : Following are the required environment variables\n    - **MONGO_URI** (string) : MongoDB connection String\n    - **BUCKET_NAME** (string) : GCS Bucket Name\n    - **ENV** (string) and **APP_NAME** (string) : Using these env variables backup storage path will be `/\u003cAPP_NAME\u003e/\u003cENV\u003e/\u003cFILE_NAME\u003e`\n- Dockerfile Path : `Dockerfile-Full-Backup`\n- Kubernetes Manifests : `kubernetes-manifests/full-backup-cronjob-template.yml`\n\n## MongoDB Oplog Backup\n\nWe have implemented automated MongoDB Oplog Backups using a Python script that runs as a Kubernetes CronJob. This script periodically queries the MongoDB oplog collection and uploads the extracted logs to a GCS Bucket. These oplog backups enable near Point-in-Time Recovery (PITR) when combined with full backups.\n\n### Prerequisites\n\n- A GCS Bucket to store oplog backups.\n- Necessary permissions to access the GCS Bucket.\n- MongoDB must be running as a replica set (as oplogs are only available in replica sets).\n\n### Setup Details\n\n- The script queries the `local.oplog.rs` collection to extract oplogs generated within a specific time interval.\n- This interval is controlled by the environment variable `INTERVAL_IN_MINS` (default: 60).\n- **Timestamp Calculation** :\n\n  - The script determines the end timestamp by rounding the current UTC time down to the nearest hour.\n  - The start timestamp is calculated by subtracting `INTERVAL_IN_MINS` from the end timestamp.\n  - This ensures a clean, non-overlapping window and helps align oplog backups with monitoring and recovery strategies.\n  - For Example, If the script runs at `11:17 AM` UTC and `INTERVAL_IN_MINS=60`, the backup window is from `10:00:00` to `11:00:00` UTC.\n\n- Oplogs are backed up using the `mongodump` (version 100.11.0) utility and stored in a GCS bucket in BSON format.\n- Backup file path format: `/\u003cENV\u003e/YYYY/MM/DD/HH/MM/oplog.bson`.\n- **Script Details** :\n  - _Script Path_: `oplog-backup/oplog-backup.py`\n  - _Environment Variables_ :\n    - **MONGO_URI** (string) : MongoDB Connection String\n    - **BUCKET_NAME** (string) : GCS Bucket Name\n    - **ENV** (string) : Environment name (used in the GCS path)\n    - **INTERVAL_IN_MINS** (string,optional) : Duration (in minutes) of the oplog window to back up. Default is `60`.\n- Dockerfile Path : `Dockerfile-Oplog-Backup`\n- Kubernetes Manifests : `kubernetes-manifests/oplog-backup-cronjob-template.yml`\n\n## MongoDB Restore Script\n\nThis script automates the restoration of MongoDB databases from a full dump and optional oplog replay using backups stored in Google Cloud Storage (GCS).\n\nIt supports two restoration modes:\n\n- **FULL_RESTORE** : Restore from full `mongodump` archive.\n- **OPLOG_REPLAY** : Apply oplogs after the full restore to catch up to a more recent point in time.\n\n### Prerequisites\n\n- GCS buckets storing full dump archives and oplog backups.\n- Kubernetes service account with required IAM permissions to access the GCS buckets.\n- MongoDB URI with appropriate permissions to perform restores and oplog replays.\n\n### Setup Details\n\n- The script pulls the full dump from a GCS bucket using the provided URI, and optionally downloads and applies oplogs based on timestamp-based folder structure.\n- Script automatically cleans up temporary files after each restore step.\n- **Modes of Operation** :\n  1. **FULL_RESTORE** : Restores the entire MongoDB dump from an archived file (created using `mongodump --archive --gzip`) and drops existing collections.\n  2. **OPLOG_REPLAY** : Fetches and applies relevant oplog files from the configured oplog bucket between:\n  - Full dump timestamp (extracted from `FULL_DUMP_URI`)\n  - Target `END_TIME` (defaults to current time if not provided)\n- **Script Details** :\n  - _Script Path_: `oplog_restore/oplog_restore.py`\n  - _Environment Variables_ :\n    - **MONGO_URI** (string) : (Required) MongoDB Connection String\n    - **FULL_DUMP_URI** (string) : (Required) GCS URI to full backup file (e.g., `gs://\u003cbucket\u003e/\u003cpath\u003e/mongodb_backup_20250404_010203.gz`)\n    - **ACTION** (string) : (Required) One of: `FULL_RESTORE` or `OPLOG_REPLAY`.\n    - **OPLOG_BUCKET_NAME** (string) : (Optional, Required when `ACTION==OPLOG_REPLAY`) Name of the GCS bucket where oplogs are stored.\n    - **ENV** (string) : (Optional, Required when `ACTION==OPLOG_REPLAY`) Environment folder prefix (e.g., `preprod`, `prod`)\n    - **END_TIME** (string) : (Optional) Timestamp (UTC) till which oplogs should be replayed. Format: `YYYY/MM/DD HH:MM`. Defaults to current UTC time.\n- Dockerfile Path : `Dockerfile-Restore`","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fone2nc%2Fmongodb-backup-scripts","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fone2nc%2Fmongodb-backup-scripts","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fone2nc%2Fmongodb-backup-scripts/lists"}