{"id":13530772,"url":"https://github.com/stefanprodan/mgob","last_synced_at":"2025-04-01T19:30:38.116Z","repository":{"id":38726205,"uuid":"89944219","full_name":"stefanprodan/mgob","owner":"stefanprodan","description":"MongoDB dockerized backup agent. Runs schedule backups with retention, S3 \u0026 SFTP upload, notifications, instrumentation with Prometheus and more.","archived":true,"fork":false,"pushed_at":"2022-11-03T10:23:21.000Z","size":5588,"stargazers_count":769,"open_issues_count":0,"forks_count":152,"subscribers_count":18,"default_branch":"master","last_synced_at":"2024-11-15T00:49:19.486Z","etag":null,"topics":["backup","docker","kubernetes","mongodb","s3-storage","sftp"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/stefanprodan.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2017-05-01T17:08:26.000Z","updated_at":"2024-10-07T05:33:27.000Z","dependencies_parsed_at":"2022-09-15T08:41:05.476Z","dependency_job_id":null,"html_url":"https://github.com/stefanprodan/mgob","commit_stats":null,"previous_names":[],"tags_count":14,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/stefanprodan%2Fmgob","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/stefanprodan%2Fmgob/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/stefanprodan%2Fmgob/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/stefanprodan%2Fmgob/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/stefanprodan","download_url":"https://codeload.github.com/stefanprodan/mgob/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":246700040,"owners_count":20819815,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["backup","docker","kubernetes","mongodb","s3-storage","sftp"],"created_at":"2024-08-01T07:00:55.017Z","updated_at":"2025-04-01T19:30:37.557Z","avatar_url":"https://github.com/stefanprodan.png","language":"Go","funding_links":[],"categories":["Go","Tools"],"sub_categories":["Administration"],"readme":"# mgob\n\nThis project is no longer maintained.\nI lost interest in MongoDB some years ago and I can't drive an OSS project forward in my spare time without being passioned about it.\n\nBut MGOB lives on thanks to [@maxisam](https://github.com/maxisam). The new home of MGOB is https://github.com/maxisam/mgob\n\n---\n\n[![Build Status](https://travis-ci.org/stefanprodan/mgob.svg?branch=master)](https://travis-ci.org/stefanprodan/mgob)\n[![Docker Pulls](https://img.shields.io/docker/pulls/stefanprodan/mgob)](https://hub.docker.com/r/stefanprodan/mgob/)\n\nMGOB is a MongoDB backup automation tool built with Go.\n\n#### Features\n\n- schedule backups\n- local backups retention\n- upload to S3 Object Storage (Minio, AWS, Google Cloud, Azure)\n- upload to gcloud storage\n- upload to SFTP\n- upload to any [Rclone](https://rclone.org/) supported storage\n- notifications (Email, Slack)\n- instrumentation with Prometheus\n- http file server for local backups and logs\n- distributed as an Alpine Docker image\n\n#### Install\n\nMGOB is available on Docker Hub at [stefanprodan/mgob](https://hub.docker.com/r/stefanprodan/mgob/).\n\nSupported tags:\n\n- `stefanprodan/mgob:latest` latest stable [release](https://github.com/stefanprodan/mgob/releases)\n- `stefanprodan/mgob:edge` master branch latest successful [build](https://travis-ci.org/stefanprodan/mgob)\n\nCompatibility matrix:\n\n| MGOB                     | MongoDB |\n| ------------------------ | ------- |\n| `stefanprodan/mgob:0.9`  | 3.4     |\n| `stefanprodan/mgob:0.10` | 3.6     |\n| `stefanprodan/mgob:1.0`  | 4.0     |\n| `stefanprodan/mgob:1.1`  | 4.2     |\n\nDocker:\n\n```bash\ndocker run -dp 8090:8090 --name mgob \\\n    -v \"/mgob/config:/config\" \\\n    -v \"/mgob/storage:/storage\" \\\n    -v \"/mgob/tmp:/tmp\" \\\n    -v \"/mgob/data:/data\" \\\n    stefanprodan/mgob \\\n    -LogLevel=info\n```\n\nKubernetes:\n\nA step by step guide on running MGOB as a StatefulSet with PersistentVolumeClaims can be found [here](https://github.com/stefanprodan/mgob/tree/master/k8s).\n\n#### Configure\n\nDefine a backup plan (yaml format) for each database you want to backup inside the `config` dir.\nThe yaml file name is being used as the backup plan ID, no white spaces or special characters are allowed.\n\n_Backup plan_\n\n```yaml\nscheduler:\n  # run every day at 6:00 and 18:00 UTC\n  cron: \"0 6,18 */1 * *\"\n  # number of backups to keep locally\n  retention: 14\n  # backup operation timeout in minutes\n  timeout: 60\ntarget:\n  # mongod IP or host name\n  host: \"172.18.7.21\"\n  # mongodb port\n  port: 27017\n  # mongodb database name, leave blank to backup all databases\n  database: \"test\"\n  # leave blank if auth is not enabled\n  username: \"admin\"\n  password: \"secret\"\n  # add custom params to mongodump (eg. Auth or SSL support), leave blank if not needed\n  params: \"--ssl --authenticationDatabase admin\"\n# Encryption (optional)\nencryption:\n  # At the time being, only gpg asymmetric encryption is supported\n  # Public key file or at least one recipient is mandatory\n  gpg:\n    # optional path to a public key file, only the first key is used.\n    keyFile: /secret/mgob-key/key.pub\n    # optional key server, defaults to hkps://keys.openpgp.org\n    keyServer: hkps://keys.openpgp.org\n    # optional list of recipients, they will be looked up on key server\n    recipients:\n      - example@example.com\n# S3 upload (optional)\ns3:\n  url: \"https://play.minio.io:9000\"\n  bucket: \"backup\"\n  # accessKey and secretKey are optional for AWS, if your Docker image has awscli\n  accessKey: \"Q3AM3UQ867SPQQA43P2F\"\n  secretKey: \"zuf+tfteSlswRu7BJ86wekitnifILbZam1KYY3TG\"\n  # Optional, only used for AWS (when awscli is present)\n  # The customer-managed AWS Key Management  Service (KMS) key ID that should be used to\n  # server-side encrypt the backup in S3\n  #kmsKeyId:\n  # Optional, only used for AWS (when awscli is present)\n  # Valid choices are: STANDARD | REDUCED_REDUNDANCY | STANDARD_IA  |  ONE-\n  #     ZONE_IA  |  INTELLIGENT_TIERING  |  GLACIER | DEEP_ARCHIVE.\n  # Defaults to 'STANDARD'\n  #storageClass: STANDARD\n  # For Minio and AWS use S3v4 for GCP use S3v2\n  api: \"S3v4\"\n# GCloud upload (optional)\ngcloud:\n  bucket: \"backup\"\n  keyFilePath: /path/to/service-account.json\n# Azure blob storage upload (optional)\nazure:\n  containerName: \"backup\"\n  connectionString: \"DefaultEndpointsProtocol=https;AccountName=...;AccountKey=...;EndpointSuffix=core.windows.net\"\n# Rclone upload (optional)\nrclone:\n  bucket: \"my-backup-bucket\"\n  # See https://rclone.org/docs/ for details on how to configure rclone\n  configFilePath: /etc/rclone.conf\n  configSection: \"myrclonesection\"\n# SFTP upload (optional)\nsftp:\n  host: sftp.company.com\n  port: 2022\n  username: user\n  password: secret\n  # you can also specify path to a private key and a passphrase\n  private_key: /etc/ssh/ssh_host_rsa_key\n  passphrase: secretpassphrase\n  # dir must exist on the SFTP server\n  dir: backup\n# Email notifications (optional)\nsmtp:\n  server: smtp.company.com\n  port: 465\n  username: user\n  password: secret\n  from: mgob@company.com\n  to:\n    - devops@company.com\n    - alerts@company.com\n# Slack notifications (optional)\nslack:\n  url: https://hooks.slack.com/services/xxxx/xxx/xx\n  channel: devops-alerts\n  username: mgob\n  # 'true' to notify only on failures\n  warnOnly: false\n```\n\nReplicaSet example:\n\n```yaml\ntarget:\n  host: \"mongo-0.mongo.db,mongo-1.mongo.db,mongo-2.mongo.db\"\n  port: 27017\n  database: \"test\"\n```\n\nSharded cluster with authentication and SSL example:\n\n```yaml\ntarget:\n  host: \"mongos-0.db,mongos-1.db\"\n  port: 27017\n  database: \"test\"\n  username: \"admin\"\n  password: \"secret\"\n  params: \"--ssl --authenticationDatabase admin\"\n```\n\n#### Web API\n\n- `mgob-host:8090/storage` file server\n- `mgob-host:8090/status` backup jobs status\n- `mgob-host:8090/metrics` Prometheus endpoint\n- `mgob-host:8090/version` mgob version and runtime info\n- `mgob-host:8090/debug` pprof endpoint\n\nOn demand backup:\n\n- HTTP POST `mgob-host:8090/backup/:planID`\n\n```bash\ncurl -X POST http://mgob-host:8090/backup/mongo-debug\n```\n\n```json\n{\n  \"plan\": \"mongo-debug\",\n  \"file\": \"mongo-debug-1494256295.gz\",\n  \"duration\": \"3.635186255s\",\n  \"size\": \"455 kB\",\n  \"timestamp\": \"2017-05-08T15:11:35.940141701Z\"\n}\n```\n\nScheduler status:\n\n- HTTP GET `mgob-host:8090/status`\n- HTTP GET `mgob-host:8090/status/:planID`\n\n```bash\ncurl -X GET http://mgob-host:8090/status/mongo-debug\n```\n\n```json\n{\n  \"plan\": \"mongo-debug\",\n  \"next_run\": \"2017-05-13T14:32:00+03:00\",\n  \"last_run\": \"2017-05-13T11:31:00.000622589Z\",\n  \"last_run_status\": \"200\",\n  \"last_run_log\": \"Backup finished in 2.339055539s archive mongo-debug-1494675060.gz size 527 kB\"\n}\n```\n\n#### Logs\n\nView scheduler logs with `docker logs mgob`:\n\n```bash\ntime=\"2017-05-05T16:50:55+03:00\" level=info msg=\"Next run at 2017-05-05 16:51:00 +0300 EEST\" plan=mongo-dev\ntime=\"2017-05-05T16:50:55+03:00\" level=info msg=\"Next run at 2017-05-05 16:52:00 +0300 EEST\" plan=mongo-test\ntime=\"2017-05-05T16:51:00+03:00\" level=info msg=\"Backup started\" plan=mongo-dev\ntime=\"2017-05-05T16:51:02+03:00\" level=info msg=\"Backup finished in 2.359901432s archive size 448 kB\" plan=mongo-dev\ntime=\"2017-05-05T16:52:00+03:00\" level=info msg=\"Backup started\" plan=mongo-test\ntime=\"2017-05-05T16:52:02+03:00\" level=info msg=\"S3 upload finished `/storage/mongo-test/mongo-test-1493992320.gz` -\u003e `bktest/mongo-test-1493992320.gz` Total: 1.17 KB, Transferred: 1.17 KB, Speed: 2.96 KB/s \" plan=mongo-test\ntime=\"2017-05-05T16:52:02+03:00\" level=info msg=\"Backup finished in 2.855078717s archive size 1.2 kB\" plan=mongo-test\n```\n\nThe success/fail logs will be sent via SMTP and/or Slack if notifications are enabled.\n\nThe mongodump log is stored along with the backup data (gzip archive) in the `storage` dir:\n\n```bash\naleph-mbp:test aleph$ ls -lh storage/mongo-dev\ntotal 4160\n-rw-r--r--  1 aleph  staff   410K May  3 17:46 mongo-dev-1493822760.gz\n-rw-r--r--  1 aleph  staff   1.9K May  3 17:46 mongo-dev-1493822760.log\n-rw-r--r--  1 aleph  staff   410K May  3 17:47 mongo-dev-1493822820.gz\n-rw-r--r--  1 aleph  staff   1.5K May  3 17:47 mongo-dev-1493822820.log\n```\n\n#### Metrics\n\nSuccessful backups counter\n\n```bash\nmgob_scheduler_backup_total{plan=\"mongo-dev\",status=\"200\"} 8\n```\n\nSuccessful backups duration\n\n```bash\nmgob_scheduler_backup_latency{plan=\"mongo-dev\",status=\"200\",quantile=\"0.5\"} 2.149668417\nmgob_scheduler_backup_latency{plan=\"mongo-dev\",status=\"200\",quantile=\"0.9\"} 2.39848413\nmgob_scheduler_backup_latency{plan=\"mongo-dev\",status=\"200\",quantile=\"0.99\"} 2.39848413\nmgob_scheduler_backup_latency_sum{plan=\"mongo-dev\",status=\"200\"} 17.580484907\nmgob_scheduler_backup_latency_count{plan=\"mongo-dev\",status=\"200\"} 8\n```\n\nFailed jobs count and duration (status 500)\n\n```bash\nmgob_scheduler_backup_latency{plan=\"mongo-test\",status=\"500\",quantile=\"0.5\"} 2.4180213\nmgob_scheduler_backup_latency{plan=\"mongo-test\",status=\"500\",quantile=\"0.9\"} 2.438254775\nmgob_scheduler_backup_latency{plan=\"mongo-test\",status=\"500\",quantile=\"0.99\"} 2.438254775\nmgob_scheduler_backup_latency_sum{plan=\"mongo-test\",status=\"500\"} 9.679809477\nmgob_scheduler_backup_latency_count{plan=\"mongo-test\",status=\"500\"} 4\n```\n\n#### Restore\n\nIn order to restore from a local backup you have two options:\n\nBrowse `mgob-host:8090/storage` to identify the backup you want to restore.\nLogin to your MongoDB server and download the archive using `curl` and restore the backup with `mongorestore` command line.\n\n```bash\ncurl -o /tmp/mongo-test-1494056760.gz http://mgob-host:8090/storage/mongo-test/mongo-test-1494056760.gz\nmongorestore --gzip --archive=/tmp/mongo-test-1494056760.gz --drop\n```\n\nYou can also restore a backup from within mgob container.\nExec into mgob, identify the backup you want to restore and use `mongorestore` to connect to your MongoDB server.\n\n```bash\ndocker exec -it mgob sh\nls /storage/mongo-test\nmongorestore --gzip --archive=/storage/mongo-test/mongo-test-1494056760.gz --host mongohost:27017 --drop\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fstefanprodan%2Fmgob","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fstefanprodan%2Fmgob","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fstefanprodan%2Fmgob/lists"}