{"id":15373857,"url":"https://github.com/boly38/node-mongotools","last_synced_at":"2025-04-15T11:32:54.972Z","repository":{"id":37101297,"uuid":"311088826","full_name":"boly38/node-mongotools","owner":"boly38","description":"node mongoTools wrapper","archived":false,"fork":false,"pushed_at":"2024-04-11T19:16:14.000Z","size":295,"stargazers_count":12,"open_issues_count":6,"forks_count":9,"subscribers_count":1,"default_branch":"main","last_synced_at":"2024-04-14T12:21:45.383Z","etag":null,"topics":["backup","dropbox","hacktoberfest","hacktoberfest2021","javascript","javascript-library","mongo","mongodb","mongodump","mongorestore","nodejs"],"latest_commit_sha":null,"homepage":"","language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/boly38.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":".github/CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null}},"created_at":"2020-11-08T15:08:10.000Z","updated_at":"2024-04-17T07:34:27.732Z","dependencies_parsed_at":"2023-02-08T06:15:23.234Z","dependency_job_id":"f7c6f253-e01c-4730-a771-c2a1f0160321","html_url":"https://github.com/boly38/node-mongotools","commit_stats":{"total_commits":66,"total_committers":6,"mean_commits":11.0,"dds":0.4242424242424242,"last_synced_commit":"1b6afe374bfc90b55c0a6219299879a6564cdb01"},"previous_names":[],"tags_count":29,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/boly38%2Fnode-mongotools","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/boly38%2Fnode-mongotools/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/boly38%2Fnode-mongotools/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/boly38%2Fnode-mongotools/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/boly38","download_url":"https://codeload.github.com/boly38/node-mongotools/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":223673050,"owners_count":17183618,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["backup","dropbox","hacktoberfest","hacktoberfest2021","javascript","javascript-library","mongo","mongodb","mongodump","mongorestore","nodejs"],"created_at":"2024-10-01T13:56:34.356Z","updated_at":"2024-11-08T11:02:52.552Z","avatar_url":"https://github.com/boly38.png","language":"JavaScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# node-mongotools\r\n[![NPM](https://nodei.co/npm/node-mongotools.png?compact=true)](https://npmjs.org/package/node-mongotools)\r\n\r\nThis project provides 2 wrappers for :\r\n- **mongodump**,\r\n- **mongorestore**.\r\n\r\nThis project also include **dropbox** integration feature to \r\n- dump and upload onto dropbox,\r\n- restore from a dropbox hosted mongo backup.\r\n\r\nThere is an autonomous feature called **rotation** that provide a backup file rotation mechanism\r\n- remove N oldest deprecated backups.\r\n\r\nThis readme contains some samples. Other samples are under [examples/](./examples). \r\n\r\nYou may notice that `mongodump` and `mongorestore` binaries are part of [database-tools](https://www.mongodb.com/docs/database-tools/installation/installation/) (follow standard installation for common OS). These binaries are mandatory for node-mongotools nodejs project as we are talking here of \"wrapper\" : a more convenient way to launch command + arguments from NodeJS code).\r\n\r\nFrom a GitHub Actions context, you can install them via [action-mongo-tools](https://github.com/boly38/action-mongo-tools). There is an example in this project in [main workflow](.github/workflows/main.yml) that execute tests.\r\n\r\n\r\n## Command line usage\r\nYou should decide to install it globally without getting source or by cloning repository.\r\n\r\n### Install globally (beta)\r\n```bash\r\nnpm install -g node-mongotools@latest\r\n```\r\nWhen using a global install `node-mongotools` command must be directly available (ie. you can replace `node mt` by `node-mongotools` in examples).\r\n\r\nYou could revert thins by using\r\n```bash\r\nnpm uninstall -g node-mongotools\r\n```\r\n\r\n### Install by getting source \r\n```bash\r\ngit clone https://github.com/boly38/node-mongotools.git\r\n# install dependencies\r\nnpm install\r\n```\r\n\r\n### Environment setup - first time only\r\n```bash\r\n#~ setup environment variables\r\ncp env/initEnv.template.sh env/initEnv.dontpush.sh\r\n# you must update env/initEnv.dontpush.sh\r\n```\r\n\r\n### Set your preferences\r\nsource your options\r\n```bash\r\n. ./env/initEnv.dontpush.sh\r\n```\r\n\r\n### Basic feature\r\n```bash\r\n# create a mongo dump\r\nnode mt dump\r\n# or if you rely on a global install\r\n# node-mongotools dump\r\n# apply the same logic for next examples \r\n\r\n# create a encrypted mongo dump\r\nnode mt dumpz\r\n\r\n# list backups\r\nnode mt list\r\n\r\n# restore a mongo local dump\r\n# please note that mt restore use following options : dropBeforeRestore: true, deleteDumpAfterRestore: true\r\nnode mt restore backup/myDatabase__2020-11-08_150102.gz\r\n\r\n# rotate backup files\r\nnode mt rotation\r\n\r\n# Helper : show current options values\r\nnode mt options\r\n```\r\n\r\n### Add in-line extra options\r\n\r\nYou could play with env default options plus in-line command extra options\r\n```bash\r\n# create dump of a given 'shippingprices' collection, provide a target backup name as '2023_01_shippingPrices.gz', and show mongodump command\r\nMT_COLLECTION=shippingprices MT_FILENAME=2023_01_shippingPrices.gz MT_SHOW_COMMAND=true node mt dump\r\n\r\n# show backup in list\r\nnode mt list\r\n\r\n# using mongo: drop a given collection\r\nmongo myDb --eval \"db.shippingprices.drop()\"\r\n\r\n# restore collection\r\nMSYS_NO_PATHCONV=1 MT_COLLECTION=shippingprices MT_SHOW_COMMAND=true node mt restore /backup/2023_01_shippingprices.gz\r\n# Note that collection option will produce wildcard in nsInclude arg '--nsInclude myDb.*'\r\n```\r\n\r\n\r\n### Dropbox feature\r\n\r\nYou have a dropbox access token in your preferences, (cf. \"Mongo tools options\")\r\n\r\n```\r\n# create a mongo dump is the same command\r\nnode mt dump\r\n# restore a mongo dropbox dump \r\nnode mt restore /backup/myDatabase__2020-11-08_150102.gz\r\n\r\n# git bash for windows users having absolute path issue could use the following command\r\nunalias node\r\nMSYS_NO_PATHCONV=1 node mt restore /backup/myDatabase__2020-11-08_150102.gz\r\n\r\n# rotate local and dropbox backup files\r\nnode mt rotation\r\n```\r\n\r\n## Library use\r\n\r\n### Install dependency\r\n\r\nYou have to import as dependency\r\n```\r\nnpm install node-mongotools\r\n```\r\n\r\n### Define the requirements, example:\r\n``` \r\nimport { MongoTools, MTOptions, MTCommand } from \"node-mongotools\";\r\n\r\nvar mongoTools = new MongoTools();\r\nconst mtOptions = {\r\n        db: 'myDb',\r\n        port: 17017,\r\n        path: '/opt/myapp/backups',\r\n        dropboxToken: process.env.MYAPP_DROPBOX_SECRET_TOKEN\r\n      };\r\n```\r\n\r\nThis project is compatible with ES Module projects that rely on `import`.\r\nFor now it's not compatible with CommonJS (`require`). But contribution are welcome.\r\n\r\n### List dumps\r\n```\r\nvar promiseResult = mongoTools.list(mtOptions);\r\n```\r\n\r\n### Dump\r\n```\r\nvar promiseResult = mongoTools.mongodump(mtOptions);\r\n```\r\n\r\n### Restore\r\n```\r\nvar promiseResult = mongoTools.mongorestore(mtOptions);\r\n```\r\n\r\n### Rotation\r\n```\r\nvar promiseResult = mongoTools.rotation(mtOptions);\r\n```\r\n\r\n## Mongo tools options\r\n\r\nEach mongotools feature rely on Mongo tools options (aka. [MTOption](./lib/MTOptions.js)).\r\n\r\nOptions precedence is the following:\r\n- take `options` attribute if set,\r\n- else take related environment variable if any,\r\n- else take default value if any,\r\n- else if it's a mandatory option, throw an error.\r\n\r\nTIP: you could also show current options by doing:\r\n```\r\nconsole.log(new MTOptions());\r\n```\r\n\r\n### shared options\r\nThese options are used by dump and restore.\r\n\r\nEither `uri` or `host`/`port`/`db`:\r\n\r\n| option | env          | required | default value | description                                                                                                                                                   |\r\n|--------|--------------|----------|---------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------|\r\n| `uri`  | `MT_MONGO_URI` | **true** | (none)        | mongodump uri, example `mongodb+srv://granted-user:MySecretHere@cluster0.xzryx.mongodb.net/tMyDatababse`. You could omit database name to dump all databases. |\r\n\r\nor\r\n\r\n| option     | env              | required | default value | description                                                                         |\r\n|------------|------------------|----------|---------------|-------------------------------------------------------------------------------------|\r\n| `db`       | `MT_MONGO_DB`      | **true** | (none)        | mongo database name. For dump only, you could set it to `*` to dump all databases   |\r\n| `host`     | `MT_MONGO_HOST`    | false    | `127.0.0.1`   | mongo database hostname                                                             |\r\n| `port`     | `MT_MONGO_PORT`    | false    | `27017`       | mongo database port                                                                 |\r\n| `username` | `MT_MONGO_USER`    | false    | (none)        | mongo database username                                                             |\r\n| `password` | `MT_MONGO_PWD`     | false    | (none)        | mongo database password                                                             |\r\n| `authDb`   | `MT_MONGO_AUTH_DB` | false    | `admin`       | mongo auth database                                                                 |\r\n\r\n#### ssl options\r\n\r\nOptional ssl related options\r\n\r\n| option               | env                           | required | default value  | description                                                       |\r\n|----------------------|-------------------------------|----------|----------------|-------------------------------------------------------------------|\r\n| `ssl`                | `MT_MONGO_SSL`                  |  false   | (none)         | if \"1\" then add `--ssl` option                                    |\r\n| `sslCAFile`          | `MT_MONGO_SSL_CA_FILE`          |  false   | (none)         | .pem file containing the root certificate chain                   |\r\n| `sslPEMKeyFile`      | `MT_MONGO_SSL_PEM_KEY_FILE`     |  false   | (none)         | .pem file containing the certificate and key                      |\r\n| `sslPEMKeyPassword`  | `MT_MONGO_SSL_PEM_KEY_PASSWORD` |  false   | (none)         | password to decrypt the sslPEMKeyFile, if necessary               |\r\n| `sslCRLFile`         | `MT_MONGO_SSL_CRL_FILE`         |  false   | (none)         | pem file containing the certificate revocation list               |\r\n| `sslFIPSMode`        | `MT_MONGO_SSL_FIPS`             |  false   | (none)         | if \"1\" then use FIPS mode of the installed openssl library        |\r\n| `tlsInsecure`        | `MT_MONGO_TLS_INSECURE`         |  false   | (none)         | if \"1\" then  bypass the validation for server's certificate chain |\r\n\r\n### mongodump options\r\n\r\n| option                   | env                    | required | default value           | description                                                                  |\r\n|--------------------------|------------------------|----------|-------------------------|------------------------------------------------------------------------------|\r\n| `path`                   | `MT_PATH`                | false    | `backup`                | dump target directory, created if it doesn't exist                           |\r\n| `dumpCmd `               | `MT_MONGODUMP`           | false    | `mongodump`             | mongodump binary                                                             |\r\n| `fileName`               | `MT_FILENAME`            | false    | `\u003cdbName_date_time.gz\u003e` | dump target filename                                                         |\r\n| `encrypt`                |                        | false    | false                   | encrypt the dump using secret                                                |\r\n| `secret`                 | `MT_SECRET`              | false    | null                    | secret to use if encrypt is enabled (aes-256-ctr require 32 byte length key) |\r\n| `encryptSuffix`          |                        | false    | `.enc`                  | encrypt file suffix                                                          |\r\n| `includeCollections`     |                        | false    | (none)                  | **Deprecated** - please use `collection`                                     |\r\n| `collection`             | `MT_COLLECTION`          | false    | (none)                  | Collection to include, if not specified all collections are included         |\r\n| `excludeCollections`     | `MT_EXCLUDE_COLLECTIONS` | false    | (none)                  | Collections to exclude, if not specified all collections are included        |\r\n| `numParallelCollections` |                        | false    | 4                       | Number of collections mongodump should export in parallel.                   |\r\n| `viewsAsCollections`     |                        | false    | false                   | When specified, mongodump exports read-only views as collections.            |\r\n\r\nSimple example:\r\n```\r\nimport { MongoTools, MTOptions, MTCommand } from \"node-mongotools\";\r\nvar mongoTools = new MongoTools();\r\n\r\nmongoTools.mongodump({ \r\n   db:'myDatabase',\r\n   path:'backup',\r\n   username:'root', password:'mypass', authDb:'admin'\r\n})\r\n.then((success) =\u003e console.info(\"success\", success) )\r\n.catch((err) =\u003e console.error(\"error\", err) );\r\n```\r\n\r\n### mongorestore options\r\n\r\n| option                   | env              | required | default value  | description                                                    |\r\n|--------------------------|------------------|----------|----------------|----------------------------------------------------------------|\r\n| `dbFrom`               | `MT_MONGO_DB_FROM` | false     | (none)         | name of the source db (if `db` is not specified)\r\n| `dbTo`                 | `MT_MONGO_DB_TO`   | false     | (none)         | name of the target db (if `db` is not specified)                                            |\r\n| `dumpFile`               | `MT_DUMP_FILE`     | true     | (none)         | dump file to restore                                           |\r\n| `restoreCmd`             | `MT_MONGORESTORE`  | false    | `mongorestore` | mongorestore binary                                            |\r\n| `dropBeforeRestore`      |                  | false    | false          | set it to `true` to append `--drop` option                     |\r\n| `deleteDumpAfterRestore` |                  | false    | false          | set it to `true` to remove restored backup file                |\r\n| `decrypt`                |                  | false    | false          | decrypt the dump using secret. Activated if suffix is detected |\r\n| `secret`                 | `MT_SECRET`        | false    | null           | secret to use if decrypt is enabled                            |\r\n\r\nSimple example:\r\n```\r\nimport { MongoTools, MTOptions, MTCommand } from \"node-mongotools\";\r\nvar mongoTools = new MongoTools();\r\n\r\nmongoTools.mongorestore({ \r\n   dumpFile:'backup/myDatabase__2020-11-8_160011.gz',\r\n   username:'root', password:'mypass', authDb:'admin'\r\n})\r\n.then((success) =\u003e {\r\n  console.info(\"success\", success.message);\r\n  if (success.stderr) {\r\n    console.info(\"stderr:\\n\", success.stderr);// mongorestore binary write details on stderr\r\n  }\r\n})\r\n.catch((err) =\u003e console.error(\"error\", err) );\r\n```\r\n\r\n### Dropbox options\r\nYou could create a dropbox app to get a token : cf. https://www.dropbox.com/developers/apps/  \"Generated access token\"\r\n\r\n| option                | env                        |  required | default value   | description                                                          |\r\n|-----------------------|----------------------------|-----------|-----------------|----------------------------------------------------------------------|\r\n| `dropboxLocalPath`    | `MT_DROPBOX_LOCAL_PATH`    | false     | \"dropbox\"       | local directory to receive dropbox dump                              |\r\n| `dropboxAppKey`       | `MT_DROPBOX_APP_KEY`       | false     | (none)          | (refresh token based) dropbox feature application key. (*1)          |\r\n| `dropboxAppSecret`    | `MT_DROPBOX_APP_SECRET`    | false     | (none)          | (refresh token based) dropbox feature application secret. (*1)       |\r\n| `dropboxRefreshToken` | `MT_DROPBOX_REFRESH_TOKEN` | false     | (none)          | (refresh token based) dropbox feature application refreshToken. (*1) |\r\n| `dropboxToken` (*2)   | `MT_DROPBOX_TOKEN`         | false     | (none)          | DEPRECATED - activate dropbox feature if present. (*2)               |\r\n\r\n(*1) `dropboxAppKey`, `dropboxAppSecret` and `dropboxRefreshToken` (long-lived offline refresh token) are required to activate optional dropbox feature. You have some help about how to get them in [dropbox-refresh-token](https://github.com/boly38/dropbox-refresh-token) dedicated helper repository.\r\n(*2) `dropboxToken` option is DEPRECATED : This was legacy old-long-lived access-token - this kind of token are no more available from dropbox developers portal ([src](https://dropbox.tech/developers/migrating-app-permissions-and-access-tokens)). Please switch to `dropboxAppKey`, `dropboxAppSecret`, and `dropboxRefreshToken` instead. For now this is still supported.\r\n\r\nWhen a token is set, \r\n- the `list` feature will list the `/` + `path` dropbox directory\r\n- the `mongodump` feature will upload the dump onto `/` + `path` dropbox directory (in addition to spawn it locally),\r\n- the `mongorestore` feature will use `dumpFile` as dropbox dump location \r\n  and retrieve it into `dropboxLocalPath` before doing the mongorestore action.\r\n\r\n\r\n### Rotation options\r\nA safe time windows is defined by :\r\n * `now - rotationWindowsDays day(s)` ===\u003e `now`  \r\nwhere backups can't be removed.\r\n\r\nBackup out of safe time windows are called `deprecated backup`.\r\n\r\n- `rotationMinCount`: minimum deprecated backups to keep,\r\n- `rotationCleanCount`: number of (oldest) deprecated backups to delete.\r\n\r\n| option                | env                      | required | default value | description                                            |\r\n|-----------------------|--------------------------|----------|---------------|--------------------------------------------------------|\r\n| `rotationDryMode`     | `MT_ROTATION_DRY_MODE`     | false    | false         | dont do delete actions, just print it                  |\r\n| `rotationWindowsDays` | `MT_ROTATION_WINDOWS_DAYS` | true     | 15            | safe time windows in days since now                    |\r\n| `rotationMinCount`    | `MT_ROTATION_MIN_COUNT`    | true     | 2             | minimum deprecated backups to keep.                    |\r\n| `rotationCleanCount`  | `MT_ROTATION_CLEAN_COUNT`  | true     | 10            | number of (oldest first) deprecated backups to delete. |\r\n\r\nSimple example:\r\n```\r\nMT_ROTATION_CLEAN_COUNT=2 \\\r\nMT_ROTATION_DRY_MODE=true \\\r\nMT_ROTATION_WINDOWS_DAYS=3 \\ node mt rotation\r\n```\r\nExample details: if there is a backup that is more than 3 days old, keep 2 newer ones and delete the 10 oldest.\r\n\r\nDropbox limits:\r\n- rotation feature will not apply if dropbox backup target directory content contains more than 2000 files.\r\n\r\n## How to contribute\r\nYou're not a dev ? just submit an issue (bug, improvements, questions). Or else:\r\n* Clone\r\n* Install deps\r\n* Then mocha tests\r\n```\r\ngit clone https://github.com/boly38/node-mongotools.git\r\nnpm install\r\nnpm run test\r\n```\r\n* you could also fork, feature branch, then submit a pull request.\r\n\r\n### Services or activated bots\r\n\r\n| badge                                                                                                                                                              | name                            | description                                               |\r\n|--------------------------------------------------------------------------------------------------------------------------------------------------------------------|---------------------------------|:----------------------------------------------------------|\r\n| ![CI/CD](https://github.com/boly38/node-mongotools/workflows/mongotools-ci/badge.svg)                                                                              | Github actions                  | Continuous tests.                                         |\r\n| [![Audit](https://github.com/boly38/node-mongotools/actions/workflows/audit.yml/badge.svg)](https://github.com/boly38/node-mongotools/actions/workflows/audit.yml) | Github actions                  | Continuous vulnerability audit.                           |\r\n| [![Reviewed by Hound](https://img.shields.io/badge/Reviewed_by-Hound-8E64B0.svg)](https://houndci.com)                                                             | [Houndci](https://houndci.com/) | JavaScript  automated review (configured by `.hound.yml`) |\r\n\r\n### Contributions\r\n\r\n![Repobeats](https://repobeats.axiom.co/api/embed/6406b5b306aa531758e7787bbffb71cda2c4e4c7.svg \"Repobeats analytics image\")\r\n\r\n\u003csmall\u003eprovided by [Repobeats](https://repobeats.axiom.co/)\u003c/small\u003e \r\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fboly38%2Fnode-mongotools","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fboly38%2Fnode-mongotools","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fboly38%2Fnode-mongotools/lists"}