Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/boly38/node-mongotools
node mongoTools wrapper
https://github.com/boly38/node-mongotools
backup dropbox hacktoberfest hacktoberfest2021 javascript javascript-library mongo mongodb mongodump mongorestore nodejs
Last synced: 12 days ago
JSON representation
node mongoTools wrapper
- Host: GitHub
- URL: https://github.com/boly38/node-mongotools
- Owner: boly38
- License: mit
- Created: 2020-11-08T15:08:10.000Z (about 4 years ago)
- Default Branch: main
- Last Pushed: 2024-04-11T19:16:14.000Z (7 months ago)
- Last Synced: 2024-04-14T12:21:45.383Z (7 months ago)
- Topics: backup, dropbox, hacktoberfest, hacktoberfest2021, javascript, javascript-library, mongo, mongodb, mongodump, mongorestore, nodejs
- Language: JavaScript
- Homepage:
- Size: 288 KB
- Stars: 12
- Watchers: 1
- Forks: 9
- Open Issues: 6
-
Metadata Files:
- Readme: README.md
- Contributing: .github/CONTRIBUTING.md
- License: LICENSE
Awesome Lists containing this project
README
# node-mongotools
[](https://npmjs.org/package/node-mongotools)
This project provides 2 wrappers for :
- **mongodump**,
- **mongorestore**.
This project also include **dropbox** integration feature to
- dump and upload onto dropbox,
- restore from a dropbox hosted mongo backup.
There is an autonomous feature called **rotation** that provide a backup file rotation mechanism
- remove N oldest deprecated backups.
This readme contains some samples. Other samples are under [examples/](./examples).
You may notice that `mongodump` and `mongorestore` binaries are part of [database-tools](https://www.mongodb.com/docs/database-tools/installation/installation/) (follow standard installation for common OS). These binaries are mandatory for node-mongotools nodejs project as we are talking here of "wrapper" : a more convenient way to launch command + arguments from NodeJS code).
From a GitHub Actions context, you can install them via [action-mongo-tools](https://github.com/boly38/action-mongo-tools). There is an example in this project in [main workflow](.github/workflows/main.yml) that execute tests.
## Command line usage
You should decide to install it globally without getting source or by cloning repository.
### Install globally (beta)
```bash
npm install -g node-mongotools@latest
```
When using a global install `node-mongotools` command must be directly available (ie. you can replace `node mt` by `node-mongotools` in examples).
You could revert thins by using
```bash
npm uninstall -g node-mongotools
```
### Install by getting source
```bash
git clone https://github.com/boly38/node-mongotools.git
# install dependencies
npm install
```
### Environment setup - first time only
```bash
#~ setup environment variables
cp env/initEnv.template.sh env/initEnv.dontpush.sh
# you must update env/initEnv.dontpush.sh
```
### Set your preferences
source your options
```bash
. ./env/initEnv.dontpush.sh
```
### Basic feature
```bash
# create a mongo dump
node mt dump
# or if you rely on a global install
# node-mongotools dump
# apply the same logic for next examples
# create a encrypted mongo dump
node mt dumpz
# list backups
node mt list
# restore a mongo local dump
# please note that mt restore use following options : dropBeforeRestore: true, deleteDumpAfterRestore: true
node mt restore backup/myDatabase__2020-11-08_150102.gz
# rotate backup files
node mt rotation
# Helper : show current options values
node mt options
```
### Add in-line extra options
You could play with env default options plus in-line command extra options
```bash
# create dump of a given 'shippingprices' collection, provide a target backup name as '2023_01_shippingPrices.gz', and show mongodump command
MT_COLLECTION=shippingprices MT_FILENAME=2023_01_shippingPrices.gz MT_SHOW_COMMAND=true node mt dump
# show backup in list
node mt list
# using mongo: drop a given collection
mongo myDb --eval "db.shippingprices.drop()"
# restore collection
MSYS_NO_PATHCONV=1 MT_COLLECTION=shippingprices MT_SHOW_COMMAND=true node mt restore /backup/2023_01_shippingprices.gz
# Note that collection option will produce wildcard in nsInclude arg '--nsInclude myDb.*'
```
### Dropbox feature
You have a dropbox access token in your preferences, (cf. "Mongo tools options")
```
# create a mongo dump is the same command
node mt dump
# restore a mongo dropbox dump
node mt restore /backup/myDatabase__2020-11-08_150102.gz
# git bash for windows users having absolute path issue could use the following command
unalias node
MSYS_NO_PATHCONV=1 node mt restore /backup/myDatabase__2020-11-08_150102.gz
# rotate local and dropbox backup files
node mt rotation
```
## Library use
### Install dependency
You have to import as dependency
```
npm install node-mongotools
```
### Define the requirements, example:
```
import { MongoTools, MTOptions, MTCommand } from "node-mongotools";
var mongoTools = new MongoTools();
const mtOptions = {
db: 'myDb',
port: 17017,
path: '/opt/myapp/backups',
dropboxToken: process.env.MYAPP_DROPBOX_SECRET_TOKEN
};
```
This project is compatible with ES Module projects that rely on `import`.
For now it's not compatible with CommonJS (`require`). But contribution are welcome.
### List dumps
```
var promiseResult = mongoTools.list(mtOptions);
```
### Dump
```
var promiseResult = mongoTools.mongodump(mtOptions);
```
### Restore
```
var promiseResult = mongoTools.mongorestore(mtOptions);
```
### Rotation
```
var promiseResult = mongoTools.rotation(mtOptions);
```
## Mongo tools options
Each mongotools feature rely on Mongo tools options (aka. [MTOption](./lib/MTOptions.js)).
Options precedence is the following:
- take `options` attribute if set,
- else take related environment variable if any,
- else take default value if any,
- else if it's a mandatory option, throw an error.
TIP: you could also show current options by doing:
```
console.log(new MTOptions());
```
### shared options
These options are used by dump and restore.
Either `uri` or `host`/`port`/`db`:
| option | env | required | default value | description |
|--------|--------------|----------|---------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------|
| `uri` | `MT_MONGO_URI` | **true** | (none) | mongodump uri, example `mongodb+srv://granted-user:[email protected]/tMyDatababse`. You could omit database name to dump all databases. |
or
| option | env | required | default value | description |
|------------|------------------|----------|---------------|-------------------------------------------------------------------------------------|
| `db` | `MT_MONGO_DB` | **true** | (none) | mongo database name. For dump only, you could set it to `*` to dump all databases |
| `host` | `MT_MONGO_HOST` | false | `127.0.0.1` | mongo database hostname |
| `port` | `MT_MONGO_PORT` | false | `27017` | mongo database port |
| `username` | `MT_MONGO_USER` | false | (none) | mongo database username |
| `password` | `MT_MONGO_PWD` | false | (none) | mongo database password |
| `authDb` | `MT_MONGO_AUTH_DB` | false | `admin` | mongo auth database |
#### ssl options
Optional ssl related options
| option | env | required | default value | description |
|----------------------|-------------------------------|----------|----------------|-------------------------------------------------------------------|
| `ssl` | `MT_MONGO_SSL` | false | (none) | if "1" then add `--ssl` option |
| `sslCAFile` | `MT_MONGO_SSL_CA_FILE` | false | (none) | .pem file containing the root certificate chain |
| `sslPEMKeyFile` | `MT_MONGO_SSL_PEM_KEY_FILE` | false | (none) | .pem file containing the certificate and key |
| `sslPEMKeyPassword` | `MT_MONGO_SSL_PEM_KEY_PASSWORD` | false | (none) | password to decrypt the sslPEMKeyFile, if necessary |
| `sslCRLFile` | `MT_MONGO_SSL_CRL_FILE` | false | (none) | pem file containing the certificate revocation list |
| `sslFIPSMode` | `MT_MONGO_SSL_FIPS` | false | (none) | if "1" then use FIPS mode of the installed openssl library |
| `tlsInsecure` | `MT_MONGO_TLS_INSECURE` | false | (none) | if "1" then bypass the validation for server's certificate chain |
### mongodump options
| option | env | required | default value | description |
|--------------------------|------------------------|----------|-------------------------|------------------------------------------------------------------------------|
| `path` | `MT_PATH` | false | `backup` | dump target directory, created if it doesn't exist |
| `dumpCmd ` | `MT_MONGODUMP` | false | `mongodump` | mongodump binary |
| `fileName` | `MT_FILENAME` | false | `` | dump target filename |
| `encrypt` | | false | false | encrypt the dump using secret |
| `secret` | `MT_SECRET` | false | null | secret to use if encrypt is enabled (aes-256-ctr require 32 byte length key) |
| `encryptSuffix` | | false | `.enc` | encrypt file suffix |
| `includeCollections` | | false | (none) | **Deprecated** - please use `collection` |
| `collection` | `MT_COLLECTION` | false | (none) | Collection to include, if not specified all collections are included |
| `excludeCollections` | `MT_EXCLUDE_COLLECTIONS` | false | (none) | Collections to exclude, if not specified all collections are included |
| `numParallelCollections` | | false | 4 | Number of collections mongodump should export in parallel. |
| `viewsAsCollections` | | false | false | When specified, mongodump exports read-only views as collections. |
Simple example:
```
import { MongoTools, MTOptions, MTCommand } from "node-mongotools";
var mongoTools = new MongoTools();
mongoTools.mongodump({
db:'myDatabase',
path:'backup',
username:'root', password:'mypass', authDb:'admin'
})
.then((success) => console.info("success", success) )
.catch((err) => console.error("error", err) );
```
### mongorestore options
| option | env | required | default value | description |
|--------------------------|------------------|----------|----------------|----------------------------------------------------------------|
| `dbFrom` | `MT_MONGO_DB_FROM` | false | (none) | name of the source db (if `db` is not specified)
| `dbTo` | `MT_MONGO_DB_TO` | false | (none) | name of the target db (if `db` is not specified) |
| `dumpFile` | `MT_DUMP_FILE` | true | (none) | dump file to restore |
| `restoreCmd` | `MT_MONGORESTORE` | false | `mongorestore` | mongorestore binary |
| `dropBeforeRestore` | | false | false | set it to `true` to append `--drop` option |
| `deleteDumpAfterRestore` | | false | false | set it to `true` to remove restored backup file |
| `decrypt` | | false | false | decrypt the dump using secret. Activated if suffix is detected |
| `secret` | `MT_SECRET` | false | null | secret to use if decrypt is enabled |
Simple example:
```
import { MongoTools, MTOptions, MTCommand } from "node-mongotools";
var mongoTools = new MongoTools();
mongoTools.mongorestore({
dumpFile:'backup/myDatabase__2020-11-8_160011.gz',
username:'root', password:'mypass', authDb:'admin'
})
.then((success) => {
console.info("success", success.message);
if (success.stderr) {
console.info("stderr:\n", success.stderr);// mongorestore binary write details on stderr
}
})
.catch((err) => console.error("error", err) );
```
### Dropbox options
You could create a dropbox app to get a token : cf. https://www.dropbox.com/developers/apps/ "Generated access token"
| option | env | required | default value | description |
|-----------------------|----------------------------|-----------|-----------------|----------------------------------------------------------------------|
| `dropboxLocalPath` | `MT_DROPBOX_LOCAL_PATH` | false | "dropbox" | local directory to receive dropbox dump |
| `dropboxAppKey` | `MT_DROPBOX_APP_KEY` | false | (none) | (refresh token based) dropbox feature application key. (*1) |
| `dropboxAppSecret` | `MT_DROPBOX_APP_SECRET` | false | (none) | (refresh token based) dropbox feature application secret. (*1) |
| `dropboxRefreshToken` | `MT_DROPBOX_REFRESH_TOKEN` | false | (none) | (refresh token based) dropbox feature application refreshToken. (*1) |
| `dropboxToken` (*2) | `MT_DROPBOX_TOKEN` | false | (none) | DEPRECATED - activate dropbox feature if present. (*2) |
(*1) `dropboxAppKey`, `dropboxAppSecret` and `dropboxRefreshToken` (long-lived offline refresh token) are required to activate optional dropbox feature. You have some help about how to get them in [dropbox-refresh-token](https://github.com/boly38/dropbox-refresh-token) dedicated helper repository.
(*2) `dropboxToken` option is DEPRECATED : This was legacy old-long-lived access-token - this kind of token are no more available from dropbox developers portal ([src](https://dropbox.tech/developers/migrating-app-permissions-and-access-tokens)). Please switch to `dropboxAppKey`, `dropboxAppSecret`, and `dropboxRefreshToken` instead. For now this is still supported.
When a token is set,
- the `list` feature will list the `/` + `path` dropbox directory
- the `mongodump` feature will upload the dump onto `/` + `path` dropbox directory (in addition to spawn it locally),
- the `mongorestore` feature will use `dumpFile` as dropbox dump location
and retrieve it into `dropboxLocalPath` before doing the mongorestore action.
### Rotation options
A safe time windows is defined by :
* `now - rotationWindowsDays day(s)` ===> `now`
where backups can't be removed.
Backup out of safe time windows are called `deprecated backup`.
- `rotationMinCount`: minimum deprecated backups to keep,
- `rotationCleanCount`: number of (oldest) deprecated backups to delete.
| option | env | required | default value | description |
|-----------------------|--------------------------|----------|---------------|--------------------------------------------------------|
| `rotationDryMode` | `MT_ROTATION_DRY_MODE` | false | false | dont do delete actions, just print it |
| `rotationWindowsDays` | `MT_ROTATION_WINDOWS_DAYS` | true | 15 | safe time windows in days since now |
| `rotationMinCount` | `MT_ROTATION_MIN_COUNT` | true | 2 | minimum deprecated backups to keep. |
| `rotationCleanCount` | `MT_ROTATION_CLEAN_COUNT` | true | 10 | number of (oldest first) deprecated backups to delete. |
Simple example:
```
MT_ROTATION_CLEAN_COUNT=2 \
MT_ROTATION_DRY_MODE=true \
MT_ROTATION_WINDOWS_DAYS=3 \ node mt rotation
```
Example details: if there is a backup that is more than 3 days old, keep 2 newer ones and delete the 10 oldest.
Dropbox limits:
- rotation feature will not apply if dropbox backup target directory content contains more than 2000 files.
## How to contribute
You're not a dev ? just submit an issue (bug, improvements, questions). Or else:
* Clone
* Install deps
* Then mocha tests
```
git clone https://github.com/boly38/node-mongotools.git
npm install
npm run test
```
* you could also fork, feature branch, then submit a pull request.
### Services or activated bots
| badge | name | description |
|--------------------------------------------------------------------------------------------------------------------------------------------------------------------|---------------------------------|:----------------------------------------------------------|
|  | Github actions | Continuous tests. |
| [](https://github.com/boly38/node-mongotools/actions/workflows/audit.yml) | Github actions | Continuous vulnerability audit. |
| [](https://houndci.com) | [Houndci](https://houndci.com/) | JavaScript automated review (configured by `.hound.yml`) |
### Contributions
provided by [Repobeats](https://repobeats.axiom.co/)