Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/motleyagency/influxdb-incremental-restore
InfluxDB incremental restore script
https://github.com/motleyagency/influxdb-incremental-restore
Last synced: 21 days ago
JSON representation
InfluxDB incremental restore script
- Host: GitHub
- URL: https://github.com/motleyagency/influxdb-incremental-restore
- Owner: motleyagency
- License: mit
- Created: 2018-05-22T15:48:35.000Z (over 6 years ago)
- Default Branch: master
- Last Pushed: 2020-06-16T12:46:50.000Z (over 4 years ago)
- Last Synced: 2024-12-02T12:38:14.698Z (about 1 month ago)
- Language: JavaScript
- Size: 457 KB
- Stars: 14
- Watchers: 10
- Forks: 0
- Open Issues: 9
-
Metadata Files:
- Readme: README.md
- License: LICENSE
- Code of conduct: CODE_OF_CONDUCT.md
Awesome Lists containing this project
README
# InfluxDB incremental restore
> `influxdb-incremental-restore` InfluxDB incremental restore script with NodeJS
## Why
Currently there's [no way to incrementally restore](https://github.com/influxdata/influxdb/issues/9593#issuecomment-387693856)
incremental backups in InfluxDB (1.5.x). This script does just that for InfluxDB OSS instances.
### Prerequisites
* [NodeJS](https://nodejs.org/en/) > 8.9.0
* `influx` CLI tool (comes with InfluxDB)
* `influxd` CLI tool
* InfluxDB `-portable` backups (`meta`, `manifest` and `sX.tar.gz` (shards/data)) in a single level deep folder
* A running InfluxDB installation that does not contain a database with the given name.
* (OPTIONAL) Disable [`query timeout`](https://docs.influxdata.com/influxdb/v1.6/troubleshooting/query_management/#query-timeout)s if any.
#### Folder structure
```bash
backups/
├── .meta
├── .manifest
├── .s1.tar.gz
├── .s200.tar.gz
├── .sX.tar.gz
├── .meta
├── .manifest
├── .s1.tar.gz
├── .s66.tar.gz
├── .s199.tar.gz
└── .sX.tar.gz
```
### How it works
The scripts runs in three different steps. First it creates a `tmp` folder and organises the data into groups according to their
prefixes. Next it uses `influxd` to restore the data to the database in the format of `${dbname}_${backup-prefix}`. If the restore
was successful, we create an empty database (by the name of `-db` value or `-newdb` if provided) and select all the measurements into
the database incrementally and drop the restored, post-fixed ones.
## Installation
```bash
npm install -g @motleyagency/influxdb-incremental-restore
// or
yarn global add @motleyagency/influxdb-incremental-restore
```
### Usage
```bash
CLI for incrementally restoring incremental InfluxDB backups
Usage
$ influxdb-incremental-restore
Options
[ -host ]: Host and port for InfluxDB OSS instance. Default value is '127.0.0.1'. Required for remote connections. Example: -host 127.0.0.1
[ -port ]: Host and port for InfluxDB OSS instance. Default value is '8088'. Required for restore/backup connections. Example: -port 8088
[ -portHttp ]: Host and port for InfluxDB OSS instance. Default value is '8086'. Required for Api connections. Example: -port 8086
[ -db ]: Name of the database to be restored from the backup. Required.
[ -newdb ]: Name of the database into which the archived data will be imported on the target system. If not specified, then the value for -db is used. The new database name must be unique to the target system.
[ -rp ]: Name of the retention policy from the backup that will be restored. Requires that -db is set. If not specified, all retention policies will be used.
[ -newrp ]: Name of the retention policy to be created on the target system. Requires that -rp is set. If not specified, then the -rp value is used.
[ -shard ]: Shard ID of the shard to be restored. If specified, then -db and -rp are required.
[ -password ]: Password to connect to the server.
[ -username ]: Username to connect to the server.
[ -ssl ]: Use https for requests.
[ -unsafeSsl ]: Set this when connecting to the cluster using https and not use SSL verification.
[ -pps ] How many points per second the import will allow. By default it is zero and will not throttle importing.
[ -useTargetMeasurements] Use measurements from target database, use if you get errors like '... input field "" on measurement "" is type float, already exists as type integer... '
[ -concurrency ]: Amount of concurrent requests to the database. Default is 1.
[ --version ]: Display version and exit
[ --help ]: Display this help
Examples
$ influxdb-incremental-restore -db old-database ./backups
$ influxdb-incremental-restore -db old-database ./backups # restores old-database
$ influxdb-incremental-restore -db old-database -newdb new-database # restores old-database as new-database
$ influxdb-incremental-restore -db old-database -useTargetMeasurements
$ influxdb-incremental-restore --version
$ influxdb-incremental-restore --help
```
> You can also use npx with `npx --package @motleyagency/influxdb-incremental-restore influx-incremental-restore ` if you don't want to install it globally.
### Versions
* `0.0.7`: Added flag `useTargetMeasurements` (#12, @mgronbar)
* `0.0.6`: You can now pick which measurements and fields you want to restore (#11, @mgronbar)
* `0.0.5`: Increased logging for better UX
* `0.0.4`: Fix `portHttp` flag, few typo fixes
* `0.0.3`: Separate API and RPC port configurations
* `0.0.2`: Fix bin/-link.
* `0.0.1`: Initial release catered for our needs. Seems to work but YMMV.
### Contributions
Contributions are welcome! Send a PR or file a bug. Note that we follow a [Code Of Conduct](./CODE_OF_CONDUCT.md).
### License
MIT