Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/Quiq/influxdb-tools
InfluxDB Tools
https://github.com/Quiq/influxdb-tools
backup clickhouse influxdb
Last synced: 3 months ago
JSON representation
InfluxDB Tools
- Host: GitHub
- URL: https://github.com/Quiq/influxdb-tools
- Owner: Quiq
- License: apache-2.0
- Created: 2017-09-11T20:04:33.000Z (over 7 years ago)
- Default Branch: master
- Last Pushed: 2023-10-18T13:07:35.000Z (over 1 year ago)
- Last Synced: 2023-10-18T14:27:48.558Z (over 1 year ago)
- Topics: backup, clickhouse, influxdb
- Language: Python
- Homepage:
- Size: 37.1 KB
- Stars: 46
- Watchers: 8
- Forks: 26
- Open Issues: 1
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
- awesome-clickhouse - Quiq/influxdb-tools - InfluxDB Tools provides scripts for backing up InfluxDB data, migrating it to ClickHouse, and restoring data using the InfluxDB HTTP API. (Integrations / ETL and Data Processing)
README
# InfluxDB Tools
## Scripts to migrate off influxdb to a better database :)
* schema-influx-to-clickhouse.py - Generate table schemas for Clickhouse based on influxdb measurements.
* line-protocol-to-clickhouse.py - Load influxdb line-protocol backup data into Clickhouse.
## influx-backup.py
InfluxDB backup/restore script using HTTP API and line-protocol format.
* Use InfluxDB HTTP API
* Backup raw data into text files in line-protocol format
* Restore from a backup
* Chunked read/write
* Separate file for each measurement
* Backup/restore individual measurements
* Backup/restore specific retention
* Incremental backups using "since", "until" date/time arguments
* Delayed restore
* Gzip support for backup/restore process
It is recommended to do a delayed restore using `--restore-chunk-delay`, `--restore-measurement-delay`
so your InfluxDB instance does not run out of memory or IO pretty fast.
### Usage
```
usage: influx-backup.py [-h] --url URL --user USER --dir DIR
[--measurements MEASUREMENTS]
[--from-measurement FROM_MEASUREMENT]
[--retention RETENTION] [--gzip] [--dump]
[--dump-db DUMP_DB] [--dump-since DUMP_SINCE]
[--dump-until DUMP_UNTIL] [--restore]
[--restore-db RESTORE_DB]
[--restore-chunk-delay RESTORE_CHUNK_DELAY]
[--restore-measurement-delay RESTORE_MEASUREMENT_DELAY]
InfluxDB backup script
optional arguments:
-h, --help show this help message and exit
--url URL InfluxDB URL including schema and port
--user USER InfluxDB username. Password must be set as env var
INFLUX_PW, otherwise will be asked.
--dir DIR directory name for backup or restore form
--measurements MEASUREMENTS
comma-separated list of measurements to dump/restore
--from-measurement FROM_MEASUREMENT
dump/restore from this measurement and on (ignored
when using --measurements)
--retention RETENTION
retention to dump/restore
--gzip dump/restore into/from gzipped files automatically
--dump create a backup
--dump-db DUMP_DB database to dump
--dump-since DUMP_SINCE
start date in the format YYYY-MM-DD (starting
00:00:00) or YYYY-MM-DDTHH:MM:SSZ
--dump-until DUMP_UNTIL
end date in the format YYYY-MM-DD (exclusive)
or YYYY-MM-DDTHH:MM:SSZ
--restore restore from a backup
--restore-db RESTORE_DB
database target of restore
--restore-chunk-delay RESTORE_CHUNK_DELAY
restore delay in sec or subsec between chunks of 5000
points
--restore-measurement-delay RESTORE_MEASUREMENT_DELAY
restore delay in sec or subsec between measurements
```
### Examples
Dump `stats` db:
```
./influx-backup.py --url https://influxdb.localhost:8086 --user admin --dump --dump-db stats --dir stats
```
Dump `heartbeat` measurement from `stats` db with data until 2017-09-01:
```
./influx-backup.py --url https://influxdb.localhost:8086 --user admin --dump --dump-db stats --dir stats \
--dump-until 2017-09-01 --measurements heartbeat
```
NOTE: If you get `ChunkedEncodingError` on dump, try to limit the data set using "since", "until" arguments.
Restore from `stats` dir into `stats_new` db:
```
./influx-backup.py --url https://influxdb.localhost:8086 --user admin --restore --restore-db stats_new \
--dir stats
```
Restore only `heartbeat` measurement from `stats` dir into `stats_new` db:
```
./influx-backup.py --url https://influxdb.localhost:8086 --user admin --restore --restore-db stats_new \
--dir stats --measurements heartbeat
```