{"id":26032612,"url":"https://github.com/apache/iotdb-backup-tool","last_synced_at":"2025-04-10T00:20:50.833Z","repository":{"id":65979481,"uuid":"598585820","full_name":"apache/iotdb-backup-tool","owner":"apache","description":"Apache IoTDB Backup Tool","archived":false,"fork":false,"pushed_at":"2023-02-17T01:35:04.000Z","size":356,"stargazers_count":5,"open_issues_count":0,"forks_count":2,"subscribers_count":21,"default_branch":"master","last_synced_at":"2025-04-03T04:16:59.545Z","etag":null,"topics":["backup","iotdb","java","timeseries"],"latest_commit_sha":null,"homepage":"https://iotdb.apache.org/","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/apache.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null}},"created_at":"2023-02-07T12:17:52.000Z","updated_at":"2023-12-28T06:21:37.000Z","dependencies_parsed_at":"2023-03-04T03:45:13.171Z","dependency_job_id":null,"html_url":"https://github.com/apache/iotdb-backup-tool","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/apache%2Fiotdb-backup-tool","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/apache%2Fiotdb-backup-tool/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/apache%2Fiotdb-backup-tool/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/apache%2Fiotdb-backup-tool/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/apache","download_url":"https://codeload.github.com/apache/iotdb-backup-tool/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247899548,"owners_count":21014918,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["backup","iotdb","java","timeseries"],"created_at":"2025-03-06T22:18:53.887Z","updated_at":"2025-04-10T00:20:50.814Z","avatar_url":"https://github.com/apache.png","language":"Java","readme":"\u003c!--\n\n    Licensed to the Apache Software Foundation (ASF) under one\n    or more contributor license agreements.  See the NOTICE file\n    distributed with this work for additional information\n    regarding copyright ownership.  The ASF licenses this file\n    to you under the Apache License, Version 2.0 (the\n    \"License\"); you may not use this file except in compliance\n    with the License.  You may obtain a copy of the License at\n    \n        http://www.apache.org/licenses/LICENSE-2.0\n    \n    Unless required by applicable law or agreed to in writing,\n    software distributed under the License is distributed on an\n    \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n    KIND, either express or implied.  See the License for the\n    specific language governing permissions and limitations\n    under the License.\n\n--\u003e\n\n# Backup-Tool\nA tool for import and export data from iotdb,including timeseries structure and those data.\n\n## Project structure\nThis project is divided into two sub-projects:backup-core and backup-command projects;  \nThe backup-core project contains the main business logic and implementation methods, this project can provides external services in the form of jar packages.    \nThe backup-command project is a tool class package,The function implementation depends on the core project and provides a command line tool.  \n\n## how to compile\nPrecondition:\n1. Java \u003e= 1.8, the JAVA_HOME environment variable needs to be configured  \n2. Maven \u003e= 3.6  \nyou can use the command below to compile this project or install it:  \n````\nmvn clean package;  \nmvn install;  \nmvn test;    \n````\nps: you can add '-DskipTests' to skip those tests.    \nSince the core-jar has not been placed on the remote repository, to compile the tools smoothly, you first need to execute mvn install -DskipTests.    \nThe integration tests need a real iotdbserver，you can config the server in \"test/resource/sesseinConfig.properties\"\n\n## Features    \n- based on iotdb-session  \n- Pipeline-based import and export tool, easy to expand  \n    1. ````\n       Pipeline design document: https://apache-iotdb.feishu.cn/wiki/wikcn8zaxOegcQ4vBASwtmbmE4f\n       ````\n- Based on project reactor framework, multi-track execution, support back pressure mechanism  \n- Data export takes the device as the smallest unit  \n  1. The data export takes the device as the smallest unit, that is, one device corresponds to one file, and this file contains all the measurement data of the device. Because the smallest export unit is the device, the corresponding path should be the smallest device-level path.  \n- Data can be exported from one or more storage groups  \n- Guarantee the consistency of time series structure  \n  1. It can ensure that the structure of the time series will not change during the import and export process. When exporting data, you can choose whether to export the structure information of the time series. This information will be saved in a file separately; it records the properties of the time series in detail. The type of each field, the compression format, whether it is an aligned time series, etc., create a corresponding time series according to this file when importing, to ensure that the time series structure will not change.  \n- Support incremental/full export  \n  1. The \"where\" parameter supports the selected time range, and currently does not support non-time parameters  \n- Supports the export of more than 1000 time series  \n  1. Since this tool relies on iotdb-session, and session queries are limited, the time series results of a one-time query are not allowed to exceed 1000. This tool solves this problem and is the time series in the query results. More than 1000 can also be exported normally.  \n- The core function provides jar package, which is convenient for third-party integration  \n- Rich export formats:  \n  1. The export file that can be viewed, provides two file formats : sql and csv, which can be viewed directly  \n  2. Unviewable export files, export files in compressed format, take up less disk space, suitable for some backup work; now supported compression formats TSFILE、SNAPPY, GZIP, etc.  \n\n## Export\nExport command tool: data-export.bat/data-export.sh\n````\n -h  // iotdb host address\n -p  // port\n -u  // username\n -pw // password\n -f  // fileFolder\n -i  // iotdbPath\n -sy // file generation strategy, exclude export tsfile,others could use this param\n -se // Whether a time series structure is required,if true the tool will export a new file which will record the timeseries structure\n -c  // compress type : SQL、CSV、SNAPPY、GZIP、LZ4\n -w  // where \n -vn // virtualNum export tsfile could use this param\n -pi // partitionInterval export tsfile could use this param\n````\n\n## Import\nImport command tool: data-import.bat/data-import.sh\n````\n -h  // iotdb host address\n -p  // port\n -u  // username\n -pw // password\n -f  // fileFolder\n -se // Whether a time series structure is required,if true the tool will export a new file which will record the timeseries structure\n -c  // compress type : SQL、CSV、SNAPPY、GZIP、LZ4\n````\n\n## example\nScenario description:\n- Storage group:\n  1. root.ln.company1\n  2. root.ln.company2\n- Timeseries:  \n  1.root.ln.company1.diggingMachine.d1.position  \n  2.root.ln.company1.diggingMachine.d1.weight  \n  3.root.ln.company1.diggingMachine.d2.position  \n  4.root.ln.company1.diggingMachine.d2.weight  \n  5.root.ln.company2.taxiFleet.team1.mileage  \n  6.root.ln.company2.taxiFleet.team1.oilConsumption  \n  \n \u003e There are two excavators d1 and d2 under the storage group root.ln.company1\n \u003e There is a taxi fleet team1 under the storage group root.ln.company2\n ---\n \u003e Export the d1 device, export the file to d:/company1/machine, select the second file strategy (extra file), need to generate a time series structure, the file format is gzip\n ````\ndata-export.bat -h 127.0.0.1 -p 6667 -u root -pw root -f d:\\\\company1\\\\machine -i root.ln.company1.diggingMachine.d1 -sy true -se true -c gzip\n ````\n\u003e Export results:\n\u003e TIMESERIES_STRUCTURE.STRUCTURE records the time series structure\n\u003e CATALOG_COMPRESS.CATALOG records the correspondence between file names and device paths\n\u003e 1.gz.bin\n\n--- \n\u003e Export all devices under company1, export the file to d:/company1/machine, select the first file strategy, do not need to generate a time series structure, the file format is csv\n````\ndata-export.bat -h 127.0.0.1 -p 6667 -u root -pw root -f d:\\\\company1\\\\machine -i root.ln.company1.** -sy false -se false -c csv\n````\n\u003e Export results:\n\u003e d2.csv\n\u003e d1.csv\n\n--- \n\u003e Export all devices under root.ln, export files to d:/all/devices, select the first file strategy, need to generate time series structure, file format csv\n````\ndata-export.bat -h 127.0.0.1 -p 6667 -u root -pw root -f d:\\\\all\\\\devices -i root.ln.** -sy false -se true -c csv -w \"time \u003e 1651036025230\"\n````\n\u003e Export results:\n\u003e TIMESERIES_STRUCTURE.STRUCTURE records the time series structure\n\u003e d2.csv\n\u003e d1.csv\n\u003e team1.csv\n\n--- \n\u003e Import data, specify the folder d:/all/devices to export data, need to import time series, file format csv\n````\ndata-import.bat -h 127.0.0.1 -p 6667 -u root -pw root -f d:\\\\all\\\\devices -se true -c csv\n````\n--- \nps: export tsfile\n````\ndata-export.bat -h 127.0.0.1 -p 6667 -u root -pw root -f d:\\\\tsfileexport\\\\A106 -i  root.yyy.** -se false -c tsfile -vn 3 -pi 604800\n````\nif you want to import tsfile,please use [Load External TsFile Tool](https://iotdb.apache.org/zh/UserGuide/V0.13.x/Write-And-Delete-Data/Load-External-Tsfile.html).","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fapache%2Fiotdb-backup-tool","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fapache%2Fiotdb-backup-tool","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fapache%2Fiotdb-backup-tool/lists"}