{"id":18521421,"url":"https://github.com/thingsboard/database-migrator","last_synced_at":"2025-04-09T09:33:09.981Z","repository":{"id":77772945,"uuid":"428980800","full_name":"thingsboard/database-migrator","owner":"thingsboard","description":null,"archived":false,"fork":false,"pushed_at":"2024-07-11T11:36:14.000Z","size":37,"stargazers_count":10,"open_issues_count":3,"forks_count":5,"subscribers_count":14,"default_branch":"master","last_synced_at":"2025-03-24T04:43:23.243Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/thingsboard.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-11-17T09:21:25.000Z","updated_at":"2025-02-03T22:55:05.000Z","dependencies_parsed_at":null,"dependency_job_id":"f8b404d3-54af-4ad6-accf-5be214233a0a","html_url":"https://github.com/thingsboard/database-migrator","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/thingsboard%2Fdatabase-migrator","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/thingsboard%2Fdatabase-migrator/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/thingsboard%2Fdatabase-migrator/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/thingsboard%2Fdatabase-migrator/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/thingsboard","download_url":"https://codeload.github.com/thingsboard/database-migrator/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248012821,"owners_count":21033247,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-06T17:25:49.245Z","updated_at":"2025-04-09T09:33:09.294Z","avatar_url":"https://github.com/thingsboard.png","language":"Java","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Prerequisites\n\n**IMPORTANT! Requires Java 8 SDK**\n\nPlease make sure you have JDK 8 installed:\n\n```\nubuntu@pc:~$ java -version\nopenjdk version \"1.8.0_292\"\nOpenJDK Runtime Environment (build 1.8.0_292-8u292-b10-0ubuntu1~20.04-b10)\nOpenJDK 64-Bit Server VM (build 25.292-b10, mixed mode)\n```\n\n# Description\nThis tool used for migrating ThingsBoard setup from PostgreSQL only into Hybrid mode.\n\nYou can use this tool in two scenarios.\n\n## Scenario #1\nThis is a scenario when rule engine requires historical data to work properly (e.g. you have rule nodes that are fetching historical data during processing new messages from devices). \nIn this case system downtime requires for the migration.\n \n## Scenario #2 \nThis is a scenario when rule engine DOES NOT require historical data to work properly. \nIn this case system downtime not required (in case you are running cluster you can do this migration with zero downtime), or you'll need to restart ThingsBoard service in case of monolith setup.\nYou'll need to upgrade configuration of ThingsBoard to use Cassandra for timeseries instead PostgreSQL - old historical data will be added to Cassandra from PostgreSQL later. \nIn this scenario historical data will not be available after the reconfiguration of ThingsBoard (for instance on dashboards), but you'll see after migration.  \n\n# Performance\n\nPerformance of this tool depends on disk type and instance type (mostly on CPU resources).\nHere are few benchmarks in general:\n1. Creating Dump of the postgres ts_kv table (if it's size is 100 GB) ~ 1-2 hours\n2. Generation SSTables from dump 100 GB ~ 3-4 hours\n3. 100 GB dump file will be converted into SSTable with size about ~ 20-30 GB\n\n**NOTE** Recommended instance type in AWS is m5.xlarge or larger. Avoid using instance with burst CPUs.\n\n# Tool build instructions:\nTo build the project execute: \n\n```\nmvn clean compile assembly:single\n```\n    \nIt will generate single jar file with all required dependencies inside `target` dir -\u003e `database-migrator-1.0-SNAPSHOT-jar-with-dependencies.jar`.\n\n# Prepare required files and run tool\n\n## Scenario #1\n\n### Cassandra setup\n\nInstall Cassandra - in this scenario you can install only single instance.\n \nUsing `cqlsh` create `thingsboard` keyspace and required tables.\n\n**NOTE** You can use *schema-ts.cql* and *schema-ts-latest.cql* files, that are located in main/resources folder of this project.\n\n### Dump data from the Postgres DB to files\n\n**Do not use compression if possible because tool can only work with uncompressed files**\n\n* Stop ThingsBoard instance\n* Dump related tables (entities) that used to validate telemetry:\n  ```bash\n  pg_dump -h localhost -U postgres -d thingsboard --exclude-table=admin_settings \\\n  --exclude-table=attribute_kv --exclude-table=audit_log --exclude-table=component_discriptor \\\n  --exclude-table=device_credentials --exclude-table=event --exclude-table=oauth2_client_registration \\\n  --exclude-table=oauth2_client_registration_info --exclude-table=oauth2_client_registration_template \\\n  --exclude-table=relation --exclude-table=rule_node_state --exclude-table=tb_schema_settings \\\n  --exclude-table=user_credentials --exclude-table=ts_kv* --exclude-table=_timescaledb_internal.* \u003e /home/user/dump/related_entities.dmp`\n  ```\n* Dump `ts_kv_dictionary`:\n  ```bash\n  pg_dump -h localhost -U postgres -d thingsboard --table=ts_kv_dictionary --table=key_dictionary \u003e /home/user/dump/ts_kv_dictionary.dmp\n  ```\n   \n* Dump `ts_kv` and all partitions:\n  ```bash\n  pg_dump -h localhost -U postgres -d thingsboard --load-via-partition-root --data-only \\\n  --table=ts_kv* --table=_timescaledb_internal.* \u003e /home/user/dump/ts_kv_all.dmp  \n  ```\n  - If only part of the data needs to be migrated, it will be necessary to create a custom table in Postgres:\n    ```\n    create table ts_kv_custom as (\n      select ts_kv.* \n      from ts_kv \n      join ts_kv_dictionary on ts_kv.key=ts_kv_dictionary.key_id\n      join device on device.id=ts_kv.entity_id\n      join device_profile on device_profile.id=device.device_profile_id\n      where ts_kv_dictionary.key in ('pulseCounter', 'temperature', 'flow')\n        and device_profile.name in ('default')\n    );\n    ```\n    Such tables can be created several times using different queries. The table name must start with `ts_kv_custom`. Like `ts_kv_custom1`, `ts_kv_custom_tenant_x`\n    \n    To create a dump, use the command:\n    ```bash\n    pg_dump -h localhost -U postgres -d thingsboard --load-via-partition-root --data-only \\\n    --table=ts_kv_custom* \u003e /home/user/dump/ts_kv_all.dmp  \n    ```\n\n\n\u003e [Optional] Move table dumps to the instance where cassandra will be hosted\n\n### Prepare directory structure for SSTables\nTool use 3 different directories for saving SSTables - `ts_kv_cf`, `ts_kv_latest_cf`, `ts_kv_partitions_cf`.\n\nCreate 3 empty directories. For example:\n\n```\n/home/user/migration/ts\n/home/user/migration/ts_latest\n/home/user/migration/ts_partition\n```\n    \n### Run tool\n\n**IMPORTANT! If you run this tool on the remote instance - don't forget to execute this command in `screen` to avoid unexpected termination**\n\n```\njava -jar ./target/database-migrator-1.0-SNAPSHOT-jar-with-dependencies.jar \\\n        -telemetryFrom /home/user/dump/ts_kv_all.dmp \\\n        -relatedEntities /home/user/dump/related_entities.dmp \\\n        -dictionary /home/user/dump/ts_kv_dictionary.dmp \\\n        -latestOut /home/user/migration/ts_latest \\\n        -tsOut /home/user/migration/ts \\\n        -partitionsOut /home/user/migration/ts_partition \\ \n        -castEnable false \\\n        -partitioning MONTHS \\\n        -linesToSkip 0 \u003e /tmp/migration.log \u0026\n```\n\n*If you want to migrate just `ts_kv` without `ts_kv_latest`, execute next:*\n\n```\njava -jar ./target/database-migrator-1.0-SNAPSHOT-jar-with-dependencies.jar \\\n        -telemetryFrom /home/user/dump/ts_kv_all.dmp \\\n        -relatedEntities /home/user/dump/related_entities.dmp \\\n        -dictionary /home/user/dump/ts_kv_dictionary.dmp \\\n        -tsOut /home/user/migration/ts \\\n        -partitionsOut /home/user/migration/ts_partition \\ \n        -castEnable false \\\n        -partitioning MONTHS \\\n        -linesToSkip 0 \u003e /tmp/migration.log \u0026\n```\n  \n*Use your paths for program arguments*\n\nTool execution time depends on DB size, CPU resources and disk throughput.\n\n### Loading SSTables into Cassandra\n\n**Note that this part works only for single node Cassandra cluster**\n\n* Stop Cassandra\n* Look at `/var/lib/cassandra/data/thingsboard` and check for names of data folders\n* Copy generated SSTable files into cassandra data dir using next command:\n\n```\nsudo find /home/user/migration/ts -name '*.*' -exec mv {} /var/lib/cassandra/data/thingsboard/ts_kv_cf-0e9aaf00ee5511e9a5fa7d6f489ffd13/ \\;\nsudo find /home/user/migration/ts_latest -name '*.*' -exec mv {} /var/lib/cassandra/data/thingsboard/ts_kv_latest_cf-161449d0ee5511e9a5fa7d6f489ffd13/ \\;\nsudo find /home/user/migration/ts_partition -name '*.*' -exec mv {} /var/lib/cassandra/data/thingsboard/ts_kv_partitions_cf-12e8fa80ee5511e9a5fa7d6f489ffd13/ \\;\n```\n  \n*Pay attention! Data folders have similar name  `ts_kv_cf-0e9aaf00ee5511e9a5fa7d6f489ffd13`, but you have to use own*\n  \n* Start Cassandra service and trigger compaction\n    * Trigger compactions: `nodetool compact thingsboard`\n    * Check compaction status: `nodetool compactionstats`\n\n* Switch ThingsBoard into Hybrid Mode\n\nModify ThingsBoard properties file `thingsboard.conf` and add next lines:\n```\nexport DATABASE_TS_TYPE=cassandra\nexport TS_KV_PARTITIONING=MONTHS\nexport CASSANDRA_URL=YOUR_CASSANDRA_URL\nexport CASSANDRA_CLUSTER_NAME=YOUR_CASSANDRA_CLUSTER_NAME\nexport CASSANDRA_USE_CREDENTIALS=true # false if credentials not required \nexport CASSANDRA_USERNAME=YOUR_CASSANDRA_USERNAME\nexport CASSANDRA_PASSWORD=YOUR_CASSANDRA_PASSWORD\n```\n\n### Final steps\n\nStart ThingsBoard instance and verify migration.\n\n## Scenario #2\n\n### Cassandra setup\n\nInstall Cassandra - you can install single cluster, cluster of N nodes. Cluster can be in docker or k8s or bare metal.\n \nUsing `cqlsh` create `thingsboard` keyspace and required tables.\n\n**NOTE** You can use *schema-ts.cql* and *schema-ts-latest.cql* files, that are located in main/resources folder of this project.\n\n### Switch ThingsBoard into Hybrid Mode\n\nModify ThingsBoard properties file `thingsboard.conf` and add next lines:\n```\nexport DATABASE_TS_TYPE=cassandra\nexport TS_KV_PARTITIONING=MONTHS\nexport CASSANDRA_URL=YOUR_CASSANDRA_URL\nexport CASSANDRA_CLUSTER_NAME=YOUR_CASSANDRA_CLUSTER_NAME\nexport CASSANDRA_USE_CREDENTIALS=true # false if credentials not required \nexport CASSANDRA_USERNAME=YOUR_CASSANDRA_USERNAME\nexport CASSANDRA_PASSWORD=YOUR_CASSANDRA_PASSWORD\n```\n\nRe-start ThingsBoard and verify that new timeseries data written into Cassandra.\n\n### Dump data from the Postgres DB to files\n\n**Do not use compression if possible because tool can only work with uncompressed files**\n\n* Dump related tables (entities) that used to validate telemetry:\n  ```bash\n  pg_dump -h localhost -U postgres -d thingsboard --exclude-table=admin_settings \\\n  --exclude-table=attribute_kv --exclude-table=audit_log --exclude-table=component_discriptor \\\n  --exclude-table=device_credentials --exclude-table=event --exclude-table=oauth2_client_registration \\\n  --exclude-table=oauth2_client_registration_info --exclude-table=oauth2_client_registration_template \\\n  --exclude-table=relation --exclude-table=rule_node_state --exclude-table=tb_schema_settings \\\n  --exclude-table=user_credentials --exclude-table=ts_kv* --exclude-table=_timescaledb_internal.* \u003e /home/user/dump/related_entities.dmp`\n  ```\n* Dump `ts_kv_dictionary`:\n  ```bash\n  pg_dump -h localhost -U postgres -d thingsboard --table=ts_kv_dictionary --table=key_dictionary \u003e /home/user/dump/ts_kv_dictionary.dmp\n  ```\n\n* Dump `ts_kv` and all partitions:\n  ```bash\n  pg_dump -h localhost -U postgres -d thingsboard --load-via-partition-root --data-only \\\n  --table=ts_kv* --table=_timescaledb_internal.* \u003e /home/user/dump/ts_kv_all.dmp  \n  ```\n  - If only part of the data needs to be migrated, it will be necessary to create a custom table in Postgres:\n    ```\n    create table ts_kv_custom as (\n      select ts_kv.* \n      from ts_kv \n      join ts_kv_dictionary on ts_kv.key=ts_kv_dictionary.key_id\n      join device on device.id=ts_kv.entity_id\n      join device_profile on device_profile.id=device.device_profile_id\n      where ts_kv_dictionary.key in ('pulseCounter', 'temperature', 'flow')\n        and device_profile.name in ('default')\n    );\n    ```\n    Such tables can be created several times using different queries. The table name must start with `ts_kv_custom`. Like `ts_kv_custom1`, `ts_kv_custom_tenant_x`\n\n    To create a dump, use the command:\n    ```bash\n    pg_dump -h localhost -U postgres -d thingsboard --load-via-partition-root --data-only \\\n    --table=ts_kv_custom* \u003e /home/user/dump/ts_kv_all.dmp  \n    ```\n\n\n\u003e [Optional] Move table dumps to the instance where cassandra will be hosted\n\n### Prepare directory structure for SSTables\nTool use 3 different directories for saving SSTables - `ts_kv_cf`, `ts_kv_latest_cf`, `ts_kv_partitions_cf`\n\nCreate 3 empty directories. \n\n**IMPORTANT** directories MUST follow pattern *.../KEYSPACE_NAME/COLUMN_FAMILY_NAME/*\n\nFor example:\n\n```\n/home/user/migration/thingsboard/ts_kv_cf/\n/home/user/migration/thingsboard/ts_kv_partitions_cf/\n/home/user/migration/thingsboard/ts_kv_latest_cf/\n```\n\n### Run tool\n\n**IMPORTANT! If you run this tool on the remote instance - don't forget to execute this command in `screen` to avoid unexpected termination**\n\n```\njava -jar ./target/database-migrator-1.0-SNAPSHOT-jar-with-dependencies.jar \\\n        -telemetryFrom /home/user/dump/ts_kv_all.dmp \\\n        -relatedEntities /home/user/dump/related_entities.dmp \\ \n        -dictionary /home/user/dump/ts_kv_dictionary.dmp \\\n        -latestOut /home/user/migration/thingsboard/ts_kv_latest_cf \\\n        -tsOut /home/user/migration/thingsboard/ts_kv_cf \\\n        -partitionsOut /home/user/migration/thingsboard/ts_kv_partitions_cf \\\n        -castEnable false \\\n        -partitioning MONTHS \\\n        -linesToSkip 0 \u003e /tmp/migration.log \u0026\n```\n\n*If you want to migrate just `ts_kv` without `ts_kv_latest`, execute next:*\n\n```\njava -jar ./target/database-migrator-1.0-SNAPSHOT-jar-with-dependencies.jar \\\n        -telemetryFrom /home/user/dump/ts_kv_all.dmp \\\n        -relatedEntities /home/user/dump/related_entities.dmp \\ \n        -dictionary /home/user/dump/ts_kv_dictionary.dmp \\\n        -tsOut /home/user/migration/thingsboard/ts_kv_cf \\\n        -partitionsOut /home/user/migration/thingsboard/ts_kv_partitions_cf \\\n        -castEnable false \\\n        -partitioning MONTHS \\\n        -linesToSkip 0 \u003e /tmp/migration.log \u0026\n```\n  \n*Use your paths for program arguments*\n\nTool execution time depends on DB size, CPU resources and disk throughput.\n\n### Loading SSTables into Cassandra\n\nUsing [sstableloader](https://docs.datastax.com/en/cassandra-oss/3.x/cassandra/tools/toolsBulkloader.html) start loading data into Cassandra:\n\n```\nsstableloader --verbose --nodes CASSANDRA_NODES --username cassandra --password CASSANDRA_PASSWORD /home/user/migration/thingsboard/ts_kv_partitions_cf/\n\nsstableloader --verbose --nodes CASSANDRA_NODES --username cassandra --password CASSANDRA_PASSWORD /home/user/migration/thingsboard/ts_kv_cf/\n```\n\n### Final steps\n\nVerify that historical data available in ThingsBoard.\n\n# Troubleshooting\n\n## Continue migration in case of failure on particular migration line\n\n**IMPORTANT: works only in case of Scenario #2**  \n\nTool is able to continue creation of SSTables from the particular line.\nLet's image that tool has stopped at particular line - XXXXXXX:\n```\n2021-11-30 13:55:22,648 [main] INFO  o.t.c.t.m.writer.AbstractTbWriter - Lines processed 408000000, castOk 0, castErr 0, skippedLines 68956935\n2021-11-30 13:55:25,625 [main] INFO  o.t.c.t.m.writer.AbstractTbWriter - Lines migrated 738000000, castOk 0, castErr 0, skippedLines 68966040\n2021-11-30 13:55:32,037 [main] INFO  o.t.c.t.m.writer.AbstractTbWriter - Lines processed 409000000, castOk 0, castErr 0, skippedLines 68975104\n2021-11-30 13:55:34,974 [main] INFO  o.t.c.t.m.writer.AbstractTbWriter - Lines migrated 739000000, castOk 0, castErr 0, skippedLines 68984191\n2021-11-30 13:55:37,762 [main] INFO  o.t.c.t.m.writer.AbstractTbWriter - Lines processed 410000000, castOk 0, castErr 0, skippedLines 6899318\n```\n\nYou'll need to find the latest entry of `Lines migrated XXXXXX`. You can start migration tool from XXXXXX line.\n\nTo use this feature please do next steps:\n* Migrate already created SSTable to the Cassandra by following steps in the guide below for Scenario #2 \n    *You can copy already created SSTables into some other folder, and later migrate with sstableloader folder by folder*\n* Create new 3 different directories for saving SSTables, or reuse the ones used in the previous step \n```\n/home/user/migration/thingsboard/ts_kv_cf/\n/home/user/migration/thingsboard/ts_kv_partitions_cf/\n/home/user/migration/thingsboard/ts_kv_latest_cf/\n```\n* Start tool according to instructions above, but with an additional parameter **linesToSkip**:\n\n```\njava -jar ./target/database-migrator-1.0-SNAPSHOT-jar-with-dependencies.jar \\\n        -telemetryFrom /home/user/dump/ts_kv_all.dmp \\\n        -relatedEntities /home/user/dump/related_entities.dmp \\ \n        -dictionary /home/user/dump/ts_kv_dictionary.dmp \\\n        -latestOut /home/user/migration/thingsboard/ts_kv_latest_cf \\\n        -tsOut /home/user/migration/thingsboard/ts_kv_cf \\\n        -partitionsOut /home/user/migration/thingsboard/ts_kv_partitions_cf \\\n        -castEnable false \\\n        -partitioning MONTHS \\\n        -linesToSkip XXXXXX \u003e /tmp/migration.log \u0026\n```\n\nwhere XXXXXXX is the number of the line, that script should continue.\n\n* Continue migration according to the steps above in Scenario #2","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fthingsboard%2Fdatabase-migrator","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fthingsboard%2Fdatabase-migrator","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fthingsboard%2Fdatabase-migrator/lists"}