{"id":22520518,"url":"https://github.com/apecloud/ape-dts","last_synced_at":"2025-05-16T01:04:52.230Z","repository":{"id":264587073,"uuid":"587995044","full_name":"apecloud/ape-dts","owner":"apecloud","description":"ApeCloud's Data Transfer Suite, written in Rust. Provides ultra-fast data replication between MySQL, PostgreSQL, Redis, MongoDB, Kafka and ClickHouse, ideal for disaster recovery (DR) and migration scenarios.","archived":false,"fork":false,"pushed_at":"2025-04-23T08:40:55.000Z","size":68442,"stargazers_count":428,"open_issues_count":52,"forks_count":58,"subscribers_count":8,"default_branch":"main","last_synced_at":"2025-04-23T09:36:12.657Z","etag":null,"topics":["binlog","cdc","clickhouse","data-migration","data-transfer","doris","etl","kafka-connect","mongo","mysql","postgres","redis","starrocks"],"latest_commit_sha":null,"homepage":"","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"agpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/apecloud.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2023-01-12T04:16:22.000Z","updated_at":"2025-04-23T08:40:59.000Z","dependencies_parsed_at":"2025-04-23T09:38:36.679Z","dependency_job_id":null,"html_url":"https://github.com/apecloud/ape-dts","commit_stats":{"total_commits":383,"total_committers":6,"mean_commits":"63.833333333333336","dds":"0.21409921671018273","last_synced_commit":"798fe6e660ea249c49d23f5b9aa0902bdf96b8f9"},"previous_names":["apecloud/ape-dts"],"tags_count":11,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/apecloud%2Fape-dts","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/apecloud%2Fape-dts/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/apecloud%2Fape-dts/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/apecloud%2Fape-dts/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/apecloud","download_url":"https://codeload.github.com/apecloud/ape-dts/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254448579,"owners_count":22072764,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["binlog","cdc","clickhouse","data-migration","data-transfer","doris","etl","kafka-connect","mongo","mysql","postgres","redis","starrocks"],"created_at":"2024-12-07T05:07:39.637Z","updated_at":"2025-05-16T01:04:52.203Z","avatar_url":"https://github.com/apecloud.png","language":"Rust","readme":"\n# English | [中文](README_ZH.md)\n\n# Introduction\n\n- ape-dts is a data migration tool enabling any-to-any data transfers. \n- It also provides data subscription and data processing.\n- It is lightweight, efficient and standalone, requiring no third-party components or extra storage.\n- In Rust.\n\n## Key features\n- Supports data migration between various databases, both homogeneous and heterogeneous.\n- Supports snapshot and cdc tasks with resume from breakpoint.\n- Supports checking and revising data.\n- Supports filtering and routing at the database, table, and column levels.\n- Implements different parallel algorithms for different sources, targets, and task types to improve performance.\n- Allows loading user-defined Lua scripts to modify the data.\n- Supports starting ape-dts as an HTTP server to pull data from the source, allowing users to use any language's HTTP client to retrieve and consume the data.\n\n## Supported task types\n|  | mysql -\u003e mysql | pg -\u003e pg | mongo -\u003e mongo | redis -\u003e redis | mysql -\u003e kafka | pg -\u003e kafka| mysql -\u003e starrocks | mysql -\u003e clickhouse | mysql -\u003e tidb | pg -\u003e starrocks | pg -\u003e clickhouse | mysql -\u003e doris | pg -\u003e doris |\n| :-------- | :-------- | :-------- | :-------- | :-------- | :-------- | :-------- | :-------- | :-------- | :-------- | :-------- | :-------- | :-------- | :-------- |\n| Snapshot | \u0026#10004; | \u0026#10004; | \u0026#10004; | \u0026#10004; | \u0026#10004; | \u0026#10004; | \u0026#10004; | \u0026#10004; | \u0026#10004; | \u0026#10004; | \u0026#10004; | \u0026#10004; | \u0026#10004; |\n| CDC | \u0026#10004; | \u0026#10004; | \u0026#10004; | \u0026#10004; | \u0026#10004; | \u0026#10004; | \u0026#10004; | \u0026#10004; | \u0026#10004; | \u0026#10004; | \u0026#10004; | \u0026#10004; | \u0026#10004; |\n| Data check/revise/review | \u0026#10004; | \u0026#10004; | \u0026#10004; | | | | | | \u0026#10004; | | | | |\n| Structure migration | \u0026#10004; | \u0026#10004; | | | | | \u0026#10004; | \u0026#10004; | \u0026#10004; | \u0026#10004; | \u0026#10004; | \u0026#10004; | \u0026#10004; |\n\n\n# Quick starts\n\n## Tutorial\n- [prerequisites](./docs/en/tutorial/prerequisites.md)\n- [mysql -\u003e mysql](./docs/en/tutorial/mysql_to_mysql.md)\n- [pg -\u003e pg](./docs/en/tutorial/pg_to_pg.md)\n- [mongo -\u003e mongo](./docs/en/tutorial/mongo_to_mongo.md)\n- [redis -\u003e redis](./docs/en/tutorial/redis_to_redis.md)\n- [mysql -\u003e starrocks](./docs/en/tutorial/mysql_to_starrocks.md)\n- [mysql -\u003e doris](./docs/en/tutorial/mysql_to_doris.md)\n- [mysql -\u003e clickhouse](./docs/en/tutorial/mysql_to_clickhouse.md)\n- [mysql -\u003e tidb](./docs/en/tutorial/mysql_to_tidb.md)\n- [mysql -\u003e kafka -\u003e consumer](./docs/en/tutorial/mysql_to_kafka_consumer.md)\n- [pg -\u003e starrocks](./docs/en/tutorial/pg_to_starrocks.md)\n- [pg -\u003e doris](./docs/en/tutorial/pg_to_doris.md)\n- [pg -\u003e clickhouse](./docs/en/tutorial/pg_to_clickhouse.md)\n- [pg -\u003e kafka -\u003e consumer](./docs/en/tutorial/pg_to_kafka_consumer.md)\n- [mysql -\u003e ape_dts(HTTP server) -\u003e consumer](./docs/en/tutorial/mysql_to_http_server_consumer.md)\n- [pg -\u003e ape_dts(HTTP server) -\u003e consumer](./docs/en/tutorial/pg_to_http_server_consumer.md)\n- [snapshot + cdc without data loss](./docs/en/tutorial/snapshot_and_cdc_without_data_loss.md)\n- [modify data by lua](./docs/en/tutorial/etl_by_lua.md)\n\n## Run tests\n\nRefer to [test docs](./dt-tests/README.md) for details.\n\n# More docs\n- Configurations\n    - [config details](./docs/en/config.md)\n- Structure tasks\n    - [migration](./docs/en/structure/migration.md)\n    - [check](./docs/en/structure/check.md)\n    - [check by Liquibase](./docs/en/structure/check_by_liquibase.md)\n- Snapshot tasks\n    - [data migration](./docs/en/snapshot/migration.md)\n    - [data check](./docs/en/snapshot/check.md)\n    - [data revise](./docs/en/snapshot/revise.md)\n    - [data review](./docs/en/snapshot/review.md)\n    - [resume at breakpoint](./docs/en/snapshot/resume.md)\n    - [multiple tables in parallel](./docs/en/snapshot/tb_in_parallel.md)\n- CDC tasks\n    - [data sync](./docs/en/cdc/sync.md)\n    - [heartbeat to source database](./docs/en/cdc/heartbeat.md)\n    - [two-way data sync](./docs/en/cdc/two_way.md)  \n    - [generate sqls from CDC](./docs/en/cdc/to_sql.md)\n    - [resume at breakpoint](./docs/en/cdc/resume.md)\n- Custom consumers\n    - [mysql/pg -\u003e kafka -\u003e consumer](./docs/en/consumer/kafka_consumer.md)\n    - [mysql/pg -\u003e ape_dts(HTTP server) -\u003e consumer](./docs/en/consumer/http_consumer.md)\n- Data processing\n    - [modify data by lua](./docs/en/etl/lua.md)\n- Monitor\n    - [monitor info](./docs/en/monitor/monitor.md)\n    - [position info](./docs/en/monitor/position.md)\n- Task templates\n    - [mysql -\u003e mysql](./docs/templates/mysql_to_mysql.md)\n    - [pg -\u003e pg](./docs/templates/pg_to_pg.md)\n    - [mongo -\u003e mongo](./docs/templates/mongo_to_mongo.md)\n    - [redis -\u003e redis](./docs/templates/redis_to_redis.md)\n    - [mysql/pg -\u003e kafka](./docs/templates/rdb_to_kafka.md)\n    - [mysql/pg -\u003e ape_dts(HTTP server)](./docs/templates/rdb_to_http_server.md)\n    - [mysql -\u003e starrocks](./docs/templates/mysql_to_starrocks.md)\n    - [mysql -\u003e doris](./docs/templates/mysql_to_doris.md)\n    - [mysql -\u003e clickhouse](./docs/templates/mysql_to_clickhouse.md)\n    - [pg -\u003e starrocks](./docs/templates/pg_to_starrocks.md)\n    - [pg -\u003e doris](./docs/templates/pg_to_doris.md)\n    - [pg -\u003e clickhouse](./docs/templates/pg_to_clickhouse.md)\n\n# Benchmark\n- MySQL -\u003e MySQL, Snapshot\n\n| Method | Node Specs | RPS(rows per second) | Source MySQL Load (CPU/Memory) | Target MySQL Load (CPU/Memory) |\n| :-------- | :-------- | :-------- | :-------- | :-------- | \n| ape_dts | 1c2g | 71428 | 8.2% / 5.2% | 211% / 5.1% |\n| ape_dts | 2c4g | 99403 | 14.0% / 5.2% | 359% / 5.1% |\n| ape_dts | 4c8g | 126582 | 13.8% / 5.2% | 552% / 5.1% |\n| debezium | 4c8g |\t4051 | 21.5% / 5.2% | 51.2% / 5.1% |\n\n- MySQL -\u003e MySQL, CDC\n\n| Method | Node Specs | RPS(rows per second) | Source MySQL Load (CPU/Memory) | Target MySQL Load (CPU/Memory) |\n| :-------- | :-------- | :-------- | :-------- | :-------- |\n| ape_dts | 1c2g | 15002 | 18.8% / 5.2% | 467% / 6.5% | \n| ape_dts | 2c4g | 24692 | 18.1% / 5.2% | 687% / 6.5% | \n| ape_dts | 4c8g | 26287 | 18.2% / 5.2% | 685% / 6.5% |\n| debezium | 4c8g | 2951 | 20.4% / 5.2% | 98% / 6.5% |\n\n- Image size\n\n| ape_dts:2.0.14\t| debezium/connect:2.7 |\n| :-------- | :-------- |\n| 86.4 MB |\t1.38 GB |\n\n- more benchmark [details](./docs/en/benchmark.md)\n\n# Contributing\n\n## Structure\n\n![Structure](docs/img/structure.png)\n\n## Modules\n- dt-main: program entry\n- dt-connector: extractors + sinkers for databases\n- dt-pipeline: pipeline to connect extractors and sinkers\n- dt-parallelizer: parallel algorithms\n- dt-task: create extractors + sinkers + pipelines + parallelizers according to configurations\n- dt-common: common utils, basic data structures, metadata management\n- dt-tests: integration tests\n\n- related sub module: [mysql binlog connector in rust](https://github.com/apecloud/mysql-binlog-connector-rust)\n\n## Build\n- cargo build\n- [build images](./docs/en/build_images.md)\n\n## Checklist\n- run `cargo clippy --all-targets --all-features --workspace` fix all clippy issues.\n\n# Contact us\n\n[Slack Community](https://join.slack.com/t/kubeblocks/shared_invite/zt-22cx2f84x-BPZvnLRqBOGdZ_XSjELh4Q)\n\nWeChat Group \n- The WeChat group has reached its limit of 200 members and cannot be scanned to join in. \n- Please add this WeChat, and you will be manually pulled into the group. \n- When adding the WeChat, please note: **ape-dts**.\n\n\u003cimg src=\"./docs/img/wechat_group.png\" width=\"55%\" style=\"display: block; margin-left: 0;\"/\u003e","funding_links":[],"categories":["Recently Updated","Applications","Rust","Integrations"],"sub_categories":["[Who Wants to Be a Millionare](https://www.boardgamecapital.com/who-wants-to-be-a-millionaire-rules.htm)","Database","Data Transfer and Synchronization"],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fapecloud%2Fape-dts","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fapecloud%2Fape-dts","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fapecloud%2Fape-dts/lists"}