{"id":26611061,"url":"https://github.com/sqlpub/qin-cdc","last_synced_at":"2025-04-10T00:43:43.051Z","repository":{"id":241699330,"uuid":"779673228","full_name":"sqlpub/qin-cdc","owner":"sqlpub","description":"Real-time data synchronization component, supports MySQL -\u003e StarRocks, MySQL, Doris, Kafka, etc.), used to replace Aliyun DTS","archived":false,"fork":false,"pushed_at":"2024-07-27T08:01:18.000Z","size":140,"stargazers_count":16,"open_issues_count":3,"forks_count":5,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-04-10T00:43:34.934Z","etag":null,"topics":["doris","dts","kafka","mysql","sqlserver","starrocks","sync"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"agpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/sqlpub.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-03-30T13:36:10.000Z","updated_at":"2025-02-13T06:41:15.000Z","dependencies_parsed_at":"2024-05-29T18:51:30.582Z","dependency_job_id":"f59c9822-db81-4456-8987-feec8dc0738a","html_url":"https://github.com/sqlpub/qin-cdc","commit_stats":null,"previous_names":["sqlpub/qin-cdc"],"tags_count":5,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sqlpub%2Fqin-cdc","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sqlpub%2Fqin-cdc/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sqlpub%2Fqin-cdc/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sqlpub%2Fqin-cdc/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/sqlpub","download_url":"https://codeload.github.com/sqlpub/qin-cdc/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248137998,"owners_count":21053775,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["doris","dts","kafka","mysql","sqlserver","starrocks","sync"],"created_at":"2025-03-24T02:25:22.249Z","updated_at":"2025-04-10T00:43:43.026Z","avatar_url":"https://github.com/sqlpub.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"## qin-cdc data sync\nSimple, efficient, real-time, stable, scalable, highly available, AI, open source\n\n![LICENSE](https://img.shields.io/badge/license-AGPLv3%20-blue.svg)\n![](https://img.shields.io/github/languages/top/sqlpub/qin-cdc)\n[![Release](https://img.shields.io/github/release/sqlpub/qin-cdc.svg?style=flat-square)](https://github.com/sqlpub/qin-cdc/releases)\n\nEnglish | [简体中文](README.zh-CN.md)\n\n### Support sync database\n#### source\n1. mysql\n2. TODO sqlserver\n3. TODO mongo\n4. TODO oracle\n\n#### target\n\n1. mysql \n2. starrocks \n3. doris\n4. kafka json\n5. TODO kafka canal\n6. kafka aliyun_dts_canal\n\n### Quick start\n#### 1. Install\n[Download](https://github.com/sqlpub/qin-cdc/releases/latest) the latest release and extract it.\n\n#### 2. Create a synchronization account\n```sql\nCREATE USER 'qin_cdc'@'%' IDENTIFIED BY 'xxxxxx';\nGRANT SELECT, REPLICATION CLIENT, REPLICATION SLAVE ON *.* TO 'qin_cdc'@'%';\n```\n#### 3. Create a configuration file\nmysql-to-starrocks.toml\n```toml\n# name is required and must be globally unique when multiple instances are running\nname = \"mysql2starrocks\"\n\n[input]\ntype = \"mysql\"\n\n[input.config.source]\nhost = \"127.0.0.1\"\nport = 3306\nusername = \"root\"\npassword = \"root\"\n\n[input.config.source.options]\n#start-gtid = \"3ba13781-44eb-2157-88a5-0dc879ec2221:1-123456\"\n#server-id = 1001\n\n[[transforms]]\ntype = \"rename-column\"\n[transforms.config]\nmatch-schema = \"mysql_test\"\nmatch-table = \"tb1\"\ncolumns = [\"col_1\", \"col_2\"]\nrename-as = [\"col_11\", \"col_22\"]\n\n[[transforms]]\ntype = \"delete-column\"\n[transforms.config]\nmatch-schema = \"mysql_test\"\nmatch-table = \"tb1\"\ncolumns = [\"phone\"]\n\n[output]\ntype = \"starrocks\"\n\n[output.config.target]\nhost = \"127.0.0.1\"\nport = 9030\nload-port = 8040 # support fe httpPort:8030 or be httpPort:8040\nusername = \"root\"\npassword = \"\"\n\n[input.config.target.options]\nbatch-size = 1000\nbatch-interval-ms = 1000\nparallel-workers = 4\n\n[[output.config.routers]]\nsource-schema = \"mysql_test\"\nsource-table = \"tb1\"\ntarget-schema = \"sr_test\"\ntarget-table = \"ods_tb1\"\n\n[[output.config.routers]]\nsource-schema = \"mysql_test\"\nsource-table = \"tb2\"\ntarget-schema = \"sr_test\"\ntarget-table = \"ods_tb2\"\n# mapper column, optional, if empty, same name mapping\n# [output.config.routers.columns-mapper]\n# source-columns = []\n# target-columns = []\n```\n\n#### 4. View Help\n```shell\n[sr@ ~]$ ./qin-cdc-linux-xxxxxx -h\n```\n\n#### 5. Startup\n```shell\n[sr@ ~]$ ./qin-cdc-linux-xxxxxx -config mysql-to-starrocks.toml -log-file mysql2starrocks.log -level info -daemon\n```\n\n#### 6. View logs\n```shell\n[sr@ ~]$ tail -f mysql2starrocks.log\n```\n\n#### TODO AI functional points\n1. Intelligent data synchronization and migration\n2. Data security and monitoring\n3. Intelligent operation and maintenance management\n4. User experience optimization\n5. Automated data mapping\n6. Natural language processing\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsqlpub%2Fqin-cdc","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsqlpub%2Fqin-cdc","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsqlpub%2Fqin-cdc/lists"}