{"id":13695633,"url":"https://github.com/zeromicro/cds","last_synced_at":"2025-05-16T13:04:04.436Z","repository":{"id":37364124,"uuid":"315848971","full_name":"zeromicro/cds","owner":"zeromicro","description":"Data syncing in golang for ClickHouse.","archived":false,"fork":false,"pushed_at":"2023-08-08T12:30:55.000Z","size":7347,"stargazers_count":969,"open_issues_count":4,"forks_count":139,"subscribers_count":38,"default_branch":"master","last_synced_at":"2025-05-07T13:02:07.356Z","etag":null,"topics":["bigdata","clickhouse","go","golang","kafka-consumer"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/zeromicro.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2020-11-25T06:28:58.000Z","updated_at":"2025-04-15T06:28:27.000Z","dependencies_parsed_at":"2024-01-17T16:08:24.500Z","dependency_job_id":"cf48b036-558d-4fa0-b431-deae8ec353f7","html_url":"https://github.com/zeromicro/cds","commit_stats":null,"previous_names":["tal-tech/cds"],"tags_count":4,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zeromicro%2Fcds","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zeromicro%2Fcds/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zeromicro%2Fcds/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zeromicro%2Fcds/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/zeromicro","download_url":"https://codeload.github.com/zeromicro/cds/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254043268,"owners_count":22004913,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["bigdata","clickhouse","go","golang","kafka-consumer"],"created_at":"2024-08-02T18:00:31.454Z","updated_at":"2025-05-16T13:04:04.379Z","avatar_url":"https://github.com/zeromicro.png","language":"Go","readme":"\u003cp align=\"left\"\u003e\n  \u003cimg width =\"200px\" src=\"https://github.com/zeromicro/cds/raw/master/doc/logo.png\"\u003e\n\u003c/p\u003e\n\n# ClickHouse Data Synchromesh\nData syncing in golang for ClickHouse.\n\n\nbased on [go-zero](https://github.com/zeromicro/go-zero) \n\n### ARCH\n\nData workflow of a typical data warehouse architecture \n\n![avatar](https://github.com/zeromicro/cds/raw/master/doc/clickhouse_arch.png)\n\n### Design of Data Sync\n\nAutomatically synchronizing data from MySQL/MongoDB data source to ClickHouse cluster in real time(\u003c 2min).\n\n![sync](https://github.com/zeromicro/cds/raw/master/doc/sync.png)\n\n\n# start up\n\n```bash\ngit clone https://github.com/zeromicro/cds.git\ncd cds\nmake up\n```\n\nAfter the build , pay attention to check if any container exits abnormally.\n\nclick http://localhost:3414/cds.html to accessing the web interface.\n\n\nusing username and password below to login\n```\nuser: admin@email.com\npassword: 123456\n```\n## create table in ClickHouse for syncing MySQL(or MongoDB) data\n\nChose \"create table\" tab\n\n```\n1. Click \"Target ClickHouse Database Info\",\n2. Click “connect”\n3. Select the schema (\"default\") synchronized to Clickhouse\n4. switch to \"Data Source\" \n5. MySQL input connection string root:root@tcp(mysql:3306)/test_mysql\nor\nMongoDB input connection string mongodb://mongo1:30001/test_mongo\n6. Click “connect”\n7. Select the table, such as the default example_mysql (MySQL) or example (MongoDB)\n8. click \"Generate create Table SQL\" \n Note: select the partition field as needed, here'PARTITION BY toYYYYMM()' can be deleted, or replaced with'PARTITION BY toYYYYMM(dt)'\n9. . Click \"send SQL to ClickHouse\", and the successful execution will pop up below\n```\n\n\n## One-time full data synchronization:\nChose \"full sync\" tab\n```\n1. Click \"+\" in the upper right corner\n2. MySQL input connection string root:root@tcp(mysql:3306)/test_mysql\nor\nMongoDB input connection string mongodb://mongo1:30001/test_mongo\n3. Click “connect”\n4. Select the table, such as the default example_mysql (MySQL) or example (MongoDB)\n5. Click \"Target ClickHouse Database Info\"\n6. Click “connect”\n7. Select the schema (\"default\") synchronized in Clickhouse\n10. Click \"Add\", a pop-up below shows successful execution\n```\n\n\n### check task status\nRefresh the page \n\n\n### Confirm data in Clickhouse\n\n![image-20201118135156133](doc/image-20201118135156133.png)\n\n## Turn on real-time incremental synchronization\n\nTake mysql as an example\n\nchose \"Connector\" tab\n```\n1. Click \"+\" in the upper right corner\n2. MySQL input connection string root:root@tcp(mysql:3306)/test_mysql\n3. select table \n4. Click \"Add\"\n```\nchose \"Incremental Sync\" tab\n\n```\n1. Click \"+\" in the upper right corner\n2. MySQL input connection string root:root@tcp(mysql:3306)/test_mysql\n3. select table \n4. Click \"Target ClickHouse Database Info\"\n5. Click “connect”\n6. Select the schema (\"default\") synchronized in Clickhouse\n7. Click \"Add\", a pop-up below shows successful execution\n```\nrefresh page\n\n### Verify incremental update\nExecute the initialization database script again, you can insert 100000 rows of data again.\n\n```\ncd sit/docker/\nsh ./init.sh\n```\n\nVerify the incremental data of mysql in clickhouse：\n\n![image-20201118135503830](doc/image-20201118135503830.png)\n\n## clean up\nTo clean up all the docker containers started above and restore the initial state, you can ：\n\n```\ncd cds\nmake down\n```\n\nonly clean \n\n```\ncd cds\nmake docker_clean\n```\n\n\n### data model in clickhouse\n[CDS中ClickHouse使用的建表方案](doc/CDS中ClickHouse使用的建表方案.md)\n\n## help\n\n[提问的智慧](https://github.com/ryanhanwu/How-To-Ask-Questions-The-Smart-Way/blob/main/README-zh_CN.md)\n\n[如何有效的报告bug](https://www.chiark.greenend.org.uk/~sgtatham/bugs-cn.html)\n\n---\n\nif you like this project and want to support it，please `star` 🤝\n\n","funding_links":[],"categories":["Repositories","Integrations"],"sub_categories":["Data Transfer and Synchronization"],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fzeromicro%2Fcds","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fzeromicro%2Fcds","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fzeromicro%2Fcds/lists"}