{"id":26109492,"url":"https://github.com/dyrnq/cdc-vagrant","last_synced_at":"2026-04-18T22:32:21.510Z","repository":{"id":64060673,"uuid":"570891421","full_name":"dyrnq/cdc-vagrant","owner":"dyrnq","description":"CDC(Change Data Capture) is made up of two components, the CDD and the CDT. CDD is stand for Change Data Detection and CDT is stand for Change Data Transfer.","archived":false,"fork":false,"pushed_at":"2026-02-06T12:20:27.000Z","size":157,"stargazers_count":0,"open_issues_count":30,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2026-02-06T20:15:01.905Z","etag":null,"topics":["apache-doris","cdc","debezium","etl","flink","flink-cdc","flink-doris-connector","flink-sql","flink-stream-processing","hadoop","minio-cluster","vagrantfile"],"latest_commit_sha":null,"homepage":"https://nightlies.apache.org/flink/flink-cdc-docs-stable/","language":"Shell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/dyrnq.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2022-11-26T13:19:59.000Z","updated_at":"2026-02-06T12:20:32.000Z","dependencies_parsed_at":"2024-11-30T09:24:13.352Z","dependency_job_id":"02a3c4b4-3a67-4a83-ad1d-58c02f1234a9","html_url":"https://github.com/dyrnq/cdc-vagrant","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/dyrnq/cdc-vagrant","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dyrnq%2Fcdc-vagrant","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dyrnq%2Fcdc-vagrant/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dyrnq%2Fcdc-vagrant/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dyrnq%2Fcdc-vagrant/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/dyrnq","download_url":"https://codeload.github.com/dyrnq/cdc-vagrant/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dyrnq%2Fcdc-vagrant/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":31987798,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-18T20:23:30.271Z","status":"ssl_error","status_checked_at":"2026-04-18T20:23:29.375Z","response_time":103,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["apache-doris","cdc","debezium","etl","flink","flink-cdc","flink-doris-connector","flink-sql","flink-stream-processing","hadoop","minio-cluster","vagrantfile"],"created_at":"2025-03-09T23:11:12.445Z","updated_at":"2026-04-18T22:32:21.455Z","avatar_url":"https://github.com/dyrnq.png","language":"Shell","funding_links":[],"categories":[],"sub_categories":[],"readme":"# cdc-vagrant\n\n\u003c!-- TOC --\u003e\n\n- [cdc-vagrant](#cdc-vagrant)\n  - [introduce](#introduce)\n  - [architecture](#architecture)\n    - [zookeeper cluster](#zookeeper-cluster)\n    - [flink cluster](#flink-cluster)\n    - [flink cdc](#flink-cdc)\n    - [flink-cdc version matrix](#flink-cdc-version-matrix)\n  - [ref](#ref)\n\n\u003c!-- /TOC --\u003e\n\n## introduce\n\nThis project is for experiment of flink-cdc and doris.\n\nCDC(Change Data Capture) is made up of two components, the CDD and the CDT. CDD is stand for Change Data Detection and CDT is stand for Change Data Transfer.\n\nExtract, Load, Transform (ELT) is a data integration process for transferring raw data from a source server to a data system (such as a data warehouse or data lake) on a target server and then preparing the information for downstream uses.\n\nStreaming ETL (Extract, Transform, Load) is the processing and movement of real-time data from one place to another. ETL is short for the database functions extract, transform, and load.\n\n## architecture\n\n### zookeeper cluster\n\n| vm    | role      | ip             | xxx_home       |\n|-------|-----------|----------------|----------------|\n| vm116 | zookeeper | 192.168.56.116 | /opt/zookeeper |\n| vm117 | zookeeper | 192.168.56.117 | /opt/zookeeper |\n| vm118 | zookeeper | 192.168.56.118 | /opt/zookeeper |\n\n```bash\ncd /opt/zookeeper\n./bin/zkServer.sh status\n```\n\n```bash\necho stat | nc 127.0.0.1 2181\nZookeeper version: 3.8.4-9316c2a7a97e1666d8f4593f34dd6fc36ecc436c, built on 2024-02-12 22:16 UTC\nClients:\n /127.0.0.1:39370[0](queued=0,recved=1,sent=0)\n\nLatency min/avg/max: 0/6.375/41\nReceived: 10\nSent: 9\nConnections: 1\nOutstanding: 0\nZxid: 0x100000003\nMode: follower\nNode count: 5\n```\n\n\n\n### flink cluster\n\n| vm    | role   | ip             | xxx_home   |\n|-------|--------|----------------|------------|\n| vm116 | minio  | 192.168.56.116 | /opt/minio |\n| vm117 | minio  | 192.168.56.117 | /opt/minio |\n| vm118 | minio  | 192.168.56.118 | /opt/minio |\n| vm119 | minio  | 192.168.56.119 | /opt/minio |\n\n| vm    | role                             | ip             | xxx_home   |\n|-------|----------------------------------|----------------|------------|\n| vm116 | sidekick, flink(masters+workers) | 192.168.56.116 | /opt/flink |\n| vm117 | sidekick, flink(masters+workers) | 192.168.56.117 | /opt/flink |\n| vm118 | sidekick, flink(masters+workers) | 192.168.56.118 | /opt/flink |\n| vm119 | sidekick, flink(workers)         | 192.168.56.119 | /opt/flink |\n\n\u003e ssh init\n\n```bash\nvagrant ssh vm116\n## vm116 vm117 vm118 vm119 四台分别执行\nsu -l root\nbash /vagrant/scripts/ssh-copy-id.sh --iface enp0s8 --ips \"192.168.56.116,192.168.56.117,192.168.56.118,192.168.56.119\"\n```\n\n\u003e minio init\n\n```bash\nvagrant ssh vm116\n## 只需要在vm116上执行，且执行一次即可\ncurl -o /usr/local/bin/mc -# -fSL https://files.m.daocloud.io/dl.min.io/client/mc/release/linux-amd64/mc\nchmod +x /usr/local/bin/mc\n\nmc alias set myminio http://localhost:9000 minioadmin minioadmin\nmc admin user svcacct add --access-key \"u5SybesIDVX9b6Pk\" --secret-key \"lOpH1v7kdM6H8NkPu1H2R6gLc9jcsmWM\" myminio minioadmin\n# mc admin user svcacct add --access-key \"myuserserviceaccount\" --secret-key \"myuserserviceaccountpassword\" myminio minioadmin\n\nmc mb myminio/flink\nmc mb myminio/flink-state\n```\n\n```bash\n## flink集群启动\nsu -l hduser\ncd /opt/flink\n\n## start-cluster\nbin/start-cluster.sh\n## stop-cluster\nbin/stop-cluster.sh\n## 测试一下\nbin/flink run /opt/flink/examples/streaming/WordCount.jar\n```\n\n### flink cdc\n\nvagrant ssh vm211\n\n```bash\ncd /vagrant/doris\ndocker compose exec doris-fe mysql -uroot -P9030 -h127.0.0.1 -e \"show backends; show frontends;\"\n```\n\n```bash\nvagrant ssh vm116\n\nflink_cdc_home=\"/opt/flink-cdc\"\npushd $flink_cdc_home || exit 1\n./bin/flink-cdc.sh /vagrant/doris/mysql-to-doris.yaml --jar lib/mysql-connector-java-8.0.27.jar --flink-home /opt/flink\npopd || exit 1\n\n```\n\n### fink-sql-client\n\n```bash\nbin/sql-client.sh\nSET execution.checkpointing.interval = 6000;\n\n```\n\n### flink-cdc version matrix\n\nsee \u003chttps://nightlies.apache.org/flink/flink-cdc-docs-master/docs/connectors/flink-sources/overview/#supported-flink-versions\u003e\n\n## ref\n\n- [Deprecated Properties](https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/DeprecatedProperties.html)\n- [HDFS High Availability](https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/HDFSHighAvailabilityWithNFS.html)\n- [HDFS High Availability Using the Quorum Journal Manager](https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/HDFSHighAvailabilityWithQJM.html)\n- [ResourceManager High Availability](https://hadoop.apache.org/docs/stable/hadoop-yarn/hadoop-yarn-site/ResourceManagerHA.html)\n- [High Performance HTTP Sidecar Load Balancer](https://github.com/minio/sidekick)\n- [https://github.com/apache/doris/tree/master/samples/doris-demo](https://github.com/apache/doris/tree/master/samples/doris-demo)\n- [Streaming ELT from MySQL to Doris](https://nightlies.apache.org/flink/flink-cdc-docs-release-3.2/docs/get-started/quickstart/mysql-to-doris/)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdyrnq%2Fcdc-vagrant","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdyrnq%2Fcdc-vagrant","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdyrnq%2Fcdc-vagrant/lists"}