{"id":37021684,"url":"https://github.com/wangzihaogithub/dts","last_synced_at":"2026-01-14T02:34:23.511Z","repository":{"id":244781637,"uuid":"815101851","full_name":"wangzihaogithub/dts","owner":"wangzihaogithub","description":"dts","archived":false,"fork":false,"pushed_at":"2025-09-08T15:29:20.000Z","size":1124,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-09-08T17:31:26.472Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/wangzihaogithub.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2024-06-14T11:02:06.000Z","updated_at":"2025-09-08T15:29:23.000Z","dependencies_parsed_at":"2024-08-26T03:57:31.392Z","dependency_job_id":"5e36873b-48d8-48c7-8393-34ea8c1dec4c","html_url":"https://github.com/wangzihaogithub/dts","commit_stats":null,"previous_names":["wangzihaogithub/dts"],"tags_count":20,"template":false,"template_full_name":null,"purl":"pkg:github/wangzihaogithub/dts","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wangzihaogithub%2Fdts","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wangzihaogithub%2Fdts/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wangzihaogithub%2Fdts/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wangzihaogithub%2Fdts/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/wangzihaogithub","download_url":"https://codeload.github.com/wangzihaogithub/dts/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wangzihaogithub%2Fdts/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28408711,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-14T01:52:23.358Z","status":"online","status_checked_at":"2026-01-14T02:00:06.678Z","response_time":107,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-01-14T02:34:22.921Z","updated_at":"2026-01-14T02:34:23.498Z","avatar_url":"https://github.com/wangzihaogithub.png","language":"Java","funding_links":[],"categories":[],"sub_categories":[],"readme":"# dts\n\n#### 介绍\n数据同步 \n- 延迟低至5ms（修改数据库至elasticsearch可以搜索到）\n- 支持 同步elasticsearch-Nested字段，会收集sql的join相关表，自动反向更新，支持自定义处理字段，解析url转换为文本\n- 支持 直连数据库binglog，支持redis记忆offset，不丢binlog\n- 支持 连阿里云-kafka-binlog\n- 支持 自定义监听\n- 支持 将Row变更转化为SQL语句对象\n- 支持 报警消息\n- 支持 es8写入\n- 支持 es8的向量存储，dense_vector类型\n\n\n### 代码例子 demo如下\n\nhttps://github.com/wangzihaogithub/dts-demo\n\n\n[![Maven Central](https://img.shields.io/maven-central/v/com.github.wangzihaogithub/dts.svg?label=Maven%20Central)](https://search.maven.org/search?q=g:com.github.wangzihaogithub%20AND%20a:dts)\n\n```xml\n\u003c!-- https://github.com/wangzihaogithub/dts --\u003e\n\u003c!-- https://mvnrepository.com/artifact/com.github.wangzihaogithub/dts --\u003e\n\u003cdependency\u003e\n  \u003cgroupId\u003ecom.github.wangzihaogithub\u003c/groupId\u003e\n  \u003cartifactId\u003edts\u003c/artifactId\u003e\n  \u003cversion\u003e1.1.25\u003c/version\u003e\n\u003c/dependency\u003e\n```\n    \n-  1.仅导入上面的maven包就行\n\n    \n        @SpringBootApplication\n        public class Application {\n            public static void main(String[] args) {\n                SpringApplication.run(Application.class, args);\n            }\n        }\n\n\n- 2.application-dev.yaml 配置\n\n        `\n        cnwy.llm-api-key: ${OPEN_API_KEY}\n        canal.conf:\n          enable-pull: true\n          srcDataSources:\n            defaultDS:\n              url: 'jdbc:mysql://${MYSQL_HOST}:3306/cnwy?useUnicode=true\u0026characterEncoding=utf-8\u0026allowMultiQueries=true\u0026rewriteBatchedStatements=true\u0026zeroDateTimeBehavior=CONVERT_TO_NULL'\n              username: 'test_cnwy'\n              password: ${MYSQL_PASSWORD}\n          canalAdapters:\n            - destination: 'cnwybinlog'\n              topics:\n                - 'cnwy\\\\.job.*'\n                - 'cnwy\\\\.corp.*'\n                - 'cnwy.region'\n                - 'cnwy.kn_question'\n                - 'cnwy.kn_scene'\n              properties: {\n                dataSource: 'defaultDS',\n                maxDumpThread: 10,\n                enableGTID: true\n              }\n              groups:\n                - outerAdapters:\n                    - name: 'adapterES716'\n                      es7x:\n                        concurrentBulkRequest: 4\n                        address: 'https://my-elasticsearch-project-f10e79.es.ap-southeast-1.aws.elastic.cloud:443'\n                        api-key: ${ES_API_KEY}\n        spring:\n          redis:\n            host: ${REDIS_HOST}\n            password: ${REDIS_PASSWORD}\n\n\n          `\n\n\n-  3.数据关系配置\n\n        `\n            dataSourceKey: defaultDS\n            destination: cnwybinlog\n            esMapping:\n              env: dev\n              pk: id\n              _index: cnwy_job_test_index_alias\n              upsert: false\n              writeNull: false\n              indexUpdatedTime: 'indexUpdatedTime'\n              sql: \"SELECT\n                    job.id as id,\n                    job.type as type,\n                    job.education as education,\n                    job.`name` as name,\n                    job.notice_url as noticeUrl,\n                    job.content as content,\n                    job.content as contentVector,\n                    job.`status` as status,\n                    job.company_name as companyName,\n                    job.create_time as createTime,\n                    job.update_time as updateTime\n                  FROM job job\n                  \"\n              objFields:\n                education:\n                  type: array\n                  paramArray:\n                    split: '[、]'\n                corp$aliasNames:\n                  type: array\n                  paramArray:\n                    split: ','\n                contentVector:\n                  type: llm-vector\n                  paramLlmVector:\n                    apiKey: ${cnwy.llm-api-key}\n                    baseUrl: https://dashscope.aliyuncs.com/compatible-mode/v1\n                    modelName: text-embedding-v3\n                    dimensions: 1024\n                corp:\n                  type: object-sql\n                  paramSql:\n                    sql: \"SELECT\n                            corp.id ,\n                            corp.`name`,\n                            GROUP_CONCAT(if(corpName.type = 2, corpName.`name`, null)) as aliasNames,\n                            GROUP_CONCAT(if(corpName.type = 3, corpName.`name`, null)) as historyNames,\n                            corp.group_name as groupCompany,\n                            corp.data_type as dataType,\n                            corp.central_country_ent_flag as centralCountryEntFlag\n                          FROM corp corp\n                          LEFT JOIN corp_name corpName on corpName.corp_id = corp.id \n                    \"\n                    onMainTableChangeWhereSql: 'WHERE corp.id = #{corp_id} '\n                    onSlaveTableChangeWhereSql: 'WHERE corp.id = #{id} '\n                corpTagList$tagIds:\n                  type: array\n                  paramArray:\n                    split: ','\n                corpTagList:\n                  type: array-sql\n                  paramSql:\n                    sql: \"SELECT\n                            group_concat(corpRelationTag.tag_id) as tagIds,\n                            corpTag.category_id as categoryId\n                        FROM corp_relation_tag corpRelationTag\n                        INNER JOIN corp_tag corpTag on corpTag.id = corpRelationTag.tag_id\n                        GROUP BY corpRelationTag.corp_id, corpTag.category_id\n              \"\n                    onMainTableChangeWhereSql: 'WHERE corpRelationTag.corp_id = #{corp_id} '\n                    onSlaveTableChangeWhereSql: 'WHERE corpRelationTag.corp_id = #{corp_id} '\n                regionList:\n                  type: array-sql\n                  paramSql:\n                    sql: \"SELECT\n                            region.id as id,\n                            region.region_id as regionId,\n                            region.province_id as provinceId,\n                            region.province_name as provinceName,\n                            region.city_id as cityId,\n                            region.city_name as cityName,\n                            region.district_id as districtId,\n                            region.district_name as districtName,\n                            region.address as address,\n                            region.region_id_colloquial as regionIdColloquial,\n                            region.region_name_colloquial as regionNameColloquial,\n                            concat(if(region.lat\u003c region.lng, region.lat,region.lng), ',', if(region.lat \u003e region.lng, region.lat,region.lng)) as geo\n                        FROM job_region region \"\n                    onMainTableChangeWhereSql: 'WHERE region.job_id = #{id} '\n                    onSlaveTableChangeWhereSql: 'WHERE region.job_id = #{job_id} '\n                specialtyList:\n                  type: array-sql\n                  paramSql:\n                    sql: \"SELECT\n                            specialty.id as id,\n                            specialty.specialty_id as specialtyId,\n                            specialty.specialty_name as specialtyName,\n                            specialty.specialty_id_colloquial as specialtyIdColloquial,\n                            specialty.specialty_name_colloquial as specialtyNameColloquial\n                        FROM job_specialty specialty \"\n                    onMainTableChangeWhereSql: 'WHERE specialty.job_id = #{id} '\n                    onSlaveTableChangeWhereSql: 'WHERE specialty.job_id = #{job_id} '\n                corpRegionList:\n                  type: array-sql\n                  paramSql:\n                    sql: \"SELECT\n                                 corpRegion.province_id as provinceId,\n                                 corpRegion.city_id as cityId,\n                                 corpRegion.district_id as districtId,\n                                 corpRegion.region_id as regionId,\n                                 province.`name` as provinceName,\n                                 city.`name` as cityName,\n                                 district.`name` as districtName,\n                                 concat(if(region.lat\u003c region.lng, region.lat,region.lng), ',', if(region.lat \u003e region.lng, region.lat,region.lng)) as geo\n                             FROM corp_region corpRegion\n                             LEFT JOIN region region on region.id = corpRegion.region_id\n                             LEFT JOIN region province on province.id = corpRegion.province_id\n                             LEFT JOIN region city on city.id = corpRegion.city_id\n                             LEFT JOIN region district on district.id = corpRegion.district_id\n                             GROUP BY corpRegion.id\n                  \"\n                    onMainTableChangeWhereSql: 'WHERE corpRegion.corp_id = #{corp_id} '\n                    onSlaveTableChangeWhereSql: 'WHERE corpRegion.corp_id = #{corp_id} '\n\n\n           \n       `\n\n\n - 启动springboot 项目用mysql执行SQL： show processlist， 即可看到 binlog dump 线程已启动\n\n\n### dts-sdk使用\n\n```xml\n\u003c!-- https://github.com/wangzihaogithub/dts-sdk --\u003e\n\u003c!-- https://mvnrepository.com/artifact/com.github.wangzihaogithub/dts-sdk --\u003e\n\u003cdependency\u003e\n  \u003cgroupId\u003ecom.github.wangzihaogithub\u003c/groupId\u003e\n  \u003cartifactId\u003edts-sdk\u003c/artifactId\u003e\n  \u003cversion\u003e1.1.25\u003c/version\u003e\n\u003c/dependency\u003e\n```\n\n-  1.仅导入上面的maven包就行\n\n   yaml配置\n\n        server:\n          dts:\n            sdk:\n              cluster:\n                discovery: redis\n        spring:\n          redis:\n            host: localhost\n            password: xxx\n    \n\n        @Autowired\n        private DtsSdkClient dtsSdkClient;\n        @PostMapping(\"/save\")\n        public CompletableFuture\u003cAjaxResult\u003e save(@RequestBody OrderRequest request) {\n            Integer orderId = orderService.save(request);\n            return dtsSdkClient.listenEs(\"cnwy_order_test_index_alias\", orderId, 500)\n                    .handle((listenEsResponse, throwable) -\u003e AjaxResult.success(orderId));\n        }\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fwangzihaogithub%2Fdts","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fwangzihaogithub%2Fdts","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fwangzihaogithub%2Fdts/lists"}