{"id":27645954,"url":"https://github.com/devlive-community/dameng-connector","last_synced_at":"2025-04-24T01:14:14.849Z","repository":{"id":280747105,"uuid":"940449497","full_name":"devlive-community/dameng-connector","owner":"devlive-community","description":"Dameng Connector 是一个专门为达梦数据库(DM Database)设计的变更数据捕获(CDC)解决方案。该项目通过扩展 Debezium 和 Flink CDC，实现了对达梦数据库的实时数据变更监控、捕获和处理能力，为数据集成、数据同步、实时分析等场景提供强大支持。","archived":false,"fork":false,"pushed_at":"2025-04-11T09:17:24.000Z","size":321,"stargazers_count":14,"open_issues_count":1,"forks_count":9,"subscribers_count":1,"default_branch":"dev","last_synced_at":"2025-04-24T01:14:07.257Z","etag":null,"topics":["cdc","dameng","dameng-exporter","dameng8","debezium","debezium-connector","flink","flink-cdc","flink-connector"],"latest_commit_sha":null,"homepage":"","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/devlive-community.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-02-28T07:38:06.000Z","updated_at":"2025-04-23T12:31:07.000Z","dependencies_parsed_at":"2025-04-11T10:58:00.509Z","dependency_job_id":null,"html_url":"https://github.com/devlive-community/dameng-connector","commit_stats":null,"previous_names":["devlive-community/dameng-connector"],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/devlive-community%2Fdameng-connector","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/devlive-community%2Fdameng-connector/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/devlive-community%2Fdameng-connector/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/devlive-community%2Fdameng-connector/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/devlive-community","download_url":"https://codeload.github.com/devlive-community/dameng-connector/tar.gz/refs/heads/dev","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":250540916,"owners_count":21447427,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cdc","dameng","dameng-exporter","dameng8","debezium","debezium-connector","flink","flink-cdc","flink-connector"],"created_at":"2025-04-24T01:14:14.242Z","updated_at":"2025-04-24T01:14:14.841Z","avatar_url":"https://github.com/devlive-community.png","language":"Java","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Dameng Connector\n\n基于 Debezium 和 Flink CDC 的达梦数据库变更数据捕获解决方案\n\n## 项目简介\n\nDameng Connector 是一个专门为达梦数据库(DM Database)设计的变更数据捕获(CDC)解决方案。该项目通过扩展 Debezium 和 Flink CDC，实现了对达梦数据库的实时数据变更监控、捕获和处理能力，为数据集成、数据同步、实时分析等场景提供强大支持。\n\n## 快速开始\n\n### 前置条件\n\n- Java 8+\n- 达梦数据库 8.x+\n- Maven 3.6+\n\n### 安装配置\n\n1. **克隆仓库**\n\n```bash\ngit clone https://github.com/yourusername/dameng-connector.git\ncd dameng-connector\n```\n\n2. **编译项目**\n\n```bash\n./mvnw clean package -DskipTests\n```\n\n### 使用示例\n\n#### 1. 独立模式运行 Debezium\n\n```bash\n# 启动 Debezium 独立服务器\ndebezium-server/run.sh\n\n# 注册达梦连接器\ncurl -X POST -H \"Content-Type: application/json\" --data @dm-connector.json http://localhost:8083/connectors\n```\n\n## 配置选项\n\n### Debezium 连接器配置\n\n| 选项                   | 描述       | 默认值                                  | 必填 |\n|----------------------|----------|--------------------------------------|----|\n| connector.class      | 连接器类名    | io.debezium.connector.dm.DMConnector | 是  |\n| database.hostname    | 达梦数据库主机名 | -                                    | 是  |\n| database.port        | 达梦数据库端口号 | 5236                                 | 是  |\n| database.user        | 数据库用户名   | -                                    | 是  |\n| database.password    | 数据库密码    | -                                    | 是  |\n| database.dbname      | 数据库名称    | -                                    | 是  |\n| database.server.name | 连接器唯一名称  | -                                    | 是  |\n| table.include.list   | 需要捕获的表列表 | -                                    | 否  |\n| snapshot.mode        | 快照模式     | initial                              | 否  |\n| transforms           | 转换配置     | -                                    | 否  |\n\n## 版本兼容性\n\n| 本项目版本    | Debezium 版本 | 达梦数据库版本 |\n|----------|-------------|---------|\n| 2025.1.0 | 1.9.x       | 8.x     |\n\n## 贡献指南\n\n我们非常欢迎社区贡献！如果您想参与项目开发，请遵循以下步骤：\n\n1. Fork 本仓库\n2. 创建您的特性分支 (`git checkout -b feature/amazing-feature`)\n3. 提交您的更改 (`git commit -m 'Add some amazing feature'`)\n4. 推送到分支 (`git push origin feature/amazing-feature`)\n5. 开启一个 Pull Request\n\n## 许可证\n\n本项目采用 MIT 许可证。详情请参阅 [LICENSE](LICENSE) 文件。\n\n## 联系方式\n\n- 项目维护者：您的姓名\n- 电子邮件：support@devlive.org\n- 项目问题跟踪：[GitHub Issues](https://github.com/devlive-community/dameng-connector/issues)\n\n---\n\n# 达梦数据库 CDC 配置指南\n\n本文档介绍如何为达梦数据库配置变更数据捕获(CDC)功能，以便与 Debezium 等工具集成。\n\n## 1. 准备测试环境\n\n首先，需要创建测试数据库和表，并插入一些测试数据：\n\n```sql\n-- 测试数据库\nCREATE SCHEMA TEST;\n\n-- 测试数据表\nCREATE TABLE \"TEST\".\"T00003\"\n(\n    \"ID\" NUMBER PRIMARY KEY,\n    \"NAME\" VARCHAR(8188),\n    \"AGE\" NUMBER\n);\n\n-- 测试数据\nINSERT INTO \"TEST\".\"T00003\" (\"ID\", \"NAME\", \"AGE\") VALUES (1, '张三', 25);\nINSERT INTO \"TEST\".\"T00003\" (\"ID\", \"NAME\", \"AGE\") VALUES (2, '李四', 30);\nINSERT INTO \"TEST\".\"T00003\" (\"ID\", \"NAME\", \"AGE\") VALUES (3, '王五', 28);\nINSERT INTO \"TEST\".\"T00003\" (\"ID\", \"NAME\", \"AGE\") VALUES (4, '赵六', 35);\nINSERT INTO \"TEST\".\"T00003\" (\"ID\", \"NAME\", \"AGE\") VALUES (5, '钱七', 22);\nINSERT INTO \"TEST\".\"T00003\" (\"ID\", \"NAME\", \"AGE\") VALUES (6, '孙八', 40);\nINSERT INTO \"TEST\".\"T00003\" (\"ID\", \"NAME\", \"AGE\") VALUES (7, '周九', 29);\nINSERT INTO \"TEST\".\"T00003\" (\"ID\", \"NAME\", \"AGE\") VALUES (8, '吴十', 31);\nINSERT INTO \"TEST\".\"T00003\" (\"ID\", \"NAME\", \"AGE\") VALUES (9, '刘十一', 27);\nINSERT INTO \"TEST\".\"T00003\" (\"ID\", \"NAME\", \"AGE\") VALUES (10, '陈十二', 33);\n```\n\n## 2. 配置兼容模式\n\n需要开启兼容模式，以确保 CDC 功能正常工作：\n\n```sql\n-- 开启兼容模式，需要重启服务器，或者修改 dm.ini\nSP_SET_PARA_VALUE(2,'compatible_mode',2);\nSELECT para_name,para_type,para_value FROM V$DM_INI WHERE PARA_NAME = 'COMPATIBLE_MODE';\n```\n\n**注意**：修改兼容模式后，需要重启数据库服务才能生效。您也可以直接在 dm.ini 文件中修改配置项。\n\n## 3. 开启归档日志\n\nCDC 功能依赖于数据库的归档日志模式：\n\n```sql\n-- 开启归档\nALTER DATABASE MOUNT;\nALTER DATABASE ARCHIVELOG;\nALTER DATABASE ADD ARCHIVELOG 'DEST = /data/dameng/data/DAMENG/archivelog, TYPE = local, FILE_SIZE = 1024, SPACE_LIMIT = 2048';\nALTER DATABASE OPEN;\n```\n\n## 4. 开启闪回功能\n\n配置闪回功能以支持 CDC 操作：\n\n```sql\n-- 设置回闪\nSP_SET_PARA_VALUE(1,'ENABLE_FLASHBACK',1);\nSELECT para_name,para_value FROM V$DM_INI WHERE PARA_NAME ='ENABLE_FLASHBACK';\n```\n\n## 5. 开启日志追加模式（关键步骤）\n\n**重要**：此步骤是确保 CDC 正常工作的关键，如果不配置，`V$LOGMNR_CONTENTS` 视图可能始终为空，导致无法捕获数据变更。\n\n```sql\n-- 注意：这里一定要处理，设置为 1 否则 V$LOGMNR_CONTENTS 一直是 0 导致获取不到数据\n-- 开启日志追加：修改 dm.ini 配置项 RLOG_APPEND_LOGIC=1 重启服务\n\n-- 查询是否是追加模式\nSELECT para_name, para_value FROM v$dm_ini WHERE para_name = 'RLOG_APPEND_LOGIC';\n```\n\n您需要直接修改 dm.ini 配置文件，添加或修改以下配置项：\n\n```\nRLOG_APPEND_LOGIC=1\n```\n\n**注意**：修改 dm.ini 后需要重启数据库服务才能生效。\n\n## 6. 数据库服务启动\n\n使用后台方式启动数据库服务（生产环境推荐）：\n\n```bash\nnohup ./dmserver /data/dameng/data/DAMENG/dm.ini \u003e /path/to/logfile.log 2\u003e\u00261 \u0026\n```\n\n启动后，可以通过以下命令确认进程是否在运行：\n\n```bash\nps -ef | grep dmserver\n```\n\n## 7. 验证配置\n\n完成上述配置后，您可以尝试在表中插入新数据，然后观察 Debezium 连接器的日志。如果配置正确，应该能够在日志中看到类似以下内容的输出：\n\n```\n[debezium-damengconnector-my-dameng-connector-2025-0226-change-event-source-coordinator] INFO io.debezium.connector.dameng.logminer.LogMinerQueryResultProcessor - 1 Rows, 0 DMLs, 0 Commits, 0 Rollbacks, 0 Inserts, 0 Updates, 0 Deletes. Processed in 0 millis. Lag:0. Offset scn:50684. Offset commit scn:null. Active transactions:0. Sleep time:3000\n```\n\n如果看到 `0 Rows` 的输出，请检查是否正确配置了所有步骤，特别是日志追加模式设置。\n\n## 8. 常见问题排查\n\n如果遇到问题，可以检查以下几点：\n\n1. 确认兼容模式设置为 2\n2. 确认数据库处于归档日志模式\n3. 确认闪回功能已开启\n4. 确认 RLOG_APPEND_LOGIC 设置为 1\n5. 确认所有需要重启服务的配置都已生效\n\n*备注：本项目不是达梦数据库官方项目，由社区维护和开发。*","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdevlive-community%2Fdameng-connector","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdevlive-community%2Fdameng-connector","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdevlive-community%2Fdameng-connector/lists"}