{"id":32942794,"url":"https://github.com/codingapi/dbstream-driver","last_synced_at":"2025-11-12T13:03:23.291Z","repository":{"id":323308588,"uuid":"1092848144","full_name":"codingapi/dbstream-driver","owner":"codingapi","description":"dbstream-driver","archived":false,"fork":false,"pushed_at":"2025-11-12T12:02:48.000Z","size":270,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-11-12T13:02:58.521Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/codingapi.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-11-09T12:42:28.000Z","updated_at":"2025-11-12T12:02:52.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/codingapi/dbstream-driver","commit_stats":null,"previous_names":["codingapi/dbstream-driver"],"tags_count":10,"template":false,"template_full_name":null,"purl":"pkg:github/codingapi/dbstream-driver","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/codingapi%2Fdbstream-driver","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/codingapi%2Fdbstream-driver/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/codingapi%2Fdbstream-driver/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/codingapi%2Fdbstream-driver/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/codingapi","download_url":"https://codeload.github.com/codingapi/dbstream-driver/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/codingapi%2Fdbstream-driver/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":284037609,"owners_count":26936683,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-11-12T02:00:06.336Z","response_time":59,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-11-12T13:01:04.381Z","updated_at":"2025-11-12T13:03:23.282Z","avatar_url":"https://github.com/codingapi.png","language":"Java","readme":"[![License](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](https://github.com/codingapi/dbstream-driver/blob/main/LICENSE)\n[![Maven Central](https://img.shields.io/maven-central/v/com.codingapi.dbstream/dbstream-driver.svg?label=Maven%20Central)](https://search.maven.org/search?q=g:%22com.codingapi.dbstream%22%20AND%20a:%22dbstream-driver%22)\n\n# dbstream-driver\n\n一个基于 JDBC 代理驱动的数据库变更事件推送框架，通过代理 JDBC 驱动实现监控数据库的持久化操作，自动捕获 INSERT/UPDATE/DELETE 操作并推送结构化数据变更事件。本框架支持JDK1.8及以上的java环境。\n\n## ✨ 核心特性\n\n### 🚀 无侵入性（Non-invasive）\n- **零代码修改**：无需修改任何业务代码，只需替换 JDBC 驱动类名\n- **透明代理**：自动委派给真实 JDBC 驱动，保持原有功能完全不变\n- **配置即用**：修改数据源配置即可启用，无需额外代码\n\n### 💡 方便性（Convenience）\n- **自动元数据扫描**：首次连接自动扫描并缓存数据库表结构、字段、主键等元数据信息\n- **内置解析器**：内置 INSERT/UPDATE/DELETE SQL 解析器，自动提取变更数据\n- **事务感知**：自动识别事务边界，支持自动提交和手动事务模式\n- **多数据源支持**：通过 jdbcKey 区分不同数据源，支持多数据源场景\n\n### 🔧 扩展性（Extensibility）\n- **监听器机制**：支持自定义 `SQLExecuteListener` 监听 SQL 执行前后事件\n- **事件推送器**：支持自定义 `DBEventPusher` 实现事件推送逻辑（如对接消息队列）\n- **插件化架构**：基于接口设计，易于扩展和定制\n\n\n## 📋 应用场景\n\n- **数据宽表查询优化**：实时同步数据到宽表，提升查询性能\n- **统一数据统计口径**：实时捕获数据变更，统一数据统计逻辑\n- **数据实时备份**：监听数据变更，实现实时数据备份\n- **数据缓存同步**：数据库变更时自动更新缓存\n- **数据变更审计**：记录所有数据变更操作，用于审计和追溯\n- **数据同步**：实现数据库之间的实时数据同步\n\n## 🔧 依赖环境\n\n- **JDK**: 8 +\n- **Maven**: 3.8 +\n\n## 🚀 快速开始\n\n### 1. 引入依赖\n\n在项目的 `pom.xml` 中添加依赖：\n\n```xml\n\u003cdependency\u003e\n    \u003cgroupId\u003ecom.codingapi.dbstream\u003c/groupId\u003e\n    \u003cartifactId\u003edbstream-driver\u003c/artifactId\u003e\n    \u003cversion\u003e${latest.version}\u003c/version\u003e\n\u003c/dependency\u003e\n```\n\n### 2. 配置数据源\n\n将数据源驱动类配置为 `com.codingapi.dbstream.driver.DBStreamProxyDriver`，URL 保持原有 JDBC URL 不变：\n\n**Spring Boot 配置示例（application.properties）：**\n\n```properties\nspring.datasource.driver-class-name=com.codingapi.dbstream.driver.DBStreamProxyDriver\nspring.datasource.url=jdbc:mysql://localhost:3306/example?createDatabaseIfNotExist=true\u0026useUnicode=true\u0026characterEncoding=utf-8\u0026useSSL=false\u0026serverTimezone=Asia/Shanghai\u0026allowPublicKeyRetrieval=true\nspring.datasource.username=root\nspring.datasource.password=12345678\n```\n\n**说明**：代理驱动会在运行时自动识别 JDBC URL 并委派给真实 JDBC 驱动（如 MySQL、PostgreSQL、H2 等），无需额外配置。\n\n### 3. 订阅数据库变更事件（可选）\n\n实现并注册 `DBEventPusher` 接口，接收结构化的数据库变更事件：\n\n```java\n\nimport com.codingapi.dbstream.event.DBEvent;\nimport com.codingapi.dbstream.event.DBEventPusher;\n\n// 在应用启动时注册事件推送器（如 @PostConstruct、@Configuration 等）\nDBStreamContext.getInstance().\n\naddEventPusher(new DBEventPusher() {\n    @Override\n    public void push (List \u003c DBEvent \u003e events) {\n        // 处理数据库变更事件\n        for (DBEvent event : events) {\n            System.out.println(\"表名: \" + event.getTableName());\n            System.out.println(\"操作类型: \" + event.getType()); // INSERT/UPDATE/DELETE\n            System.out.println(\"变更数据: \" + event.getData());\n            System.out.println(\"主键: \" + event.getPrimaryKeys());\n            System.out.println(\"事务标识: \" + event.getTransactionKey());\n            System.out.println(\"时间戳: \" + event.getTimestamp());\n\n            // 可以对接消息队列（如 Kafka、RocketMQ 等）\n            // kafkaProducer.send(event);\n        }\n    }\n});\n```\n\n### 4. 订阅 SQL 执行回调（可选）\n\n实现并注册 `SQLExecuteListener`，可以获取原始 SQL 和参数信息：\n\n```java\nimport com.codingapi.dbstream.DBStreamContext;\nimport com.codingapi.dbstream.interceptor.SQLExecuteState;\nimport com.codingapi.dbstream.listener.SQLExecuteListener;\n\npublic class MySQLListener implements SQLExecuteListener {\n\n    @Override\n    public int order() {\n        return 0;\n    }\n    \n    @Override\n    public void before(SQLExecuteState executeState) {\n        System.out.println(\"执行前 - SQL: \" + executeState.getSql());\n        System.out.println(\"执行前 - 参数: \" + executeState.getListParams());\n    }\n\n    @Override\n    public void after(SQLExecuteState executeState, Object result) {\n        System.out.println(\"执行后 - SQL: \" + executeState.getSql());\n        System.out.println(\"执行后 - 参数: \" + executeState.getListParams());\n        System.out.println(\"执行后 - 耗时: \" + executeState.getExecuteTimestamp());\n        System.out.println(\"执行后 - 结果: \" + result);\n    }\n}\n\n// 注册监听器\nDBStreamContext.getInstance().addListener(new MySQLListener());\n```\n\n### 5. 通过设置DBTableSupportProvider订阅对那些表进行监听（可选）\n\n```\n \nimport com.codingapi.dbstream.scanner.DbTable;\n\nimport java.util.Properties;\n\n/**\n * 默认DB事件判断类 \n * 规则为满足条件的全部支持\n */\npublic class DefaultDBEventSupporter implements DBEventSupporter {\n\n    @Override\n    public boolean support(Properties info, DbTable dbTable) {\n        return true;\n    }\n}\n\n\n// 添加 SQL 表执行判断\nDBStreamContext.getInstance().setDbEventSupporter(new DefaultDBEventSupporter());\n```\n\n根据表名等信息来决定是否进行数据事件解析。仅当返回true的才会进行事件推送。DefaultDBTableSupportProvider为默认的实现机制。\n\n### 6. 查看表或情况表缓存数据（可选）\n\n在项目启动以后，会在项目的根路径下创建.dbstream文件夹，文件夹中存储的内容为数据库的表扫描缓存数据。  \n文件夹的名称为jdbcKey的字段，文件夹下的内容为缓存的表结构信息，当表结构发生变化以后可以删除对应的文件进行更新。  \n也可以在系统中通过执行 `DBStreamContext.getInstance().clear(String jdbcKey);`进行清空元数据信息。   \njdbcKey是通过sha256(jdbcUrl+schema)计算得来。\n```\n.\n└── beefae7e00deb825a3a591ab7a22791a4df799afba9fed71f8b549665508c7ee\n    └── M_USER\n\n```\n\n### 7. 主键关系手动维护（可选）\n\n在数据库中存在不存在物理的主键字段，但是存在业务主键字段，可通过手动配置的方式，手动标记字段为主键字段。在扫描后的配置文件下增加对应表名的.key文件，例如：M_USER.key\n```\n.\n└── beefae7e00deb825a3a591ab7a22791a4df799afba9fed71f8b549665508c7ee\n    ├── M_USER\n    ├── M_USER.key\n    └── M_USER_2\n```\n\n写法如下，填写字段的名称，多个用英文,分割。\n```\nUSERNAME,ID\n```\n\n## 📖 API 文档\n\n### DBStreamContext\n\n框架的核心上下文类，提供所有对外能力：\n\n#### 事件推送相关\n\n```java\n// 添加数据库事件推送器\nDBStreamContext.getInstance().addEventPusher(DBEventPusher pusher);\n```\n\n#### 监听器相关\n\n```java\n// 添加 SQL 执行监听器\nDBStreamContext.getInstance().addListener(SQLExecuteListener listener);\n```\n\n#### 数据库表支持判断\n\n```java\n// 添加 SQL 表执行判断\nDBStreamContext.getInstance().setDbEventSupporter(DBEventSupporter dbEventSupporter);\n```\n\n#### 元数据管理\n\n```java\n// 获取所有数据库的元数据信息列表\nList\u003cDBMetaData\u003e metaDataList = DBStreamContext.getInstance().metaDataList();\n\n// 通过 jdbcKey 获取指定数据库的元数据信息\nDBMetaData metaData = DBStreamContext.getInstance().getMetaData(String jdbcKey);\n\n// 主动更新表的元数据信息，在使用改变时将会先更新metadata然后再执行业务。（动态更新表的元数据）\nmetaData.addUpdateTableMateList(String tableName);\n\n// 获取所有已缓存的数据库连接 jdbcKey 列表\nList\u003cString\u003e dbKeys = DBStreamContext.getInstance().loadDbKeys();\n\n// 清理指定数据库的元数据缓存（清空后下次访问会自动重新加载，动态更新表的元数据）\nDBStreamContext.getInstance().clear(String jdbcKey);\n\n// 清理所有数据库的元数据缓存（动态更新表的元数据）\nDBStreamContext.getInstance().clearAll();\n```\n\n### DBEvent 事件模型\n\n数据库变更事件包含以下信息：\n\n- `tableName`: 表名\n- `type`: 操作类型（INSERT/UPDATE/DELETE）\n- `data`: 变更的数据（Map 格式，key 为字段名，value 为字段值）\n- `primaryKeys`: 主键列表\n- `jdbcUrl`: 数据库连接 URL\n- `transactionKey`: 事务标识（同一事务内的操作共享相同标识）\n- `timestamp`: 事件产生时间戳\n- `pushTimestamp`: 事件推送时间戳\n\n## 🧪 运行测试\n\n项目已包含完整的单元测试示例，运行测试：\n\n```bash\nmvn clean test -P travis\n```\n\n## ⚠️ 注意事项\n\n1. **事件推送时机**：\n   - 仅在使用 `Statement`/`PreparedStatement` 执行 INSERT/UPDATE/DELETE 操作时才会产生事件\n   - SELECT 查询操作不会推送事件\n   - 事件推送是同步触发回调，请在实现中避免耗时阻塞，必要时交给异步处理\n\n2. **事务支持**：\n   - 框架自动识别事务边界，支持自动提交和手动事务模式\n   - 手动事务模式下，事件会在 `commit()` 时批量推送\n   - 事务回滚时，相关事件会被丢弃\n\n3. **使用场景限制**：\n   - 数据库表必须有主键的定义，在DELETE事件需要明确主键信息，主键物理表不存在时可通过外部key文件配置的方式添加。\n   - 若INSERT INTO SELECT 语句中，采用主键自增模式，受限于JDBC的支持将无法解析到自增ID，建议修改单条保存或修改ID为手动传递。\n\n4. **元数据缓存**：\n   - 数据库元数据会在首次连接时自动扫描并缓存\n   - 如果数据库表结构发生变化，可以调用 `clear()`或`metaData.addUpdateTableMateList(String tableName);` 方法清理缓存，下次访问时会自动重新加载\n\n## 📄 许可证\n\n本项目采用 [Apache License 2.0](./LICENSE) 许可证。\n\n## 🤝 贡献\n\n欢迎提交 Issue 和 Pull Request！\n\n\n## 📞 联系方式\n\n- 项目地址: https://github.com/codingapi/dbstream-driver\n- 问题反馈: https://github.com/codingapi/dbstream-driver/issues\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcodingapi%2Fdbstream-driver","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcodingapi%2Fdbstream-driver","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcodingapi%2Fdbstream-driver/lists"}