{"id":19993115,"url":"https://github.com/kekingcn/kkbinlog","last_synced_at":"2025-05-04T12:30:49.239Z","repository":{"id":45034720,"uuid":"130146472","full_name":"kekingcn/kkbinlog","owner":"kekingcn","description":"支持MySQL、MongoDB数据变更订阅分发 ","archived":false,"fork":false,"pushed_at":"2022-01-13T05:28:31.000Z","size":338,"stargazers_count":147,"open_issues_count":1,"forks_count":54,"subscribers_count":15,"default_branch":"master","last_synced_at":"2024-11-13T04:55:58.542Z","etag":null,"topics":["binlog","binlog2sql","etcd","kafka","mongodb","mysql","oplog","oplog-tailing","rabbitmq","spring","spring-boot"],"latest_commit_sha":null,"homepage":"","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/kekingcn.png","metadata":{"files":{"readme":"readme.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2018-04-19T02:14:03.000Z","updated_at":"2024-08-24T01:56:48.000Z","dependencies_parsed_at":"2022-08-27T16:31:04.422Z","dependency_job_id":null,"html_url":"https://github.com/kekingcn/kkbinlog","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kekingcn%2Fkkbinlog","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kekingcn%2Fkkbinlog/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kekingcn%2Fkkbinlog/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kekingcn%2Fkkbinlog/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/kekingcn","download_url":"https://codeload.github.com/kekingcn/kkbinlog/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":252334121,"owners_count":21731338,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["binlog","binlog2sql","etcd","kafka","mongodb","mysql","oplog","oplog-tailing","rabbitmq","spring","spring-boot"],"created_at":"2024-11-13T04:52:27.967Z","updated_at":"2025-05-04T12:30:44.231Z","avatar_url":"https://github.com/kekingcn.png","language":"Java","funding_links":[],"categories":["Java"],"sub_categories":[],"readme":"# 1 概述\n\u003emysql、MongoDB数据变动监听分发\n\u003e本项目意在简化监听mysql、MongoDB数据库的不同表的各种数据变动\n\u003e项目依赖redis,mysql\n\u003e使用场景：刷新缓存、异构系统...\n\n![示意图](Picture1.png)\n# 2 使用方式\n\u003e从bin-log-distributor-app到client数据分发方式的默认实现为redis，如果要开发基于其他的比如mq，只需要分别实现`bin-log-distributor-pub`下的`DataPublisher`接口与`bin-log-distributor-client`下的即可\n## 2.1 服务端\n服务端是项目中`bin-log-distributor-app`模块，在[mysql-binlog-connector-java](服务端是项目中`bin-log-distributor-app`模块，在[mysql-binlog-connector-java]()提供了监听mysql数据库二进制日志的功能\n)基础上提供了监听mysql数据库二进制日志并进行分发的功能\n\n### 2.1.1 参考配置\n```\n# redis地址\nspring.redisson.address=redis://192.168.1.204:6379\n\n# mysql日志同步账户,\nbinaryLog.host = 192.168.1.204\nbinaryLog.port = 3306\nbinaryLog.username = aa\nbinaryLog.password = aa\nbinaryLog.serverId = 1\n\n# 读取列名，进行映射\nspring.datasource.url = jdbc:mysql://${binaryLog.host}:${binaryLog.port}/mysql?autoReconnect=true\u0026amp;useUnicode=true\u0026amp;characterEncoding=utf-8\nspring.datasource.username = ${binaryLog.username}\nspring.datasource.password = ${binaryLog.password}\nspring.datasource.driverClassName = com.mysql.jdbc.Driver\n\n# rabbitmq连接相关信息(如果只有redis客户端可以不需要使用)\nspring.rabbit.host = 192.168.1.204\nspring.rabbit.port = 5672\nspring.rabbit.username = aa\nspring.rabbit.password = aa\nspring.rabbit.virtualHost = /binlog\n```\n\n### 2.1.2 启动方式\n编译打包项目，直接通过（java -jar）启动bin-log-distributor-app-${version}-SNAPSHOT.jar，可参考[spring boot手册](https://docs.spring.io/spring-boot/docs/current-SNAPSHOT/reference/htmlsingle/#using-boot-running-your-application)\n\n## 2.2 客户端\n### 2.2.1 redis客户端\n#### 2.2.1.1 引入依赖包\n```\n\u003cdependency\u003e\n    \u003cgroupId\u003ecn.keking.project\u003c/groupId\u003e\n    \u003cartifactId\u003ebin-log-distributor-client-redis\u003c/artifactId\u003e\n    \u003cversion\u003e${version}\u003c/version\u003e\n\u003c/dependency\u003e\n```\n\n#### 2.2.1.2 添加客户端配置\n```\n#自动注册客户端(2.1中服务端的地址)\ndatabaseEventServerUrl=http://localhost:8885/client/addAll\n#本应用命名\nappName=lbt-service-ext-redis\n# redis地址\nspring.redisson.address=redis://192.168.1.204:6379\n```\n\n#### 2.2.1.3 写handler，实现 DatabaseEventHandler 接口，并加上注解 @HandleDatabaseEven \n```\n/**\n * LockLevel为保持顺序的级别,None为默认\n * TABLE -\u003e 同表按顺序执行\n * COLUMN -\u003e 某列值一致的按顺序执行\n * NONE -\u003e 无序\n */\n@Service\n@HandleDatabaseEvent(database = \"Rana_G\", table = \"PTP_PROJ_REP_HIS\", events = {DatabaseEvent.UPDATE_ROWS, DatabaseEvent.DELETE_ROWS},lockLevel = LockLevel.TABLE)\npublic class ProjRepHisDatabaseEventHandler implements DatabaseEventHandler {\n    private static final Logger logger = LoggerFactory.getLogger(ProjRepHisDatabaseEventHandler.class);\n\n    @Override\n    public void handle(EventBaseDTO eventBaseDTO) {\n        logger.info(JSON.toJSONString(eventBaseDTO));\n        // todo 在这里写相关逻辑\n    }\n}\n```\n\n#### 2.2.1.4 启动监听，当2里的handler是由容器管理时需要通过registerHandler(projRepHisDatabaseEventHandler)手动注册，如果不需要容器管理，可以直接通过autoScanHandler()自动扫描添加\n```\n@Component\npublic class DatabaseEventListener {\n    private static Logger logger = LoggerFactory.getLogger(DatabaseEventListener.class);\n    \n    @Autowired\n    private RedissonClient redissonClient;\n\n    @Value(\"#{env.databaseEventServerUrl}\")\n    private String serverUrl;\n    \n    @Value(\"#{env.appName}\")\n    private String appName;\n    \n    //将第3步里的service注入\n    @Autowired\n    private ProjRepHisDatabaseEventHandler projRepHisDatabaseEventHandler; \n\n    @PostConstruct\n    public void start() {\n        //初始化订阅的实现\n        DataSubscriber dataSubscriber = new DataSubscriberRedisImpl(redissonClient);\n        new BinLogDistributorClient(appName, dataSubscriber)\n                //在binlog中注册handler\n                .setQueueType(ClientInfo.QUEUE_TYPE_REDIS) \n                .registerHandler(projRepHisDatabaseEventHandler)\n                .setServerUrl(serverUrl).autoRegisterClient().start();\n    }\n}\n```\n\n### 2.2.2 rabbitmq客户端\n#### 2.2.2.1 引入依赖包\n```\n\u003cdependency\u003e\n    \u003cgroupId\u003ecn.keking.project\u003c/groupId\u003e\n    \u003cartifactId\u003ebin-log-distributor-client-rabbitmq\u003c/artifactId\u003e\n    \u003cversion\u003e${version}\u003c/version\u003e\n\u003c/dependency\u003e\n```\n\n#### 2.2.2.2 添加客户端配置\n```\n#自动注册客户端(2.1中服务端的地址)\ndatabaseEventServerUrl=http://localhost:8885/client/addAll\n#本应用命名\nappName=lbt-service-ext-rabbit\n# redis地址(rabbitmq实现也要使用redis作为分布式锁)\nspring.redisson.address=redis://192.168.1.204:6379\n# RabbitMQ配置\nspring.rabbit.host=192.168.1.204\nspring.rabbit.port=5672\nspring.rabbit.username=aa\nspring.rabbit.password=aa\nspring.rabbit.virtualHost=/binlog\nspring.rabbit.apiUrl=http://192.168.1.204:15672/api/\n```\n\n#### 2.2.2.3 写handler，实现 DatabaseEventHandler 接口，并加上注解 @HandleDatabaseEven \n```\n/**\n * LockLevel为保持顺序的级别,None为默认\n * TABLE -\u003e 同表按顺序执行\n * COLUMN -\u003e 某列值一致的按顺序执行\n * NONE -\u003e 无序\n */\n@Service\n@HandleDatabaseEvent(database = \"Rana_G\", table = \"PTP_PROJ_REP_HIS\", events = {DatabaseEvent.UPDATE_ROWS, DatabaseEvent.DELETE_ROWS},lockLevel = LockLevel.TABLE)\npublic class ProjRepHisDatabaseEventHandler implements DatabaseEventHandler {\n    private static final Logger logger = LoggerFactory.getLogger(ProjRepHisDatabaseEventHandler.class);\n\n    @Override\n    public void handle(EventBaseDTO eventBaseDTO) {\n        logger.info(JSON.toJSONString(eventBaseDTO));\n        // todo 在这里写相关逻辑\n    }\n}\n```\n\n#### 2.2.2.4 启动监听，当2里的handler是由容器管理时需要通过registerHandler(projRepHisDatabaseEventHandler)手动注册，如果不需要容器管理，可以直接通过autoScanHandler()自动扫描添加\n```\n@Component\npublic class DatabaseEventListener {\n    private static Logger logger = LoggerFactory.getLogger(DatabaseEventListener.class);\n    \n    @Autowired\n    private RedissonClient redissonClient;\n\n    @Value(\"#{env.databaseEventServerUrl}\")\n    private String serverUrl;\n    \n    @Value(\"#{env.appName}\")\n    private String appName;\n    //将第3步里的service注入\n    \n    @Autowired\n    private ProjRepHisDatabaseEventHandler projRepHisDatabaseEventHandler; \n    \n    //org.springframework.amqp.rabbit.connection.ConnectionFactory 自行创建spring bean\n    @Autowired\n    private ConnectionFactory connectionFactory; \n\n    //com.rabbitmq.http.client.Client 自行创建spring bean\n    @Autowired\n    private Client rabbitHttpClient; \n\n    @PostConstruct\n    public void start() {\n        //初始化订阅的实现\n        DataSubscriber dataSubscriber = new DataSubscriberRabbitMQImpl(connectionFactory, rabbitHttpClient, redissonClient);\n        new BinLogDistributorClient(appName, dataSubscriber)\n                //默认为redis实现，使用mq实现这里一定要指定QueueType\n                .setQueueType(ClientInfo.QUEUE_TYPE_RABBIT)\n                .registerHandler(projRepHisDatabaseEventHandler)\n                .setServerUrl(serverUrl).autoRegisterClient().start();\n    }\n}\n```\n\n# 2.3 前端管理模块\n\u003e前端管理服务模块是基于vue的管理各个应用监听状况的管理界面\n#### 新增MySQL数据源\n\n#### 数据源管理\n![](https://oscimg.oschina.net/oscnet/up-d00682b927661b6a9da13536546d89ffc1e.png)\n\n#### 日志进度\n![](https://oscimg.oschina.net/oscnet/up-0b311ee0cf6b0851a6ee9de998f0d03fede.png)\n\n#### 分发队列监控\n![](https://oscimg.oschina.net/oscnet/up-828e4558f279bed409ca583572f14aa3bb3.png)\n\n#### 新增数据订阅\n![](https://oscimg.oschina.net/oscnet/up-7c094efbc9fdcccc8d5e4f1e6b40b346ea8.png)\n# 3 其他\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkekingcn%2Fkkbinlog","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fkekingcn%2Fkkbinlog","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkekingcn%2Fkkbinlog/lists"}