{"id":19291681,"url":"https://github.com/dianping/puma","last_synced_at":"2025-04-10T03:56:46.896Z","repository":{"id":5816340,"uuid":"7031510","full_name":"dianping/puma","owner":"dianping","description":null,"archived":false,"fork":false,"pushed_at":"2016-04-11T04:02:19.000Z","size":26527,"stargazers_count":180,"open_issues_count":2,"forks_count":108,"subscribers_count":37,"default_branch":"master","last_synced_at":"2025-04-10T03:56:41.331Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/dianping.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2012-12-06T07:16:29.000Z","updated_at":"2024-12-09T10:53:39.000Z","dependencies_parsed_at":"2022-08-31T16:44:00.957Z","dependency_job_id":null,"html_url":"https://github.com/dianping/puma","commit_stats":null,"previous_names":[],"tags_count":2,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dianping%2Fpuma","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dianping%2Fpuma/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dianping%2Fpuma/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dianping%2Fpuma/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/dianping","download_url":"https://codeload.github.com/dianping/puma/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248154999,"owners_count":21056542,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-09T22:27:06.547Z","updated_at":"2025-04-10T03:56:46.873Z","avatar_url":"https://github.com/dianping.png","language":"Java","funding_links":[],"categories":["中间件","数据库中间件"],"sub_categories":[],"readme":"### 项目简介\n\n随着网站业务不断发展，各业务对数据的实时性，数据库的可用性要求越来越高。\n\n本系统可以实时获得数据库的变更并通过消息方式发布出来，供各业务线订阅。\n\n同时，本系统还会实现数据库同步（同构和异构），以满足数据库冗余备份，数据迁移的需求。\n\n\u0026nbsp;\n\n### 项目依赖\n\n```\n\u003cdependency\u003e\n    \u003cgroupId\u003ecom.dianping.puma\u003c/groupId\u003e\n    \u003cartifactId\u003epuma-client\u003c/artifactId\u003e\n    \u003cversion\u003e${version}\u003c/version\u003e\n\u003c/dependency\u003e\n```\n\n最新版本为`2.0.0`\n\n\u0026nbsp;\n\n### 接入申请\n发邮件给`xiaotian.li@dianping.com`。\n\n未申请的 ClientName 无法正常使用。\n\n邮件标题：PumaClient 申请\n\n邮件内容：\n\n* ClientName\n* 需要监听的库和表\n* 上线时间\n\n\u0026nbsp;\n\n**申请完成在本地开发时，可以用`test`结尾的`ClientName`来做开发，避免相互影响。**\n\n\u0026nbsp;\n\n### 使用方法简介\n\n```\nPumaClient client = new PumaClientConfig()\n\t.setClientName(\"your-client-name\")\n\t.setDatabase(\"database\")\n\t.setTables(Lists.newArrayList(\"table0\", \"table1\"))\n\t.buildClusterPumaClient();\n\nwhile(!Thread.currentThread().isInterrupted()) {\n\ttry {\n\t\tBinlogMessage binlogMessage = client.get(10, 1, TimeUnit.SECOND);\n\t\t//todo: 处理数据\n\t\tclient.ack(binlogMessage.getLastBinlogInfo());\n\t} catch(Exception e) {\n\t\t//这里的异常主要是用来打点的，便于及时发现问题\n\t}\n}\n```\n\n`PumaClient`所有操作都是同步操作，并且线程不安全。\n\n**如果是`job`项目可以直接写上述代码。但是如果是在`service`或者`web`项目中，一定要新启一个线程来跑上述代码。**\n\n\u0026nbsp;\n\n### PumaClientConfig API \u0026\u0026 PumaClient API\n\n代码即是文档。\n\n建议直接在项目中看源码，或者到这里看源码：\n\n* [PumaClientConfig](http://code.dianpingoa.com/arch/puma/blob/master/puma-client/src/main/java/com/dianping/puma/api/PumaClientConfig.java)\n* [PumaClient](http://code.dianpingoa.com/arch/puma/blob/master/puma-client/src/main/java/com/dianping/puma/api/PumaClient.java)\n\n\u0026nbsp;\n\n### 最佳实践\n```\nimport com.dianping.cat.Cat;\nimport com.dianping.puma.api.PumaClient;\nimport com.dianping.puma.api.PumaClientConfig;\nimport com.dianping.puma.api.PumaClientException;\nimport com.dianping.puma.core.dto.BinlogMessage;\nimport com.dianping.puma.core.event.Event;\nimport com.dianping.puma.core.event.RowChangedEvent;\nimport com.google.common.collect.Lists;\n\nimport java.util.concurrent.TimeUnit;\n\npublic class Example1 {\n\n    /**\n     * 假设这是一个 web 或 service 项目\n     * 可以尝试运行多个,并随机关掉其中正在运行的那个,来模拟 failover\n     *\n     * @param args\n     */\n    public static final void main(String... args) {\n\n        //不要阻塞主线程,需要自己令起线程\n        Thread pumaClientThread = new Thread(new Runnable() {\n            @Override\n            public void run() {\n\n                PumaClient client = new PumaClientConfig()\n                        .setClientName(\"puma-client-example-test\")\n                        .setDatabase(\"Puma\")\n                        .setTables(Lists.newArrayList(\"PumaServer\"))\n                        .buildClusterPumaClient();\n\n                while (!Thread.currentThread().isInterrupted()) {\n                    try {\n                        BinlogMessage message = client.get(100, 1, TimeUnit.SECONDS);\n\n                        for (Event event : message.getBinlogEvents()) {\n                            if (event instanceof RowChangedEvent) {\n\n                                RowChangedEvent rowChangedEvent = (RowChangedEvent) event;\n                                System.out.println(rowChangedEvent.toString());\n\n                                //todo: 处理数据\n                            }\n                        }\n\n                        client.ack(message.getLastBinlogInfo());\n                    } catch (PumaClientException e) {\n                        Cat.logError(e.getMessage(), e);\n                    }\n                }\n            }\n        });\n\n\n        pumaClientThread.setName(\"puma-client-example-test\");\n        pumaClientThread.setDaemon(true);\n        pumaClientThread.start();\n\n\n        while (true) {\n            try {\n                Thread.sleep(1);\n            } catch (InterruptedException ignore) {\n            }\n        }\n    }\n}\n```\n\n\u0026nbsp;\n\n### FAQ\n\n\u0026nbsp;\n\n#### ack 方法有什么用\n\n`PumaClient`的同步进度会通过`ack`方法同步到云端，重启时，会读取最后一次`ack`的位置来进行同步。\n\n\u0026nbsp;\n\n#### rollback 方法有什么用\n`rollback`方法可以用来强制定位后续数据的起始位置。\n\n例如 DW 团队不需要实时监听数据，而是需要在每天凌晨同步昨天的数据，那么可以在启动的时候调用`rollback`方法将任务的起始位置定位到昨天0点，然后开始同步。\n\n使用方法是：`client.rollback(new BinlogInfo().setTimestamp(1447800000))`\n\n备注：这里的`timestamp`是秒数，不是毫秒数。(`mysql`底层`binlog`只精确到秒)","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdianping%2Fpuma","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdianping%2Fpuma","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdianping%2Fpuma/lists"}