{"id":16714179,"url":"https://github.com/flier/zkpipe","last_synced_at":"2025-03-15T07:11:42.163Z","repository":{"id":140004648,"uuid":"84268340","full_name":"flier/zkpipe","owner":"flier","description":"Consume Zookeeper binary log to sync filtered transactions to Kafka topic.","archived":false,"fork":false,"pushed_at":"2017-03-08T02:29:57.000Z","size":100,"stargazers_count":3,"open_issues_count":0,"forks_count":1,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-01-21T22:09:41.089Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Scala","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/flier.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2017-03-08T02:19:08.000Z","updated_at":"2023-07-28T15:55:56.000Z","dependencies_parsed_at":"2023-05-01T03:49:21.297Z","dependency_job_id":null,"html_url":"https://github.com/flier/zkpipe","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/flier%2Fzkpipe","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/flier%2Fzkpipe/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/flier%2Fzkpipe/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/flier%2Fzkpipe/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/flier","download_url":"https://codeload.github.com/flier/zkpipe/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":243695589,"owners_count":20332629,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-12T20:49:46.821Z","updated_at":"2025-03-15T07:11:42.143Z","avatar_url":"https://github.com/flier.png","language":"Scala","funding_links":[],"categories":[],"sub_categories":[],"readme":"# zkpipe\n\nConsume Zookeeper binary log to sync filtered transactions to Kafka topic.\n\n# Features\n\n- [Sync](#sync) transactions from `Zookeeper` binary log/snapshot files to `Kafka` topic. \n- [Watch](#watch) log directory and continue to sync changed transaction logs. \n- [Filter](#filter) transaction with `zxid range`, `path prefix` or `path pattern`.\n- [Encode](#encoding-format) transaction as `JSON`, `XML`, `ProtoBuf` or `Raw` format.\n- Resume [from latest](#from-latest) transaction in the `Kafka` topic.\n- Report [metrics](#metrics) to `Prometheus`, `Graphite` or `Ganglia`.\n- Low memory footprint base on the streaming mode\n\n# Usage\n\n[watch](#watch) and [sync](#sync) command consume changed or [filtered](#filter) transactions from the Zookeeper binary log files, \nsend [encoded](#encoding-format) message to Kafka topic that defined in the `--kafka-uri` option.\n\n## Watch \n`watch` log directory and continue to sync changed transactions to kafka that defined in the `--kafka-uri` option.\n\n\u003e zkpipe watch \\\u003clog dir\\\u003e --kafka-uri \\\u003curi\\\u003e\n\n```bash\n$ zkpipe watch /usr/local/var/run/zookeeper/data/version-2/ --kafka-uri=kafka://localhost/zkpipe \n```\n\n### From Latest\nBy default, `zkpipe` will auto resume the sync progress from the latest transaction in the Kafka topic, \n`--from-latest` option skip the sync progress to the latest zxid in the Zookeeper binary logs. \n\n\u003e zkpipe watch \\\u003clog dir\\\u003e [options] --from-latest\n\n```bash\n$ zkpipe watch /usr/local/var/run/zookeeper/data/version-2/ --kafka-uri=kafka://localhost/zkpipe --from-latest\n```\n\n## Sync \n`sync` transactions from the binary log files to Kafka topic.\n\n\u003e zkpipe sync \\\u003clog/snapshot files\\\u003e\n\n```bash\n$ zkpipe sync /usr/local/var/run/zookeeper/data/version-2/log.c7\n```\n\nWhen the syntax is \"glob\" then the `log files` representation of the path is matched using a limited pattern language that resembles regular expressions but with a simpler syntax. For example:\n\n- *.java\tMatches a path that represents a file name ending in .java\n- *.*\tMatches file names containing a dot\n- *.{java,class}\tMatches file names ending with .java or .class\n- foo.?\tMatches file names starting with foo. and a single character extension\n- /home/*/*\tMatches /home/gus/data on UNIX platforms\n- /home/**\tMatches /home/gus and /home/gus/data on UNIX platforms\n- C:\\\\*\tMatches C:\\foo and C:\\bar on the Windows platform (note that the backslash is escaped; as a string literal in the Java Language the pattern would be \"C:\\\\\\\\*\")\n\nPlease check [FileSystem.getPathMatcher](https://docs.oracle.com/javase/7/docs/api/java/nio/file/FileSystem.html#getPathMatcher(java.lang.String)) document for more detail. \n\n## Filter \n\n`watch` or `sync` command can filter transactions with filters.\n\n### zxid Range\n\n`--range \u003czxid:zxid\u003e` option filter transaction in the `zxid` range\n- `\u003clow\u003e:` matches transactions `zxid` lager than `low` value\n- `:\u003chigh\u003e` matches transactions `zxid` less than `high` value\n- `\u003clow\u003e:\u003chigh\u003e` matches transactions `zxid` between `low` and `high` value\n- `:` matches all transactions\n\n\u003e zkpipe [options] --range 238:250\n\n### Path Prefix\n\n`--prefix \u003cpath\u003e` option filter transaction act on the node that have same path prefix.\n\n\u003e zkpipe [options] --prefix /broker\n\n### Match Pattern\n\n`--match \u003cpattern\u003e` option filter transaction act on the node that path matches pattern in the regular expression. \n\n\u003e zkpipe [options] --match ^/broker/.*\n\n### Session ID\n\n`--session-id \u003cid\u003e` option filter transaction with the session ID\n\n\u003e zkpipe [options] --session-id 97295928260820996\n\n### Ignore Session Events\n\n`--ignore-session` option ignore all the Zookeeper session events, including CreateSession, CloseSession etc.\n\n\u003e zkpipe [options] --ignore-session\n\n## Encoding Format\n\n`--encode \u003cformat\u003e` support the following encoding\n\n- `pb` encode transaction in ProtoBuf format\n- `json` encode transaction in JSON format\n- `xml` encode transaction in XML format\n- `raw` transaction in raw Zookeeper format \n\n\u003e zkpipe [options] --encode pb\n\n# Metrics\n\n`zkpipe` will serve metrics for Prometheus scrape or reports metrics to Graphite or Ganglia server.\n\n## Prometheus\n\n`zkpipe` serve metrics for [Prometheus](https://prometheus.io/) scrape in pull mode, that defined in the `--metrics-uri \u003curi\u003e` option defined. \n\n\u003e zkpipe watch \\\u003clog dir\\\u003e [options] --metrics-uri http://\u003chost\u003e\\[:port\\]/\u003ctopic\u003e\n\nWhen deploy `zkpipe` behind firewall or without public IP address, use `--push-gateway \u003caddr\u003e` option to push metrics to the Prometheus [push gateway](https://prometheus.io/docs/practices/pushing/).\n\n\u003e zkpipe watch \\\u003clog dir\\\u003e [options] --push-gateway \u003chost\u003e\\[:port\\]\n\n## Graphite\n\n`zkpipe` report metrics to the [Graphite](https://graphiteapp.org/) server that defined in the `--report-uri \u003curi\u003e` option.\n\n\u003e zkpipe watch \\\u003clog dir\\\u003e --kafka-uri \\\u003curi\\\u003e --report-uri graphite://\u003chost\u003e\\[:port\\]\n\nThe [pickle protocol](http://graphite.readthedocs.io/en/latest/feeding-carbon.html#the-pickle-protocol) is a much more efficient take on the plaintext protocol, and supports sending batches of metrics.\n\n\u003e zkpipe watch \\\u003clog dir\\\u003e [options] --report-uri pickle://\u003chost\u003e\\[:port\\]\n\n## Ganglia\n\n`zkpipe` report metrics to the [Ganglia](http://ganglia.info/) server that defined in the `--report-uri \u003curi\u003e` option.\n\n\u003e zkpipe watch \\\u003clog dir\\\u003e [options] --report-uri ganglia://\u003chost\u003e\\[:port\\]\n\n# Records\n\n`zkpipe` support all the Zookeeper transaction type, please check ProtoBuf schema for more detail.\n\n## Create Session\n```json\n{\n  \"cxid\": 0,\n  \"record\": {\n    \"create-session\": {\n      \"timeout\": 30000\n    }\n  },\n  \"session\": 97250703064170497,\n  \"time\": 1484217490213,\n  \"type\": \"CreateSession\",\n  \"zxid\": 10\n}\n```\n```xml\n\u003crecord\nsession=\"97250703064170497\" cxid=\"0\" zxid=\"10\" time=\"1484217490213\" type=\"CreateSession\"\u003e\n  \u003ccreate-session timeout=\"30000\"/\u003e\n\u003c/record\u003e\n```\n## Create Node\n```json\n{\n  \"cxid\": 5,\n  \"path\": \"/test/abc\",\n  \"record\": {\n    \"create\": {\n      \"acl\": [\n        {\n          \"id\": \"anyone\",\n          \"perms\": 31,\n          \"scheme\": \"world\"\n        }\n      ],\n      \"data\": \"\",\n      \"ephemeral\": false,\n      \"parentCVersion\": 1,\n      \"path\": \"/test/abc\"\n    }\n  },\n  \"session\": 97223867455045632,\n  \"time\": 1483674202631,\n  \"type\": \"Create\",\n  \"zxid\": 3\n}\n```\n```xml\n\u003crecord\nsession=\"97223867455045632\" cxid=\"5\" zxid=\"3\" time=\"1483674202631\" path=\"/test/abc\" type=\"Create\"\u003e\n  \u003ccreate path=\"/test/abc\" ephemeral=\"false\" parentCVersion=\"1\"\u003e\n    \u003cdata\u003e\u003c![CDATA[]]\u003e\u003c/data\u003e\n    \u003cacl scheme=\"world\" id=\"anyone\" perms=\"31\"/\u003e\n  \u003c/create\u003e\n\u003c/record\u003e\n```\n\n## Set Data\n```json\n{\n  \"cxid\": 2,\n  \"path\": \"/test/abc\",\n  \"record\": {\n    \"set-data\": {\n      \"data\": \"aGVsbG8=\",\n      \"path\": \"/test/abc\",\n      \"version\": 3\n    }\n  },\n  \"session\": 97250703064170497,\n  \"time\": 1484218456704,\n  \"type\": \"SetData\",\n  \"zxid\": 12\n}\n```\n```xml\n\u003crecord\nsession=\"97250703064170497\" cxid=\"2\" zxid=\"12\" time=\"1484218456704\" path=\"/test/abc\" type=\"SetData\"\u003e\n  \u003cset-data path=\"/test/abc\" version=\"3\"\u003e\n    \u003cdata\u003e\u003c![CDATA[aGVsbG8=]]\u003e\u003c/data\u003e\n  \u003c/set-data\u003e\n\u003c/record\u003e\n```\n\n# Build\n\nFirst, use `compile` command to generate source codes and compile classes.\n\n```bash\n$ sbt compile\n```\n\nAnd then, use `assembly` command to package all the code and dependencies to one fat jar.\n\n```bash\n$ sbt assembly \n```\n\nThe fat jar will be prepend with shell script, so we could use it as a shell command\n\n```bash\n$ ./zkpipe-0.1.1 -h\n```","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fflier%2Fzkpipe","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fflier%2Fzkpipe","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fflier%2Fzkpipe/lists"}