https://github.com/flier/zkpipe
Consume Zookeeper binary log to sync filtered transactions to Kafka topic.
https://github.com/flier/zkpipe
Last synced: over 1 year ago
JSON representation
Consume Zookeeper binary log to sync filtered transactions to Kafka topic.
- Host: GitHub
- URL: https://github.com/flier/zkpipe
- Owner: flier
- Created: 2017-03-08T02:19:08.000Z (over 9 years ago)
- Default Branch: master
- Last Pushed: 2017-03-08T02:29:57.000Z (over 9 years ago)
- Last Synced: 2025-01-21T22:09:41.089Z (over 1 year ago)
- Language: Scala
- Size: 97.7 KB
- Stars: 3
- Watchers: 3
- Forks: 1
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# zkpipe
Consume Zookeeper binary log to sync filtered transactions to Kafka topic.
# Features
- [Sync](#sync) transactions from `Zookeeper` binary log/snapshot files to `Kafka` topic.
- [Watch](#watch) log directory and continue to sync changed transaction logs.
- [Filter](#filter) transaction with `zxid range`, `path prefix` or `path pattern`.
- [Encode](#encoding-format) transaction as `JSON`, `XML`, `ProtoBuf` or `Raw` format.
- Resume [from latest](#from-latest) transaction in the `Kafka` topic.
- Report [metrics](#metrics) to `Prometheus`, `Graphite` or `Ganglia`.
- Low memory footprint base on the streaming mode
# Usage
[watch](#watch) and [sync](#sync) command consume changed or [filtered](#filter) transactions from the Zookeeper binary log files,
send [encoded](#encoding-format) message to Kafka topic that defined in the `--kafka-uri` option.
## Watch
`watch` log directory and continue to sync changed transactions to kafka that defined in the `--kafka-uri` option.
> zkpipe watch \ --kafka-uri \
```bash
$ zkpipe watch /usr/local/var/run/zookeeper/data/version-2/ --kafka-uri=kafka://localhost/zkpipe
```
### From Latest
By default, `zkpipe` will auto resume the sync progress from the latest transaction in the Kafka topic,
`--from-latest` option skip the sync progress to the latest zxid in the Zookeeper binary logs.
> zkpipe watch \ [options] --from-latest
```bash
$ zkpipe watch /usr/local/var/run/zookeeper/data/version-2/ --kafka-uri=kafka://localhost/zkpipe --from-latest
```
## Sync
`sync` transactions from the binary log files to Kafka topic.
> zkpipe sync \
```bash
$ zkpipe sync /usr/local/var/run/zookeeper/data/version-2/log.c7
```
When the syntax is "glob" then the `log files` representation of the path is matched using a limited pattern language that resembles regular expressions but with a simpler syntax. For example:
- *.java Matches a path that represents a file name ending in .java
- *.* Matches file names containing a dot
- *.{java,class} Matches file names ending with .java or .class
- foo.? Matches file names starting with foo. and a single character extension
- /home/*/* Matches /home/gus/data on UNIX platforms
- /home/** Matches /home/gus and /home/gus/data on UNIX platforms
- C:\\* Matches C:\foo and C:\bar on the Windows platform (note that the backslash is escaped; as a string literal in the Java Language the pattern would be "C:\\\\*")
Please check [FileSystem.getPathMatcher](https://docs.oracle.com/javase/7/docs/api/java/nio/file/FileSystem.html#getPathMatcher(java.lang.String)) document for more detail.
## Filter
`watch` or `sync` command can filter transactions with filters.
### zxid Range
`--range ` option filter transaction in the `zxid` range
- `:` matches transactions `zxid` lager than `low` value
- `:` matches transactions `zxid` less than `high` value
- `:` matches transactions `zxid` between `low` and `high` value
- `:` matches all transactions
> zkpipe [options] --range 238:250
### Path Prefix
`--prefix ` option filter transaction act on the node that have same path prefix.
> zkpipe [options] --prefix /broker
### Match Pattern
`--match ` option filter transaction act on the node that path matches pattern in the regular expression.
> zkpipe [options] --match ^/broker/.*
### Session ID
`--session-id ` option filter transaction with the session ID
> zkpipe [options] --session-id 97295928260820996
### Ignore Session Events
`--ignore-session` option ignore all the Zookeeper session events, including CreateSession, CloseSession etc.
> zkpipe [options] --ignore-session
## Encoding Format
`--encode ` support the following encoding
- `pb` encode transaction in ProtoBuf format
- `json` encode transaction in JSON format
- `xml` encode transaction in XML format
- `raw` transaction in raw Zookeeper format
> zkpipe [options] --encode pb
# Metrics
`zkpipe` will serve metrics for Prometheus scrape or reports metrics to Graphite or Ganglia server.
## Prometheus
`zkpipe` serve metrics for [Prometheus](https://prometheus.io/) scrape in pull mode, that defined in the `--metrics-uri ` option defined.
> zkpipe watch \ [options] --metrics-uri http://\[:port\]/
When deploy `zkpipe` behind firewall or without public IP address, use `--push-gateway ` option to push metrics to the Prometheus [push gateway](https://prometheus.io/docs/practices/pushing/).
> zkpipe watch \ [options] --push-gateway \[:port\]
## Graphite
`zkpipe` report metrics to the [Graphite](https://graphiteapp.org/) server that defined in the `--report-uri ` option.
> zkpipe watch \ --kafka-uri \ --report-uri graphite://\[:port\]
The [pickle protocol](http://graphite.readthedocs.io/en/latest/feeding-carbon.html#the-pickle-protocol) is a much more efficient take on the plaintext protocol, and supports sending batches of metrics.
> zkpipe watch \ [options] --report-uri pickle://\[:port\]
## Ganglia
`zkpipe` report metrics to the [Ganglia](http://ganglia.info/) server that defined in the `--report-uri ` option.
> zkpipe watch \ [options] --report-uri ganglia://\[:port\]
# Records
`zkpipe` support all the Zookeeper transaction type, please check ProtoBuf schema for more detail.
## Create Session
```json
{
"cxid": 0,
"record": {
"create-session": {
"timeout": 30000
}
},
"session": 97250703064170497,
"time": 1484217490213,
"type": "CreateSession",
"zxid": 10
}
```
```xml
```
## Create Node
```json
{
"cxid": 5,
"path": "/test/abc",
"record": {
"create": {
"acl": [
{
"id": "anyone",
"perms": 31,
"scheme": "world"
}
],
"data": "",
"ephemeral": false,
"parentCVersion": 1,
"path": "/test/abc"
}
},
"session": 97223867455045632,
"time": 1483674202631,
"type": "Create",
"zxid": 3
}
```
```xml
```
## Set Data
```json
{
"cxid": 2,
"path": "/test/abc",
"record": {
"set-data": {
"data": "aGVsbG8=",
"path": "/test/abc",
"version": 3
}
},
"session": 97250703064170497,
"time": 1484218456704,
"type": "SetData",
"zxid": 12
}
```
```xml
```
# Build
First, use `compile` command to generate source codes and compile classes.
```bash
$ sbt compile
```
And then, use `assembly` command to package all the code and dependencies to one fat jar.
```bash
$ sbt assembly
```
The fat jar will be prepend with shell script, so we could use it as a shell command
```bash
$ ./zkpipe-0.1.1 -h
```