{"id":13631579,"url":"https://github.com/twitter/zktraffic","last_synced_at":"2025-07-09T22:43:12.724Z","repository":{"id":18775733,"uuid":"21988855","full_name":"twitter/zktraffic","owner":"twitter","description":"ZooKeeper protocol analyzer and stats gathering daemon","archived":false,"fork":false,"pushed_at":"2023-04-10T11:49:30.000Z","size":280,"stargazers_count":168,"open_issues_count":4,"forks_count":54,"subscribers_count":113,"default_branch":"master","last_synced_at":"2025-06-25T22:52:01.206Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/twitter.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGES.md","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2014-07-18T17:39:42.000Z","updated_at":"2025-05-13T12:34:06.000Z","dependencies_parsed_at":"2024-01-19T11:26:19.421Z","dependency_job_id":null,"html_url":"https://github.com/twitter/zktraffic","commit_stats":{"total_commits":163,"total_committers":13,"mean_commits":"12.538461538461538","dds":0.5766871165644172,"last_synced_commit":"82db04d9aafa13f694d4f5c7265069db42c0307c"},"previous_names":[],"tags_count":4,"template":false,"template_full_name":null,"purl":"pkg:github/twitter/zktraffic","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/twitter%2Fzktraffic","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/twitter%2Fzktraffic/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/twitter%2Fzktraffic/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/twitter%2Fzktraffic/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/twitter","download_url":"https://codeload.github.com/twitter/zktraffic/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/twitter%2Fzktraffic/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":262925062,"owners_count":23385470,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-01T22:02:30.564Z","updated_at":"2025-07-09T22:43:12.704Z","avatar_url":"https://github.com/twitter.png","language":"Python","funding_links":[],"categories":["Python"],"sub_categories":[],"readme":"# ZKTraffic [![Build Status](https://travis-ci.org/twitter/zktraffic.svg?branch=master)](https://travis-ci.org/twitter/zktraffic) [![Coverage Status](https://coveralls.io/repos/twitter/zktraffic/badge.png)](https://coveralls.io/r/twitter/zktraffic) [![PyPI version](https://badge.fury.io/py/zktraffic.svg)](http://badge.fury.io/py/zktraffic)\n\n**Table of Contents**\n\n- [tl;dr](#tldr)\n- [Installing](#installing)\n- [What is ZKTraffic?](#what-is-zktraffic)\n- [Contributing and Testing](#contributing-and-testing)\n- [More tools!](#more-tools)\n- [OS X](#os-x)\n- [Dependencies](#dependencies)\n\n### tl;dr ###\n\nZooKeeper protocol analyzer and stats gathering daemon\n\n### Installing ###\n\nYou can install ZKTraffic via pip:\n\n.. code-block:: bash\n\n   $ pip install zktraffic\n\nOr run it from source (if you have the dependencies installed, see below):\n\n.. code-block:: bash\n\n   $ git clone https://github.com/twitter/zktraffic.git\n   $ cd zktraffic\n   $ sudo ZKTRAFFIC_SOURCE=1 bin/zk-dump --iface=eth0\n\nTo get a quick count of requests by path:\n\n.. code-block:: bash\n\n   $ sudo ZKTRAFFIC_SOURCE=1 bin/zk-dump --iface=eth0 --count-requests 10000 --sort-by path\n   / 1749\n   /services/prod/search 846\n   /configs/teleportation/features 843\n\nOr by type:\n\n.. code-block:: bash\n\n   $ sudo ZKTRAFFIC_SOURCE=1 bin/zk-dump --iface=eth0 --count-requests 10000 --sort-by type\n   GetChildrenRequest 9044\n   ExistsRequest 958\n\nYou can also measure latencies by path (avg, p95 and p99):\n\n.. code-block:: bash\n\n   $ sudo ZKTRAFFIC_SOURCE=1 bin/zk-dump --measure-latency 1000 --group-by path --aggregation-depth 2 --sort-by p99\n   path                     avg         p95         p99\n   ---------------  -----------  ----------  ----------\n   /party/services  0.000199077  0.00048846  0.00267805\n   /party           0.000349498  0.00136839  0.00201204\n   /party/configs   0.000157728  0.00036664  0.00122663\n\nOr by type:\n\n.. code-block:: bash\n\n   $ sudo ZKTRAFFIC_SOURCE=1 bin/zk-dump --measure-latency 1000 --group-by type --sort-by p99\n   type                            avg          p95          p99\n   ----------------------  -----------  -----------  -----------\n   CreateEphemeralRequest  0.000735009  0.000978041  0.0032404\n   GetChildrenRequest      0.000182547  0.000453258  0.00220628\n   ExistsRequest           0.000162728  0.000430155  0.000862937\n\nOr by client:\n\n.. code-block:: bash\n\n   $ sudo ZKTRAFFIC_SOURCE=1 bin/zk-dump --measure-latency 1000 --group-by client --sort-by p99\n   client                          avg          p95          p99\n   ----------------------  -----------  -----------  -----------\n   10.0.1.3:44308          0.000735009  0.000978041  0.0032404\n   10.0.1.6:34305          0.000182547  0.000453258  0.00220628\n   10.0.1.9:36110          0.000162728  0.000430155  0.000862937\n\nOr use the stats gathering daemon:\n\n.. code-block:: bash\n\n   $ sudo ZKTRAFFIC_SOURCE=1 bin/zk-stats-daemon --iface=eth0 --http-port=9090\n\nOr you can build PEX files — from the source — for any of the available tools:\n\n.. code-block:: bash\n\n   $ pip install pex\n\n   # zk-dump\n   $ pex -v -e zktraffic.cli.zk -o zk-dump.pex .\n\n   # zk-stats-daemon\n   $ pex -v -e zktraffic.cli.stats_daemon -o stats-daemon.pex .\n\n   # zab-dump\n   $ pex -v -e zktraffic.cli.zab -o zab-dump.pex .\n\n   # fle-dump\n   $ pex -v -e zktraffic.cli.fle -o fle-dump.pex .\n\nMore info about PEX [here](https://pex.readthedocs.org \"PEX\").\n\n### What is ZKTraffic? ###\n\nAn {iptraf,top}-esque traffic monitor for ZooKeeper. Right now it exports\nper-path (and global) stats. Eventually it'll be made to export per-user\nstats too.\n\nIt has a front-end, zk-dump, that can be used in interactive mode to dump traffic:\n\n```\n# need root or CAP_NET_ADMIN \u0026 CAP_NET_RAW\n$ sudo zk-dump --iface eth0\n21:08:05:991542 ConnectRequest(ver=0, zxid=0, timeout=10000, session=0x0, readonly=False, client=127.0.0.1:50049)\n————————►21:08:06:013513 ConnectReply(ver=0, timeout=10000, session=0x148cf0aedc60000, readonly=False, client=127.0.0.1:50049)\n21:08:07:432361 ExistsRequest(xid=1, path=/, watch=False, size=14, client=127.0.0.1:50049)\n————————►21:08:07:447353 ExistsReply(xid=1, zxid=31, error=0, client=127.0.0.1:50049)\n21:08:07:448033 GetChildrenRequest(xid=2, path=/, watch=False, size=14, client=127.0.0.1:50049)\n————————►21:08:07:456169 GetChildrenReply(xid=2, zxid=31, error=0, count=1, client=127.0.0.1:50049)\n...\n```\n\nOr, it can work in daemon mode from which it exposes HTTP/JSON endpoints with\nstats that can be fed into your favourite data collection system:\n\n.. code-block:: bash\n\n   $ sudo zk-stats-daemon.pex --app_daemonize --aggregation-depth=5\n\n   # Wait for 1 min and:\n\n   $ sleep 60 \u0026\u0026 curl http://localhost:7070/json/paths | python -mjson.tool\n   {\n    \"ConnectRequest\": 2,\n    \"ConnectRequestBytes\": 90,\n    \"CreateRequest/configs\": 2,\n    \"CreateRequest/configs/server\": 2,\n    \"CreateRequest/discovery\": 2,\n    \"CreateRequest/discovery/hosts\": 2,\n    \"CreateRequest/discovery/services\": 2,\n    \"CreateRequestBytes/configs\": 110,\n    \"CreateRequestBytes/configs/server\": 124,\n    \"CreateRequestBytes/discovery\": 114,\n    \"CreateRequestBytes/discovery/hosts\": 126,\n    \"CreateRequestBytes/discovery/services\": 132,\n    \"ExistsRequest/\": 1574,\n    \"ExistsRequest/configs\": 3,\n    \"ExistsRequest/configs/server\": 2,\n    \"ExistsRequest/discovery\": 4,\n    \"ExistsRequest/discovery/hosts\": 2,\n    \"ExistsRequest/discovery/services\": 2,\n    \"ExistsRequestBytes/\": 22036,\n    \"ExistsRequestBytes/configs\": 63,\n    \"ExistsRequestBytes/configs/server\": 56,\n    \"ExistsRequestBytes/discovery\": 92,\n    \"ExistsRequestBytes/discovery/hosts\": 58,\n    \"ExistsRequestBytes/discovery/services\": 64,\n    \"GetChildrenRequest/configs\": 1285,\n    \"GetChildrenRequest/configs/server\": 1242,\n    \"GetChildrenRequest/discovery\": 1223,\n    \"GetChildrenRequest/discovery/hosts\": 1250,\n    \"GetChildrenRequest/discovery/services\": 1222,\n    \"GetChildrenRequest/zookeeper/config\": 1285,\n    \"GetChildrenRequest/zookeeper/quota/limits\": 1228,\n    \"GetChildrenRequest/zookeeper/quota/limits/by-path\": 1269,\n    \"GetChildrenRequest/zookeeper/quota/limits/global\": 1230,\n    \"GetChildrenRequest/zookeeper/quota/stats/by-path\": 1222,\n    \"GetChildrenRequestBytes/discovery/hosts\": 36250,\n    \"GetChildrenRequestBytes/discovery/services\": 39104,\n    \"GetChildrenRequestBytes/zookeeper/config\": 38550,\n    \"GetChildrenRequestBytes/zookeeper/quota/limits\": 44208,\n    \"GetChildrenRequestBytes/zookeeper/quota/limits/by-path\": 55836,\n    \"GetChildrenRequestBytes/zookeeper/quota/limits/global\": 52890,\n    \"GetChildrenRequestBytes/zookeeper/quota/limits/slices\": 51815,\n    \"GetChildrenRequestBytes/zookeeper/quota/stats\": 42630,\n    \"GetChildrenRequestBytes/zookeeper/quota/stats/by-path\": 52546,\n    \"GetChildrenRequestBytes/zookeeper/quota/stats/global\": 50568,\n    \"reads/\": 2761,\n    \"reads/configs\": 1288,\n    \"reads/configs/server\": 1244,\n    \"reads/discovery\": 1227,\n    \"reads/discovery/hosts\": 1252,\n    \"reads/discovery/services\": 1224,\n    \"reads/zookeeper/config\": 1285,\n    \"reads/zookeeper/quota/limits\": 1228,\n    \"reads/zookeeper/quota/limits/by-path\": 1269,\n    \"reads/zookeeper/quota/limits/global\": 1230,\n    \"readsBytes/\": 38654,\n    \"readsBytes/discovery/services\": 39168,\n    \"readsBytes/zookeeper/config\": 38550,\n    \"readsBytes/zookeeper/quota/limits\": 44208,\n    \"readsBytes/zookeeper/quota/limits/by-path\": 55836,\n    \"readsBytes/zookeeper/quota/limits/global\": 52890,\n    \"readsBytes/zookeeper/quota/limits/slices\": 51815,\n    \"readsBytes/zookeeper/quota/stats\": 42630,\n    \"readsBytes/zookeeper/quota/stats/by-path\": 52546,\n    \"readsBytes/zookeeper/quota/stats/global\": 50568,\n    \"total/readBytes\": 655586,\n    \"total/reads\": 21251,\n    \"total/writeBytes\": 606,\n    \"total/writes\": 10,\n    \"writes/\": 0,\n    \"writes/configs\": 2,\n    \"writes/configs/server\": 2,\n    \"writes/discovery\": 2,\n    \"writes/discovery/hosts\": 2,\n    \"writes/discovery/services\": 2,\n    \"writesBytes/\": 0,\n    \"writesBytes/configs\": 110,\n    \"writesBytes/configs/server\": 124,\n    \"writesBytes/discovery\": 114,\n    \"writesBytes/discovery/hosts\": 126,\n    \"writesBytes/discovery/services\": 132\n   }\n\nOther relevant endpoints for stats are:\n\n* /json/ips: top-N per-ip stats\n* /json/auths: per-auth stats\n* /json/auths-dump: a full dump of known auths\n* /json/info: process uptime and introspection info\n* /threads: stacks for all threads\n\n### Contributing and Testing ###\n\nPlease see [CONTRIBUTING.md](CONTRIBUTING.md).\n\n### More tools! ###\n\nAlong with zk-dump and zk-stats-daemon, you can find fle-dump which allows you\nto inspect FastLeaderElection traffic (i.e.: the protocol by which ZooKeeper decides\nwho will lead and the mechanism by which the leader is subsequently discovered):\n\n.. code-block:: bash\n\n   $ sudo fle-dump --iface eth0 -c\n   Notification(\n     timestamp=00:57:12:593254,\n     src=10.0.0.1:32938,\n     dst=10.0.0.2:3888,\n     state=following,\n     leader=3,\n     zxid=0,\n     election_epoch=0,\n     peer_epoch=0,\n     config=\n          server.0=10.0.0.1:2889:3888:participant;0.0.0.0:2181\n          server.1=10.0.0.2:2889:3888:participant;0.0.0.0:2181\n          server.2=10.0.0.3:2889:3888:participant;0.0.0.0:2181\n          server.3=10.0.0.4:2889:3888:participant;0.0.0.0:2181\n          server.4=10.0.0.5:2889:3888:participant;0.0.0.0:2181\n          version=10010d4d6\n   )\n   Notification(\n     timestamp=00:57:12:595525,\n     src=10.0.0.2:3888,\n     dst=10.0.0.1:32938,\n     state=looking,\n     leader=1,\n     zxid=4296326153,\n     election_epoch=1,\n     peer_epoch=1,\n     config=\n          server.0=10.0.0.1:2889:3888:participant;0.0.0.0:2181\n          server.1=10.0.0.2:2889:3888:participant;0.0.0.0:2181\n          server.2=10.0.0.3:2889:3888:participant;0.0.0.0:2181\n          server.3=10.0.0.4:2889:3888:participant;0.0.0.0:2181\n          server.4=10.0.0.5:2889:3888:participant;0.0.0.0:2181\n          version=10010d4d6\n   )\n   ...\n\nNote: for initial messages to be visible you'll need the patch available\nat [ZOOKEEPER-2098](https://issues.apache.org/jira/browse/ZOOKEEPER-2098 \"ZOOKEEPER-2098\"),\nif you are using ZooKeeper prior to ZooKeeper 3.5.1-rc2.\n\nNote: if you are using Linux 3.14 or later, you'll need to disable [TCP Auto Corking](http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=f54b311142a92ea2e42598e347b84e1655caf8e3) by running `echo 0 \u003e /proc/sys/net/ipv4/tcp_autocorking`.\n\nIf you are interested in debugging ZAB (ZooKeeper Atomic Broadcast protocol), you can use\nzab-dump:\n\n.. code-block:: bash\n\n   $ sudo zab-dump --iface eth0\n\n   Request(\n    cxid=6,\n    dst=10.0.0.1:2889,\n    length=112,\n    req_type=CreateRequest,\n    session_id=0x34e4d23b0d70001,\n    src=10.0.0.2:48604,\n    timestr=22:54:31:995353,\n    zxid=-1,\n   )\n   Proposal(\n    cxid=6,\n    dst=10.0.0.2:48603,\n    length=110,\n    session_id=0x34e4d23b0d70001,\n    src=10.0.0.1:2889,\n    timestr=22:54:31:995753,\n    txn_time=1435816471995,\n    txn_type=CreateRequest,\n    txn_zxid=8589934619,\n    zxid=8589934619,\n   )\n   Proposal(\n    cxid=6,\n    dst=10.0.0.1:48604,\n    length=110,\n    session_id=0x34e4d23b0d70001,\n    src=10.0.0.1:2889,\n    timestr=22:54:31:995755,\n    txn_time=1435816471995,\n    txn_type=CreateRequest,\n    txn_zxid=8589934619,\n    zxid=8589934619,\n   )\n   Proposal(\n    cxid=6,\n    dst=10.0.0.3:48605,\n    length=110,\n    session_id=0x34e4d23b0d70001,\n    src=10.0.0.1:2889,\n    timestr=22:54:31:995770,\n    txn_time=1435816471995,\n    txn_type=CreateRequest,\n    txn_zxid=8589934619,\n    zxid=8589934619,\n   )\n   Ack(\n    dst=10.0.0.1:2889,\n    length=20,\n    src=10.0.0.1:48603,\n    timestr=22:54:31:996068,\n    zxid=8589934619,\n   )\n   Ack(\n    dst=10.0.0.1:2889,\n    length=20,\n    src=10.0.0.1:48604,\n    timestr=22:54:31:996316,\n    zxid=8589934619,\n   )\n   Ack(\n    dst=10.0.0.1:2889,\n    length=20,\n    src=10.0.0.1:48604,\n    timestr=22:54:31:996318,\n    zxid=8589934619,\n   )\n   Commit(\n    dst=10.0.0.1:48603,\n    length=20,\n    src=10.0.0.1:2889,\n    timestr=22:54:31:996193,\n    zxid=8589934619,\n   )\n   Commit(\n    dst=10.0.0.2:48604,\n    length=20,\n    src=10.0.0.1:2889,\n    timestr=22:54:31:996195,\n    zxid=8589934619,\n   )\n   Commit(\n    dst=10.0.0.2:48605,\n    length=20,\n    src=10.0.0.1:2889,\n    timestr=22:54:31:996442,\n    zxid=8589934619,\n   )\n\n### OS X ###\nAlthough no one has tried running this on OS X in production, it can be used for some parts of development and unit testing. If you are running on OS X, please run the following to install the correct dependencies:\n\n.. code-block:: bash\n\n   $ pip install -r ./osx_requirements.txt\n\n### Dependencies ###\n* Python 2.7 (Py3K soon)\n* ansicolors\n* dpkt-fix\n* hexdump\n* psutil\u003e=2.1.0\n* scapy==2.4.2\n* six\n* twitter.common.app\n* twitter.common.collections\n* twitter.common.exceptions\n* twitter.common.http\n* twitter.common.log\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftwitter%2Fzktraffic","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftwitter%2Fzktraffic","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftwitter%2Fzktraffic/lists"}