{"id":13646795,"url":"https://github.com/pinterest/DoctorK","last_synced_at":"2025-04-21T21:31:27.589Z","repository":{"id":45403754,"uuid":"100966553","full_name":"pinterest/DoctorK","owner":"pinterest","description":"DoctorK is a service for Kafka cluster auto healing and workload balancing","archived":true,"fork":false,"pushed_at":"2021-12-15T14:54:32.000Z","size":1632,"stargazers_count":633,"open_issues_count":42,"forks_count":92,"subscribers_count":45,"default_branch":"master","last_synced_at":"2024-04-14T12:12:23.334Z","etag":null,"topics":["kafka"],"latest_commit_sha":null,"homepage":"","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/pinterest.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2017-08-21T15:30:24.000Z","updated_at":"2024-04-03T23:18:07.000Z","dependencies_parsed_at":"2022-08-29T23:00:36.868Z","dependency_job_id":null,"html_url":"https://github.com/pinterest/DoctorK","commit_stats":null,"previous_names":["pinterest/doctorkafka"],"tags_count":9,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pinterest%2FDoctorK","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pinterest%2FDoctorK/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pinterest%2FDoctorK/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pinterest%2FDoctorK/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/pinterest","download_url":"https://codeload.github.com/pinterest/DoctorK/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":223880284,"owners_count":17219094,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["kafka"],"created_at":"2024-08-02T01:03:06.390Z","updated_at":"2024-11-09T20:30:34.223Z","avatar_url":"https://github.com/pinterest.png","language":"Java","funding_links":[],"categories":["Java","进程间通信"],"sub_categories":[],"readme":"#  \u003cimg src=\"docs/doctork_logo.svg\" alt=\"DoctorK logo\" width=\"48\"\u003e \u0026nbsp;\u0026nbsp; Pinterest DoctorK\n\n### Open Sourcing Orion\nBased on learning from DoctorK we have created and open sourced [Orion](https://github.com/pinterest/orion), a more capable system for management of Kafka and other distributed systems. Orion addresses the [shortcomings](https://github.com/pinterest/orion/blob/master/docs/Motivation.md) of DoctorK and also adds new features like topic management, rolling restarts, rolling upgrades, stuck consumer remediation etc. Orion has been stabily managing our entire kafka fleet for \u003e6months.\n\n---\n\nDoctorK is a service for [Kafka] cluster auto healing and workload balancing.  DoctorK can automatically detect broker failure and reassign the workload on the failed nodes to other nodes. DoctorK can also perform load balancing based on topic partitions's network usage, and makes sure that broker network usage does not exceed the defined settings. DoctorK sends out alerts when it is not confident on taking actions.\n\n#### Features   \n\n * Automated cluster healing by moving partitions on failed brokers to other brokers\n * Workload balancing among brokers\n * Centralized management of multiple kafka clusters\n\n#### Detailed design\n\nDesign details are available in [docs/DESIGN.md](docs/DESIGN.md).\n\n## Setup Guide\n\n##### Get DoctorK code\n```sh\ngit clone [git-repo-url] doctork\ncd doctork\n```\n\n##### Build kafka stats collector and deployment it to kafka brokers \n\n```sh\nmvn package -pl kafkastats -am\n```\n\nKafkastats is a kafka broker stats collector that runs on kafka brokers and reports broker stats\nto some kafka topic based on configuration. The following is the kafkastats usage.\n\n```sh\nusage: KafkaMetricsCollector\n -broker \u003carg\u003e                      kafka broker\n -disable_ec2metadata               Disable the collection of host information using ec2metadata\n -jmxport \u003ckafka jmx port number\u003e   kafka jmx port number\n -kafka_config \u003carg\u003e                kafka server properties file path\n -ostrichport \u003carg\u003e                 ostrich port\n -pollingintervalinseconds \u003carg\u003e    polling interval in seconds\n -primary_network_ifacename \u003carg\u003e   network interface used by kafka\n -producer_config \u003carg\u003e             kafka_stats producer config\n -topic \u003carg\u003e                       kafka topic for metric messages\n -tsdhostport \u003carg\u003e                 tsd host and port, e.g.\n                                    localhost:18621\n -uptimeinseconds \u003carg\u003e             uptime in seconds\n -zookeeper \u003carg\u003e                   zk url for metrics topic\n```\n\nThe following is a sample command line for running kafkastats collector:\n\n```\njava -server \\\n    -Dlog4j.configurationFile=file:./log4j2.xml \\\n    -cp lib/*:kafkastats-0.2.4.10.jar \\\n    com.pinterest.doctork.stats.KafkaStatsMain \\\n        -broker 127.0.0.1 \\\n        -jmxport 9999 \\\n        -topic brokerstats \\\n        -zookeeper zookeeper001:2181/cluster1 \\\n        -uptimeinseconds 3600 \\\n        -pollingintervalinseconds 60 \\\n        -ostrichport 2051 \\\n        -tsdhostport localhost:18126 \\\n        -kafka_config /etc/kafka/server.properties \\\n        -producer_config /etc/kafka/producer.properties \\\n        -primary_network_ifacename eth0\n```\n\nUsing the above command as an example, after the kafkastats process is up, we can check the process stats using ```curl -s ``` command, and view the logs under /var/log/kafkastats.\n\n```\ncurl -s localhost:2051/stats.txt\n```\n\nThe following is a sample upstart scripts for automatically restarting kafkastats if it fails:\n\n```description \"KafkaStats\"\n   start on runlevel [2345]\n   respawn\n   respawn limit 20 5\n\n   env NAME=kafkastats\n   env JAVA_HOME=/usr/lib/jvm/java-8-oracle\n   env STATSCOLLECTOR_HOME=/opt/kafkastats\n   env LOG_DIR=/var/log/kafkastats\n   env HOSTNAME=$(hostname)\n\n   script\n       DAEMON=$JAVA_HOME/bin/java\n       CLASSPATH=$STATSCOLLECTOR_HOME:$STATSCOLLECTOR_HOME/*:$STATSCOLLECTOR_HOME/lib/*\n       DAEMON_OPTS=\"-server -Xmx800M -Xms800M -verbosegc -Xloggc:$LOG_DIR/gc.log \\\n       -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=20 -XX:GCLogFileSize=20M \\\n       -XX:+UseG1GC -XX:MaxGCPauseMillis=250 -XX:G1ReservePercent=10 -XX:ConcGCThreads=4 \\\n       -XX:ParallelGCThreads=4 -XX:G1HeapRegionSize=8m -XX:InitiatingHeapOccupancyPercent=70 \\\n       -XX:ErrorFile=$LOG_DIR/jvm_error.log \\\n       -cp $CLASSPATH\"\n       exec $DAEMON $DAEMON_OPTS -Dlog4j.configuration=${LOG_PROPERTIES} \\\n                    com.pinterest.doctork.stats.KafkaStatsMain \\\n                    -broker 127.0.0.1 \\\n                    -jmxport 9999 \\\n                    -topic brokerstats \\\n                    -zookeeper zookeeper001:2181/cluster1 \\\n                    -uptimeinseconds 3600 \\\n                    -pollingintervalinseconds 60 \\\n                    -ostrichport 2051 \\\n                    -tsdhostport localhost:18126 \\\n                    -kafka_config /etc/kafka/server.properties \\\n                    -producer_config /etc/kafka/producer.properties \\\n                    -primary_network_ifacename eth0\n```\n\n\n##### Customize doctork configuration parameters\n\nEdit `doctork/config/*.properties` files to specify parameters describing the environment. Those files contain\ncomments describing the meaning of individual parameters.\n\n\n#### Create and install jars\n\n```\nmvn package -pl doctork -am \n```\n\n```sh\nmvn package\nmkdir ${DOCTORK_INSTALL_DIR} # directory to place DoctorK binaries in.\ntar -zxvf target/doctork-0.2.4.10-bin.tar.gz -C ${DOCTORK_INSTALL_DIR}\n```\n\n##### Run DoctorK\n```sh\ncd ${DOCTORK_INSTALL_DIR}\n\njava -server \\\n    -cp lib/*:doctork-0.2.4.10.jar \\\n    com.pinterest.doctork.DoctorKMain \\\n        server dropwizard_yaml_file\n```\n\nThe above `dropwizard_yaml_file` is the path to a standard [DropWizard configuration file ](https://www.dropwizard.io/1.0.0/docs/manual/configuration.html)\nthat only requires the following line pointing to your `doctork.properties` path.\n\n```\nconfig:  $doctork_config_properties_file_path\n```\n\n##### Customize configuration parameters\n\nEdit `src/main/config/*.properties` files to specify parameters describing the environment.\nThose files contain comments describing the meaning of individual parameters.\n\n\n## Tools\nDoctorK comes with a number of tools implementing interactions with the environment.\n\n##### Cluster Load Balancer\n\n```bash\ncd ${DOCTORK_INSTALL_DIR}\njava -server \\\n    -Dlog4j.configurationFile=file:doctork/config/log4j2.xml \\\n    -cp doctork/target/lib/*:doctork/target/doctork-0.2.4.10.jar \\\n    com.pinterest.doctork.tools.ClusterLoadBalancer \\\n        -brokerstatstopic  brokerstats \\\n        -brokerstatszk zookeeper001:2181/cluster1 \\\n        -clusterzk zookeeper001:2181,zookeeper002:2181,zookeeper003:2181/cluster2 \\\n        -config ./doctork/config/doctork.properties \\\n        -seconds 3600\n```\nCluster load balancer balances the workload among brokers to make sure the broker network\nusage does not exceed the threshold.\n\n\n## DoctorK UI \n\nDoctorK uses [dropwizard-core module](https://www.dropwizard.io/1.3.5/docs/manual/core.html) and [serving assets](https://www.dropwizard.io/1.3.5/docs/manual/core.html#serving-assets) to provide a web UI. The following is the screenshot from a demo:\n\n![doctork UI](docs/doctork_ui.png)\n\u003cimg src=\"docs/doctork_ui.png\" width=\"160\"\u003e\n\n## DoctorK APIs\n\nThe following APIs are available for DoctorK:\n\n    - List Cluster\n    - Maintenance Mode\n\nDetailed description of APIs can be found [docs/APIs.md](docs/APIs.md)\n\n## Maintainers\n  * [Yu Yang](https://github.com/yuyang08)\n  * [Henry Cai](https://github.com/HenryCaiHaiying)\n  * [Ambud Sharma](https://github.com/ambud)\n  * [Ping-Min Lin](https://github.com/kabochya)\n\n## Contributors\n  * [Brian Gallew](https://github.com/BrianGallew)\n  * [Georgios Andrianakis](https://github.com/geoand)\n  * [Jaakko Aro](https://github.com/jaakkoo)\n  * [Patrick Double](https://github.com/double16)\n  * [Paul Kiernan](https://github.com/paulkiernan)\n  * [Brian Sang](https://github.com/Baisang)\n  * [Nitin Agrawal](https://github.com/nitin456)\n\n## License\n\nDoctorK is distributed under [Apache License, Version 2.0](http://www.apache.org/licenses/LICENSE-2.0.html).\n\n[Kafka]:http://kafka.apache.org/\n[Ostrich]: https://github.com/twitter/ostrich\n[OpenTSDB]: http://opentsdb.net/\n[statsD]: https://github.com/etsy/statsd/\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpinterest%2FDoctorK","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fpinterest%2FDoctorK","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpinterest%2FDoctorK/lists"}