{"id":13697574,"url":"https://github.com/uber-archive/chaperone","last_synced_at":"2025-05-03T20:30:37.597Z","repository":{"id":54930402,"uuid":"73840074","full_name":"uber-archive/chaperone","owner":"uber-archive","description":"A Kafka audit system","archived":true,"fork":false,"pushed_at":"2021-01-20T23:05:56.000Z","size":193,"stargazers_count":640,"open_issues_count":24,"forks_count":140,"subscribers_count":1783,"default_branch":"master","last_synced_at":"2024-08-28T16:07:52.552Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/uber-archive.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2016-11-15T17:51:11.000Z","updated_at":"2024-08-22T08:16:09.000Z","dependencies_parsed_at":"2022-08-14T06:50:41.327Z","dependency_job_id":null,"html_url":"https://github.com/uber-archive/chaperone","commit_stats":null,"previous_names":["uber/chaperone"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/uber-archive%2Fchaperone","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/uber-archive%2Fchaperone/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/uber-archive%2Fchaperone/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/uber-archive%2Fchaperone/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/uber-archive","download_url":"https://codeload.github.com/uber-archive/chaperone/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":252252493,"owners_count":21718749,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-02T18:01:00.297Z","updated_at":"2025-05-03T20:30:33.255Z","avatar_url":"https://github.com/uber-archive.png","language":"Java","funding_links":[],"categories":["Libraries","进程间通信"],"sub_categories":["Kafka"],"readme":"Chaperone\n============\n(This project is deprecated and not maintained.)\n\nAs Kafka audit system, Chaperone monitors the completeness and latency of data stream. The audit metrics are persisted in database for Kafka users to quantify the loss of their topics if any.\n\nBasically, Chaperone cuts timeline into 10min buckets and assigns message to corresponding bucket according to its event time. The stats of the bucket are updated accordingly, like the total message count. Periodically, the stats are sent out to a dedicated Kafka topic, say 'chaperone-audit'. ChaperoneCollector consumes those stats from this topic and persists them into database.\n\n============\nChaperone is made of several components:\n\n1. ChaperoneClient is a library that can be put in like Kafka Producer or Consumer to audit messages as they flow through. The audit stats are sent to a dedicated Kafka topic, say 'chaperone-audit'.\n2. ChaperoneCollector consumes audit stats from 'chaperone-audit' and persists them into database.\n3. ChaperoneService audits messages kept in Kafka. Since it's built upon [uReplicator](https://github.com/uber/uReplicator), it consists of two subsystems: ChaperoneServiceController to auto-detect topics in Kafka and assign the topic-partitions to workers to audit; ChaperoneServiceWorker to audit messages from assigned topic-partitions. In particular, ChaperoneService and ChaperoneCollector together ensure each message is audited exactly once.\n\n# Chaperone Quick Start\n\n## Get the Code\nCheck out the Chaperone project:\n```\ngit clone git@github.com:uber/chaperone.git\ncd chaperone\n```\nThis project contains everything you’ll need to run Chaperone.\n\n\n## Build Chaperone\nBefore you can run Chaperone, you need to build a package for it.\n```\nmvn clean package\n```\nOr command below to skip tests\n```\nmvn clean package -DskipTests\n```\n\n\n## Set Up Local Test Environment\nTo test Chaperone locally, you need two systems: [Kafka](http://kafka.apache.org/), and [ZooKeeper](http://zookeeper.apache.org/). The script “grid” is to help you set up these systems.\n- The command below will download, install, and start ZooKeeper and Kafka (named cluster1)\n```\nbin/grid bootstrap\n```\n\n\n## Start ChaperoneService\n- Start ChaperoneService Controller\n```\n./ChaperoneDistribution/target/ChaperoneDistribution-pkg/bin/start-chaperone-controller.sh\n```\n\n- Start ChaperoneService Worker \n```\n./ChaperoneDistribution/target/ChaperoneDistribution-pkg/bin/start-chaperone-worker.sh\n```\n\n\n## Generate Load \n- Create a dummyTopic in Kafka and produce some dummy data:\n```\n./bin/produce-data-to-kafka-topic-dummyTopic.sh\n```\n- Check if the data is successfully produced to Kafka by console-consumer as below:\n```\n./deploy/kafka/bin/kafka-console-consumer.sh --zookeeper localhost:2181/cluster1 --topic dummyTopic\n```\nYou should get this data:\n```\nKafka topic dummy topic data 1\nKafka topic dummy topic data 2\nKafka topic dummy topic data 3\nKafka topic dummy topic data 4\n…\n```\n\n\n## Check Audit Stats\nIn this example, the topic dummyTopic will be auto-detected and assigned to worker to audit. Periodically, the audit stats are sent to a topic called 'chaperone-audit'. \n```\n./deploy/kafka/bin/kafka-console-consumer.sh --zookeeper localhost:2181/cluster1 --topic chaperone-audit \n```\nOne can also manually add topic to audit by command below:\n```\ncurl -X POST -d '{\"topic\":\"dummyTopic\", \"numPartitions\":\"1\"}' http://localhost:9000/topics\n```\n\n\n## Start ChaperoneCollector\nTo start ChaperoneCollector, MySQL is required and Redis is optional. MySQL is used to persist audit stats and Redis is used to deduplicate. Deduplication can be turned off. The configuration file for ChaperoneCollector is ./config/chaperonecollector.properties, which might be updated to connect to MySQL and Redis.\n- Start ChaperoneCollector\n```\n./ChaperoneDistribution/target/ChaperoneDistribution-pkg/bin/start-chaperone-collector.sh\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fuber-archive%2Fchaperone","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fuber-archive%2Fchaperone","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fuber-archive%2Fchaperone/lists"}