{"id":13558485,"url":"https://github.com/yahoo/CMAK","last_synced_at":"2025-04-03T13:31:20.056Z","repository":{"id":26528766,"uuid":"29981861","full_name":"yahoo/CMAK","owner":"yahoo","description":"CMAK is a tool for managing Apache Kafka clusters","archived":false,"fork":false,"pushed_at":"2023-08-02T19:26:23.000Z","size":4066,"stargazers_count":11877,"open_issues_count":520,"forks_count":2508,"subscribers_count":529,"default_branch":"master","last_synced_at":"2025-03-04T12:01:51.767Z","etag":null,"topics":["big-data","cluster-management","kafka","scala"],"latest_commit_sha":null,"homepage":"","language":"Scala","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/yahoo.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2015-01-28T18:33:21.000Z","updated_at":"2025-03-04T11:22:22.000Z","dependencies_parsed_at":"2023-01-14T04:51:50.561Z","dependency_job_id":"c0da85d8-c792-4880-a9f6-a3caa379b402","html_url":"https://github.com/yahoo/CMAK","commit_stats":null,"previous_names":["yahoo/kafka-manager"],"tags_count":43,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yahoo%2FCMAK","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yahoo%2FCMAK/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yahoo%2FCMAK/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yahoo%2FCMAK/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/yahoo","download_url":"https://codeload.github.com/yahoo/CMAK/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":246735300,"owners_count":20825223,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["big-data","cluster-management","kafka","scala"],"created_at":"2024-08-01T12:04:58.763Z","updated_at":"2025-04-03T13:31:20.033Z","avatar_url":"https://github.com/yahoo.png","language":"Scala","readme":"CMAK (Cluster Manager for Apache Kafka, previously known as Kafka Manager)\n=============\n\nCMAK (previously known as Kafka Manager) is a tool for managing [Apache Kafka](http://kafka.apache.org) clusters.\n_See below for details about the name change._\n\nCMAK supports the following:\n\n - Manage multiple clusters\n - Easy inspection of cluster state (topics, consumers, offsets, brokers, replica distribution, partition distribution)\n - Run preferred replica election\n - Generate partition assignments with option to select brokers to use\n - Run reassignment of partition (based on generated assignments)\n - Create a topic with optional topic configs (0.8.1.1 has different configs than 0.8.2+)\n - Delete topic (only supported on 0.8.2+ and remember set delete.topic.enable=true in broker config)\n - Topic list now indicates topics marked for deletion (only supported on 0.8.2+)\n - Batch generate partition assignments for multiple topics with option to select brokers to use\n - Batch run reassignment of partition for multiple topics\n - Add partitions to existing topic\n - Update config for existing topic\n - Optionally enable JMX polling for broker level and topic level metrics.\n - Optionally filter out consumers that do not have ids/ owners/ \u0026 offsets/ directories in zookeeper.\n\nCluster Management\n\n![cluster](/img/cluster.png)\n\n***\n\nTopic List\n\n![topic](/img/topic-list.png)\n\n***\n\nTopic View\n\n![topic](/img/topic.png)\n\n***\n\nConsumer List View\n\n![consumer](/img/consumer-list.png)\n\n***\n\nConsumed Topic View\n\n![consumer](/img/consumed-topic.png)\n\n***\n\nBroker List\n\n![broker](/img/broker-list.png)\n\n***\n\nBroker View\n\n![broker](/img/broker.png)\n\n***\n\nRequirements\n------------\n\n1. [Kafka 0.8.*.* or 0.9.*.* or 0.10.*.* or 0.11.*.*](http://kafka.apache.org/downloads.html)\n2. Java 11+\n\nConfiguration\n-------------\n\nThe minimum configuration is the zookeeper hosts which are to be used for CMAK (pka kafka manager) state.\nThis can be found in the application.conf file in conf directory.  The same file will be packaged\nin the distribution zip file; you may modify settings after unzipping the file on the desired server.\n\n    cmak.zkhosts=\"my.zookeeper.host.com:2181\"\n\nYou can specify multiple zookeeper hosts by comma delimiting them, like so:\n\n    cmak.zkhosts=\"my.zookeeper.host.com:2181,other.zookeeper.host.com:2181\"\n\nAlternatively, use the environment variable `ZK_HOSTS` if you don't want to hardcode any values.\n\n    ZK_HOSTS=\"my.zookeeper.host.com:2181\"\n\nYou can optionally enable/disable the following functionality by modifying the default list in application.conf :\n\n    application.features=[\"KMClusterManagerFeature\",\"KMTopicManagerFeature\",\"KMPreferredReplicaElectionFeature\",\"KMReassignPartitionsFeature\"]\n\n - KMClusterManagerFeature - allows adding, updating, deleting cluster from CMAK (pka Kafka Manager)\n - KMTopicManagerFeature - allows adding, updating, deleting topic from a Kafka cluster\n - KMPreferredReplicaElectionFeature - allows running of preferred replica election for a Kafka cluster\n - KMReassignPartitionsFeature - allows generating partition assignments and reassigning partitions\n\nConsider setting these parameters for larger clusters with jmx enabled :\n\n - cmak.broker-view-thread-pool-size=\u003c 3 * number_of_brokers\u003e\n - cmak.broker-view-max-queue-size=\u003c 3 * total # of partitions across all topics\u003e\n - cmak.broker-view-update-seconds=\u003c cmak.broker-view-max-queue-size / (10 * number_of_brokers) \u003e\n\nHere is an example for a kafka cluster with 10 brokers, 100 topics, with each topic having 10 partitions giving 1000 total partitions with JMX enabled :\n\n - cmak.broker-view-thread-pool-size=30\n - cmak.broker-view-max-queue-size=3000\n - cmak.broker-view-update-seconds=30\n\nThe follow control consumer offset cache's thread pool and queue :\n\n - cmak.offset-cache-thread-pool-size=\u003c default is # of processors\u003e\n - cmak.offset-cache-max-queue-size=\u003c default is 1000\u003e\n - cmak.kafka-admin-client-thread-pool-size=\u003c default is # of processors\u003e\n - cmak.kafka-admin-client-max-queue-size=\u003c default is 1000\u003e\n\nYou should increase the above for large # of consumers with consumer polling enabled.  Though it mainly affects ZK based consumer polling.\n\nKafka managed consumer offset is now consumed by KafkaManagedOffsetCache from the \"__consumer_offsets\" topic.  Note, this has not been tested with large number of offsets being tracked.  There is a single thread per cluster consuming this topic so it may not be able to keep up on large # of offsets being pushed to the topic.\n\n### Authenticating a User with LDAP\nWarning, you need to have SSL configured with CMAK (pka Kafka Manager) to ensure your credentials aren't passed unencrypted.\nAuthenticating a User with LDAP is possible by passing the user credentials with the Authorization header.\nLDAP authentication is done on first visit, if successful, a cookie is set.\nOn next request, the cookie value is compared with credentials from Authorization header.\nLDAP support is through the basic authentication filter.\n\n1. Configure basic authentication\n- basicAuthentication.enabled=true\n- basicAuthentication.realm=\u003c basic authentication realm\u003e\n\n2. Encryption parameters (optional, otherwise randomly generated on startup) :\n- basicAuthentication.salt=\"some-hex-string-representing-byte-array\"\n- basicAuthentication.iv=\"some-hex-string-representing-byte-array\"\n- basicAuthentication.secret=\"my-secret-string\"\n\n3. Configure LDAP / LDAP + StartTLS / LDAPS authentication\n\n_Note: LDAP is unencrypted and insecure. LDAPS is a commonly implemented \nextension that implements an encryption layer in a manner similar to how \nHTTPS adds encryption to an HTTP. LDAPS has not been documented, and the \nspecification is not formally defined anywhere. LDAP + StartTLS is the \ncurrently recommended way to start an encrypted channel, and it upgrades \nan existing LDAP connection to achieve this encryption._\n\n- basicAuthentication.ldap.enabled=\u003c Boolean flag to enable/disable ldap authentication \u003e\n- basicAuthentication.ldap.server=\u003c fqdn of LDAP server \u003e\n- basicAuthentication.ldap.port=\u003c port of LDAP server (typically 389 for LDAP and LDAP + StartTLS and typically 636 for LDAPS) \u003e\n- basicAuthentication.ldap.username=\u003c LDAP search username \u003e\n- basicAuthentication.ldap.password=\u003c LDAP search password \u003e\n- basicAuthentication.ldap.search-base-dn=\u003c LDAP search base \u003e\n- basicAuthentication.ldap.search-filter=\u003c LDAP search filter \u003e\n- basicAuthentication.ldap.connection-pool-size=\u003c maximum number of connection to LDAP server \u003e\n- basicAuthentication.ldap.ssl=\u003c Boolean flag to enable/disable LDAPS (usually incompatible with StartTLS) \u003e\n- basicAuthentication.ldap.starttls=\u003c Boolean flat to enable StartTLS (usually incompatible with SSL) \u003e\n\n4. (Optional) Limit access to a specific LDAP Group\n- basicAuthentication.ldap.group-filter=\u003c LDAP group filter \u003e\n- basicAuthentication.ldap.ssl-trust-all=\u003c Boolean flag to allow non-expired invalid certificates \u003e\n\n#### Example (Online LDAP Test Server):\n\n- basicAuthentication.ldap.enabled=true\n- basicAuthentication.ldap.server=\"ldap.forumsys.com\"\n- basicAuthentication.ldap.port=389\n- basicAuthentication.ldap.username=\"cn=read-only-admin,dc=example,dc=com\"\n- basicAuthentication.ldap.password=\"password\"\n- basicAuthentication.ldap.search-base-dn=\"dc=example,dc=com\"\n- basicAuthentication.ldap.search-filter=\"(uid=$capturedLogin$)\"\n- basicAuthentication.ldap.group-filter=\"cn=allowed-group,ou=groups,dc=example,dc=com\"\n- basicAuthentication.ldap.connection-pool-size=10\n- basicAuthentication.ldap.ssl=false\n- basicAuthentication.ldap.ssl-trust-all=false\n- basicAuthetication.ldap.starttls=false\n\n\nDeployment\n----------\n\nThe command below will create a zip file which can be used to deploy the application.\n\n    ./sbt clean dist\n\nPlease refer to play framework documentation on [production deployment/configuration](https://www.playframework.com/documentation/2.4.x/ProductionConfiguration).\n\nIf java is not in your path, or you need to build against a specific java version,\nplease use the following (the example assumes zulu java11):\n\n    $ PATH=/usr/lib/jvm/zulu-11-amd64/bin:$PATH \\\n      JAVA_HOME=/usr/lib/jvm/zulu-11-amd64 \\\n      /path/to/sbt -java-home /usr/lib/jvm/zulu-11-amd64 clean dist\n\nThis ensures that the 'java' and 'javac' binaries in your path are first looked up in the\ncorrect location. Next, for all downstream tools that only listen to JAVA_HOME, it points\nthem to the java11 location. Lastly, it tells sbt to use the java11 location as\nwell.\n\nStarting the service\n--------------------\n\nAfter extracting the produced zipfile, and changing the working directory to it, you can\nrun the service like this:\n\n    $ bin/cmak\n\nBy default, it will choose port 9000. This is overridable, as is the location of the\nconfiguration file. For example:\n\n    $ bin/cmak -Dconfig.file=/path/to/application.conf -Dhttp.port=8080\n\nAgain, if java is not in your path, or you need to run against a different version of java,\nadd the -java-home option as follows:\n\n    $ bin/cmak -java-home /usr/lib/jvm/zulu-11-amd64\n\nStarting the service with Security\n----------------------------------\n\nTo add JAAS configuration for SASL, add the config file location at start:\n\n    $ bin/cmak -Djava.security.auth.login.config=/path/to/my-jaas.conf\n\nNOTE: Make sure the user running CMAK (pka kafka manager) has read permissions on the jaas config file\n\n\nPackaging\n---------\n\nIf you'd like to create a Debian or RPM package instead, you can run one of:\n\n    sbt debian:packageBin\n\n    sbt rpm:packageBin\n\nCredits\n-------\n\nMost of the utils code has been adapted to work with [Apache Curator](http://curator.apache.org) from [Apache Kafka](http://kafka.apache.org).\n\nName and Management\n-------\n\nCMAK was renamed from its previous name due to [this issue](https://github.com/yahoo/kafka-manager/issues/713). CMAK is designed to be used with Apache Kafka and is offered to support the needs of the Kafka community. This project is currently managed by employees at Verizon Media and the community who supports this project. \n\nLicense\n-------\n\nLicensed under the terms of the Apache License 2.0. See accompanying LICENSE file for terms.\n\nConsumer/Producer Lag\n-------\n\nProducer offset is polled.  Consumer offset is read from the offset topic for Kafka based consumers.  This means the reported lag may be negative since we are consuming offset from the offset topic faster then polling the producer offset.  This is normal and not a problem.\n\nMigration from Kafka Manager to CMAK\n-------\n\n1. Copy config files from old version to new version (application.conf, consumer.properties)\n2. Change start script to use bin/cmak instead of bin/kafka-manager\n\n","funding_links":[],"categories":["Scala","ML frameworks \u0026 applications","Reinforcement Learning Tools, Libraries, and Frameworks","Parallel Computing Tools, Libraries, and Frameworks","Flutter Tools","ML Frameworks, Libraries, and Tools","Apache Spark Tools, Libraries, and Frameworks","scala","其他__大数据","Distributed Systems Tools","Deep Learning Tools, Libraries, and Frameworks","Tools","Scala Tools, Libraries, and Frameworks","进程间通信","\u003ca name=\"Scala\"\u003e\u003c/a\u003eScala","Management \u0026 Monitoring"],"sub_categories":["Interfaces","viii. Linear Regression","网络服务_其他","Parallel Systems Courses \u0026 Tutorials","Winetricks","Objective-C Tools, Libraries, and Frameworks","Mesh networks","JavaScript Libraries for Machine Learning","Web UIs"],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fyahoo%2FCMAK","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fyahoo%2FCMAK","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fyahoo%2FCMAK/lists"}