{"id":19249296,"url":"https://github.com/sous-chefs/kafka","last_synced_at":"2025-04-09T20:14:21.277Z","repository":{"id":9364947,"uuid":"11219679","full_name":"sous-chefs/kafka","owner":"sous-chefs","description":"Development repository for the kafka cookbook","archived":false,"fork":false,"pushed_at":"2024-12-04T18:13:59.000Z","size":938,"stargazers_count":91,"open_issues_count":6,"forks_count":104,"subscribers_count":30,"default_branch":"main","last_synced_at":"2025-04-09T20:14:12.012Z","etag":null,"topics":["chef","chef-cookbook","chef-resource","hacktoberfest","kafka","managed-by-terraform"],"latest_commit_sha":null,"homepage":"https://supermarket.chef.io/cookbooks/kafka","language":"Ruby","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":"pinterest/teletraan","license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/sous-chefs.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":".github/CODEOWNERS","security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null},"funding":{"open_collective":"sous-chefs"}},"created_at":"2013-07-06T15:01:06.000Z","updated_at":"2025-03-25T14:04:53.000Z","dependencies_parsed_at":"2024-02-20T17:14:02.392Z","dependency_job_id":null,"html_url":"https://github.com/sous-chefs/kafka","commit_stats":{"total_commits":860,"total_committers":31,"mean_commits":"27.741935483870968","dds":0.2755813953488372,"last_synced_commit":"e90f60deaf1c130e28edae4c63f41e460888e231"},"previous_names":[],"tags_count":44,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sous-chefs%2Fkafka","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sous-chefs%2Fkafka/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sous-chefs%2Fkafka/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sous-chefs%2Fkafka/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/sous-chefs","download_url":"https://codeload.github.com/sous-chefs/kafka/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248103872,"owners_count":21048245,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["chef","chef-cookbook","chef-resource","hacktoberfest","kafka","managed-by-terraform"],"created_at":"2024-11-09T18:13:35.581Z","updated_at":"2025-04-09T20:14:21.252Z","avatar_url":"https://github.com/sous-chefs.png","language":"Ruby","funding_links":["https://opencollective.com/sous-chefs"],"categories":[],"sub_categories":[],"readme":"# kafka cookbook\n\n[![Cookbook Version](https://img.shields.io/cookbook/v/kafka.svg)](https://supermarket.chef.io/cookbooks/kafka)\n[![Build Status](https://img.shields.io/circleci/project/github/sous-chefs/kafka/master.svg)](https://circleci.com/gh/sous-chefs/kafka)\n[![OpenCollective](https://opencollective.com/sous-chefs/backers/badge.svg)](#backers)\n[![OpenCollective](https://opencollective.com/sous-chefs/sponsors/badge.svg)](#sponsors)\n[![License](https://img.shields.io/badge/License-Apache%202.0-green.svg)](https://opensource.org/licenses/Apache-2.0)\n\nInstalls and configures Kafka `\u003e= v0.8.1`.\n\nInitially based on the Kafka cookbook released by Webtrends (thanks!), but with a few\nnotable differences:\n\n- does not depend on runit cookbook.\n- does not depend on zookeeper cookbook, thus it will not search for nodes with\n  a specific role or such, that is left up to you to decide.\n- intended to be used by wrapper cookbooks.\n\n## Maintainers\n\nThis cookbook is maintained by the Sous Chefs. The Sous Chefs are a community of Chef cookbook maintainers working together to maintain important cookbooks. If you’d like to know more please visit [sous-chefs.org](https://sous-chefs.org/) or come chat with us on the Chef Community Slack in [#sous-chefs](https://chefcommunity.slack.com/messages/C2V7B88SF).\n\n## Requirements\n\nThis cookbook does not depend on any specific cookbooks, but it requires that\njava is installed on the system, thus the `java` cookbook is recommended.\n\nFurthermore, Kafka requires ZooKeeper for coordination, and this cookbook does\nnot install or manage ZooKeeper to any extent.\nA general recommendation is to not run Kafka and ZooKeeper on the same hardware.\n\nChef 13+\n\n### Platform\n\n- Amazon Linux\n- CentOS 6 and 7\n- Debian 7 and 8\n- Fedora 21\n- Ubuntu 14.04\n\nThe platforms / versions listed above are the ones that are included in\n`.kitchen.yml` and/or tested in the wild, so it might work on other platforms as\nwell, YMMV.\n\n## Attributes\n\nIn order to keep the README in some kind of manageable state (and thus in sync\nwith attributes), attributes are documented inline (in the `attribute` files\nthat is).\n\nAttributes concerning configuration of a Kafka broker are to be set under the\n`broker` namespace, as such:\n\n```ruby\nnode.default['kafka']['broker']['log.dirs'] = %w[/tmp/kafka-logs]\nnode.default['kafka']['broker']['num.io.threads'] = 4\nnode.default['kafka']['broker']['zookeeper.connect'] = %w[localhost:2181]\n```\n\nThe attribute names match the configuration names that Kafka uses to make it\neasier to support new and old versions of Kafka simultaneously, by avoiding\n\"hardcoded\" attribute names for configuration options, so please refer to the\nofficial documentation for the release at your hand.\n\nRefer to the official documentation for the version of Kafka that you're\ninstalling.\nDocumentation for the latest release can be found [over here](https://kafka.apache.org/documentation.html#brokerconfigs).\n\n## Recipes\n\nThis section describes the different recipes that are available.\n\n### default\n\nDownloads and installs Kafka from the official binary releases.\nDefaults to installing `v1.1.1` of Kafka.\n\n## Controlling restart of Kafka brokers in a cluster\n\nAny changes made to the broker configuration could result in a restart of the\nKafka broker, if the `kafka.automatic_restart` attribute is set to `true`.\nIf Chef runs as a daemon on all of the nodes this could result in all of the Kafka\nbrokers being brought down at the same time, resulting in unavailability of\nservice.\n\nIf unavailability is an issue, this cookbook provides an option to implement custom\nlogic to control the restart of Kafka brokers so that not all of the brokers in\na cluster are stopped at the same time.\nFor example the custom logic can be something along the lines of acquiring a lock\nin ZooKeeper and when held the broker is allowed to restart.\nBe aware that a restart might take quite some time if you're using controlled\nshutdown and have a lot of partitions, and Chef usually have some timeout for\neach resource.\n\nBy default the resources in the [`_coordinate`](https://github.com/mthssdrbrg/kafka-cookbook/blob/master/recipes/_coordinate.rb)\nrecipe performs the start/restart of the `kafka` service.\nIf custom logic needs to be implemented, this recipe can be replaced with\nanother recipe, but don't forget to update the `kafka.start_coordination.recipe`\nattribute.\n\nThe only requirement is that the new recipe has a `ruby_block` resource with\n`'coordinate-kafka-start'` as ID.\nThe following is a sample recipe that shows roughly what one can do with this\nfeature.\n\n```ruby\nruby_block 'coordinate-kafka-start' do\n  block do\n    Chef::Log.info 'Custom recipe to coordinate Kafka start/restart'\n  end\n  action :nothing\n  notifies :create, 'ruby_block[restart-coordination]', :delayed\nend\n\nruby_block 'restart-coordination' do\n  block do\n    Chef::Log.info 'Implement the process to coordinate the restart, like using ZK'\n  end\n  action :nothing\n  # `kafka_service_resource` is a helper method that will return the correct\n  # resource name depending on whether you're using `runit` or not.\n  notifies :restart, kafka_service_resource, :delayed\n  notifies :create, 'ruby_block[restart-coordination-cleanup]', :delayed\nend\n\nruby_block 'restart-coordination-cleanup' do\n  block do\n    Chef::Log.info 'Implement any cleanup logic required after restart like releasing locks'\n  end\n  action :nothing\nend\n```\n\nPlease refer to [issue #58](https://github.com/mthssdrbrg/kafka-cookbook/issues/58) for background of this feature.\n\n## FAQ\n\n### Kafka dies for no apparent reason (ulimit)\n\nDepending on your system / infrastructure setup you might run into issues with\nKafka just sporadically dying for no obvious reason.\nOne thing you might want to look into is the file handle limit as Kafka tend to\nkeep a lot file handles open due to socket connections (depends on the number of\nbrokers, producers and consumers) and the actual data log files (depends on\nthe number of partitions and log segment and/or log roll settings).\n\nIt's possible to set a specific `ulimit` for Kafka using the `node['kafka']['ulimit_file']`\nattribute.\nIf this value is not set, Kafka will use whatever the system default is, which\nas stated previously might not be enough, so it might be wise to set a higher\nlimit.\n\n### How do I get started locally? (minimal required setup)\n\nIf you want to hit the ground running and just setup a single broker (or a\ncluster for that matter) locally, these are the necessary `broker` attributes\nthat needs to be set (assumes that you have ZooKeeper running on port 2181\nlocally):\n\n```ruby\nnode.default['kafka']['broker']['zookeeper.connect'] = 'localhost:2181'\n# This shouldn't normally be necessary, but might need to be set explicitly\n# if you're having trouble connecting to the brokers.\nnode.default['kafka']['broker']['hostname'] = '127.0.0.1' # or perhaps 'localhost'\n```\n\nIf you plan on running a cluster locally you will want to set separate\nvalues for the following configuration options:\n\n```ruby\nnode.default['kafka']['broker']['broker.id'] = \u003cid\u003e\nnode.default['kafka']['broker']['port'] = \u003cport\u003e\nnode.default['kafka']['broker']['log.dirs'] = \u003cdir path\u003e\n```\n\nOther than that things should work as expected, though depending on what\nplatform you're running on, you might want to change the install and config\ndirectories as well. See `attributes/default.rb` and `recipes/_defaults.rb` for\nthe default path regarding directories that Kafka will use.\n\n### Kafka killed prematurely (kill timeout)\n\nWhen using `controlled shutdown` and either `systemd` or `upstart` as init\nsystem you might run into issues with Kafka being killed before it has managed\nto shutdown completely, resulting in long recovery times.\n\nNot sure if it's possible to configure either `systemd` or `upstart` to not\nautomatically kill processes, but a workaround is to set `kafka.kill_timeout` to\na sufficiently high value.\n\n## Contributing\n\n- Fork the repository on Github\n- Create a named feature branch (like `add-component-x`)\n- Write your change\n- Check that your change works, for example with Vagrant\n- Submit a Pull Request using Github\n\n## Contributors\n\nThis project exists thanks to all the people who [contribute.](https://opencollective.com/sous-chefs/contributors.svg?width=890\u0026button=false)\n\n### Backers\n\nThank you to all our backers!\n\n![https://opencollective.com/sous-chefs#backers](https://opencollective.com/sous-chefs/backers.svg?width=600\u0026avatarHeight=40)\n\n### Sponsors\n\nSupport this project by becoming a sponsor. Your logo will show up here with a link to your website.\n\n![https://opencollective.com/sous-chefs/sponsor/0/website](https://opencollective.com/sous-chefs/sponsor/0/avatar.svg?avatarHeight=100)\n![https://opencollective.com/sous-chefs/sponsor/1/website](https://opencollective.com/sous-chefs/sponsor/1/avatar.svg?avatarHeight=100)\n![https://opencollective.com/sous-chefs/sponsor/2/website](https://opencollective.com/sous-chefs/sponsor/2/avatar.svg?avatarHeight=100)\n![https://opencollective.com/sous-chefs/sponsor/3/website](https://opencollective.com/sous-chefs/sponsor/3/avatar.svg?avatarHeight=100)\n![https://opencollective.com/sous-chefs/sponsor/4/website](https://opencollective.com/sous-chefs/sponsor/4/avatar.svg?avatarHeight=100)\n![https://opencollective.com/sous-chefs/sponsor/5/website](https://opencollective.com/sous-chefs/sponsor/5/avatar.svg?avatarHeight=100)\n![https://opencollective.com/sous-chefs/sponsor/6/website](https://opencollective.com/sous-chefs/sponsor/6/avatar.svg?avatarHeight=100)\n![https://opencollective.com/sous-chefs/sponsor/7/website](https://opencollective.com/sous-chefs/sponsor/7/avatar.svg?avatarHeight=100)\n![https://opencollective.com/sous-chefs/sponsor/8/website](https://opencollective.com/sous-chefs/sponsor/8/avatar.svg?avatarHeight=100)\n![https://opencollective.com/sous-chefs/sponsor/9/website](https://opencollective.com/sous-chefs/sponsor/9/avatar.svg?avatarHeight=100)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsous-chefs%2Fkafka","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsous-chefs%2Fkafka","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsous-chefs%2Fkafka/lists"}