{"id":21131255,"url":"https://github.com/g-corp/kafe","last_synced_at":"2025-07-09T01:33:41.572Z","repository":{"id":48301075,"uuid":"39394769","full_name":"G-Corp/kafe","owner":"G-Corp","description":"A Kafka client in pure Erlang","archived":false,"fork":false,"pushed_at":"2021-08-02T15:55:28.000Z","size":4216,"stargazers_count":7,"open_issues_count":1,"forks_count":9,"subscribers_count":6,"default_branch":"master","last_synced_at":"2025-07-03T09:54:35.555Z","etag":null,"topics":["elixir","erlang","kafka"],"latest_commit_sha":null,"homepage":null,"language":"Erlang","has_issues":false,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/G-Corp.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2015-07-20T16:19:49.000Z","updated_at":"2021-03-11T23:38:01.000Z","dependencies_parsed_at":"2022-09-26T16:31:02.405Z","dependency_job_id":null,"html_url":"https://github.com/G-Corp/kafe","commit_stats":null,"previous_names":[],"tags_count":4,"template":false,"template_full_name":null,"purl":"pkg:github/G-Corp/kafe","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/G-Corp%2Fkafe","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/G-Corp%2Fkafe/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/G-Corp%2Fkafe/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/G-Corp%2Fkafe/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/G-Corp","download_url":"https://codeload.github.com/G-Corp/kafe/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/G-Corp%2Fkafe/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":264375601,"owners_count":23598413,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["elixir","erlang","kafka"],"created_at":"2024-11-20T05:52:27.278Z","updated_at":"2025-07-09T01:33:40.451Z","avatar_url":"https://github.com/G-Corp.png","language":"Erlang","funding_links":[],"categories":[],"sub_categories":[],"readme":"\n\n# A Kafka client for Erlang and Elixir #\n\nCopyright (c) 2014, 2015 Finexkap, 2015, 2016, 2017, 2018 G-Corp, 2015, 2016, 2017 BotsUnit\n\n__Version:__ 2.2.0\n\n__Authors:__ Gregoire Lejeune ([`gregoire.lejeune@finexkap.com`](mailto:gregoire.lejeune@finexkap.com)), Gregoire Lejeune ([`greg@g-corp.io`](mailto:greg@g-corp.io)), Gregoire Lejeune ([`gregoire.lejeune@botsunit.com`](mailto:gregoire.lejeune@botsunit.com)).\n\n[![Hex.pm version](https://img.shields.io/hexpm/v/kafe.svg?style=flat-square)](https://hex.pm/packages/kafe)\n[![Hex.pm downloads](https://img.shields.io/hexpm/dt/kafe.svg?style=flat-square)](https://hex.pm/packages/kafe)\n[![License](https://img.shields.io/hexpm/l/kafe.svg?style=flat-square)](https://hex.pm/packages/kafe)\n[![Build Status](https://travis-ci.org/G-Corp/kafe.svg?branch=master)](https://travis-ci.org/G-Corp/kafe)\n\n__Version 2.0.0 cause changes in the following APIs :__\n\n* [`kafe:start_consumer/3`](https://github.com/G-Corp/kafe/blob/master/doc/kafe.md#start_consumer-3)\n\n* [`kafe:fetch/3`](https://github.com/G-Corp/kafe/blob/master/doc/kafe.md#fetch-3)\n\n\n__Kafe__ has been tested with Kafka 0.9 and above\n\nYou can also use it with Kafka 0.8 but [`kafe_consumer`](https://github.com/G-Corp/kafe/blob/master/doc/kafe_consumer.md) is not compatible with this version.\n\n\n### Links ###\n* [Apache Kafka](http://kafka.apache.org)\n* [Apache Kafka Protocol](https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocol)\n\n\n\n### Configuration ###\n\n\n\u003ctable width=\"100%\" border=\"0\" summary=\"configuration\"\u003e\n\u003ctr\u003e\u003ctd\u003ebrokers\u003c/td\u003e\u003ctd\u003e\u003ctt\u003e[{inet:hostname(), inet:port_number()}]\u003c/tt\u003e\u003c/td\u003e\u003ctd\u003eList of brokers\u003c/td\u003e\u003ctd\u003e\u003ctt\u003e[{\"localhost\", 9092}]\u003c/tt\u003e\u003c/td\u003e\u003c/tr\u003e\n\u003ctr\u003e\u003ctd\u003epool_size\u003c/td\u003e\u003ctd\u003e\u003ctt\u003einteger()\u003c/tt\u003e\u003c/td\u003e\u003ctd\u003eInitial connection pool/brocker\u003c/td\u003e\u003ctd\u003e\u003ctt\u003e5\u003c/tt\u003e\u003c/td\u003e\u003c/tr\u003e\n\u003ctr\u003e\u003ctd\u003echunk_pool_size\u003c/td\u003e\u003ctd\u003e\u003ctt\u003einteger()\u003c/tt\u003e\u003c/td\u003e\u003ctd\u003eSize of new connection pool/brocker\u003c/td\u003e\u003ctd\u003e\u003ctt\u003e10\u003c/tt\u003e\u003c/td\u003e\u003c/tr\u003e\n\u003ctr\u003e\u003ctd\u003ebrokers_update_frequency\u003c/td\u003e\u003ctd\u003e\u003ctt\u003einteger()\u003c/tt\u003e\u003c/td\u003e\u003ctd\u003eFrequency (ms) for brokers update\u003c/td\u003e\u003ctd\u003e\u003ctt\u003e60000\u003c/tt\u003e\u003c/td\u003e\u003c/tr\u003e\n\u003ctr\u003e\u003ctd\u003eprotocol_timeout\u003c/td\u003e\u003ctd\u003e\u003ctt\u003einteger()\u003c/tt\u003e\u003c/td\u003e\u003ctd\u003eAPI call timeout\u003c/td\u003e\u003ctd\u003e\u003ctt\u003e60000\u003c/tt\u003e\u003c/td\u003e\u003c/tr\u003e\n\u003ctr\u003e\u003ctd\u003eclient_id\u003c/td\u003e\u003ctd\u003e\u003ctt\u003ebinary()\u003c/tt\u003e\u003c/td\u003e\u003ctd\u003eClient ID Name\u003c/td\u003e\u003ctd\u003e\u003ctt\u003e\u003c\u003c\"kafe\"\u003e\u003e\u003c/tt\u003e\u003c/td\u003e\u003c/tr\u003e\n\u003ctr\u003e\u003ctd\u003eapi_version\u003c/td\u003e\u003ctd\u003e\u003ctt\u003e[{integer(), integer()}] | integer() | auto\u003c/tt\u003e\u003c/td\u003e\u003ctd\u003eAPI Version\u003c/td\u003e\u003ctd\u003e\u003ctt\u003eauto\u003csup\u003e*\u003c/sup\u003e\u003c/tt\u003e\u003c/td\u003e\u003c/tr\u003e\n\u003ctr\u003e\u003ctd\u003ecorrelation_id\u003c/td\u003e\u003ctd\u003e\u003ctt\u003einteger()\u003c/tt\u003e\u003c/td\u003e\u003ctd\u003eCorrelation ID\u003c/td\u003e\u003ctd\u003e\u003ctt\u003e0\u003c/tt\u003e\u003c/td\u003e\u003c/tr\u003e\n\u003ctr\u003e\u003ctd\u003esocket\u003c/td\u003e\u003ctd\u003e\u003ctt\u003e[{sndbuf, integer()}, {recbuf, integer()}, {buffer, integer()}]\u003c/tt\u003e\u003c/td\u003e\u003ctd\u003eSocker configuration\u003c/td\u003e\u003ctd\u003e\u003ctt\u003e[{sndbuf, 4194304}, {recbuf, 4194304}, {buffer, 4194304}]\u003c/tt\u003e\u003c/td\u003e\u003c/tr\u003e\n\u003c/table\u003e\n\n\n\u003csup\u003e*\u003c/sup\u003e\n use `0` with Kafka \u003e= 0.8 \u003c 0.9 ; `auto` with Kafka \u003e= 0.9 \u003c 0.10\n\nExample :\n\n```\n\n[\n  {kafe, [\n    {brokers, [\n      {\"localhost\", 9092},\n      {\"localhost\", 9093},\n      {\"localhost\", 9094}\n    ]},\n    {pool_size, 1},\n    {chunk_pool_size, 2},\n    {brokers_update_frequency, 10000},\n    {client_id, \u003c\u003c\"kafe\"\u003e\u003e},\n    {api_version, auto},\n    {correlation_id, 0},\n    {socket, [\n      {sndbuf, 4194304},\n      {recbuf, 4194304},\n      {buffer, 4194304}\n    ]},\n  ]}\n]\n\n```\n\n__Kafe__ use [lager](https://github.com/basho/lager) ; see also how to [configure](https://github.com/basho/lager#configuration) it.\n\n\n#### Custom API version ####\n\nIf you need to, you can specify each version of the protocol APIs. To do so, use a list as value for the `api_version`'s configuration. This list must be a list of tuples where the first element is the api key and the second is the api version.\n\nExample :\n\n```\n\n...\n{api_version, [{0, 0}, {0, 1}, {0, 0}, ...]}\n...\n\n```\n\n\n### Create a consumer ###\n\n\n#### Using a function ####\n\nTo create a consumer, create a function with 6 parameters :\n\n```\n\n-module(my_consumer).\n\n-export([consume/6]).\n\nconsume(CommitID, Topic, Partition, Offset, Key, Value) -\u003e\n  % Do something with Topic/Partition/Offset/Key/Value\n  ok.\n\n```\n\nThe `consume` function must return `ok` if the message was treated, or `{error, term()}` on error.\n\nThen start a new consumer :\n\n```\n\n...\nkafe:start(),\n...\nkafe:start_consumer(my_group, fun my_consumer:consume/6, Options),\n...\n\n```\n\nSee [`kafe:start_consumer/3`](https://github.com/G-Corp/kafe/blob/master/doc/kafe.md#start_consumer-3) for the available `Options`.\n\nIn the `consume` function, if you didn't start the consumer in autocommit mode (using `before_processing | after_processing` in the `commit` options),\nyou need to commit manually when you have finished to treat the message. To do so, use [`kafe_consumer:commit/4`](https://github.com/G-Corp/kafe/blob/master/doc/kafe_consumer.md#commit-4).\n\nWhen you are done with your consumer, stop it :\n\n```\n\n...\nkafe:stop_consumer(my_group),\n...\n\n```\n\n\n#### Using the `kafe_consumer_subscriber` behaviour ####\n\n```\n\n-module(my_consumer).\n-behaviour(kafe_consumer_subscriber).\n\n-export([init/4, handle_message/2]).\n-include_lib(\"kafe/include/kafe_consumer.hrl\").\n\n-record(state, {\n               }).\n\ninit(Group, Topic, Partition, Args) -\u003e\n  % Do something with Group, Topic, Partition, Args\n  {ok, #state{}}.\n\nhandle_message(Message, State) -\u003e\n  % Do something with Message\n  % And update your State (if needed)\n  {ok, NewState}.\n\n```\n\nThen start a new consumer :\n\n```\n\n...\nkafe:start().\n...\nkafe:start_consumer(my_group, {my_consumer, Args}, Options).\n% Or\nkafe:start_consumer(my_group, my_consumer, Options).\n...\n\n```\n\nTo commit a message (if you need to), use [`kafe_consumer:commit/4`](https://github.com/G-Corp/kafe/blob/master/doc/kafe_consumer.md#commit-4).\n\n\n### Using with Elixir ###\n\nElixir' users can use `Kafe` and `Kafe.Consumer` instead of `:kafe` and `:kafe_consumer`.\n\n```\n\ndefmodule My.Consumer do\n  def consume(commit_id, topic, partition, offset, key, value) do\n    # Do something with topic/partition/offset/key/value\n    :ok\n  end\nend\n\ndefmodule My.Consumer.Subscriber do\n  @behaviour Kafe.Consumer.Subscriber\n  require Kafe.Records\n\n  def init(group, topic, partition, args) do\n    % Do something with group/topic/partition/args\n    % and create the state\n    {:ok, state}\n  end\n\n  def handle_message(message, state) do\n    msg = Kafe.Records.message(message)\n    % Do something with msg and update (or not) the state\n    {:ok, new_state}\n  end\nend\n\n```\n\n```\n\n...\nKafe.start()\n...\nKafe.start_consumer(:my_group, \u0026My.Consumer.consume/6, options)\n# or\nKafe.start_consumer(:my_group, {My.Consumer.Subscriber, args}, options)\n# or\nKafe.start_consumer(:my_group, My.Consumer.Subscriber, options)\n...\nKafe.stop_consumer(:my_group)\n...\n\n```\n\n\n### Metrics ###\n\nTo enable metrics :\n\n1/ Add [metrics \u003e 2.2](https://hex.pm/packages/metrics) in your dependencies.\n\n2/ Set `enable_metrics` to true in the `kafe` configuration :\n\n```\n\n{kafe, [\n  ...\n  {enable_metrics, true},\n  ...\n]}\n\n```\n\n3/ Adding a metrics module in your configuration :\n\n```\n\n{metrics, [\n  {metrics_mod, metrics_folsom}\n]}\n\n```\n\nYou can choose between [Folsom](https://github.com/folsom-project/folsom) (`{metrics_mod, metrics_folsom}`), [Exometer](https://github.com/Feuerlabs/exometer) (`{metrics_mod, metrics_exometer}`) or [Grapherl](https://github.com/processone/grapherl) (`{metrics_mod, metrics_grapherl}`).\n\nBe sure that's Folsom, Exometer or Grapherl and metrics is started before starting Kafe.\n\n```\n\napplication:ensure_all_started(folsom).\napplication:ensure_all_started(metrics).\napplication:ensure_all_started(kafe).\n\n```\n\nMetrics are disabled by default.\n\nKafe offers the following metrics :\n\n\n\u003ctable\u003e\n\u003ctr\u003e\u003cth\u003eName\u003c/th\u003e\u003cth\u003eType\u003c/th\u003e\u003cth\u003eDescription\u003c/th\u003e\u003c/tr\u003e\n\u003ctr\u003e\u003ctd\u003ekafe_consumer.CONSUMER_GROUP.messages.fetch\u003c/td\u003e\u003ctd\u003egauge\u003c/td\u003e\u003ctd\u003eNumber of received messages on the last fetch for the CONSUMER_GROUP\u003c/td\u003e\u003c/tr\u003e\n\u003ctr\u003e\u003ctd\u003ekafe_consumer.CONSUMER_GROUP.TOPIC.PARTITION.messages.fetch\u003c/td\u003e\u003ctd\u003egauge\u003c/td\u003e\u003ctd\u003eNumber of received messages on the last fetch for the {TOPIC, PARTITION} and CONSUMER_GROUP\u003c/td\u003e\u003c/tr\u003e\n\u003ctr\u003e\u003ctd\u003ekafe_consumer.CONSUMER_GROUP.messages\u003c/td\u003e\u003ctd\u003ecounter\u003c/td\u003e\u003ctd\u003eTotal number of received messages for the CONSUMER_GROUP\u003c/td\u003e\u003c/tr\u003e\n\u003ctr\u003e\u003ctd\u003ekafe_consumer.CONSUMER_GROUP.TOPIC.PARTITION.messages\u003c/td\u003e\u003ctd\u003ecounter\u003c/td\u003e\u003ctd\u003eTotal number of received messages for the {TOPIC, PARTITION} and CONSUMER_GROUP\u003c/td\u003e\u003c/tr\u003e\n\u003ctr\u003e\u003ctd\u003ekafe_consumer.CONSUMER_GROUP.TOPIC.PARTITION.duration.fetch\u003c/td\u003e\u003ctd\u003egauge\u003c/td\u003e\u003ctd\u003eFetch duration (ms) per message, for the {TOPIC, PARTITION} and CONSUMER_GROUP\u003c/td\u003e\u003c/tr\u003e\n\u003ctr\u003e\u003ctd\u003ekafe_consumer.CONSUMER_GROUP.TOPIC.PARTITION.pending_commits\u003c/td\u003e\u003ctd\u003egauge\u003c/td\u003e\u003ctd\u003eNumber of pending commits, for the {TOPIC, PARTITION} and CONSUMER_GROUP\u003c/td\u003e\u003c/tr\u003e\n\u003c/table\u003e\n\n\nYou can add a prefix to all metrics by adding a `metrics_prefix` in the `metrics` configuration :\n\n```\n\n{metrics, [\n  {metrics_mod, metrics_folsom},\n  {metrics_prefix, my_bot}\n]}\n\n```\n\n\n### Build and tests ###\n\n__Kafe__ use [rebar3](http://www.rebar3.org) and [bu.mk](https://github.com/G-Corp/bu.mk). So, you can use :\n\n* `./rebar3 compile` to compile Kafe.\n\n* `./rebar3 eunit` to run tests.\n\n* `./rebar3 ct` to run (integration) tests.\n\n* `./rebar3 edoc` to build documentation.\n\n* `./rebar3 elixir generate_mix` to generate `mix.exs` file.\n\n* `./rebar3 elixir generate_lib` to generate Elixir bindings.\n\n\nOr\n\n* `make release` Tag and release to hex.pm\n\n* `make integ` Run integration tests\n\n* `make docker-compose.yml` Create docker-compose.yml\n\n* `make docker-start` Start docker\n\n* `make docker-stop` Stop docker\n\n* `make elixir` Generate Elixir bindings (mix.exs and libs)\n\n* `make tests` Run tests\n\n* `make doc` Generate doc\n\n* `make dist` Create a distribution\n\n* `make clean` Clean\n\n* `make distclean` Clean the distribution\n\n* `make info` Display application informations\n\n* `make tag` Create a git tag\n\n* `make local.hex` Install hexfor Mix\n\n* `make local.rebar` Install rebar for Mix\n\n* `make bu-mk` Update bu.mk\n\n* `make help` Show this help.\n\n\nTo run the integration tests, you must start zookeeper and a kafka cluster (3 brokers) and have the following three topics :\n\n* `testone` : replication factor: 1, partitions: 1\n\n* `testtwo` : replication factor: 2, partitions: 2\n\n* `testthree` : replication factor: 3, partitions: 3\n\n\nYou can use the makefile rules `docker-compose.yml` and `docker-start` to help you to create this environment using docker (tested on Linux only).\n\n\n### API Documentation ###\n\nSee [documentation](https://github.com/G-Corp/kafe/blob/master/doc/.)\n\n\n### Contributing ###\n1. Fork it ( https://github.com/G-Corp/kafe/fork )\n1. Create your feature branch (`git checkout -b my-new-feature`)\n1. Commit your changes (`git commit -am 'Add some feature'`)\n1. Push to the branch (`git push origin my-new-feature`)\n1. Create a new Pull Request\n\n\n\n### Licence ###\n\nkafe is available for use under the following license, commonly known as the 3-clause (or \"modified\") BSD license:\n\nCopyright (c) 2014, 2015 Finexkap\u003cbr /\u003e\nCopyright (c) 2015, 2016, 2017 BotsUnit\u003cbr /\u003e\nCopyright (c) 2015, 2016, 2017, 2018 G-Corp\u003cbr /\u003e\n\nRedistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:\n\n* Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer.\n* Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution.\n* The name of the author may not be used to endorse or promote products derived from this software without specific prior written permission.\n\n\n\nTHIS SOFTWARE IS PROVIDED BY THE AUTHOR `AS IS` AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.\n\n\n## Modules ##\n\n\n\u003ctable width=\"100%\" border=\"0\" summary=\"list of modules\"\u003e\n\u003ctr\u003e\u003ctd\u003e\u003ca href=\"https://github.com/G-Corp/kafe/blob/master/doc/kafe.md\" class=\"module\"\u003ekafe\u003c/a\u003e\u003c/td\u003e\u003c/tr\u003e\n\u003ctr\u003e\u003ctd\u003e\u003ca href=\"https://github.com/G-Corp/kafe/blob/master/doc/kafe_consumer.md\" class=\"module\"\u003ekafe_consumer\u003c/a\u003e\u003c/td\u003e\u003c/tr\u003e\n\u003ctr\u003e\u003ctd\u003e\u003ca href=\"https://github.com/G-Corp/kafe/blob/master/doc/kafe_consumer_subscriber.md\" class=\"module\"\u003ekafe_consumer_subscriber\u003c/a\u003e\u003c/td\u003e\u003c/tr\u003e\u003c/table\u003e\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fg-corp%2Fkafe","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fg-corp%2Fkafe","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fg-corp%2Fkafe/lists"}