{"id":13753365,"url":"https://github.com/sevenmind/kaufmann_ex","last_synced_at":"2025-05-07T16:08:32.024Z","repository":{"id":57512541,"uuid":"124120583","full_name":"sevenmind/kaufmann_ex","owner":"sevenmind","description":"Kafka backed service library. ","archived":false,"fork":false,"pushed_at":"2023-09-12T15:40:58.000Z","size":401,"stargazers_count":92,"open_issues_count":2,"forks_count":7,"subscribers_count":9,"default_branch":"master","last_synced_at":"2025-01-02T23:10:34.758Z","etag":null,"topics":["avro","concurrency","elixir","kafka","microservice","schema"],"latest_commit_sha":null,"homepage":"","language":"Elixir","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/sevenmind.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-03-06T18:24:03.000Z","updated_at":"2024-07-15T19:45:07.000Z","dependencies_parsed_at":"2024-08-03T09:15:52.439Z","dependency_job_id":null,"html_url":"https://github.com/sevenmind/kaufmann_ex","commit_stats":{"total_commits":132,"total_committers":4,"mean_commits":33.0,"dds":"0.31060606060606055","last_synced_commit":"44225125946921850316c272db53175bb1658fb7"},"previous_names":[],"tags_count":8,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sevenmind%2Fkaufmann_ex","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sevenmind%2Fkaufmann_ex/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sevenmind%2Fkaufmann_ex/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sevenmind%2Fkaufmann_ex/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/sevenmind","download_url":"https://codeload.github.com/sevenmind/kaufmann_ex/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":233352284,"owners_count":18663267,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["avro","concurrency","elixir","kafka","microservice","schema"],"created_at":"2024-08-03T09:01:20.997Z","updated_at":"2025-01-10T13:34:36.359Z","avatar_url":"https://github.com/sevenmind.png","language":"Elixir","funding_links":[],"categories":["schema"],"sub_categories":[],"readme":"# KaufmannEx\n\n[![Build\nStatus](https://travis-ci.org/sevenmind/kaufmann_ex.svg?branch=master)](https://travis-ci.org/sevenmind/kaufmann_ex)\n[![Hex.pm](https://img.shields.io/hexpm/v/kaufmann_ex.svg)](https://hex.pm/packages/kaufmann_ex)\n[![Inline\ndocs](http://inch-ci.org/github/sevenmind/kaufmann_ex.svg)](http://inch-ci.org/github/sevenmind/kaufmann_ex)\n[![Ebert](https://ebertapp.io/github/sevenmind/kaufmann_ex.svg)](https://ebertapp.io/github/sevenmind/kaufmann_ex)\n[![codebeat badge](https://codebeat.co/badges/5a95d37f-8087-4d99-8df3-758991d602ff)](https://codebeat.co/projects/github-com-sevenmind-kaufmann_ex-master)\n\nCheck out [our blog post about\nKaufmannEx](https://medium.com/@7mind_dev/kaufmann-ex-317415c27978)\n\nThe goal of KaufmannEx is to provide a simple to use library for building kafka\nbased microservices.\n\nIt should be simple and fast to write new microservices with\n[Avro](https://avro.apache.org/docs/current/) or JSON event schemas  (or\nwhatever).\n\nTieing `KafkaEx`, `AvroEx`, and `Schemex`.\n\nKaufmannEx exists to make it easy to consume Avro encoded messages off of a\nkafka broker in a parallel, controlled, manner. \n\n## Installation\n\nIf [available in Hex](https://hex.pm/docs/publish), the package can be installed\nby adding `kaufmann_ex` to your list of dependencies in `mix.exs`:\n\n```elixir\ndef deps do\n  [\n    {:kaufmann_ex, \"~\u003e 0.4.0-dev\"}\n  ]\nend\n```\n\nDocumentation can be generated with\n[ExDoc](https://github.com/elixir-lang/ex_doc) and published on\n[HexDocs](https://hexdocs.pm). Once published, the docs can be found at\n[https://hexdocs.pm/kaufmann_ex](https://hexdocs.pm/kaufmann_ex).\n\n## Usage\n\nKaufmannEx is under _very_ active development. So it's a little more complicated\nthan is ideal at the moment.\n\nKaufmannEx needs:\n\n* to be in `mix.exs`\n* to be in your Supervision Tree\n* `config.exs`\n* `event_handler_mod`\n* schemas\n\n### Supervision Tree\n\nTo Use KaufmannEx start by adding it to your supervision tree\n\n```elixir\ndefmodule Sample.Application do\n  use Application\n  require Logger\n\n  def start(_type, _args) do\n    children = [\n      KaufmannEx.Supervisor\n    ]\n\n    opts = [strategy: :one_for_one, name: Sample.Supervisor]\n    Supervisor.start_link(children, opts)\n  end\nend\n```\n\n### Config.exs\n\nKaufmannEx expects configuration in your Application Config.\n\nKaufman depends on KafkaEx which must also be configured\n\nA `config.exs` may include something like this:\n\n```elixir\nconfig :kaufmann_ex,\n  consumer_group: \"Consumer-Group\",\n  default_topics: [\"default-topic\"],\n  event_handler_mod: MyApp.EventHandler,\n  producer_mod: KaufmannEx.Publisher,\n  schema_path: \"priv/schemas\",\n  schema_registry_uri: \"http://localhost:8081\",\n  transcoder: [\n    default: KaufmannEx.Transcoder.SevenAvro,\n    json: KaufmannEx.Transcoder.Json\n  ]\n\nconfig :kafka_ex,\n  brokers: [\n    {\n      localhost,\n      9092\n    }\n  ],\n  consumer_group: \"Consumer-Group\",\n  commit_threshold: 10,\n  commit_interval: 100,\n  sync_timeout: 10_000\n```\n\n### `event_handler_mod`\n\nKaufmannEx expects an event handler module with the callback `given_event/1`\n\n```elixir\ndefmodule MyApp.EventHandler do\n  use KaufmannEx.EventHandler\n  alias KaufmannEx.Schemas.Event\n\n  @behaviour KaufmannEx.EventHandler\n\n  @impl true\n  def given_event(%Event{name: \"test.command\", payload: payload}) do\n    message_body = do_some_work(payload)\n\n    {:reply, [{\"test.event\", message_body, topic}]}\n  end\n\n  # In the event of an error a ErrorEvent is emitted\n  def given_event(%Event{name: \"this.event.returns.error\", payload: payload}) do\n    {:error, :unhandled_event}\n  end\nend\n```\n\n## Events\n\nKaufmannEx assumes every event has a matching event Avro or JSON Event Schema.\n\nAll AVRO events are expected to include a `meta` metadata key.\n\nIf an Event causes an exception it will emit an error event with\n`\"event.error.#{event.name}\"` as the `event_name`. Events that raise exceptions\nare not retried or persisted beyond emitting this error event. If specific\nhandling of failing events is important to you, implement a dead-letter service\nor similar.\n\n## Internals\n\nKaufmannEx uses `KafkaEx.ConsumerGroup` to subscribe to kafka topic/s. Events\nare consumed by a `kafka_ex_gen_stage_consumer` stage to a `Flow` event handler\n`KaufmannEx.Consumer.Flow`.\n\n## Release Tasks\n\nThere are a few release tasks intended for use as [Distillery custom\ncommands](https://hexdocs.pm/distillery/custom-commands.html). Distillery's\ncustom commands don't provide the environment we're used to with mix tasks, so\nextra configuration and care is needed. \n\n#### `migrate_schemas`\n\nMigrate Schemas will attempt to register all schemas in the implementing\nproject's `priv/schemas` directory.\n\n#### `reinit_service`\n\nThis task is intended to be used to recover idempotent services from a\ncatastrophic failure or substantial architectural change. \n\nReInit Service will reset the configured consumer group to the earliest\navailable Kafka Offset. It will then consume all events from the Kafka broker\nuntil the specified offset is reached (Or all events are consumed).\n\nBy default message publication is disabled during reinitialization. This can be\noverridden in `KaufmannEx.ReleaseTasks.reinit_service/4`.\n\n### Configuration \u0026 Use\n\nThese tasks are intended for use with Distillery in a release environment.  In\nthese examples the application is named `Sample`. \n\n#### `release_tasks.ex` \n\n```elixir\ndefmodule Sample.ReleaseTasks do\n  def migrate_schemas do\n    Application.load(:kaufmann_ex)\n\n    KaufmannEx.ReleaseTasks.migrate_schemas(:sample)\n  end\n\n  def reinit_service do\n    Application.load(:kaufmann_ex)\n    KaufmannEx.ReleaseTasks.reinit_service(:sample)  \n  end\nend\n```\n\n#### `rel/config.exs`\n\n```elixir\n...\n\nrelease :sample do\n  set(\n      commands: [\n        migrate_schemas: \"rel/commands/migrate_schemas.sh\",\n        reinit_service: \"rel/commands/reinit_service.sh\"\n      ]\n    )\nend\n```\n\n#### `rel/commands/migrate_schemas.sh`\n\n```\n#!/bin/sh\n\n$RELEASE_ROOT_DIR/bin/sample command Elixir.Sample.ReleaseTasks migrate_schemas\n```\n\n\n## Common questions\n\n### When are offsets commited? In case of a node going down, will it lose messages?\n\nIt is possible to lose events when a node goes down. But we try to prevent that\nfrom happening.\n\n1. The backpressure in Kaufmann prevents pulling more events than processing capacity. \n2. KuafmannEx uses the default [KafkaEx GenConsumer asynchronous offset commit\n   behavior](https://hexdocs.pm/kafka_ex/KafkaEx.GenConsumer.html#module-asynchronous-offset-commits).\n   Offsets are committed asynchronously on a timer or event count.\n\nIdeally the `kafka_ex :commit_threshold` should be set somewhat larger than\n`kaufmann_ex :max_demand` (the default is 100 and 50, respectively). This should\nmake it less likely that the node will be processing already-committed messages\nwhen it goes down.\n\n### Can order of delivery be guaranteed?\n\nKafka can only guarantee event ordering within a single partition. KaufmannEx\nwill consume events in the order published, but event processing is not\nguaranteed to be sequential. KaufmannEx (as with kafka in general) does\ncannot provide strong consistency.\n\n\n## Telemetry\n\nKaufmann provides some internal\n[:temeletry](https://github.com/beam-telemetry/telemetry) events\n\n```elixir\n[:kaufmann_ex, :schemas, :decode]\n[:kaufmann_ex, :event_handler, :handle_event]\n[:kaufmann_ex, :schema, :encode]\n[:kaufmann_ex, :publisher, :publish]\n```\n\nKaufmann provides an optional logger demonstrating consumption of these events\nin `KaufmannEx.TelemetryLogger`. This logger can be started by adding\n`KaufmannEx.TelemetryLogger` to your supervision tree.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsevenmind%2Fkaufmann_ex","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsevenmind%2Fkaufmann_ex","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsevenmind%2Fkaufmann_ex/lists"}