{"id":20515794,"url":"https://github.com/kuper-tech/sbmt-kafka_consumer","last_synced_at":"2025-04-09T12:08:08.206Z","repository":{"id":230401135,"uuid":"763536660","full_name":"Kuper-Tech/sbmt-kafka_consumer","owner":"Kuper-Tech","description":"Ruby gem for consuming Kafka messages","archived":false,"fork":false,"pushed_at":"2025-03-06T13:09:05.000Z","size":390,"stargazers_count":30,"open_issues_count":0,"forks_count":0,"subscribers_count":6,"default_branch":"master","last_synced_at":"2025-04-02T11:03:25.945Z","etag":null,"topics":["gem","inbox","kafka","outbox","ruby"],"latest_commit_sha":null,"homepage":"","language":"Ruby","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Kuper-Tech.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-02-26T13:36:04.000Z","updated_at":"2025-03-06T13:09:09.000Z","dependencies_parsed_at":"2024-06-07T09:05:30.376Z","dependency_job_id":"1b957e1b-e582-4b95-bc84-5240bc8c9c51","html_url":"https://github.com/Kuper-Tech/sbmt-kafka_consumer","commit_stats":{"total_commits":95,"total_committers":8,"mean_commits":11.875,"dds":0.5368421052631579,"last_synced_commit":"713ca79ba6879152d46526c8265f589a33758467"},"previous_names":["sbermarket-tech/sbmt-kafka_consumer","kuper-tech/sbmt-kafka_consumer"],"tags_count":68,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Kuper-Tech%2Fsbmt-kafka_consumer","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Kuper-Tech%2Fsbmt-kafka_consumer/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Kuper-Tech%2Fsbmt-kafka_consumer/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Kuper-Tech%2Fsbmt-kafka_consumer/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Kuper-Tech","download_url":"https://codeload.github.com/Kuper-Tech/sbmt-kafka_consumer/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248036067,"owners_count":21037092,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["gem","inbox","kafka","outbox","ruby"],"created_at":"2024-11-15T21:24:45.510Z","updated_at":"2025-04-09T12:08:08.187Z","avatar_url":"https://github.com/Kuper-Tech.png","language":"Ruby","funding_links":[],"categories":[],"sub_categories":[],"readme":"[![Gem Version](https://badge.fury.io/rb/sbmt-kafka_consumer.svg)](https://badge.fury.io/rb/sbmt-kafka_consumer)\n[![Build Status](https://github.com/Kuper-Tech/sbmt-kafka_consumer/actions/workflows/tests.yml/badge.svg?branch=master)](https://github.com/Kuper-Tech/sbmt-kafka_consumer/actions?query=branch%3Amaster)\n\n# Sbmt-KafkaConsumer\n\nThis gem is used to consume Kafka messages. It is a wrapper over the [Karafka](https://github.com/karafka/karafka) gem, and is recommended for use as a transport with the [sbmt-outbox](https://github.com/Kuper-Tech/sbmt-outbox) gem.\n\n## Installation\n\nAdd this line to your application's Gemfile:\n\n```ruby\ngem \"sbmt-kafka_consumer\"\n```\n\nAnd then execute:\n\n```bash\nbundle install\n```\n\n## Demo\n\nLearn how to use this gem and how it works with Ruby on Rails at here https://github.com/Kuper-Tech/outbox-example-apps\n\n## Auto configuration\n\nWe recommend going through the configuration and file creation process using the following Rails generators. Each generator can be run by using the `--help` option to learn more about the available arguments.\n\n### Initial configuration\n\nIf you plug the gem into your application for the first time, you can generate the initial configuration:\n\n```shell\nrails g kafka_consumer:install\n```\n\nAs the result, the `config/kafka_consumer.yml` file will be created.\n\n### Consumer class\n\nA consumer class can be generated with the following command:\n\n```shell\nrails g kafka_consumer:consumer MaybeNamespaced::Name\n```\n\n### Inbox consumer\n\nTo generate an Inbox consumer for use with gem [sbmt-outbox](https://github.com/Kuper-Tech/sbmt-outbox), run the following command:\n\n```shell\nrails g kafka_consumer:inbox_consumer MaybeNamespaced::Name some-consumer-group some-topic\n```\n\n## Manual configuration\n\nThe `config/kafka_consumer.yml` file is a main configuration for the gem.\n\nExample config with a full set of options:\n\n```yaml\ndefault: \u0026default\n  client_id: \"my-app-consumer\"\n  concurrency: 4 # max number of threads\n  # optional Karafka options\n  max_wait_time: 1\n  shutdown_timeout: 60\n  pause_timeout: 1\n  pause_max_timeout: 30\n  pause_with_exponential_backoff: true\n  partition_assignment_strategy: cooperative-sticky\n  auth:\n    kind: plaintext\n  kafka:\n    servers: \"kafka:9092\"\n    # optional Kafka options\n    heartbeat_timeout: 5\n    session_timeout: 30\n    reconnect_timeout: 3\n    connect_timeout: 5\n    socket_timeout: 30\n    kafka_options:\n      allow.auto.create.topics: true\n  probes: # optional section\n    port: 9394\n    endpoints:\n      readiness:\n        enabled: true\n        path: \"/readiness\"\n      liveness:\n        enabled: true\n        path: \"/liveness\"\n        timeout: 15\n        max_error_count: 15 # default 10\n  metrics: # optional section\n    port: 9090\n    path: \"/metrics\"\n  consumer_groups:\n    group_ref_id_1:\n      name: cg_with_single_topic\n      topics:\n        - name: topic_with_inbox_items\n          consumer:\n            klass: \"Sbmt::KafkaConsumer::InboxConsumer\"\n            init_attrs:\n              name: \"test_items\"\n              inbox_item: \"TestInboxItem\"\n          deserializer:\n            klass: \"Sbmt::KafkaConsumer::Serialization::NullDeserializer\"\n          kafka_options:\n            auto.offset.reset: latest\n    group_ref_id_2:\n      name: cg_with_multiple_topics\n      topics:\n        - name: topic_with_json_data\n          consumer:\n            klass: \"SomeConsumer\"\n          deserializer:\n            klass: \"Sbmt::KafkaConsumer::Serialization::JsonDeserializer\"\n        - name: topic_with_protobuf_data\n          consumer:\n            klass: \"SomeConsumer\"\n          deserializer:\n            klass: \"Sbmt::KafkaConsumer::Serialization::ProtobufDeserializer\"\n            init_attrs:\n              message_decoder_klass: \"SomeDecoder\"\n              skip_decoding_error: true\n\ndevelopment:\n  \u003c\u003c: *default\n\ntest:\n  \u003c\u003c: *default\n  deliver: false\n\nproduction:\n  \u003c\u003c: *default\n```\n\n### `auth` config section\n\nThe gem supports 2 variants: plaintext (default) and SASL-plaintext\n\nSASL-plaintext:\n\n```yaml\nauth:\n  kind: sasl_plaintext\n  sasl_username: user\n  sasl_password: pwd\n  sasl_mechanism: SCRAM-SHA-512\n```\n\n### `kafka` config section\n\nThe `servers` key is required and should be in rdkafka format: without `kafka://` prefix, for example: `srv1:port1,srv2:port2,...`.\n\nThe `kafka_config` section may contain any [rdkafka option](https://github.com/confluentinc/librdkafka/blob/master/CONFIGURATION.md). Also, `kafka_options` may be redefined for each topic.\nPlease note that the `partition.assignment.strategy` option within kafka_options is not supported for topics; instead, use the global option partition_assignment_strategy. \n\n### `consumer_groups` config section\n\n```yaml\nconsumer_groups:\n  # group id can be used when starting a consumer process (see CLI section below)\n  group_id:\n    name: some_group_name # required\n    topics:\n    - name: some_topic_name # required\n      active: true # optional, default true\n      consumer:\n        klass: SomeConsumerClass # required, a consumer class inherited from Sbmt::KafkaConsumer::BaseConsumer\n        init_attrs: # optional, consumer class attributes (see below)\n          key: value\n      deserializer:\n        klass: SomeDeserializerClass # optional, default NullDeserializer, a deserializer class inherited from Sbmt::KafkaConsumer::Serialization::NullDeserializer\n        init_attrs: # optional, deserializer class attributes (see below)\n          key: value\n      kafka_options: # optional, this section allows to redefine the root rdkafka options for each topic\n        auto.offset.reset: latest\n```\n\n#### `consumer.init_attrs` options for `BaseConsumer`\n\n- `skip_on_error` - optional, default false, omit consumer errors in message processing and commit the offset to Kafka\n- `middlewares` - optional, default [], type String, add middleware before message processing\n\n```yaml\ninit_attrs:\n  middlewares: ['SomeMiddleware']\n```\n\n```ruby\nclass SomeMiddleware\n  def call(message)\n    yield if message.payload.id.to_i % 2 == 0\n  end\nend\n```\n__CAUTION__:\n- ⚠️ `yield` is mandatory for all middleware, as it returns control to the `process_message` method.\n\n#### `consumer.init_attrs` options for `InboxConsumer`\n\n- `inbox_item` - required, name of the inbox item class\n- `event_name` - optional, default nil, used when the inbox item keep several event types\n- `skip_on_error` - optional, default false, omit consumer errors in message processing and commit the offset to Kafka\n- `middlewares` - optional, default [], type String, add middleware before message processing\n\n```yaml\ninit_attrs:\n  middlewares: ['SomeMiddleware']\n```\n\n```ruby\nclass SomeMiddleware\n  def call(message)\n    yield if message.payload.id.to_i % 2 == 0\n  end\nend\n```\n__CAUTION__:\n- ⚠️ `yield` is mandatory for all middleware, as it returns control to the `process_message` method.\n- ⚠️ Doesn't work with `process_batch`.\n\n#### `deserializer.init_attrs` options\n\n- `skip_decoding_error` — don't raise an exception when cannot deserialize the message\n\n### `probes` config section\n\nIn Kubernetes, probes are mechanisms used to assess the health of your application running within a container.\n\n```yaml\nprobes:\n  port: 9394 # optional, default 9394\n  endpoints:\n    liveness:\n      enabled: true # optional, default true\n      path: /liveness # optional, default \"/liveness\"\n      timeout: 10 # optional, default 10, timeout in seconds after which the group is considered dead\n    readiness:\n      enabled: true # optional, default true\n      path: /readiness/kafka_consumer # optional, default \"/readiness/kafka_consumer\"\n```\n\n### `metrics` config section\n\nWe use [Yabeda](https://github.com/yabeda-rb/yabeda) to collect [all kind of metrics](./lib/sbmt/kafka_consumer/yabeda_configurer.rb).\n\n```yaml\nmetrics:\n  port: 9090 # optional, default is probes.port\n  path: /metrics # optional, default \"/metrics\"\n```\n\n### `Kafkafile`\n\nYou can create a `Kafkafile` in the root of your app to configure additional settings for your needs.\n\nExample:\n\n```ruby\nrequire_relative \"config/environment\"\n\nsome-extra-configuration\n```\n\n### `Process batch`\n\nTo process messages in batches, you need to add the `process_batch` method in the consumer\n\n```ruby\n# app/consumers/some_consumer.rb\nclass SomeConsumer \u003c Sbmt::KafkaConsumer::BaseConsumer\n  def process_batch(messages)\n    # some code \n  end\nend\n```\n__CAUTION__:\n- ⚠️ Inbox does not support batch insertion.\n- ⚠️ If you want to use this feature, you need to process the stack atomically (eg: insert it into clickhouse in one request).\n\n## CLI\n\nRun the following command to execute a server\n\n```shell\nkafka_consumer -g some_group_id_1 -g some_group_id_2 -c 5\n```\n\nWhere:\n- `-g` - `group`, a consumer group id, if not specified, all groups from the config will be processed\n- `-c` - `concurrency`, a number of threads, default is 4\n\n### `concurrency` argument\n\n[Concurrency and Multithreading](https://karafka.io/docs/Concurrency-and-multithreading/).\n\nDon't forget to properly calculate and set the size of the ActiveRecord connection pool:\n- each thread will utilize one db connection from the pool\n- an application can have monitoring threads which can use db connections from the pool\n\nAlso pay attention to the number of processes of the server:\n- `number_of_processes x concurrency` for topics with high data intensity can be equal to the number of partitions of the consumed topic\n- `number_sof_processes x concurrency` for topics with low data intensity can be less than the number of partitions of the consumed topic\n\n## Testing\n\nTo test your consumer with Rspec, please use [this shared context](./lib/sbmt/kafka_consumer/testing/shared_contexts/with_sbmt_karafka_consumer.rb)\n\n### for payload\n```ruby\nrequire \"sbmt/kafka_consumer/testing\"\n\nRSpec.describe OrderCreatedConsumer do\n  include_context \"with sbmt karafka consumer\"\n\n  it \"works\" do\n    publish_to_sbmt_karafka(payload, deserializer: deserializer)\n    expect { consume_with_sbmt_karafka }.to change(Order, :count).by(1)\n  end\nend\n```\n\n### for payloads\n```ruby\nrequire \"sbmt/kafka_consumer/testing\"\n\nRSpec.describe OrderCreatedConsumer do\n  include_context \"with sbmt karafka consumer\"\n\n  it \"works\" do\n    publish_to_sbmt_karafka_batch(payloads, deserializer: deserializer)\n    expect { consume_with_sbmt_karafka }.to change(Order, :count).by(1)\n  end\nend\n```\n\n## Development\n\n1. Prepare environment\n```shell\ndip provision\n```\n\n2. Run tests\n```shell\ndip rspec\n```\n\n3. Run linter\n```shell\ndip rubocop\n```\n\n4. Run Kafka server\n```shell\ndip up\n```\n\n5. Run consumer server\n```shell\ndip kafka-consumer\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkuper-tech%2Fsbmt-kafka_consumer","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fkuper-tech%2Fsbmt-kafka_consumer","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkuper-tech%2Fsbmt-kafka_consumer/lists"}