{"id":13879667,"url":"https://github.com/karafka/sidekiq-backend","last_synced_at":"2025-07-16T15:32:44.394Z","repository":{"id":24638019,"uuid":"102102942","full_name":"karafka/sidekiq-backend","owner":"karafka","description":"[Unmaintained] Karafka Sidekiq backend for background messages processing","archived":true,"fork":false,"pushed_at":"2023-01-25T15:07:51.000Z","size":515,"stargazers_count":69,"open_issues_count":0,"forks_count":12,"subscribers_count":4,"default_branch":"master","last_synced_at":"2025-06-30T13:23:07.707Z","etag":null,"topics":["kafka","karafka","karafka-application","karafka-framework","ruby","ruby-kafka","rubygem"],"latest_commit_sha":null,"homepage":"https://karafka.io","language":"Ruby","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/karafka.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"MIT-LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2017-09-01T10:37:32.000Z","updated_at":"2025-01-09T06:48:53.000Z","dependencies_parsed_at":"2023-02-14T08:31:56.873Z","dependency_job_id":null,"html_url":"https://github.com/karafka/sidekiq-backend","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/karafka/sidekiq-backend","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/karafka%2Fsidekiq-backend","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/karafka%2Fsidekiq-backend/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/karafka%2Fsidekiq-backend/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/karafka%2Fsidekiq-backend/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/karafka","download_url":"https://codeload.github.com/karafka/sidekiq-backend/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/karafka%2Fsidekiq-backend/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":264439145,"owners_count":23608472,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["kafka","karafka","karafka-application","karafka-framework","ruby","ruby-kafka","rubygem"],"created_at":"2024-08-06T08:02:28.295Z","updated_at":"2025-07-16T15:32:44.371Z","avatar_url":"https://github.com/karafka.png","language":"Ruby","funding_links":[],"categories":["Ruby"],"sub_categories":[],"readme":"# Karafka Sidekiq Backend\n\n## Deprecation notice\n\nThis backend was designed to compensate for lack of multi-threading in Karafka. Karafka `2.0` **is** multi-threaded.\n\nIt is **no longer needed** and is **no longer** maintained.\n\n## About\n\n[![Build Status](https://github.com/karafka/sidekiq-backend/workflows/ci/badge.svg)](https://github.com/karafka/sidekiq-backend/actions?query=workflow%3Aci)\n[![Gem Version](https://badge.fury.io/rb/karafka-sidekiq-backend.svg)](http://badge.fury.io/rb/karafka-sidekiq-backend)\n[![Join the chat at https://slack.karafka.io](https://raw.githubusercontent.com/karafka/misc/master/slack.svg)](https://slack.karafka.io)\n\n[Karafka Sidekiq Backend](https://github.com/karafka/sidekiq-backend) provides support for consuming (processing) received Kafka messages inside of Sidekiq workers.\n\n## Installations\n\nAdd this to your gemfile:\n\n```ruby\ngem 'karafka-sidekiq-backend'\n```\n\nand create a file called ```application_worker.rb``` inside of your ```app/workers``` directory, that looks like that:\n\n```ruby\nclass ApplicationWorker \u003c Karafka::BaseWorker\nend\n```\n\nand you are ready to go. Karafka Sidekiq Backend integrates with Karafka automatically\n\n**Note**: You can name your application worker base class with any name you want. The only thing that is required is a direct inheritance from the ```Karafka::BaseWorker``` class.\n\n## Usage\n\nIf you want to process messages with Sidekiq backend, you need to tell this to Karafka.\n\nTo do so, you can either configure that in a configuration block:\n\n```ruby\nclass App \u003c Karafka::App\n  setup do |config|\n    config.backend = :sidekiq\n    # Other config options...\n  end\nend\n```\n\nor on a per topic level:\n\n```ruby\nApp.routes.draw do\n  consumer_group :videos_consumer do\n    topic :binary_video_details do\n      backend :sidekiq\n      consumer Videos::DetailsConsumer\n      worker Workers::DetailsWorker\n      interchanger Interchangers::MyCustomInterchanger.new\n    end\n  end\nend\n```\n\nYou don't need to do anything beyond that. Karafka will know, that you want to run your consumer's ```#consume``` method in a background job.\n\n## Configuration\n\nThere are two options you can set inside of the ```topic``` block:\n\n| Option       | Value type | Description                                                                                                       |\n|--------------|------------|-------------------------------------------------------------------------------------------------------------------|\n| worker       | Class      | Name of a worker class that we want to use to schedule perform code                                               |\n| interchanger | Instance   | Instance of an interchanger class that we want to use to pass the incoming data to Sidekiq                        |\n\n\n### Workers\n\nKarafka by default will build a worker that will correspond to each of your consumers (so you will have a pair - consumer and a worker). All of them will inherit from ```ApplicationWorker``` and will share all its settings.\n\nTo run Sidekiq you should have sidekiq.yml file in *config* folder. The example of ```sidekiq.yml``` file will be generated to config/sidekiq.yml.example once you run ```bundle exec karafka install```.\n\nHowever, if you want to use a raw Sidekiq worker (without any Karafka additional magic), or you want to use SidekiqPro (or any other queuing engine that has the same API as Sidekiq), you can assign your own custom worker:\n\n```ruby\ntopic :incoming_messages do\n  consumer MessagesConsumer\n  worker MyCustomWorker\nend\n```\n\nNote that even then, you need to specify a consumer that will schedule a background task.\n\nCustom workers need to provide a ```#perform_async``` method. It needs to accept two arguments:\n\n - ```topic_id``` - first argument is a current topic id from which a given message comes\n - ```params_batch``` - all the params that came from Kafka + additional metadata. This data format might be changed if you use custom interchangers. Otherwise, it will be an instance of Karafka::Params::ParamsBatch.\n\n**Note**: If you use custom interchangers, keep in mind, that params inside params batch might be in two states: parsed or unparsed when passed to #perform_async. This means, that if you use custom interchangers and/or custom workers, you might want to look into Karafka's sources to see exactly how it works.\n\n### Interchangers\n\nCustom interchangers target issues with non-standard (binary, etc.) data that we want to store when we do ```#perform_async```. This data might be corrupted when fetched in a worker (see [this](https://github.com/karafka/karafka/issues/30) issue). With custom interchangers, you can encode/compress data before it is being passed to scheduling and decode/decompress it when it gets into the worker.\n\nTo specify the interchanger for a topic, specify the interchanger inside routes like this:\n\n```ruby\nApp.routes.draw do\n  consumer_group :videos_consumer do\n    topic :binary_video_details do\n      consumer Videos::DetailsConsumer\n      interchanger Interchangers::MyCustomInterchanger.new\n    end\n  end\nend\n```\nEach custom interchanger should define `encode` to encode params before they get stored in Redis, and `decode` to convert the params to hash format, as shown below:\n\n```ruby\nclass Base64Interchanger \u003c ::Karafka::Interchanger\n  def encode(params_batch)\n    Base64.encode64(Marshal.dump(super))\n  end\n\n  def decode(params_string)\n    Marshal.load(Base64.decode64(super))\n  end\nend\n\n```\n\n**Warning**: if you decide to use slow interchangers, they might significantly slow down Karafka.\n\n## References\n\n* [Karafka framework](https://github.com/karafka/karafka)\n* [Karafka Sidekiq Backend Actions CI](https://github.com/karafka/sidekiq-backend/actions?query=workflow%3Aci)\n* [Karafka Sidekiq Backend Coditsu](https://app.coditsu.io/karafka/repositories/karafka-sidekiq-backend)\n\n## Note on contributions\n\nFirst, thank you for considering contributing to the Karafka ecosystem! It's people like you that make the open source community such a great community!\n\nEach pull request must pass all the RSpec specs, integration tests and meet our quality requirements.\n\nFork it, update and wait for the Github Actions results.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkarafka%2Fsidekiq-backend","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fkarafka%2Fsidekiq-backend","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkarafka%2Fsidekiq-backend/lists"}