{"id":31195601,"url":"https://github.com/deadmanssnitch/kafka","last_synced_at":"2025-09-20T03:43:01.022Z","repository":{"id":41263936,"uuid":"232898809","full_name":"deadmanssnitch/kafka","owner":"deadmanssnitch","description":"Ruby client library for Apache Kafka based on librdkafka","archived":false,"fork":false,"pushed_at":"2024-09-03T22:22:57.000Z","size":350,"stargazers_count":12,"open_issues_count":2,"forks_count":0,"subscribers_count":3,"default_branch":"main","last_synced_at":"2025-08-16T07:43:46.274Z","etag":null,"topics":["kafka","librdkafka","ruby"],"latest_commit_sha":null,"homepage":"https://deadmanssnitch.com/opensource/kafka/","language":"Ruby","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/deadmanssnitch.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2020-01-09T20:28:42.000Z","updated_at":"2024-11-18T22:54:21.000Z","dependencies_parsed_at":"2023-02-04T07:17:11.275Z","dependency_job_id":null,"html_url":"https://github.com/deadmanssnitch/kafka","commit_stats":null,"previous_names":[],"tags_count":3,"template":false,"template_full_name":null,"purl":"pkg:github/deadmanssnitch/kafka","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/deadmanssnitch%2Fkafka","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/deadmanssnitch%2Fkafka/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/deadmanssnitch%2Fkafka/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/deadmanssnitch%2Fkafka/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/deadmanssnitch","download_url":"https://codeload.github.com/deadmanssnitch/kafka/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/deadmanssnitch%2Fkafka/sbom","scorecard":{"id":330696,"data":{"date":"2025-08-11","repo":{"name":"github.com/deadmanssnitch/kafka","commit":"4e61b272e91d9a98f9eb97cf044ab3064e57712a"},"scorecard":{"version":"v5.2.1-40-gf6ed084d","commit":"f6ed084d17c9236477efd66e5b258b9d4cc7b389"},"score":3.4,"checks":[{"name":"Code-Review","score":0,"reason":"Found 1/28 approved changesets -- score normalized to 0","details":null,"documentation":{"short":"Determines if the project requires human code review before pull requests (aka merge requests) are merged.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#code-review"}},{"name":"Maintained","score":0,"reason":"0 commit(s) and 0 issue activity found in the last 90 days -- score normalized to 0","details":null,"documentation":{"short":"Determines if the project is \"actively maintained\".","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#maintained"}},{"name":"Dangerous-Workflow","score":10,"reason":"no dangerous workflow patterns detected","details":null,"documentation":{"short":"Determines if the project's GitHub Action workflows avoid dangerous patterns.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#dangerous-workflow"}},{"name":"Packaging","score":-1,"reason":"packaging workflow not detected","details":["Warn: no GitHub/GitLab publishing workflow detected."],"documentation":{"short":"Determines if the project is published as a package that others can easily download, install, easily update, and uninstall.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#packaging"}},{"name":"Token-Permissions","score":0,"reason":"detected GitHub workflow tokens with excessive permissions","details":["Warn: no topLevel permission defined: .github/workflows/ci.yml:1","Info: no jobLevel write permissions found"],"documentation":{"short":"Determines if the project's workflows follow the principle of least privilege.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#token-permissions"}},{"name":"Binary-Artifacts","score":10,"reason":"no binaries found in the repo","details":null,"documentation":{"short":"Determines if the project has generated executable (binary) artifacts in the source repository.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#binary-artifacts"}},{"name":"Pinned-Dependencies","score":0,"reason":"dependency not pinned by hash detected -- score normalized to 0","details":["Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/ci.yml:12: update your workflow using https://app.stepsecurity.io/secureworkflow/deadmanssnitch/kafka/ci.yml/main?enable=pin","Warn: third-party GitHubAction not pinned by hash: .github/workflows/ci.yml:15: update your workflow using https://app.stepsecurity.io/secureworkflow/deadmanssnitch/kafka/ci.yml/main?enable=pin","Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/ci.yml:26: update your workflow using https://app.stepsecurity.io/secureworkflow/deadmanssnitch/kafka/ci.yml/main?enable=pin","Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/ci.yml:37: update your workflow using https://app.stepsecurity.io/secureworkflow/deadmanssnitch/kafka/ci.yml/main?enable=pin","Warn: third-party GitHubAction not pinned by hash: .github/workflows/ci.yml:40: update your workflow using https://app.stepsecurity.io/secureworkflow/deadmanssnitch/kafka/ci.yml/main?enable=pin","Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/ci.yml:63: update your workflow using https://app.stepsecurity.io/secureworkflow/deadmanssnitch/kafka/ci.yml/main?enable=pin","Warn: third-party GitHubAction not pinned by hash: .github/workflows/ci.yml:66: update your workflow using https://app.stepsecurity.io/secureworkflow/deadmanssnitch/kafka/ci.yml/main?enable=pin","Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/ci.yml:89: update your workflow using https://app.stepsecurity.io/secureworkflow/deadmanssnitch/kafka/ci.yml/main?enable=pin","Warn: third-party GitHubAction not pinned by hash: .github/workflows/ci.yml:95: update your workflow using https://app.stepsecurity.io/secureworkflow/deadmanssnitch/kafka/ci.yml/main?enable=pin","Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/ci.yml:120: update your workflow using https://app.stepsecurity.io/secureworkflow/deadmanssnitch/kafka/ci.yml/main?enable=pin","Warn: third-party GitHubAction not pinned by hash: .github/workflows/ci.yml:123: update your workflow using https://app.stepsecurity.io/secureworkflow/deadmanssnitch/kafka/ci.yml/main?enable=pin","Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/ci.yml:147: update your workflow using https://app.stepsecurity.io/secureworkflow/deadmanssnitch/kafka/ci.yml/main?enable=pin","Warn: third-party GitHubAction not pinned by hash: .github/workflows/ci.yml:153: update your workflow using https://app.stepsecurity.io/secureworkflow/deadmanssnitch/kafka/ci.yml/main?enable=pin","Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/ci.yml:169: update your workflow using https://app.stepsecurity.io/secureworkflow/deadmanssnitch/kafka/ci.yml/main?enable=pin","Warn: third-party GitHubAction not pinned by hash: .github/workflows/ci.yml:172: update your workflow using https://app.stepsecurity.io/secureworkflow/deadmanssnitch/kafka/ci.yml/main?enable=pin","Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/ci.yml:196: update your workflow using https://app.stepsecurity.io/secureworkflow/deadmanssnitch/kafka/ci.yml/main?enable=pin","Warn: third-party GitHubAction not pinned by hash: .github/workflows/ci.yml:202: update your workflow using https://app.stepsecurity.io/secureworkflow/deadmanssnitch/kafka/ci.yml/main?enable=pin","Info:   0 out of   9 GitHub-owned GitHubAction dependencies pinned","Info:   0 out of   8 third-party GitHubAction dependencies pinned"],"documentation":{"short":"Determines if the project has declared and pinned the dependencies of its build process.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#pinned-dependencies"}},{"name":"CII-Best-Practices","score":0,"reason":"no effort to earn an OpenSSF best practices badge detected","details":null,"documentation":{"short":"Determines if the project has an OpenSSF (formerly CII) Best Practices Badge.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#cii-best-practices"}},{"name":"Security-Policy","score":0,"reason":"security policy file not detected","details":["Warn: no security policy file detected","Warn: no security file to analyze","Warn: no security file to analyze","Warn: no security file to analyze"],"documentation":{"short":"Determines if the project has published a security policy.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#security-policy"}},{"name":"License","score":10,"reason":"license file detected","details":["Info: project has a license file: LICENSE.txt:0","Info: FSF or OSI recognized license: MIT License: LICENSE.txt:0"],"documentation":{"short":"Determines if the project has defined a license.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#license"}},{"name":"Vulnerabilities","score":10,"reason":"0 existing vulnerabilities detected","details":null,"documentation":{"short":"Determines if the project has open, known unfixed vulnerabilities.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#vulnerabilities"}},{"name":"Fuzzing","score":0,"reason":"project is not fuzzed","details":["Warn: no fuzzer integrations found"],"documentation":{"short":"Determines if the project uses fuzzing.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#fuzzing"}},{"name":"Signed-Releases","score":-1,"reason":"no releases found","details":null,"documentation":{"short":"Determines if the project cryptographically signs release artifacts.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#signed-releases"}},{"name":"Branch-Protection","score":0,"reason":"branch protection not enabled on development/release branches","details":["Warn: branch protection not enabled for branch 'main'"],"documentation":{"short":"Determines if the default and release branches are protected with GitHub's branch protection settings.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#branch-protection"}},{"name":"SAST","score":0,"reason":"SAST tool is not run on all commits -- score normalized to 0","details":["Warn: 0 commits out of 3 are checked with a SAST tool"],"documentation":{"short":"Determines if the project uses static code analysis.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#sast"}}]},"last_synced_at":"2025-08-18T03:31:09.819Z","repository_id":41263936,"created_at":"2025-08-18T03:31:09.819Z","updated_at":"2025-08-18T03:31:09.819Z"},"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":275764819,"owners_count":25524374,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-09-18T02:00:09.552Z","response_time":77,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["kafka","librdkafka","ruby"],"created_at":"2025-09-20T03:42:59.328Z","updated_at":"2025-09-20T03:43:01.016Z","avatar_url":"https://github.com/deadmanssnitch.png","language":"Ruby","readme":"# Kafka\n\n[![Build Status](https://github.com/deadmanssnitch/kafka/actions/workflows/ci.yml/badge.svg)](https://github.com/deadmanssnitch/kafka/actions/workflows/tests.yml)\n[![Gem Version](https://badge.fury.io/rb/kafka.svg)](https://badge.fury.io/rb/kafka)\n[![Documentation](https://img.shields.io/badge/-Documentation-success)](https://deadmanssnitch.com/opensource/kafka/docs/)\n\nKafka provides a Ruby client for [Apache Kafka](https://kafka.apache.org) that\nleverages [librdkafka](https://github.com/edenhill/librdkafka) for its\nperformance and general correctness.\n\n## Features\n- Thread safe Producer with sync and async delivery reporting\n- High-level balanced Consumer\n- Admin client\n- Object oriented librdkafka mappings for easy custom implementations\n\n## Installation\n\nAdd this line to your application's Gemfile:\n\n```ruby\ngem \"kafka\"\n```\n\nAnd then execute:\n\n    $ bundle install\n\nOr install it yourself as:\n\n    $ gem install kafka\n\n## Getting Started\n\nFor more examples see [the examples directory](examples/).\n\nFor a detailed introduction on librdkafka which would be useful when working\nwith `Kafka::FFI` directly, see\n[the librdkafka documentation](https://github.com/edenhill/librdkafka/blob/master/INTRODUCTION.md).\n\n### Sending a Message to a Topic\n\n```ruby\nrequire \"kafka\"\n\nconfig = Kafka::Config.new(\"bootstrap.servers\": \"localhost:9092\")\nproducer = Kafka::Producer.new(config)\n\n# Asynchronously publish a JSON payload to the events topic.\nevent = { time: Time.now, status: \"success\" }\nresult = producer.produce(\"events\", event.to_json)\n\n# Wait for the delivery to confirm that publishing was successful.\nresult.wait\nresult.successful?\n\n# Provide a callback to be called when the delivery status is ready.\nproducer.produce(\"events\", event.to_json) do |result|\n  StatsD.increment(\"kafka.total\")\n  \n  if result.error?\n    StatsD.increment(\"kafka.errors\")\n  end\nend\n```\n\n### Consuming Messages from a Topic\n\n```ruby\nrequire \"kafka\"\n\nconfig = Kafka::Config.new({\n  \"bootstrap.servers\": \"localhost:9092\",\n\n  # Required for consumers to know what consumer group to join.\n  \"group.id\": \"web.production.eventer\",\n})\n\nconsumer = Kafka::Consumer.new(config)\nconsumer.subscribe(\"events\")\n\n@run = true\ntrap(\"INT\")  { @run = false }\ntrap(\"TERM\") { @run = false }\n\nwhile @run\n  consumer.poll do |message|\n    puts message.payload\n  end\nend\n```\n\n### Configuration\n\nKafka has a lot of potential knobs to turn and dials to tweak. A\n`Kafka::Config` uses the same configuration options as librdkafka (and most or\nall from the Java client). The defaults are generally good and a fine place to\nstart.\n\n[All Configuration Options](https://github.com/edenhill/librdkafka/blob/master/CONFIGURATION.md)\n\n### Bindings\n\n`Kafka::FFI` provides bindings to functions in\n[librdkafka](https://github.com/edenhill/librdkafka/blob/master/src/rdkafka.h).\nAll of the names are the same and mostly have named parameters to help with\ncalling them. Be aware that you will need to handle some memory management to\ncall most functions exported in the bindings. See\n[rdkafka.h](https://github.com/edenhill/librdkafka/blob/master/src/rdkafka.h)\nfor any questions about usage and semantics.\n\nAll classes in `Kafka::FFI` provide an object oriented mapping to the functions\nexported on `Kafka::FFI.rd_kafka_*`. Most will require understanding memory\nmanagement but most should be easier to use and safe than calling into\nlibrdkafka directly.\n\n## Why another Kafka gem?\n\nThere are already at least two good gems for Kafka:\n[ruby-kafka](https://github.com/zendesk/ruby-kafka) and\n[rdkafka](https://github.com/appsignal/rdkafka-ruby). In fact we've used both\nof these gems on Dead Man's Snitch for quite a while and they've been great. We\nreally appreciate all of the work that has gone into them :heart:.\n\nUnfortunately, keeping up with Kafka feature and protocol changes can be a full\ntime job. Development on ruby-kafka has stalled for that reason and many\nconsumer/producer libraries are migrating away from it.\n\nAs a heartbeat and cron job monitoring service, we depend on receiving and\nprocessing reports from jobs reliably and quickly. Failing to receive a report\ncould mean waking someone up at 3AM or forcing them to take time away from\nfamily or friends to deal with a false alarm. What started as a deep dive into\nrdkafka to understand how best to use it reliably, we had ideas we wanted to\nimplement that probably wouldn't have been a good fit for rdkafka so we decided\nto start from scratch.\n\nOur goal is to provide a stable and easy to maintain Kafka consumer / producer\nfor Ruby. With time as our biggest constraint it makes sense to leverage\nlibrdkafka as it has full time maintenance and support by the team behind\nKafka. FFI makes it fast and easy to expose new librdkafka APIs as they are\nadded. A stable test suite means being able to meaningfully spend the limited\namount of time we have available to invest. Embracing memory management and\nbuilding clean separations between layers should reduce the burden to implement\nnew bindings as the rules and responsibilities of each layer are clear.\n\n## Thread / Fork Safety\n\nThe `Producer` is thread safe for publishing messages but should only be closed\nfrom a single thread. While the `Consumer` is thread safe for calls to `#poll`\nonly one message can be in flight at a time, causing the threads to serialize.\nInstead, create a single consumer for each thread.\n\nKafka _is not_ `fork` safe. Make sure to close any Producers or Consumer before\nforking and rebuild them after forking the new process.\n\n## Compatibility\n\nKafka requires Ruby 2.7+ and is tested against Ruby 2.7+ and Kafka 2.8+.\n\n## Development\n\nTo get started with development make sure to have `docker`, `docker-compose`, and\n[`kafkacat`](https://github.com/edenhill/kafkacat) installed as they make getting\nup to speed easier. Some rake tasks depend on `ctags`.\n\nBefore running the test, start a Kafka broker instance\n\n```console\nrake kafka:up\n```\n\nThen run the tests with\n```console\nrake\n```\n\nWhen you're done shut down the Kafka instance by running:\n```console\nrake kafka:down\n```\n\n## Contributing\n\nBug reports and pull requests are welcome on GitHub at\nhttps://github.com/deadmanssnitch/kafka. This project is intended to be a safe,\nwelcoming space for collaboration, and contributors are expected to adhere to\nthe [code of conduct](https://github.com/deadmanssnitch/kafka/blob/master/CODE_OF_CONDUCT.md).\n\n## License\n\nThe gem is available as open source under the terms of the\n[MIT License](https://opensource.org/licenses/MIT).\n\n## Code of Conduct\n\nEveryone interacting in the Kafka project's codebases and issue trackers are\nexpected to follow the\n[code of conduct](https://github.com/deadmanssnitch/kafka/blob/master/CODE_OF_CONDUCT.md).\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdeadmanssnitch%2Fkafka","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdeadmanssnitch%2Fkafka","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdeadmanssnitch%2Fkafka/lists"}