{"id":13879825,"url":"https://github.com/zendesk/delivery_boy","last_synced_at":"2025-10-17T09:15:03.250Z","repository":{"id":39737900,"uuid":"99909056","full_name":"zendesk/delivery_boy","owner":"zendesk","description":"A simple way to publish messages to Kafka from Ruby applications","archived":false,"fork":false,"pushed_at":"2025-04-30T13:32:00.000Z","size":132,"stargazers_count":238,"open_issues_count":1,"forks_count":37,"subscribers_count":122,"default_branch":"main","last_synced_at":"2025-04-30T14:15:02.857Z","etag":null,"topics":["kafka","kafka-producer","ruby"],"latest_commit_sha":null,"homepage":"","language":"Ruby","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/zendesk.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG","contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2017-08-10T10:01:17.000Z","updated_at":"2024-09-02T22:13:40.000Z","dependencies_parsed_at":"2024-01-19T13:42:30.822Z","dependency_job_id":"ff704b28-2b51-4550-bc33-4b3c88d1ea7d","html_url":"https://github.com/zendesk/delivery_boy","commit_stats":{"total_commits":144,"total_committers":26,"mean_commits":5.538461538461538,"dds":0.4305555555555556,"last_synced_commit":"39a13865dcbfd14855bb45e7f6066bdccf40c7b3"},"previous_names":[],"tags_count":21,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zendesk%2Fdelivery_boy","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zendesk%2Fdelivery_boy/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zendesk%2Fdelivery_boy/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zendesk%2Fdelivery_boy/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/zendesk","download_url":"https://codeload.github.com/zendesk/delivery_boy/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254430335,"owners_count":22069909,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["kafka","kafka-producer","ruby"],"created_at":"2024-08-06T08:02:34.719Z","updated_at":"2025-10-17T09:15:03.230Z","avatar_url":"https://github.com/zendesk.png","language":"Ruby","funding_links":[],"categories":["Ruby"],"sub_categories":[],"readme":"# DeliveryBoy\n\nThis library provides a dead easy way to start publishing messages to a Kafka cluster from your Ruby or Rails application!\n\n## Installation\n\nAdd this line to your application's Gemfile:\n\n```ruby\ngem 'delivery_boy'\n```\n\nAnd then execute:\n\n    $ bundle\n\nOr install it yourself as:\n\n    $ gem install delivery_boy\n\n## Usage\n\nOnce you've [installed the gem](#installation), and assuming your Kafka broker is running on localhost, you can simply start publishing messages to Kafka directly from your Rails code:\n\n```ruby\n# app/controllers/comments_controller.rb\nclass CommentsController \u003c ApplicationController\n  def create\n    @comment = Comment.create!(params)\n\n    # This will publish a JSON representation of the comment to the `comments` topic\n    # in Kafka. Make sure to create the topic first, or this may fail.\n    DeliveryBoy.deliver(comment.to_json, topic: \"comments\")\n  end\nend\n```\n\nThe above example will block the server process until the message has been delivered. If you want deliveries to happen in the background in order to free up your server processes more quickly, call `#deliver_async` instead:\n\n```ruby\n# app/controllers/comments_controller.rb\nclass CommentsController \u003c ApplicationController\n  def show\n    @comment = Comment.find(params[:id])\n\n    event = {\n      name: \"comment_viewed\",\n      data: {\n        comment_id: @comment.id,\n        user_id: current_user.id\n      }\n    }\n\n    # By delivering messages asynchronously you free up your server processes faster.\n    DeliveryBoy.deliver_async(event.to_json, topic: \"activity\")\n  end\nend\n```\n\nIn addition to improving response time, delivering messages asynchronously also protects your application against Kafka availability issues -- if messages cannot be delivered, they'll be buffered for later and retried automatically.\n\nA third method is to produce messages first (without delivering the messages to Kafka yet), and deliver them synchronously later.\n\n```ruby\n # app/controllers/comments_controller.rb\n class CommentsController \u003c ApplicationController\n   def create\n     @comment = Comment.create!(params)\n\n     event = {\n       name: \"comment_created\",\n       data: {\n         comment_id: @comment.id,\n         user_id: current_user.id\n       }\n     }\n\n     # This will queue the two messages in the internal buffer.\n     DeliveryBoy.produce(comment.to_json, topic: \"comments\")\n     DeliveryBoy.produce(event.to_json, topic: \"activity\")\n\n     # This will deliver all messages in the buffer to Kafka.\n     # This call is blocking.\n     DeliveryBoy.deliver_messages\n   end\n end\n```\n\nThe methods `deliver`, `deliver_async` and `produce` take the following options:\n\n* `topic` – the Kafka topic that should be written to (required).\n* `key` – the key that should be set on the Kafka message (optional).\n* `partition` – a specific partition number that should be written to (optional).\n* `partition_key` – a string that can be used to deterministically select the partition that should be written to (optional).\n\nRegarding `partition` and `partition_key`: if none are specified, DeliveryBoy will pick a partition at random. If you want to ensure that e.g. all messages related to a user always get written to the same partition, you can pass the user id to `partition_key`. Don't use `partition` directly unless you know what you're doing, since it requires you to know exactly how many partitions each topic has, which can change and cause you pain and misery. Just use `partition_key` or let DeliveryBoy choose at random.\n\n### Configuration\n\nYou configure DeliveryBoy in three different ways: in a YAML config file, in a Ruby config file, or by setting environment variables.\n\nIf you're using Rails, the fastest way to get started is to execute the following in your terminal:\n\n```\n$ bundle exec rails generate delivery_boy:install\n```\n\nThis will create a config file at `config/delivery_boy.yml` with configurations for each of your Rails environments. Open that file in order to make changes.\n\nNote that for all configuration variables, you can pass in an environment variable. These environment variables all take the form `DELIVERY_BOY_X`, where `X` is the upper-case configuration variable name, e.g. `DELIVERY_BOY_CLIENT_ID`.\n\nYou can also configure DeliveryBoy in Ruby if you prefer that. By default, the file `config/delivery_boy.rb` is loaded if present, but you can do this from anywhere – just call `DeliveryBoy.configure` like so:\n\n```ruby\nDeliveryBoy.configure do |config|\n  config.client_id = \"yolo\"\n  # ...\nend\n```\n\nThe following configuration variables can be set:\n\n#### Basic\n\n##### `brokers`\n\nA list of Kafka brokers that should be used to initialize the client. Defaults to just `localhost:9092` in development and test, but in production you need to pass a list of `hostname:port` strings.\n\n##### `client_id`\n\nThis is how the client will identify itself to the Kafka brokers. Default is `delivery_boy`.\n\n##### `log_level`\n\nThe log level for the logger.\n\n#### Message delivery\n\n##### `delivery_interval`\n\nThe number of seconds between background message deliveries. Default is 10 seconds. Disable timer-based background deliveries by setting this to 0.\n\n##### `delivery_threshold`\n\nThe number of buffered messages that will trigger a background message delivery. Default is 100 messages. Disable buffer size based background deliveries by setting this to 0.\n\n##### `required_acks`\n\nThe number of Kafka replicas that must acknowledge messages before they're considered as successfully written. Default is _all_ replicas.\n\nSee [ruby-kafka](https://github.com/zendesk/ruby-kafka#message-durability) for more information.\n\n##### `ack_timeout`\n\nA timeout executed by a broker when the client is sending messages to it. It defines the number of seconds the broker should wait for replicas to acknowledge the write before responding to the client with an error. As such, it relates to the `required_acks` setting. It should be set lower than `socket_timeout`.\n\n##### `max_retries`\n\nThe number of retries when attempting to deliver messages. The default is 2, so 3 attempts in total, but you can configure a higher or lower number.\n\n##### `retry_backoff`\n\nThe number of seconds to wait after a failed attempt to send messages to a Kafka broker before retrying. The `max_retries` setting defines the maximum number of retries to attempt, and so the total duration could be up to `max_retries * retry_backoff` seconds. The timeout can be arbitrarily long, and shouldn't be too short: if a broker goes down its partitions will be handed off to another broker, and that can take tens of seconds.\n\n#### Compression\n\nSee [ruby-kafka](https://github.com/zendesk/ruby-kafka#compression) for more information.\n\n##### `compression_codec`\n\nThe codec used to compress messages. Must be either `snappy` or `gzip`.\n\n##### `compression_threshold`\n\nThe minimum number of messages that must be buffered before compression is attempted. By default only one message is required. Only relevant if `compression_codec` is set.\n\n#### Network\n\n##### `connect_timeout`\n\nThe number of seconds to wait while connecting to a broker for the first time. When the Kafka library is initialized, it needs to connect to at least one host in `brokers` in order to discover the Kafka cluster. Each host is tried until there's one that works. Usually that means the first one, but if your entire cluster is down, or there's a network partition, you could wait up to `n * connect_timeout` seconds, where `n` is the number of hostnames in `brokers`.\n\n##### `socket_timeout`\n\nTimeout when reading data from a socket connection to a Kafka broker. Must be larger than `ack_timeout` or you risk killing the socket before the broker has time to acknowledge your messages.\n\n#### Buffering\n\nWhen using the asynhronous API, messages are buffered in a background thread and delivered to Kafka based on the configured delivery policy. Because of this, problems that hinder the delivery of messages can cause the buffer to grow. In order to avoid unlimited buffer growth that would risk affecting the host application, some limits are put in place. After the buffer reaches the maximum size allowed, calling `DeliveryBoy.deliver_async` will raise `Kafka::BufferOverflow`.\n\n##### `max_buffer_bytesize`\n\nThe maximum number of bytes allowed in the buffer before new messages are rejected.\n\n##### `max_buffer_size`\n\nThe maximum number of messages allowed in the buffer before new messages are rejected.\n\n##### `max_queue_size`\n\nThe maximum number of messages allowed in the queue before new messages are rejected. The queue is used to ferry messages from the foreground threads of your application to the background thread that buffers and delivers messages. You typically only want to increase this number if you have a very high throughput of messages and the background thread can't keep up with spikes in throughput.\n\n#### SSL Authentication and authorization\n\nSee [ruby-kafka](https://github.com/zendesk/ruby-kafka#encryption-and-authentication-using-ssl) for more information.\n\n##### `ssl_ca_cert`\n\nA PEM encoded CA cert, or an Array of PEM encoded CA certs, to use with an SSL connection.\n\n##### `ssl_ca_cert_file_path`\n\nThe path to a valid SSL certificate authority file.\n\n##### `ssl_client_cert`\n\nA PEM encoded client cert to use with an SSL connection. Must be used in combination with `ssl_client_cert_key`.\n\n##### `ssl_client_cert_key`\n\nA PEM encoded client cert key to use with an SSL connection. Must be used in combination with `ssl_client_cert`.\n\n##### `ssl_client_cert_key_password`\n\nThe password required to read the ssl_client_cert_key. Must be used in combination with ssl_client_cert_key.\n\n#### SASL Authentication and authorization\n\nSee [ruby-kafka](https://github.com/zendesk/ruby-kafka#authentication-using-sasl) for more information.\n\nUse it through `GSSAPI`, `PLAIN` _or_ `OAUTHBEARER`.\n\n##### `sasl_gssapi_principal`\n\nThe GSSAPI principal.\n\n##### `sasl_gssapi_keytab`\n\nOptional GSSAPI keytab.\n\n##### `sasl_plain_authzid`\n\nThe authorization identity to use.\n\n##### `sasl_plain_username`\n\nThe username used to authenticate.\n\n##### `sasl_plain_password`\n\nThe password used to authenticate.\n\n##### `sasl_oauth_token_provider`\n\nA instance of a class which implements the `token` method.\nAs described in [ruby-kafka](https://github.com/zendesk/ruby-kafka/tree/c3e90bc355fad1e27b9af1048966ff08d3d5735b#oauthbearer)\n\n```ruby\nclass TokenProvider\n  def token\n    \"oauth-token\"\n  end\nend\n\nDeliveryBoy.configure do |config|\n  config.sasl_oauth_token_provider = TokenProvider.new\n  config.ssl_ca_certs_from_system = true\nend\n```\n\n#### AWS MSK IAM Authentication and Authorization\n\n##### sasl_aws_msk_iam_access_key_id\n\nThe AWS IAM access key. Required.\n\n##### sasl_aws_msk_iam_secret_key_id\n\nThe AWS IAM secret access key. Required.\n\n##### sasl_aws_msk_iam_aws_region\n\nThe AWS region. Required.\n\n##### sasl_aws_msk_iam_session_token\n\nThe session token. This value can be optional.\n\n###### Examples \n\nUsing a role arn and web identity token to generate temporary credentials:\n\n```ruby\nrequire \"aws-sdk-core\"\nrequire \"delivery_boy\"\n\nrole = Aws::AssumeRoleWebIdentityCredentials.new(\n  role_arn: ENV[\"AWS_ROLE_ARN\"],\n  web_identity_token_file: ENV[\"AWS_WEB_IDENTITY_TOKEN_FILE\"]\n)\n\nDeliveryBoy.configure do |c|\n  c.sasl_aws_msk_iam_access_key_id = role.credentials.access_key_id\n  c.sasl_aws_msk_iam_secret_key_id = role.credentials.secret_access_key\n  c.sasl_aws_msk_iam_session_token = role.credentials.session_token\n  c.sasl_aws_msk_iam_aws_region    = ENV[\"AWS_REGION\"]\n  c.ssl_ca_certs_from_system       = true\nend\n```\n\n### Testing\n\nDeliveryBoy provides a test mode out of the box. When this mode is enabled, messages will be stored in memory rather than being sent to Kafka. If you use RSpec, enabling test mode is as easy as adding this to your spec helper:\n\n```ruby\n# spec/spec_helper.rb\nrequire \"delivery_boy/rspec\"\n```\n\nNow your application can use DeliveryBoy in tests without connecting to an actual Kafka cluster. Asserting that messages have been delivered is simple:\n\n```ruby\ndescribe PostsController do\n  describe \"#show\" do\n    it \"emits an event to Kafka\" do\n      post = Post.create!(body: \"hello\")\n\n      get :show, id: post.id\n\n      # Use this API to extract all messages written to a Kafka topic.\n      messages = DeliveryBoy.testing.messages_for(\"post_views\")\n\n      expect(messages.count).to eq 1\n\n      # In addition to #value, you can also pull out #key and #partition_key.\n      event = JSON.parse(messages.first.value)\n\n      expect(event[\"post_id\"]).to eq post.id\n    end\n  end\nend\n```\n\nThis takes care of clearing messages after each example, as well.\n\nIf you're not using RSpec, you can easily replicate the functionality yourself. Call `DeliveryBoy.test_mode!` at load time, and make sure that `DeliveryBoy.testing.clear` is called after each test.\n\n### Instrumentation \u0026 monitoring\n\nSince DeliveryBoy is just an opinionated API on top of ruby-kafka, you can use all the [instrumentation made available by that library](https://github.com/zendesk/ruby-kafka#instrumentation). You can also use the [existing monitoring solutions](https://github.com/zendesk/ruby-kafka#monitoring) that integrate with various monitoring services.\n\n## Contributing\n\nBug reports and pull requests are welcome on [GitHub](https://github.com/zendesk/delivery_boy). Feel free to [join our Slack team](https://ruby-kafka-slack.herokuapp.com/) and ask how best to contribute!\n\n### Releasing a new version\nA new version is published to RubyGems.org every time a change to `version.rb` is pushed to the `main` branch.\nIn short, follow these steps:\n1. Update `version.rb`,\n2. merge this change into `main`, and\n3. look at [the action](https://github.com/zendesk/delivery_boy/actions/workflows/publish.yml) for output.\n\nTo create a pre-release from a non-main branch:\n1. change the version in `version.rb` to something like `1.2.0.pre.1` or `2.0.0.beta.2`,\n2. push this change to your branch,\n3. go to [Actions → “Publish to RubyGems.org” on GitHub](https://github.com/zendesk/delivery_boy/actions/workflows/publish.yml),\n4. click the “Run workflow” button,\n5. pick your branch from a dropdown.\n\n## Support and Discussion\n\nIf you've discovered a bug, please file a [Github issue](https://github.com/zendesk/delivery_boy/issues/new), and make sure to include all the relevant information, including the version of DeliveryBoy, ruby-kafka, and Kafka that you're using.\n\nIf you have other questions, or would like to discuss best practises, how to contribute to the project, or any other ruby-kafka related topic, [join our Slack team](https://ruby-kafka-slack.herokuapp.com/)!\n\n## Copyright and license\n\nCopyright 2017 Zendesk, Inc.\n\nLicensed under the Apache License, Version 2.0 (the \"License\"); you may not use this file except in compliance with the License.\n\nYou may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0\n\nUnless required by applicable law or agreed to in writing, software distributed under the License is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fzendesk%2Fdelivery_boy","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fzendesk%2Fdelivery_boy","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fzendesk%2Fdelivery_boy/lists"}